Skip to main content

Connection profiles

Connecting to your warehouse using the command line

When you invoke dbt from the command line, dbt parses your dbt_project.yml and obtains the profile name, which dbt needs to connect to your data warehouse.

dbt_project.yml
# Example dbt_project.yml file
name: 'jaffle_shop'
profile: 'jaffle_shop'
...

dbt then checks your profiles.yml file for a profile with the same name. A profile contains all the details required to connect to your data warehouse.

This file generally lives outside of your dbt project to avoid sensitive credentials being checked in to version control, but profiles.yml can be safely checked in when using environment variables to load sensitive credentials.

~/.dbt/profiles.yml
# example profiles.yml file
jaffle_shop:
target: dev
outputs:
dev:
type: postgres
host: localhost
user: alice
password: <password>
port: 5432
dbname: jaffle_shop
schema: dbt_alice
threads: 4

About the profiles.yml file

In your profiles.yml file, you can store as many profiles as you need. Typically, you would have one profile for each warehouse you use. Most organizations only have one profile.

For information about configuring advanced options, see the profiles.yml reference page.

About profiles

A profile consists of targets, and a specified default target.

Each target specifies the type of warehouse you are connecting to, the credentials to connect to the warehouse, and some dbt-specific configurations.

The credentials you need to provide in your target varies across warehouses &mdash sample profiles for each supported warehouse are available in the Supported Data Platforms section.

Pro Tip: You may need to surround your password in quotes if it contains special characters. More details here.

Setting up your profile

To set up your profile, copy the correct sample profile for your warehouse into your profiles.yml file and update the details as follows:

  • Profile name: Replace the name of the profile with a sensible name – it’s often a good idea to use the name of your organization. Make sure that this is the same name as the profile indicated in your dbt_project.yml file.
  • target: This is the default target your dbt project will use. It must be one of the targets you define in your profile. Commonly it is set to dev.
  • Populating your target:
    • type: The type of data warehouse you are connecting to
    • Warehouse credentials: Get these from your database administrator if you don’t already have them. Remember that user credentials are very sensitive information that should not be shared.
    • schema: The default schema that dbt will build objects in.
    • threads: The number of threads the dbt project will run on.

You can find more information on which values to use in your targets below.

Validating your warehouse credentials

Use the debug command to check whether you can successfully connect to your warehouse. Simply run dbt debug from within a dbt project to test your connection.

Understanding targets in profiles

dbt supports multiple targets within one profile to encourage the use of separate development and production environments as discussed in Managing Environments.

A typical profile for an analyst using dbt locally will have a target named dev, and have this set as the default.

You may also have a prod target within your profile, which creates the objects in your production schema. However, since it's often desirable to perform production runs on a schedule, we recommend deploying your dbt project to a separate machine other than your local machine. Most dbt users only have a dev target in their profile on their local machine.

If you do have multiple targets in your profile, and want to use a target other than the default, you can do this using the --target option when issuing a dbt command.

Understanding warehouse credentials

We recommend that each dbt user has their own set of database credentials, including a separate user for production runs of dbt – this helps debug rogue queries, simplifies ownerships of schemas, and improves security.

To ensure the user credentials you use in your target allow dbt to run, you will need to ensure the user has appropriate privileges. While the exact privileges needed varies between data warehouses, at a minimum your user must be able to:

  • read source data
  • create schemas¹
  • read system tables
Running dbt without create schema privileges

If your user is unable to be granted the privilege to create schemas, your dbt runs should instead target an existing schema that your user has permission to create relations within.

Understanding target schemas

The target schema represents the default schema that dbt will build objects into, and is often used as the differentiator between separate environments within a warehouse.

Schemas in BigQuery

dbt uses the term "schema" in a target across all supported warehouses for consistency. Note that in the case of BigQuery, a schema is actually a dataset.

The schema used for production should be named in a way that makes it clear that it is ready for end-users to use for analysis – we often name this analytics.

In development, a pattern we’ve found to work well is to name the schema in your dev target dbt_<username>. Suffixing your name to the schema enables multiple users to develop in dbt, since each user will have their own separate schema for development, so that users will not build over the top of each other, and ensuring that object ownership and permissions are consistent across an entire schema.

Note that there’s no need to create your target schema beforehand – dbt will check if the schema already exists when it runs, and create it if it doesn’t.

While the target schema represents the default schema that dbt will use, it may make sense to split your models into separate schemas, which can be done by using custom schemas.

Understanding threads

When dbt runs, it creates a directed acyclic graph (DAG) of links between models. The number of threads represents the maximum number of paths through the graph dbt may work on at once – increasing the number of threads can minimize the run time of your project.

For example, if you specify threads: 1, dbt will start building only one model, and finish it, before moving onto the next. Specifying threads: 8 means that dbt will work on up to 8 models at once without violating dependencies – the actual number of models it can work on will likely be constrained by the available paths through the dependency graph.

There's no set limit of the maximum number of threads you can set – while increasing the number of threads generally decreases execution time, there are a number of things to consider:

  • Increasing the number of threads increases the load on your warehouse, which may impact other tools in your data stack. For example, if your BI tool uses the same compute resources as dbt, their queries may get queued during a dbt run.
  • The number of concurrent queries your database will allow you to run may be a limiting factor in how many models can be actively built – some models may queue while waiting for an available query slot.

Generally the optimal number of threads depends on your data warehouse and its configuration. It’s best to test different values to find the best number of threads for your project. We recommend setting this to 4 to start with.

You can use a different number of threads than the value defined in your target by using the --threads option when executing a dbt command.

Advanced: Customizing a profile directory

The parent directory for profiles.yml is determined using the following precedence:

To check the expected location of your profiles.yml file for your installation of dbt, you can run the following:

$ dbt debug --config-dir
To view your profiles.yml file, run:

open /Users/alice/.dbt

You may want to have your profiles.yml file stored in a different directory than ~/.dbt/ – for example, if you are using environment variables to load your credentials, you might choose to include this file in the root directory of your dbt project.

Note that the file always needs to be called profiles.yml, regardless of which directory it is in.

There are multiple ways to direct dbt to a different location for your profiles.yml file:

1. Use the --profiles-dir option when executing a dbt command

This option can be used as follows:

$ dbt run --profiles-dir path/to/directory

If using this method, the --profiles-dir option needs to be provided every time you run a dbt command.

2. Use the DBT_PROFILES_DIR environment variable to change the default location

Specifying this environment variable overrides the directory that dbt looks for your profiles.yml file in. You can specify this by running:

$ export DBT_PROFILES_DIR=path/to/directory

Advanced: Using environment variables

Credentials can be placed directly into the profiles.yml file or loaded from environment variables. Using environment variables is especially useful for production deployments of dbt. You can find more information about environment variables here.

0