Skip to main content

Dremio setup

Vendor plugin

Some core functionality may be limited. If you're interested in contributing, check out the source code for each repository listed below.

Overview of dbt-dremio

  • Maintained by: Dremio
  • Authors: Dremio (formerly Fabrice Etanchaud)
  • GitHub repo: dremio/dbt-dremio
  • PyPI package: dbt-dremio
  • Slack channel: n/a
  • Supported dbt Core version: v1.1.0 and newer
  • dbt Cloud support: Not Supported
  • Minimum data platform version: Dremio 22.0

Installing dbt-dremio

pip is the easiest way to install the adapter:

pip install dbt-dremio

Installing dbt-dremio will also install dbt-core and any other dependencies.

Configuring dbt-dremio

For Dremio-specifc configuration please refer to Dremio Configuration

For further info, refer to the GitHub repository: dremio/dbt-dremio

Follow the repository's link for os dependencies.

Prerequisites for Dremio Cloud

Before connecting from project to Dremio Cloud, follow these prerequisite steps:

Prerequisites for Dremio Software

  • Ensure that you are using version 22.0 or later.

  • Enable these support keys in your Dremio cluster:

    • dremio.iceberg.enabled
    • dremio.iceberg.ctas.enabled
    • dremio.execution.support_unlimited_splits

    See Support Keys in the Dremio documentation for the steps.

  • If you want to use TLS to secure the connection between dbt and Dremio Software, configure full wire encryption in your Dremio cluster. For instructions, see Configuring Wire Encryption.

Initializing a Project

  1. Run the command dbt init <project_name>.
  2. Select dremio as the database to use.
  3. Select one of these options to generate a profile for your project:
    • dremio_cloud for working with Dremio Cloud
    • software_with_username_password for working with a Dremio Software cluster and authenticating to the cluster with a username and a password
    • software_with_pat for working with a Dremio Software cluster and authenticating to the cluster with a personal access token
  4. Append these lines to the end of the content of the dbt_project.yml file at the root of your project directory:
vars:
dremio:reflections_enabled: false

Next, configure the profile for your project.

Profiles

When you initialize a project, you create one of these three profiles. You must configure it before trying to connect to Dremio Cloud or Dremio Software.

Profiles

When you initialize a project, you create one of these three profiles. You must configure it before trying to connect to Dremio Cloud or Dremio Software.

  • Profile for Dremio Cloud
  • Profile for Dremio Software with Username/Password Authentication
  • Profile for Dremio Software with Authentication Through a Personal Access Token

For descriptions of the configurations in these profiles, see Configurations.

[project name]:
outputs:
dev:
cloud_host: https://api.dremio.cloud
cloud_project_id: [project ID]
pat: [personal access token]
threads: [integer >= 1]
type: dremio
use_ssl: true
user: [email address]
target: dev

Configurations

Configurations Common to Profiles for Dremio Cloud and Dremio Software

ConfigurationRequired?Default ValueDescription
typeYesdremioAuto-populated when creating a Dremio project. Do not change this value.
threadsYes1The number of threads the dbt project runs on.

Configurations in Profiles for Dremio Cloud

ConfigurationRequired?Default ValueDescription
cloud_hostYeshttps://api.dremio.cloudUS Control Plane: https://api.dremio.cloud
EU Control Plane: https://api.eu.dremio.cloud
userYesNoneEmail address used as a username in Dremio Cloud
patYesNonePersonal Access TokenSee Personal Access Tokens for instructions about obtaining a token.
cloud_project_idYesNoneThe ID of the Sonar project in which to run transformations.
use_sslYestrueThe value must be true.

Configurations in Profiles for Dremio Software

ConfigurationRequired?Default ValueDescription
software_hostYesNoneThe hostname or IP address of the coordinator node of the Dremio cluster.
portYes9047Port for Dremio Software cluster API endpoints.
userYesNoneThe username of the account to use when logging into the Dremio cluster.
passwordYes, if you are not using the pat configuration.NoneThe password of the account to use when logging into the Dremio cluster.
patYes, if you are not using the user and password configurations.NoneThe personal access token to use for authenticating to Dremio.See Personal Access Tokens for instructions about obtaining a token.The use of a personal access token takes precedence if values for the three configurations user, password and pat are specified.
use_sslYestrueAcceptable values are true and false. If the value is set to true, ensure that full wire encryption is configured in your Dremio cluster. See Prerequisites for Dremio Software.
0