Skip to main content

Connect Redis to dbt v1

Community plugin

Some core functionality may be limited. If you're interested in contributing, check out the source code for each repository listed below.

  • Maintained by: Community
  • Authors: Albert T. Wong
  • GitHub repo: redis/redis-dbt-adapter
  • PyPI package: dbt-redis-adbc
  • Slack channel: n/a
  • Supported dbt version: v1.10.0 and newer
  • dbt support: Not Supported
  • Minimum data platform version: Redis 8.x

Installing dbt-redis-adbc

Use `pip` to install the adapter. Use the following command for installation: python -m pip install dbt-redis-adbc

Configuring dbt-redis-adbc

For Redis-specific configuration, please refer to Redis configs.

The dbt-redis-adbc adapter connects dbt v1 to Redis through the Redis Arrow Database Connectivity (ADBC) driver. It supports Python-based dbt v1. It does not support v2 or the dbt platform.

This page describes dbt-redis-adbc 0.2.0, which is tested with Redis ADBC driver 0.0.18 and Python 3.13.

Prerequisites​

  • Python 3.13 or later
  • dbt-core>=1.10,<2 and dbt-adapters>=1.16,<2
  • Redis 8.x with the Query Engine (Search) enabled; the adapter's integration tests run against a Redis Enterprise cluster with TLS and username and password authentication
  • A Redis ADBC shared library for your operating system and architecture
  • A Redis user with the driver's required access control list permissions

The driver requires Redis Search commands, including FT._LIST and FT.CREATE. Refer to the driver's server requirements for deployment compatibility. Valkey and Redis 7.2 derivatives, including Amazon ElastiCache and Google Cloud Memorystore, are not supported because they lack commands and functions that the driver requires. Redis Flex databases are also not supported.

Install the driver​

The pip command above installs the adapter. It does not install the Redis ADBC shared library.

Run dbt --version to confirm the installation. The output lists redis_adbc under plugins.

Download and extract the shared-library archive for your platform from the Redis ADBC driver releases. Adapter version 0.2.0 is tested with driver release v0.0.18. Each archive has a matching .sha256 file that you can use to verify the download. Prebuilt libraries are available for macOS Apple Silicon, Linux x86-64, and Linux arm64. Other platforms require a source build.

Configure your connection​

Export the driver location and your connection settings in the shell where you run dbt. REDIS_ADBC_DRIVER is the extracted library's absolute path; the file ends in .dylib on macOS and .so on Linux. Use a rediss:// URI for a TLS connection, or a redis:// URI for a server without TLS:

export REDIS_ADBC_DRIVER=/ABSOLUTE/PATH/TO/libadbc_driver_redis.dylib
export REDIS_URI=rediss://YOUR_REDIS_HOST:YOUR_REDIS_PORT/0
export REDIS_USERNAME=YOUR_REDIS_USERNAME
export REDIS_PASSWORD=YOUR_REDIS_PASSWORD

If a private certificate authority (CA) signed the server's certificate, also export the path to its PEM bundle. Leave it unset for a certificate that your system already trusts. The CA bundle requires a rediss:// URI.

export REDIS_TLS_CA_FILE=/ABSOLUTE/PATH/TO/redis-ca.pem

Add this profile to ~/.dbt/profiles.yml. The profile reads the variables you exported. dbt fails to read the profile if REDIS_ADBC_DRIVER or REDIS_URI is unset:

~/.dbt/profiles.yml
redis_project:
  target: dev
  outputs:
    dev:
      type: redis_adbc
      driver: "{{ env_var('REDIS_ADBC_DRIVER') }}"
      uri: "{{ env_var('REDIS_URI') }}"
      username: "{{ env_var('REDIS_USERNAME', '') }}"
      password: "{{ env_var('REDIS_PASSWORD', '') }}"
      driver_options:
        adbc.redis.tls_ca_file: "{{ env_var('REDIS_TLS_CA_FILE', '') }}"
      database: redis
      schema: analytics
      threads: 4

Then set your project's profile to redis_project:

dbt_project.yml
profile: redis_project

Use separate username and password fields instead of embedding credentials in the URI.

FieldRequiredDescription
typeYesMust be redis_adbc.
driverUnless REDIS_ADBC_DRIVER is setPath to the Redis ADBC shared library. If you omit the extension, the adapter adds .dylib on macOS, .so on Linux, or .dll on Windows.
uriNoRedis connection URI: rediss:// for TLS, or redis:// without TLS. Defaults to redis://localhost:6379/0. Use database 0 for Redis Cluster.
usernameNoRedis ACL username.
passwordNoRedis password.
databaseNoThe driver's single catalog, redis; defaults to redis.
schemaYesDefault SQL schema for dbt relations.
threadsNoNumber of concurrent dbt threads; the example uses 4.
aggregate_pushdownNoDriver aggregate pushdown mode: exact, all, or none. Unset values use the driver's default.
rename_rekeyNoBoolean that moves a renamed table's rows to keys matching its new name. This copies every row again during a dbt table build.
time_zoneNoSession time zone, such as America/New_York. The driver defaults to UTC.
read_timeoutNoTime to wait for each reply: a duration such as 30s or 10m, a number of seconds, or 0 for no timeout. The driver defaults to five minutes.
write_timeoutNoTime to send each command, using the same format and default as read_timeout.
driver_optionsNoMapping of additional driver options, such as adbc.redis.tls_ca_file, to string, numeric, or Boolean values. Options with their own profile field must use that field instead.

Run dbt debug from your project directory to verify the connection. For materializations and limitations, refer to Redis configurations.

Was this page helpful?

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

0
Loading