Skip to main content

BigQuery and Apache Iceberg

dbt supports materializing Iceberg tables on BigQuery via the catalog integration, starting with the dbt-bigquery 1.10 release.

Creating Iceberg Tables​

dbt supports creating Iceberg tables for two of the BigQuery materializations:

BigQuery Iceberg catalogs​

BigQuery supports Iceberg tables via its built-in catalog BigLake Metastore today. No setup is needed to access the BigLake Metastore. However, you will need to have a storage bucket and the required BigQuery roles configured prior to creating an Iceberg table.

dbt catalog integration configurations​

The following table outlines the configuration fields required to set up a catalog integration for Iceberg tables in Snowflake.

FieldRequiredAccepted values
nameyesName of catalog integration
catalog_nameyesThe name of the catalog integration in BigQuery. For example, biglake_metastore.
external_volumeyesgs://<bucket_name>
table_formatyesiceberg
catalog_typeyesbiglake_metastore
file_formatyesdefault,parquet

dbt has an additonal configuration: storage_uri that the user can use on the model configuration to override the catalog integration path to supply the entire storage_uri path directly.

Configure catalog integration for managed Iceberg tables​

  1. Create a catalogs.yml at the top level of your dbt project.

    An example:

catalogs:
  - name: my_bigquery_iceberg_catalog
    active_write_integration: biglake_metastore
    write_integrations:
      - name: biglake_metastore
        external_volume: 'gs://mydbtbucket'
        table_format: iceberg
        file_format: parquet
        catalog_type: biglake_metastore

  1. Apply the catalog configuration at either the model, folder, or project level:
iceberg_model.yml

{{
    config(
        materialized='table',
        catalog_name = 'my_bigquery_iceberg_catalog'

    )
}}

select * from {{ ref('jaffle_shop_customers') }}

  1. Execute the dbt model with a dbt run -s iceberg_model.

Limitations​

BigQuery today does not support connecting to external Iceberg catalogs. In terms of SQL operations and table management features, please refer to the BigQuery docs for more information.

Was this page helpful?

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

0
Edit this page