BigQuery and Apache Iceberg
dbt supports materializing Iceberg tables on BigQuery via the catalog integration, starting with the dbt-bigquery 1.10 release.
Creating Iceberg Tables
dbt supports creating Iceberg tables for two of the BigQuery materializations:
BigQuery Iceberg catalogs
BigQuery supports Iceberg tables via its built-in catalog BigLake Metastore today. No setup is needed to access the BigLake Metastore. However, you will need to have a storage bucket and the required BigQuery roles configured prior to creating an Iceberg table.
dbt catalog integration configurations
The following table outlines the configuration fields required to set up a catalog integration for Iceberg tables in Snowflake.
| Field | Required | Accepted values |
|---|---|---|
name | yes | Name of catalog integration |
catalog_name | yes | The name of the catalog integration in BigQuery. For example, biglake_metastore. |
external_volume | yes | gs://<bucket_name> |
table_format | yes | iceberg |
catalog_type | yes | biglake_metastore |
file_format | yes | default,parquet |
dbt has an additonal configuration: storage_uri that the user can use on the model configuration to override the catalog integration path to supply the entire storage_uri path directly.
Configure catalog integration for managed Iceberg tables
- Create a
catalogs.ymlat the top level of your dbt project.
An example:
catalogs:
- name: my_bigquery_iceberg_catalog
active_write_integration: biglake_metastore
write_integrations:
- name: biglake_metastore
external_volume: 'gs://mydbtbucket'
table_format: iceberg
file_format: parquet
catalog_type: biglake_metastore
- Apply the catalog configuration at either the model, folder, or project level:
{{
config(
materialized='table',
catalog_name = 'my_bigquery_iceberg_catalog'
)
}}
select * from {{ ref('jaffle_shop_customers') }}
- Execute the dbt model with a
dbt run -s iceberg_model.
Limitations
BigQuery today does not support connecting to external Iceberg catalogs. In terms of SQL operations and table management features, please refer to the BigQuery docs for more information.