About Iceberg catalogs
A technical data catalog is a metadata management layer that enables users and tools to programmatically discover, understand, and govern data assets that are available across multiple data platforms and query engines.
Background
In the early 2010s, with the introduction of Hadoop and data lakes, Hive Metastore became the standard for managing schema metadata in Hadoop ecosystems. While effective in supporting early distributed query engines (separated compute and storage), such as Apache Spark and Trino, the Hive Metastore is limited in the kinds of structural metadata it can support, and it's generally slower and costlier than modern technical catalogs.
In recent years, the emergence and consolidation around the Iceberg standard has led to the evolution of existing data catalogs (such as Unity Catalog) or the creation of new open source catalogs (including Polaris and Lakekeeper).
It's important to note that technical data catalogs serve a different purpose from business catalogs, although both benefit data teams:
-
Technical data catalogs: Focus on structural metadata, including information about data like table and column names, data types, storage locations (particularly important for open table formats), and access controls. They can be built into an existing data platform (no setup needed) — such as Databricks Unity Catalog or Snowflake Horizon Catalog — or externally managed and integrated with each query engine. Those engines use the catalog to locate, read, and write data.
-
Business data catalogs: Serve broader organizational users (such as BI analysts and product managers). They enrich technical metadata with business context in the form of metrics, business definitions, data quality indicators, usage patterns, and ownership.
Why data catalogs are important to dbt
For dbt users working in a lakehouse or multi-engine architecture, data catalogs can serve two purposes:
-
Table discovery: dbt models are materialized in an open table format (often Iceberg) and registered in a catalog. Understanding the catalog structure is critical for managing datasets and informing dbt about what has already been built and where it resides.
-
Cross-engine interoperability: Iceberg catalogs allow datasets created by one compute engine to be read by another, without replication. This is the foundation for cross-platform dbt Mesh.
Without a catalog, each Iceberg table's metadata needs to be registered with the query engine individually, requiring many additional create iceberg table and alter ... refresh queries. With a catalog, each table's metadata is registered and refreshed automatically, and there's a single endpoint to ask about every table.
Over the past year, data platform vendors have been adding support for catalog "linking" or "federation," which is when the data platform manages the synchronization of metadata between external data catalogs and its managed data catalog. This means that an Iceberg table written by one query engine into one catalog is automatically available for reading by another query engine. See: Snowflake catalog-linked databases, Databricks catalog federation, AWS Glue catalog federation, BigQuery catalog federation.
How dbt works with Iceberg catalogs
dbt uses Iceberg catalogs defined in catalogs.yml in order to:
-
Materialize models: When dbt materializes a model as a table or view, if the catalog integration is declared, the underlying adapter (such as Spark, Trino, or Snowflake) creates an Iceberg table entry in the specified catalog, both built-in or external.
-
Resolve references: When dbt reads a model materialized to a catalog, it uses the "physical" location defined by the active adapter (new spec) or write integration (old spec). This enables referencing (reading) a model (Iceberg table) in one query engine that was originally materialized (written) by another one, so long as both engines are integrated to the same catalog.
Limitations
Many data platforms, query engines, and data catalogs have added substantially more support for Iceberg standards over the past few years — but it's still not a guarantee that any given data warehouse can interoperate with any given data catalog.
When it does exist, that support can be limited. For example, most Iceberg catalogs only support tables (table + incremental materialization in dbt), not views. They may not support atomic create table as statements, meaning that dbt must update tables using multiple statements without guarantees.
Benoit (DX advocate at dbt Labs) created a useful visualization (also available as a fun interactive game) to show the current state of warehouse / catalog interoperability. (See something that looks wrong or out-of-date? Open an issue or PR in the repo.)
Was this page helpful?
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
