Events and logs
New in v1.0
While dbt has always generated logs, the eventing and structured logging system described below is new in v1.
With every task that dbt performs, it generates events. It records those events as log messages, and writes them (in real time) to two places:
- The command line terminal (
stdout), to provide interactive feedback while running dbt. - The debug log file (
logs/dbt.log), to enable detailed debugging of errors when they occur. The text-formatted log messages in this file include allDEBUG-level events, as well as contextual information, such as log level and thread name. The location of this file can be configured via thelog_pathconfig.
21:35:48 6 of 7 OK created view model dbt_testing.name_list......................... [CREATE VIEW in 0.17s]
============================== 2021-12-02 21:29:35.417263 | c83a0afc-7ed3-49e7-8c0e-797af7f9d7b6 ==============================21:29:35.417263 [info ] [MainThread]: Running with dbt=1.0.0-rc321:29:35.417955 [debug] [MainThread]: running dbt with arguments Namespace(cls=<class 'dbt.task.run.RunTask'>, debug=None, defer=None, exclude=None, fail_fast=None, full_refresh=False, log_cache_events=False, log_format=None, partial_parse=None, printer_width=None, profile=None, profiles_dir='/Users/jerco/.dbt', project_dir=None, record_timing_info=None, rpc_method='run', select=None, selector_name=None, send_anonymous_usage_stats=None, single_threaded=False, state=None, static_parser=None, target=None, threads=None, use_colors=None, use_experimental_parser=None, vars='{}', version_check=None, warn_error=None, which='run', write_json=None)...21:29:35.814348 [debug] [Thread-1 ]: On model.my_project.my_table: BEGIN
Structured logging
When json log formatting is enabled, dbt will output produce rich, structured log information which can be piped into monitoring tools for analysis, or to power applications with dbt metadata in real time.
Each log line will have the following JSON properties:
| Field | Description |
|---|---|
code | A unique identifier for each event type |
data | A dictionary containing programmatically accessible information about the log line. The contents of this dictionary vary based on the event type which generated this log message. |
invocation_id | A unique identifier for this invocation of dbt |
level | A string representation of the log level (debug, info, warn, error) |
log_version | Integer indicating version |
msg | The human-friendly log message. Note: This message is not intended for machine consumption. Log messages are bject to change in future versions of dbt, and those changes may or may not coincide with a change in log_version. |
node_info | If applicable, a dictionary of human- and machine-friendly information about a currently running resource |
pid | The process ID for the running dbt invocation which produced this log message |
thread_name | The thread in which the log message was produced, helpful for tracking queries when dbt is run with ltiple threads |
ts | When the log line was printed |
type | Always log_line |
If available, node_info will include:
| Field | Description |
|---|---|
materialized | view, table, incremental, etc. |
node_finished_at | Timestamp when node processing completed |
node_name | Name of this model/seed/test/etc |
node_path | File path to where this resource is defined |
node_started_at | Timestamp when node processing started |
node_status | Current status of the node, as defined in the result contract |
resource_type | model, test, seed, snapshot, etc. |
type | Always 'node_status' |
unique_id | The unique identifier for this resource, which can be used to look up contextual information in a manifest |
Example
{ "code": "Q033", "data": { "description": "view model dbt_testing.name_list", "index": 7, "total": 7 }, "invocation_id": "30206572-f52f-4b91-af6d-d2b18fdbbbb8", "level": "info", "log_version": 1, "msg": "7 of 7 START view model dbt_testing.name_list.............................. [RUN]", "node_info": { "materialized": "view", "node_finished_at": null, "node_name": "male_list_view", "node_path": "human/name_list.sql", "node_started_at": "2021-12-02T21:47:03.477004", "node_status": "started", "resource_type": "model", "type": "node_status", "unique_id": "model.jaffle_shop.name_list" }, "pid": 81915, "thread_name": "Thread-4", "ts": "2021-12-02T21:47:03.480384Z", "type": "log_line"}
Python interface
Be warned: While dbt-core v1 represents a significant step forward in the stability of the core framework, dbt-core's python API is still unstable and liable to change, with the exception of a few specific interfaces.
dbt-core makes available a full history of events fired during an invocation, in the form of an EVENT_HISTORY object:
from dbt.events.functions import EVENT_HISTORY
The Python interface into events is significantly less mature than the structured logging interface. For all use cases, we recommend parsing JSON-formatted logs.
For details about how the eventing system has been implemented in dbt-core, see the events module README.