Skip to main content

Events and logs

New in v1.0

While dbt has always generated logs, the eventing and structured logging system described below is new in v1.

With every task that dbt performs, it generates events. It records those events as log messages, and writes them (in real time) to two places:

  • The command line terminal (stdout), to provide interactive feedback while running dbt.
  • The debug log file (logs/dbt.log), to enable detailed debugging of errors when they occur. The text-formatted log messages in this file include all DEBUG-level events, as well as contextual information, such as log level and thread name. The location of this file can be configured via the log_path config.
CLI
21:35:48  6 of 7 OK created view model dbt_testing.name_list......................... [CREATE VIEW in 0.17s]
logs/dbt.log
============================== 2021-12-02 21:29:35.417263 | c83a0afc-7ed3-49e7-8c0e-797af7f9d7b6 ==============================
21:29:35.417263 [info ] [MainThread]: Running with dbt=1.0.0-rc3
21:29:35.417955 [debug] [MainThread]: running dbt with arguments Namespace(cls=<class 'dbt.task.run.RunTask'>, debug=None, defer=None, exclude=None, fail_fast=None, full_refresh=False, log_cache_events=False, log_format=None, partial_parse=None, printer_width=None, profile=None, profiles_dir='/Users/jerco/.dbt', project_dir=None, record_timing_info=None, rpc_method='run', select=None, selector_name=None, send_anonymous_usage_stats=None, single_threaded=False, state=None, static_parser=None, target=None, threads=None, use_colors=None, use_experimental_parser=None, vars='{}', version_check=None, warn_error=None, which='run', write_json=None)
...
21:29:35.814348 [debug] [Thread-1  ]: On model.my_project.my_table: BEGIN

Structured logging​

When json log formatting is enabled, dbt will output produce rich, structured log information which can be piped into monitoring tools for analysis, or to power applications with dbt metadata in real time.

Each log line will have the following JSON properties:

FieldDescription
codeA unique identifier for each event type
dataA dictionary containing programmatically accessible information about the log line. The contents of this dictionary vary based on the event type which generated this log message.
invocation_idA unique identifier for this invocation of dbt
levelA string representation of the log level (debug, info, warn, error)
log_versionInteger indicating version
msgThe human-friendly log message. Note: This message is not intended for machine consumption. Log messages are subject to change in future versions of dbt, and those changes may or may not coincide with a change in log_version.
node_infoIf applicable, a dictionary of human- and machine-friendly information about a currently running resource
pidThe process ID for the running dbt invocation which produced this log message
thread_nameThe thread in which the log message was produced, helpful for tracking queries when dbt is run with multiple threads
tsWhen the log line was printed
typeAlways log_line

If available, node_info will include:

FieldDescription
materializedview, table, incremental, etc.
node_finished_atTimestamp when node processing completed
node_nameName of this model/seed/test/etc
node_pathFile path to where this resource is defined
node_started_atTimestamp when node processing started
node_statusCurrent status of the node, as defined in the result contract
resource_typemodel, test, seed, snapshot, etc.
typeAlways 'node_status'
unique_idThe unique identifier for this resource, which can be used to look up contextual information in a manifest

Example​

{
    "code": "Q033",
    "data":
    {
        "description": "view model dbt_testing.name_list",
        "index": 7,
        "total": 7
    },
    "invocation_id": "30206572-f52f-4b91-af6d-d2b18fdbbbb8",
    "level": "info",
    "log_version": 1,
    "msg": "7 of 7 START view model dbt_testing.name_list.............................. [RUN]",
    "node_info":
    {
        "materialized": "view",
        "node_finished_at": null,
        "node_name": "male_list_view",
        "node_path": "human/name_list.sql",
        "node_started_at": "2021-12-02T21:47:03.477004",
        "node_status": "started",
        "resource_type": "model",
        "type": "node_status",
        "unique_id": "model.jaffle_shop.name_list"
    },
    "pid": 81915,
    "thread_name": "Thread-4",
    "ts": "2021-12-02T21:47:03.480384Z",
    "type": "log_line"
}

Python interface​

dbt-core makes available a full history of events fired during an invocation, in the form of an EVENT_HISTORY object:

from dbt.events.functions import EVENT_HISTORY

The Python interface into events is significantly less mature than the structured logging interface. For all use cases, we recommend parsing JSON-formatted logs.

For details about how the eventing system has been implemented in dbt-core, see the events module README.

0
Edit this page