Source properties

Related documentation

Overview

Source properties can be declared in .yml files in your models/ directory (as defined by the source-paths config).

You can name these files whatever_you_want.yml, and nest them arbitrarily deeply in subfolders within the models/ directory.

models/<filename>.yml
version: 2
sources:
- name: <string> # required
description: <markdown_string>
database: <database_name>
schema: <schema_name>
loader: <string>
loaded_at_field: <column_name>
meta: {<dictionary>}
tags: [<string>]
warn_after:
count: <positive_integer>
period: minute | hour | day
error_after:
count: <positive_integer>
period: minute | hour | day
filter: <where-condition>
database: true | false
schema: true | false
identifier: true | false
tables:
- name: <string> #required
description: <markdown_string>
meta: {<dictionary>}
identifier: <table_name>
loaded_at_field: <column_name>
- <test>
- ... # declare additional tests
tags: [<string>]
warn_after:
count: <positive_integer>
period: minute | hour | day
error_after:
count: <positive_integer>
period: minute | hour | day
filter: <where-condition>
database: true | false
schema: true | false
identifier: true | false
columns:
- name: <column_name> # required
description: <markdown_string>
meta: {<dictionary>}
quote: true | false
- <test>
- ... # declare additional tests
tags: [<string>]
- name: ... # declare properties of additional columns
- name: ... # declare properties of additional source tables
- name: ... # declare properties of additional sources

Example

models/<filename>.yml
version: 2
sources:
- name: jaffle_shop
database: raw
schema: public
loader: emr # informational only (free text)
loaded_at_field: _loaded_at # configure for all sources
# meta fields are rendered in auto-generated documentation
meta:
contains_pii: true
owner: "@alice"
# Add tags to this source
tags:
- ecom
- pii
quoting:
database: false
schema: false
identifier: false
tables:
- name: orders
identifier: Orders_
loaded_at_field: updated_at # override source defaults
columns:
- name: id
tests:
- unique
- name: price_in_usd
tests:
- not_null
- name: customers
quoting:
identifier: true # override source defaults
columns:
tests:
- unique