Skip to main content

Layer Profile

Overview of dbt-layer​

Maintained by: Layer
Authors: Mehmet Ecevit
Source: Github
Core version: v1.0 and newer
dbt Cloud: Not Supported
dbt Slack channel #db-layer

dbt-synapse stars latest version on PyPI

To immediately start using Layer inside your dbt DAG, install the Layer dbt Adapter for BigQuery.

pip install dbt-layer-bigquery

You don't need to install dbt separately. Installing dbt-layer will also install dbt-core and dbt-bigquery.

Profile Configuration​

Layer Bigquery targets should be set up using the following sections in your profiles.yml file.

Layer Authentication​

Add your layer_api_key to your profiles.yaml to authenticate with Layer. To get your Layer API Key:

Bigquery Authentication​

You can use any authentication method supported in the official dbt Bigquery adapter since Layer uses dbt-bigquery adapter to connect to your Bigquery instance.

A sample profile:

profiles.yml
layer-profile:
target: dev
outputs:
dev:
# Layer authentication
type: layer_bigquery
layer_api_key: [the API Key to access your Layer account (opt)]
# Bigquery authentication
method: service-account
project: [GCP project id]
dataset: [the name of your dbt dataset]
threads: [1 or more]
keyfile: [/path/to/bigquery/keyfile.json]

Description of Layer Bigquery Profile Fields​

The following fields are required:

ParameterDefaultTypeDescription
typestringSpecifies the adapter you want to use. It should be layer_bigquery.
layer_api_keystring (opt)Specifies your Layer API key. If you want to make predictions with public ML models from Layer, you don't need to have this key in your profile. It's required if you load ML models from your Layer account or train an AutoML model.
layer_projectstring (opt)Specifies your target Layer project. If you don't specify, Layer will use the project same name with your dbt project.
methodstringSpecifies the authentication type to connect to your BigQuery.

Rest of the parameters depends on the BigQuery authentication method you specified.

Usage​

AutoML​

You can automatically build state-of-art ML models using your own dbt models with plain SQL. To train an AutoML model all you have to do is pass your model type, input data (features) and target column you want to predict to layer.automl() in your SQL. The Layer AutoML will pick the best performing model and enable you to call it by its dbt model name to make predictions as shown above.

Syntax:

layer.automl("MODEL_TYPE", ARRAY[FEATURES], TARGET)

Parameters:

SyntaxDescription
MODEL_TYPEType of the model your want to train. There are two options:
- classifier: A model to predict classes/labels or categories such as spam detection
- regressor: A model to predict continious outcomes such as CLV prediction.
FEATURESInput column names as a list to train your AutoML model.
TARGETTarget column that you want to predict.

Requirements:

  • You need to put layer_api_key to your dbt profile to make AutoML work.

Example:

Check out Order Review AutoML Project:

SELECT order_id,
layer.automl(
-- This is a regression problem
'regressor',
-- Data (input features) to train our model
ARRAY[
days_between_purchase_and_delivery, order_approved_late,
actual_delivery_vs_expectation_bucket, total_order_price, total_order_freight, is_multiItems_order,seller_shipped_late],
-- Target column we want to predict
review_score
)
FROM {{ ref('training_data') }}

Prediction​

You can make predictions using any Layer ML model within your dbt models. Layer dbt Adapter helps you score your data resides on your warehouse within your dbt DAG with SQL.

Syntax:

layer.predict("LAYER_MODEL_PATH", ARRAY[FEATURES])

Parameters:

SyntaxDescription
LAYER_MODEL_PATHThis is the Layer model path in form of /[organization_name]/[project_name]/models/[model_name]. You can use only the model name if you want to use an AutoML model within the same dbt project.
FEATURESThese are the columns that this model requires to make a prediction. You should pass the columns as a list like ARRAY[column1, column2, column3].

Example:

Check out Cloth Detection Project:

SELECT
id,
layer.predict("layer/clothing/models/objectdetection", ARRAY[image])
FROM
{{ ref("products") }}