Skip to main content

Layer setup

  • Maintained by: Layer
  • Authors: Mehmet Ecevit
  • GitHub repo: layerai/dbt-layer
  • PyPI package: dbt-layer-bigquery
  • Slack channel: #tools-layer
  • Supported dbt Core version: v1.0.0 and newer
  • dbt Cloud support: Not Supported
  • Minimum data platform version: n/a

Installing dbt-layer-bigquery

Use pip to install the adapter. Before 1.8, installing the adapter would automatically install dbt-core and any additional dependencies. Beginning in 1.8, installing an adapter does not automatically install dbt-core. This is because adapters and dbt Core versions have been decoupled from each other so we no longer want to overwrite existing dbt-core installations. Use the following command for installation:

Configuring dbt-layer-bigquery

For Layer-specific configuration, please refer to Layer configs.

Profile Configuration

Layer Bigquery targets should be set up using the following sections in your profiles.yml file.

Layer Authentication

Add your layer_api_key to your profiles.yaml to authenticate with Layer. To get your Layer API Key:

Bigquery Authentication

You can use any authentication method supported in the official dbt Bigquery adapter since Layer uses dbt-bigquery adapter to connect to your Bigquery instance.

A sample profile:

profiles.yml
layer-profile:
target: dev
outputs:
dev:
# Layer authentication
type: layer_bigquery
layer_api_key: [the API Key to access your Layer account (opt)]
# Bigquery authentication
method: service-account
project: [GCP project id]
dataset: [the name of your dbt dataset]
threads: [1 or more]
keyfile: [/path/to/bigquery/keyfile.json]

Description of Layer Bigquery Profile Fields

The following fields are required:

ParameterDefaultTypeDescription
typestringSpecifies the adapter you want to use. It should be layer_bigquery.
layer_api_keystring (opt)Specifies your Layer API key. If you want to make predictions with public ML models from Layer, you don't need to have this key in your profile. It's required if you load ML models from your Layer account or train an AutoML model.
layer_projectstring (opt)Specifies your target Layer project. If you don't specify, Layer will use the project same name with your dbt project.
methodstringSpecifies the authentication type to connect to your BigQuery.

Rest of the parameters depends on the BigQuery authentication method you specified.

Usage

AutoML

You can automatically build state-of-art ML models using your own dbt models with plain SQL. To train an AutoML model all you have to do is pass your model type, input data (features) and target column you want to predict to layer.automl() in your SQL. The Layer AutoML will pick the best performing model and enable you to call it by its dbt model name to make predictions as shown above.

Syntax:

layer.automl("MODEL_TYPE", ARRAY[FEATURES], TARGET)

Parameters:

SyntaxDescription
MODEL_TYPEType of the model your want to train. There are two options:
- classifier: A model to predict classes/labels or categories such as spam detection
- regressor: A model to predict continious outcomes such as CLV prediction.
FEATURESInput column names as a list to train your AutoML model.
TARGETTarget column that you want to predict.

Requirements:

  • You need to put layer_api_key to your dbt profile to make AutoML work.

Example:

Check out Order Review AutoML Project:

SELECT order_id,
layer.automl(
-- This is a regression problem
'regressor',
-- Data (input features) to train our model
ARRAY[
days_between_purchase_and_delivery, order_approved_late,
actual_delivery_vs_expectation_bucket, total_order_price, total_order_freight, is_multiItems_order,seller_shipped_late],
-- Target column we want to predict
review_score
)
FROM {{ ref('training_data') }}

Prediction

You can make predictions using any Layer ML model within your dbt models. Layer dbt Adapter helps you score your data resides on your warehouse within your dbt DAG with SQL.

Syntax:

layer.predict("LAYER_MODEL_PATH", ARRAY[FEATURES])

Parameters:

SyntaxDescription
LAYER_MODEL_PATHThis is the Layer model path in form of /[organization_name]/[project_name]/models/[model_name]. You can use only the model name if you want to use an AutoML model within the same dbt project.
FEATURESThese are the columns that this model requires to make a prediction. You should pass the columns as a list like ARRAY[column1, column2, column3].

Example:

Check out Cloth Detection Project:

SELECT
id,
layer.predict("layer/clothing/models/objectdetection", ARRAY[image])
FROM
{{ ref("products") }}
0