Layer dbt Adapter

This adapter runs ML pipelines inside dbt dag with BigQuery or Snowflake as the backing data warehouse. With the Layer dbt Adapter you can:

• Score your data with a machine learning model from Layer.
• Train an AutoML model with your data.
• Train a custom machine learning model with your data [coming soon...]

To learn more about Layer:

https://layer.ai

Getting started

To immediately start using Layer inside your dbt DAG, install the Layer dbt Adapter for BigQuery:

pip install dbt-layer[bigquery] -U

Add the Layer dbt Adapter to your BigQuery profile. An example profile:

layer-profile:
  target: dev
  outputs:
    dev:
      type: layer_bigquery
      method: service-account
      project: [GCP project id]
      dataset: [the name of your dbt dataset]
      threads: [1 or more]
      keyfile: [/path/to/bigquery/keyfile.json]
      layer_api_key: [the API Key to access your Layer account (opt)]

Now, start making predictions directly in your dbt DAG:

select id,
       review,
       layer.predict("layer/nlptown/models/sentimentanalysis", ARRAY[review])
from {{ref('reviews')}}

Examples

Check out the examples we have prepared for you:

• Predicting survials of Titanic - Predicts the survivals of the Titanic disaster.
• Sentiment analysis of product reviews - An example that shows how to make multi-language sentiment analysis.
• Object detection in product images - Detects clothes from product images using a pre-trained computer vision model.
• Review Scores Prediction with AutoML - Train an AutoML model to predict the review scores.

Quick Tour

AutoML

You can automatically build state-of-art ML models using your own dbt models with plain SQL. To train an AutoML model, all you have to do is pass your intended model type, the input data (features), and the target column you want to predict to the layer.automl() SQL function in a dbt model.

Syntax:

layer.automl("MODEL_TYPE", ARRAY[FEATURES], TARGET)

Parameters:

Syntax	Description
MODEL_TYPE	Type of the model your want to train. There are two options: classifier: A model to predict classes/labels or categories such as spam, no-spam (ham) in the classic spam detection model regressor: A model to predict continuous outcomes such as a share price prediction.
FEATURES	Input column names as a list to train your AutoML model.
TARGET	Target column that you want to predict.

Requirements:

• For AutoML to work, you need to add the configuration layer_api_key with your Layer API key into your dbt profile.

Example:

Check out Order Review AutoML Project:

SELECT order_id,
       layer.automl(
           -- This is a regression problem
           'regressor',
           -- Data (input features) to train our model
           ARRAY[
           days_between_purchase_and_delivery, order_approved_late,
           actual_delivery_vs_expectation_bucket, total_order_price, total_order_freight, is_multiItems_order,seller_shipped_late],
           -- Target column we want to predict
           review_score
       )
FROM {{ ref('training_data') }}

Prediction

You can run predictions using any Layer ML model with your dbt models. The Layer dbt Adapter offers a SQL function that helps you score your data within your dbt DAG.

Syntax:

layer.predict("LAYER_MODEL_PATH", ARRAY[FEATURES])

Parameters:

Syntax	Description
LAYER_MODEL_PATH	This is the Layer model path in form of /[organization_name]/[project_name]/models/[model_name]. You can use only the model name if you want to use an AutoML model within the same dbt project.
FEATURES	These are the columns that this model requires to make a prediction. You should pass the columns as a list like ARRAY[column1, column2, column3].

Example:

Check out the Cloth Detection Project:

SELECT
    id,
    layer.predict("layer/clothing/models/objectdetection", ARRAY[image])
FROM
    {{ ref("products") }}

FAQ

1. Do I need a Layer account?

• If you want to use public models from Layer you don't. But if you want to create your own models with AutoML, you can always create your free Layer account.

2. How do I get my layer-api-key?

• First, create your free Layer account.
• Get your Layer API key here