This guide will teach you how to build and host a basic Python app which will add dbt Cloud job events to Datadog. To do this, when a dbt Cloud job completes it will create a log entry for each node that was run, containing all information about the node provided by the Discovery API.
In this example, we will use fly.io for hosting/running the service. fly.io is a platform for running full stack apps without provisioning servers etc. This level of usage should comfortably fit inside of the Free tier. You can also use an alternative tool such as AWS Lambda or Google Cloud Run.
This guide assumes some familiarity with:
- dbt Cloud Webhooks
- CLI apps
- Deploying code to a serverless code runner like fly.io or AWS Lambda
This repository contains the sample code for validating a webhook and creating logs in Datadog.
flyctl and sign up for fly.io
Follow the directions for your OS in the fly.io docs, then from your command line, run the following commands:
Switch to the directory containing the repo you cloned in step 1:
#example: replace with your actual path
Sign up for fly.io:
flyctl auth signup
Your console should show
successfully logged in as YOUR_EMAIL when you're done, but if it doesn't then sign in to fly.io from your command line:
flyctl auth login
Launch your fly.io app
Launching your app publishes it to the web and makes it ready to catch webhook events:
You will see a message saying that an existing
fly.tomlfile was found. Type
yto copy its configuration to your new app.
Choose an app name of your choosing, such as
YOUR_COMPANY-dbt-cloud-webhook-datadog, or leave blank and one will be generated for you. Note that your name can only contain numbers, lowercase letters and dashes.
Choose a deployment region, and take note of the hostname that is generated (normally
When asked if you would like to set up Postgresql or Redis databases, type
ywhen asked if you would like to deploy now.
Sample output from the setup wizard:
joel@Joel-Labes dbt-cloud-webhooks-datadog % flyctl launch
An existing fly.toml file was found for app dbt-cloud-webhooks-datadog
? Would you like to copy its configuration to the new app? Yes
Creating app in /Users/joel/Documents/GitHub/dbt-cloud-webhooks-datadog
Scanning source code
Detected a Dockerfile app
? Choose an app name (leave blank to generate one): demo-dbt-cloud-webhook-datadog
automatically selected personal organization: Joel Labes
Some regions require a paid plan (fra, maa).
See https://fly.io/plans to set up a plan.
? Choose a region for deployment: [Use arrows to move, type to filter]
? Choose a region for deployment: Sydney, Australia (syd)
Created app dbtlabs-dbt-cloud-webhook-datadog in organization personal
Admin URL: https://fly.io/apps/demo-dbt-cloud-webhook-datadog
? Would you like to set up a Postgresql database now? No
? Would you like to set up an Upstash Redis database now? No
Wrote config file fly.toml
? Would you like to deploy now? Yes
4. Create a Datadog API Key
Create an API Key for your Datadog account and make note of it and your Datadog site (e.g.
datadoghq.com) for later.
Configure a new webhook in dbt Cloud
- See Create a webhook subscription for full instructions. Your event should be Run completed.
- Set the webhook URL to the host name you created earlier (
- Make note of the Webhook Secret Key for later.
Do not test the endpoint; it won't work until you have stored the auth keys (next step)
The application requires four secrets to be set, using these names:
DBT_CLOUD_SERVICE_TOKEN: a dbt Cloud user token or service account token with at least the
DBT_CLOUD_AUTH_TOKEN: the Secret Key for the dbt Cloud webhook you created earlier.
DD_API_KEY: the API key you created earlier.
DD_SITE: The Datadog site for your organisation, e.g.
Set these secrets as follows, replacing
abc123 etc with actual values:
flyctl secrets set DBT_CLOUD_SERVICE_TOKEN=abc123 DBT_CLOUD_AUTH_TOKEN=def456 DD_API_KEY=ghi789 DD_SITE=datadoghq.com
Deploy your app
After you set your secrets, fly.io will redeploy your application. When it has completed successfully, go back to the dbt Cloud webhook settings and click Test Endpoint.