Create PagerDuty alarms from failed dbt Cloud tasks
This guide will teach you how to build and host a basic Python app which will monitor dbt Cloud jobs and create PagerDuty alarms based on failure. To do this, when a dbt Cloud job completes it will:
- Check for any failed nodes (e.g. non-passing tests or errored models), and
- create a PagerDuty alarm based on those nodes by calling the PagerDuty Events API. Events are deduplicated per run ID.
In this example, we will use fly.io for hosting/running the service. fly.io is a platform for running full stack apps without provisioning servers etc. This level of usage should comfortably fit inside of the Free tier. You can also use an alternative tool such as AWS Lambda or Google Cloud Run.
This guide assumes some familiarity with:
- dbt Cloud Webhooks
- CLI apps
- Deploying code to a serverless code runner like fly.io or AWS Lambda
1. Clone the
This repository contains the sample code for validating a webhook and creating events in PagerDuty.
flyctl and sign up for fly.io
Follow the directions for your OS in the fly.io docs, then from your command line, run the following commands:
Switch to the directory containing the repo you cloned in step 1:
#example: replace with your actual path
Sign up for fly.io:
flyctl auth signup
Your console should show
successfully logged in as YOUR_EMAIL when you're done, but if it doesn't then sign in to fly.io from your command line:
flyctl auth login
3. Launch your fly.io app
Launching your app publishes it to the web and makes it ready to catch webhook events:
You will see a message saying that an existing
fly.toml file was found. Type
y to copy its configuration to your new app.
Choose an app name of your choosing, such as
YOUR_COMPANY-dbt-cloud-webhook-pagerduty, or leave blank and one will be generated for you. Note that your name can only contain numbers, lowercase letters and dashes.
Choose a deployment region, and take note of the hostname that is generated (normally
When asked if you would like to set up Postgresql or Redis databases, type
n for each.
y when asked if you would like to deploy now.
Sample output from the setup wizard:
joel@Joel-Labes dbt-cloud-webhooks-pagerduty % flyctl launch
An existing fly.toml file was found for app dbt-cloud-webhooks-pagerduty
? Would you like to copy its configuration to the new app? Yes
Creating app in /Users/joel/Documents/GitHub/dbt-cloud-webhooks-pagerduty
Scanning source code
Detected a Dockerfile app
? Choose an app name (leave blank to generate one): demo-dbt-cloud-webhook-pagerduty
automatically selected personal organization: Joel Labes
Some regions require a paid plan (fra, maa).
See https://fly.io/plans to set up a plan.
? Choose a region for deployment: [Use arrows to move, type to filter]
? Choose a region for deployment: Sydney, Australia (syd)
Created app dbtlabs-dbt-cloud-webhook-pagerduty in organization personal
Admin URL: https://fly.io/apps/demo-dbt-cloud-webhook-pagerduty
? Would you like to set up a Postgresql database now? No
? Would you like to set up an Upstash Redis database now? No
Wrote config file fly.toml
? Would you like to deploy now? Yes
4. Create a PagerDuty integration application
See PagerDuty's guide for full instructions.
Make note of the integration key for later.
5. Configure a new webhook in dbt Cloud
See Create a webhook subscription for full instructions. Your event should be Run completed.
Set the webhook URL to the host name you created earlier (
Make note of the Webhook Secret Key for later.
Do not test the endpoint; it won't work until you have stored the auth keys (next step)
6. Store secrets
The application requires three secrets to be set, using these names:
DBT_CLOUD_SERVICE_TOKEN: a dbt Cloud user token or service account token with at least the
DBT_CLOUD_AUTH_TOKEN: the Secret Key for the dbt Cloud webhook you created earlier.
PD_ROUTING_KEY: the integration key for the PagerDuty integration you created earlier.
Set these secrets as follows, replacing
abc123 etc with actual values:
flyctl secrets set DBT_CLOUD_SERVICE_TOKEN=abc123 DBT_CLOUD_AUTH_TOKEN=def456 PD_ROUTING_KEY=ghi789
7. Deploy your app
After you set your secrets, fly.io will redeploy your application. When it has completed successfully, go back to the dbt Cloud webhook settings and click Test Endpoint.