dbt Cloud

Overview

Monte Carlo collects dbt Cloud runs through a webhook. When a dbt Cloud job run completes, successfully or not, dbt Cloud calls a Monte Carlo webhook, and Monte Carlo fetches the run's artifacts (manifest, run results, and logs) through the dbt Cloud API. Runs appear in Monte Carlo shortly after they finish. For what Monte Carlo does with them, see dbt Integration.

🚧

The webhook is required. Monte Carlo doesn't poll dbt Cloud on a schedule, so runs are collected only when the webhook fires.

Prerequisites

  • Admin privileges for the dbt Cloud account that hosts and runs your production dbt models
  • The warehouse your production dbt models target, connected to Monte Carlo
  • Data Collector v15069+

Step 1: Obtain a dbt Cloud API token and account ID

Please follow the instructions in dbt Cloud's API documentation to create a dbt Cloud API token. This token will be needed when creating the integration. The token only needs read-only access.

📘

Webhook check during validation

When you test the integration, Monte Carlo also reads the webhook through the dbt Cloud API to catch common setup mistakes, such as a disabled webhook or a missing job.run.completed event.

This check needs more access than the integration itself. A read-only token can't read webhooks, so the check fails and the test shows a warning. The warning doesn't mean webhook events fail.

You can ignore the warning. To run the check anyway, test with a token that has webhook access, such as the Developer permission set. After the test passes, you can change the token's permissions to read-only.

To obtain your dbt Cloud account ID, sign into dbt Cloud in your browser. Take note of the number directly following the accounts path component of the URL -- this is your account ID. For example, if the URL is https://cloud.getdbt.com/#/accounts/1234/projects/6789/dashboard/, the account ID is 1234.

Step 2: dbt Cloud Webhooks

dbt Cloud notifies Monte Carlo through an outbound webhook each time a job run completes, and Monte Carlo then collects that run. Monte Carlo verifies each webhook call with the webhook's HMAC secret, so you'll need the webhook's ID and secret in Step 3.

  1. Create an outbound webhook in the Account Settings view in dbt Cloud:

    1. Endpoint URL: in Monte Carlo, go to Settings → Integrations, click Create, and select dbt from the Orchestration section. Your webhook URL is listed in the setup form:

      The same URL is dbt_cloud_webhook_url in the response of the REST API v2 GET /api/v2/hosting/domains, and in the montecarlo_hosting_domains data source of the Terraform provider.

    2. Events: select job.run.completed. Monte Carlo ignores other events. Failed runs also trigger job.run.completed, so Monte Carlo collects failures too.

    3. Jobs: one webhook covers every project in the dbt Cloud account. To send only some runs to this integration, use the event filters in Step 3.

  2. After you save the webhook, copy its webhook ID and HMAC secret. See the dbt documentation on webhooks.

Each dbt Cloud webhook can be used by only one Monte Carlo integration.

Step 3 Option A: Create a dbt Cloud integration through the Monte Carlo UI

You can set up the dbt cloud integration all through the UI.

  1. In Monte Carlo, go to Settings → Integrations, click Create, and select dbt from the Orchestration section.

  2. Fill out the form:

    FieldValue
    Warehouse NameThe warehouse your production dbt models target.
    Cloud API tokenThe token from Step 1.
    Cloud account IDThe account ID from Step 1.
    Webhook HMAC IDThe webhook ID from Step 2.
    Webhook HMAC secretThe HMAC secret from Step 2.
    Cloud base URLOptional. Your dbt Cloud access URL followed by /api/v2, if it isn't https://cloud.getdbt.com. For example, https://emea.dbt.com/api/v2.

  3. Optionally, set Filter events by dbt connection and Filter events by dbt environment (see below).

  4. Click Test dbt Cloud Integration, then save the integration.

📘

Filter events by dbt connection

This option can limit the scope of events that are processed by this integration:

  • If left unchecked (this is the default), all events will be processed.
  • If checked, there's a further choice of automatic matching (default) or matching to a specific set of dbt connections.

Why filter

The dbt Cloud webhooks typically fire on any event within the dbt account, and depending on your dbt Cloud setup you might want to target only some events to this particular integration. For example: you might have have Dev, Stage, and Prod warehouses in your dbt account, and this particular integration is meant for Prod warehouse only. In this scenario, you would need to exclude the events originating in Dev and Stage warehouses.

Auto-matching

In this mode, Monte Carlo will inspect connections details for each webhook event and attempt to match those details with the existing Monte Carlo connections to the same warehouse. It should work reliably in most cases. Specifically, it's a good option for Snowflake and BigQuery warehouses because they do not rely on hostname matching.

Specifying dbt connections

In this mode, Monte Carlo will only process events whose connections match a given list of dbt connections (specified either with names like Redshift Prod 1 or with IDs like 4321). This is the most reliable option, and it will work even in cases where auto-matching fails. Specifically, this can be the only viable option for warehouses that (a) match by hostname (Redshift, Databricks, and so on) and (b) have private links or custom DNS setups, so that Monte Carlo and dbt Cloud use different hostnames to reach the same warehouse. Choose this mode if your runs are missing or are mis-attributed to the wrong warehouse.

To match connection on a field other than the name or ID, open Advanced and edit Connection fields. Use dbt Cloud API field names, such as:

  • account (Snowflake)
  • hostname (Redshift)
  • host (Databricks)
  • client_id (BigQuery)
  • database

An event is processed when any listed field on its dbt connection equals any listed value. A dbt connection without the field does not match.

Note: if specifying dbt Cloud connections, copy names, IDs, or other values exactly as they appear in dbt Cloud UI or API. Matching is case-sensitive.

📘

Filter events by dbt environment

To process only runs from certain dbt Cloud environments, list their names. Matching ignores case. Leave it empty to process every environment.

Now your dbt integration is set up and ready to go. Run a job in dbt Cloud, then look for it on the Assets page.

Step 3 Option B: Create a dbt Cloud integration with the Monte Carlo CLI

  1. Install or upgrade the Monte Carlo CLI: pip install -U montecarlodata
  2. Choose a warehouse to link with your dbt Cloud:
    • If you only have a single warehouse integrated with Monte Carlo, proceed to the next step.
    • Otherwise, you will need to select a warehouse to link with the dbt Cloud integration. Use montecarlo integrations list to list out all available connections, and take note of the ID of the warehouse that is used as the target for your dbt Cloud production models.
📘

EMEA and Single Tenant Setups

If your dbt Cloud access URL isn't https://cloud.getdbt.com (for example, a single-tenant or EMEA account), include --dbt-cloud-base-url set to your access URL followed by /api/v2. Please refer to the dbt documentation to find it.

Note: Make sure to include the api and version in the URL you provide (always use v2). For example, for the URL https://cloud.getdbt.com, you should provide https://cloud.getdbt.com/api/v2.

Note: If you are using dbt cloud in EMEA, please use https://emea.dbt.com/api/v2.

  1. Run montecarlo integrations add-dbt-cloud:
$ montecarlo integrations add-dbt-cloud --help
Usage: montecarlo integrations add-dbt-cloud [OPTIONS]

  Setup a dbt Cloud integration.

Options:
  --dbt-cloud-api-token TEXT   dbt Cloud API token. If you prefer a prompt
                               (with hidden input) enter -1.  [required]
  --dbt-cloud-account-id TEXT  dbt Cloud Account ID.  [required]
  --dbt-cloud-base-url TEXT    dbt Cloud Base URL.
  --webhook-hmac-secret        HMAC secret of the dbt webhook obtained
    													 after successfully creating the webhook in dbt. [required]
  --webhook-id                 ID of the webhook obtained after successfully 
                               creating the webhook in dbt.[required]
  --name TEXT                  Friendly name of the warehouse which the
                               connection will belong to.
  --collector-id UUID          ID for the data collector. To disambiguate
                               accounts with multiple collectors.
  --skip-validation            Skip all connection tests. This option cannot
                               be used with 'validate-only'.
  --validate-only              Run connection tests without adding. This
                               option cannot be used with 'skip-validation'.
  --auto-yes                   Skip any interactive approval.
  --help                       Show this message and exit.
  • montecarlo integrations add-dbt-cloud --dbt-cloud-api-token <dbt api token> --dbt-cloud-account-id <dbt account id> --webhook-hmac-secret <webhook hmac secret> --webhook-id <webhook id>
  • See section the above on how to create the Webhook
  • Monte Carlo will perform a validation step to ensure that the dbt Cloud account can be successfully accessed. If validation passes, you will be prompted to create the dbt Cloud integration. If validation fails, please double-check that you provided the correct API token and account ID.

At this point, the integration has been created.

Step 3 Option C: Create a dbt Cloud integration with Terraform

🚧

Beta

The Terraform provider and the REST API v2 are in beta.

With the Terraform provider, the integration is two resources: montecarlo_dbt_cloud_webhook_credentials stores the token and the webhook's HMAC secret, and montecarlo_connection adds the connection to the warehouse your dbt Cloud production models target. The warehouse must already exist in Monte Carlo. Together with the dbt Cloud provider, one configuration can create the webhook as well:

data "montecarlo_hosting_domains" "this" {}

resource "dbtcloud_webhook" "monte_carlo" {
  name        = "Monte Carlo"
  client_url  = data.montecarlo_hosting_domains.this.dbt_cloud_webhook_url
  event_types = ["job.run.completed"]
}

resource "montecarlo_dbt_cloud_webhook_credentials" "dbt" {
  account_id               = var.dbt_cloud_account_id
  webhook_id               = dbtcloud_webhook.monte_carlo.id
  service_token_wo         = var.dbt_cloud_service_token
  service_token_wo_version = 1
  hmac_secret_wo           = dbtcloud_webhook.monte_carlo.hmac_secret
  hmac_secret_wo_version   = 1
}

resource "montecarlo_connection" "dbt" {
  name           = "dbt Cloud"
  warehouse_id   = var.warehouse_id
  credentials_id = montecarlo_dbt_cloud_webhook_credentials.dbt.id

  dbt_cloud = {
    connection_match          = "allow_list"
    allowed_connection_fields = ["name", "id"]
    allowed_connection_values = ["Snowflake Prod"]
    allowed_environments      = []
  }
}
  • If your dbt Cloud access URL is not https://cloud.getdbt.com, also set base_url on the credentials: your access URL followed by /api/v2, such as https://emea.dbt.com/api/v2.
  • The optional dbt_cloud block filters events, with the choices described in Filter events by dbt connection above:
    • connection_match: off processes every event and is the default. auto matches events automatically. allow_list processes only events whose dbt connection matches allowed_connection_fields and allowed_connection_values.
    • allowed_connection_fields: the dbt connection fields to compare, using dbt Cloud API field names, such as name, id, or account. At least one for allow_list, and empty for the other modes.
    • allowed_connection_values: the values to match, exactly as they appear in dbt Cloud. A run is processed when one of allowed_connection_fields on its dbt connection equals one of them. At least one for allow_list, and empty for the other modes.
    • allowed_environments: dbt environment names, ignoring case. When it lists any, Monte Carlo processes only runs from those environments. Leave it empty to process every environment.
  • allowed_connection_fields and allowed_connection_values need provider v0.1.30 or later. They replace allowed_connections. To upgrade, move its entries to allowed_connection_values and set allowed_connection_fields = ["name", "id"].
  • Removing the dbt_cloud block keeps the filters as they are. To stop filtering, set connection_match = "off".
  • The Monte Carlo provider takes both secrets as write-only arguments, and its resources keep neither in state. The dbt Cloud provider does keep hmac_secret in its own state. To send Monte Carlo a new token or HMAC secret, bump its _version argument.

The REST API v2 takes the same steps: create the credentials with POST /api/v2/credentials/dbt-cloud-webhook, then the connection with POST /api/v2/connections, with the same fields as above. The REST API v2 reference documents both requests.

Step 4 (Optional): Query Tagging To Get Performance Features

Setup in this section enables MC to link your queries to dbt objects to provide additional features from the integration:

  • you will be able to filter for dbt jobs and models in the performance dashboard to drill into performance issues for a dbt job or model of interest
  • you will be able to identify the dbt job run for each query run from the performance dashboard details drawer
  • you will be able to leverage a dbt job gantt view in assets page to identify problematic models and queries in a job, then leverage MC's query-level metadata to investigate what caused the performance problems

See how you can use those features here

To achieve those, configure your dbt project with the following macro to add additional query comments.

  1. In your dbt project, create a new file macros/query_comment.sql with these contents:
{% macro query_comment(node) %}
    {%- set comment_dict = {} -%}
    {%- do comment_dict.update(
        app='dbt',
        dbt_version=dbt_version,
        profile_name=target.get('profile_name'),
        target_name=target.get('target_name'),
        invocation_id=invocation_id,
        dbt_cloud_project_id=env_var('DBT_CLOUD_PROJECT_ID', ''),
        dbt_cloud_job_id=env_var('DBT_CLOUD_JOB_ID', ''),
        dbt_cloud_run_id=env_var('DBT_CLOUD_RUN_ID', ''),
        dbt_cloud_pr_id=env_var('DBT_CLOUD_PR_ID', ''),
        dbt_cloud_git_sha=env_var('DBT_CLOUD_GIT_SHA', '')
    ) -%}
    {%- if node is not none -%}
      {%- do comment_dict.update(
        file=node.original_file_path,
        node_id=node.unique_id,
        node_name=node.name,
        resource_type=node.resource_type,
        package_name=node.package_name,
        relation={
            "database": node.database,
            "schema": node.schema,
            "identifier": node.identifier
        }
      ) -%}
    {% else %}
      {%- do comment_dict.update(node_id='internal') -%}
    {%- endif -%}
    {% do return(tojson(comment_dict)) %}
{% endmacro %}
  1. Update dbt_project.yml to include a query-comment configuration that references the macro you just added in step (1):
query-comment:
	comment: "{{ query_comment(node) }}"
	append: true # Required for Snowflake
📘

query-comment

The append: true property is REQUIRED for Snowflake, which removes leading SQL comments from queries.


Network Access Control

You can restrict access to the dbt Cloud webhook endpoint by IP address using Network Access Control Lists. See Network Access Control.

Q&A

Q: I've set up the integration following the docs. Why don't I see my dbt projects showing up on the settings page for the dbt integration?

A: A dbt project will not show up in UI until after a dbt job under that project runs, so you will see the expected projects populating in the UI overtime as their jobs run.

Q: How long does it take to see my dbt models and jobs on Assets page?

A: dbt models and jobs show up as assets shortly after the dbt job run completes and the webhook fires. dbt context in lineage can take up to 24 hours more due to batch processing.

Q: My dbt Cloud runs aren't showing up. What should I check?

A: In order of likelihood:

  1. The webhook isn't firing. Check in dbt Cloud that the webhook is active, subscribes to job.run.completed, and points to your Monte Carlo webhook URL. Without it, Monte Carlo collects nothing.
  2. The webhook ID or HMAC secret is wrong. Monte Carlo ignores calls it can't verify. Edit the integration with the webhook's current ID and secret.
  3. An event filter excludes the run. Check Filter events by dbt connection and Filter events by dbt environment.
  4. The models don't match tables in Monte Carlo. Check that the warehouse the job writes to is the one linked to this integration and that its schemas aren't excluded.

Q: Can I use dbt Cloud with a hybrid deployment?

A: Yes, with an agent hosted in AWS, GCP or Azure (agent version 445 or later). Snowflake-hosted and generic agents can't run dbt Cloud. dbt Cloud still sends its webhooks to Monte Carlo. Monte Carlo then has your agent collect the run, so the agent calls the dbt Cloud API outbound; nothing needs to reach the agent from dbt Cloud. If your account has more than one deployment, choose the one to use in Collector UUID when you create the integration.

Q: Why does the integration fail with "Only self-hosted credentials are supported"?

A: The integration is using a deployment whose agent only accepts credentials stored in your own environment, such as a Snowflake-hosted or generic agent. Monte Carlo always stores dbt Cloud credentials itself, so those deployments can't run dbt Cloud. Create the integration on another deployment using Collector UUID, such as one with no agent or with an agent hosted in AWS, GCP or Azure.

Q: What is the largest manifest.json that can be processed by Monte Carlo?
A: 300 MB, this should contemplate the vast majority of the dbt jobs. If for any reason you need larger manifest.json to be processed, please ask Bayes and we can evaluate your use case.


Did this page help you?