dbt Cloud

📘

Prerequisites

• requires admin privileges for the dbt Cloud account that hosts and runs your production dbt models
• requires Data Collector v5169+

To connect Monte Carlo to a dbt Cloud account, follow these steps:

1. Install the Monte Carlo CLI
2. Obtain a dbt Cloud API token and account ID
3. Create a dbt Cloud integration with the Monte Carlo CLI

Install the Monte Carlo CLI

Please follow this guide to install and configure the CLI.

Obtain a dbt Cloud API token and account ID

Please follow the instructions in dbt Cloud's API documentation to create a dbt Cloud API token. This token will be needed when creating the integration with the Monte Carlo CLI.

The token should have read-only access, unless you elect to use our preferred webhook integration. In that case the token should have permission to manage webhooks in your account (we'll take care of managing the webhook for you).

To obtain your dbt Cloud account ID, sign into dbt Cloud in your browser. Take note of the number directly following the accounts path component of the URL -- this is your account ID. For example, if the URL is https://cloud.getdbt.com/#/accounts/1234/projects/6789/dashboard/, the account ID is 1234.

Create a dbt Cloud integration with the Monte Carlo CLI

1. Choose a warehouse to link with your dbt Cloud:
   • If you only have a single warehouse integrated with Monte Carlo, proceed to step two.
   • Otherwise, you will need to select a warehouse to link with the dbt Cloud integration. Use montecarlo integrations list to list out all available connections, and take note of the ID of the warehouse that is used as the target for your dbt Cloud production models.

📘

EMEA

If you are using dbt cloud in EMEA, you will need to include --dbt-cloud-base-url in the command set to your base URL for the validation to work.

2. Run montecarlo integrations add-dbt-cloud:

$ montecarlo integrations add-dbt-cloud --help
Usage: montecarlo integrations add-dbt-cloud [OPTIONS]

  Setup a dbt Cloud integration.

Options:
  --dbt-cloud-api-token TEXT   dbt Cloud API token. If you prefer a prompt
                               (with hidden input) enter -1.  [required]
  --dbt-cloud-account-id TEXT  dbt Cloud Account ID.  [required]
  --dbt-cloud-base-url TEXT    dbt Cloud Base URL.
  --webhook-hmac-secret        HMAC secret of the dbt webhook that can be
                                                         after successfully creating the webhook in dbt.
  --name TEXT                  Friendly name of the warehouse which the
                               connection will belong to.
  --collector-id UUID          ID for the data collector. To disambiguate
                               accounts with multiple collectors.
  --skip-validation            Skip all connection tests. This option cannot
                               be used with 'validate-only'.
  --validate-only              Run connection tests without adding. This
                               option cannot be used with 'skip-validation'.
  --auto-yes                   Skip any interactive approval.
  --help                       Show this message and exit.

📘

dbt Cloud Webhooks

We recommend using the webhook-based dbt cloud to Monte Carlo integration using the --webhook-hmac-secret option to provide real time notifications that dbt Cloud jobs have completed and run results are available for collection by Monte Carlo. We will add a Monte Carlo integration webhook to your dbt Cloud account, configured by default to trigger for all of your dbt Cloud jobs.

Setup notes:

1. Upgrade Data Collector to v14099+
2. Upgrade Monte Carlo CLI to v0.48+
3. Creating an outbound webhook in the Account Settings view in dbt:
   1. Endpoint URL: https://integrations.getmontecarlo.com/webhooks/dbt-cloud
   2. Events: Only job.run.completed is supported by Monte Carlo.
4. Creating an inbound webhook in Monte Carlo using the Monte Carlo CLI
   1. montecarlo integrations add-dbt-cloud --dbt-cloud-api-token <dbt api token> --dbt-cloud-account-id <dbt account id> --webhook-hmac-secret <webhook hmac secret>

IMPORTANT! Webhooks are not yet fully supported for single-tenant dbt Cloud accounts. If you have a single-tenant account, please reach out to dbt to understand the latest feature availability as well as limitation of webhook integration.

• Monte Carlo will perform a validation step to ensure that the dbt Cloud account can be successfully accessed. If validation passes, you will be prompted to create the dbt Cloud integration. If validation fails, please double-check that you provided the correct API token and account ID.

At this point, the integration has been created. By default, Monte Carlo will poll your dbt Cloud account hourly to import the latest manifests and run results, across all projects and jobs.