dbt Core

Overview

Monte Carlo integrates with dbt by importing the manifest file and run results file. This will make dbt metadata available within the Monte Carlo dashboard.

Detailed Guide

You will need to use the Monte Carlo CLI to upload the dbt build artifacts whenever a dbt run or dbt test command is executed in your production workflow. Follow these steps to complete the integration:

1. Install the Monte Carlo CLI on the production machine or image that runs dbt
2. Create a Monte Carlo API key
3. Modify your production dbt CI/CD workflow to run the CLI

Install the Monte Carlo CLI

The Monte Carlo CLI is installed using pip:

pip install -U montecarlodata

You will need to install this pip library on the same image or machine that runs dbt. In most cases, this will involve updating a Dockerfile to update the machine image with this library.

Create a Monte Carlo API key

If you do not already have a Monte Carlo API key, you can create one in the "Settings" tab on the Monte Carlo dashboard. You will need both the key ID and secret.

Modify your production dbt CI/CD workflow

Whenever dbt run or dbt test is executed in your production dbt workflow, you will need to run the CLI directly afterwards. This command needs to be run within the same shell session, because when dbt run or dbt test is executed, dbt writes build artifacts to the local filesystem in the target directory of the dbt project, and the CLI will need direct access to these files.

The CLI command to run are:

MCD_DEFAULT_API_ID=${MCD_DEFAULT_API_ID} \
  MCD_DEFAULT_API_TOKEN=${MCD_DEFAULT_API_TOKEN} \        
  montecarlo import dbt-manifest \
  target/manifest.json --project-name <project name>

MCD_DEFAULT_API_ID=${MCD_DEFAULT_API_ID} \
  MCD_DEFAULT_API_TOKEN=${MCD_DEFAULT_API_TOKEN} \        
  montecarlo import dbt-run-results \
  target/run_results.json --project-name <project name>

🚧

Note: Whenever dbt run or dbt test is run, the command overwrites the existing manifest.json and run_results.json files. So you will need to run the MC CLI command after each dbt run/test command.

The project name can be set to the name of the dbt project. The project name is associated with all dbt objects imported by Monte Carlo, and is used for disambiguation between different projects and environments. For example, if you are importing from both a staging and production environment, you could set project name to be "Staging models" and "Production models".

Once your production dbt workflow is updated to run these commands, dbt metadata and run results will be available in the Monte Carlo dashboard.

📘

Performing a one-time import of the dbt manifest

It may be useful to perform a one-time import of dbt manifest for testing and evaluation purposes. To do so, simply:

1. Download manifest.json from a production run of dbt locally. You must use a manifest.json that was generated using the production target. If you use a manifest.json generated using a development target, the tables referenced in the manifest will not match up with production tables stored in Monte Carlo.
2. Install the Monte Carlo CLI (see detailed instructions here)
3. Run:

   montecarlo import dbt-manifest \
     target/manifest.json --project-name <project name>