dbt Core
Overview
Monte Carlo integrates with dbt by importing the manifest file and run results file. This will make dbt metadata available within the Monte Carlo dashboard.
Detailed Guide
You will need to use the Monte Carlo CLI to upload the dbt build artifacts whenever a dbt run
or dbt test
command is executed in your production workflow. Follow these steps to complete the integration:
- Install the Monte Carlo CLI on the production machine or image that runs dbt
- Create a Monte Carlo API key
- Modify your production dbt CI/CD workflow to run the CLI
Install the Monte Carlo CLI
The Monte Carlo CLI is installed using pip:
pip install -U montecarlodata
You will need to install this pip library on the same image or machine that runs dbt. In most cases, this will involve updating a Dockerfile to update the machine image with this library.
Create a Monte Carlo API key
If you do not already have a Monte Carlo API key, you can create one in the "Settings" tab on the Monte Carlo dashboard. You will need both the key ID and secret.
Modify your production dbt CI/CD workflow
Whenever dbt run
or dbt test
is executed in your production dbt workflow, you will need to run the CLI directly afterwards. This command needs to be run within the same shell session, because when dbt run
or dbt test
is executed, dbt writes build artifacts to the local filesystem in the target
directory of the dbt project, and the CLI will need direct access to these files.
The CLI command to run are:
MCD_DEFAULT_API_ID=${MCD_DEFAULT_API_ID} \
MCD_DEFAULT_API_TOKEN=${MCD_DEFAULT_API_TOKEN} \
montecarlo import dbt-manifest \
target/manifest.json --project-name <project name>
MCD_DEFAULT_API_ID=${MCD_DEFAULT_API_ID} \
MCD_DEFAULT_API_TOKEN=${MCD_DEFAULT_API_TOKEN} \
montecarlo import dbt-run-results \
target/run_results.json --project-name <project name>
Note: Whenever
dbt run
ordbt test
is run, the command overwrites the existingmanifest.json
andrun_results.json
files. So you will need to run the MC CLI command after each dbt run/test command.
The project name can be set to the name of the dbt project. The project name is associated with all dbt objects imported by Monte Carlo, and is used for disambiguation between different projects and environments. For example, if you are importing from both a staging and production environment, you could set project name to be "Staging models" and "Production models".
Once your production dbt workflow is updated to run these commands, dbt metadata and run results will be available in the Monte Carlo dashboard.
Performing a one-time import of the dbt manifest
It may be useful to perform a one-time import of dbt manifest for testing and evaluation purposes. To do so, simply:
- Download
manifest.json
from a production run of dbt locally. You must use a manifest.json that was generated using the production target. If you use a manifest.json generated using a development target, the tables referenced in the manifest will not match up with production tables stored in Monte Carlo.- Install the Monte Carlo CLI (see detailed instructions here)
- Run:
montecarlo import dbt-manifest \ target/manifest.json --project-name <project name>
Updated 7 days ago