Minimum dbt version
This integration requires dbt v0.22+.
Monte Carlo integrates with dbt by importing the manifest file, run results file, and optionally, a file containing the logs generated by dbt during command execution. This will make dbt metadata available within the Monte Carlo dashboard.
You will need to use the Monte Carlo CLI to upload the dbt run artifacts whenever a
dbt run, or
dbt test command is executed in your production workflow. Follow these steps to complete the integration:
- Install the Monte Carlo CLI on the production machine or image that runs dbt
- Create a Monte Carlo API key
- Modify your production dbt CI/CD workflow to run the CLI
Install the Monte Carlo CLI
The Monte Carlo CLI is installed using pip:
pip install -U montecarlodata
You will need to install this pip library on the same image or machine that runs dbt. In most cases, this will involve updating a Dockerfile to update the machine image with this library.
Create a Monte Carlo API key
If you do not already have a Monte Carlo API key, you can create one in the "Settings" tab on the Monte Carlo dashboard. You will need both the key ID and secret.
The key is required to be generated with the "Editor" or "Owner" roles, for example if you create a Service Account Key you need to select "Editors" or "Account Owners" under "Authorization Groups". If you're using a personal key, the user that generated it needs to be an "Editor" or "Owner".
Modify your production dbt CI/CD workflow (⚠️ updated)
dbt run, or
dbt test is executed in your production dbt workflow, you will need to run the CLI directly afterwards. This command needs to be run within the same shell session, because when these dbt commands are executed, dbt writes build artifacts to the local filesystem in the
target directory of the dbt project, and the CLI will need direct access to these files.
Note: Whenever a dbt command is executed, files in the
targetdirectory will be overwritten. So you will need to run the MC CLI command after each dbt command.
The CLI command to run is (⚠️ updated):
% montecarlo import dbt-run --help Usage: montecarlo import dbt-run [OPTIONS] Import dbt run artifacts. Options: --project-name TEXT Project name (perhaps a logical group of dbt models, analogous to a project in dbt Cloud) [default: default-project] --job-name TEXT Job name (perhaps a logical sequence of dbt executions, analogous to a job in dbt Cloud) [default: default-job] --manifest PATH Path to the dbt manifest file (manifest.json) [required] --run-results PATH Path to the dbt run results file (run_results.json) [required] --logs PATH Path to a file containing dbt run logs --connection-id UUID Identifier of warehouse or lake connection to use to resolve dbt models to tables. Required if you have more than one warehouse or lake connection. --help Show this message and exit.
Deprecation Warning (Migration Deadline 6/30/2023)
import dbt-runcommand was introduced in CLI v0.34.0 on 12/1/2022, and is intended to replace the following commands available in previous versions of the CLI:
These two commands are still available today but will be deprecated on 7/1/2023.
The project name is a descriptive name for your dbt project. It will be associated with all dbt objects imported by Monte Carlo, and can be used to disambiguate results from different dbt projects. If a project name is not provided, a
default-project will be created.
The job name can be used to logically group sets of dbt run results within a project. For example, if you orchestrate subsets of models within a project to be run with independent dbt commands, you could associate each subset/command with a unique job name. If a job name is not provided, a
default-job will be created.
To send the console logging produced by dbt to Monte Carlo, you will need to first write it to a file. There are several ways to do this, below is an example using the
dbt build | tee logs.txt
This command would still produce the normal dbt console output, but also copy that output to
logs.txt. You can then use the
--logs option to send that file to Monte Carlo.
Warning on the tee command and Job Failures
If you are using Airflow to monitor failures, it is important to note that the above command will not show the dbt build job has failed if the logs are successfully written via the tee command. When piping commands like this the exit code of the full pipeline is the exit code of the last command.
If the above warning is a concern you can use the following variation of the command, which will return any failed exit code as the exit code for the full pipeline:
set -o pipefail && dbt build | tee logs.txt
Note: dbt transformations run via Fivetran are currently not supported in this integration.
Updated 2 months ago