Minimum dbt version
This integration requires dbt v0.22+.
Monte Carlo integrates with dbt by importing the manifest file, run results file, and optionally, a file containing the logs generated by dbt during command execution. This will make dbt metadata available within the Monte Carlo dashboard.
You will need to use the Monte Carlo CLI to upload the dbt run artifacts whenever a
dbt run, or
dbt test command is executed in your production workflow. Follow these steps to complete the integration:
- Install the Monte Carlo CLI on the production machine or image that runs dbt
- Create a Monte Carlo API key
- Modify your production dbt CI/CD workflow to run the CLI
The Monte Carlo CLI is installed using pip:
pip install -U montecarlodata
You will need to install this pip library on the same image or machine that runs dbt. In most cases, this will involve updating a Dockerfile to update the machine image with this library.
If you do not already have a Monte Carlo API key, you can create one in the "Settings" tab on the Monte Carlo dashboard. You will need both the key ID and secret.
The key is required to be generated with the "Editor" or "Owner" roles, for example if you create a Service Account Key you need to select "Editors" or "Account Owners" under "Authorization Groups". If you're using a personal key, the user that generated it needs to be an "Editor" or "Owner".
dbt run, or
dbt test is executed in your production dbt workflow, you will need to run the CLI directly afterwards. This command needs to be run within the same shell session, because when these dbt commands are executed, dbt writes build artifacts to the local filesystem in the
target directory of the dbt project, and the CLI will need direct access to these files.
Note: Whenever a dbt command is executed, files in the
targetdirectory will be overwritten. So you will need to run the MC CLI command after each dbt command.
The CLI command to run is (⚠️ updated):
% montecarlo import dbt-run --help Usage: montecarlo import dbt-run [OPTIONS] Import dbt run artifacts. Options: --project-name TEXT Project name (perhaps a logical group of dbt models, analogous to a project in dbt Cloud) [default: default-project] --job-name TEXT Job name (perhaps a logical sequence of dbt executions, analogous to a job in dbt Cloud) [default: default-job] --manifest PATH Path to the dbt manifest file (manifest.json) [required] --run-results PATH Path to the dbt run results file (run_results.json) [required] --logs PATH Path to a file containing dbt run logs --connection-id UUID Identifier of warehouse or lake connection to use to resolve dbt models to tables. Required if you have more than one warehouse or lake connection. --help Show this message and exit.
import dbt-runcommand is introduced in CLI v0.34.0 on 12/1/2022, and is intended to replace the following commands available in previous versions of the CLI:
These two commands are still available, but will be eventually removed in a future release.
The project name is a descriptive name for your dbt project. It will be associated with all dbt objects imported by Monte Carlo, and can be used to disambiguate results from different dbt projects. If a project name is not provided, a
default-project will be created.
The job name can be used to logically group sets of dbt run results within a project. For example, if you orchestrate subsets of models within a project to be run with independent dbt commands, you could associate each subset/command with a unique job name. If a job name is not provided, a
default-job will be created.
To send the console logging produced by dbt to Monte Carlo, you will need to first write it to a file. There are several ways to do this, below is an example using the
dbt build | tee logs.txt
This command would still produce the normal dbt console output, but also copy that output to
logs.txt. You can then use the
--logs option to send that file to Monte Carlo.
Note: dbt transformations run via Fivetran are currently not supported in this integration.
Updated 11 days ago