dbt Integration

What is dbt?

dbtΒ (data build tool) is a tool that enables data analysts and engineers to transform data in their warehouses more effectively. It is the t (transform) in ELT (Extract, Load, Transform).

dbt lets teams quickly and collaboratively deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation.

dbt comes in two flavors. dbt Core is an open source command line tool where scheduling jobs relies on external tools like Airflow, CI or Github. dbt Cloud is a managed service that allows more advanced scheduling right from the web UI. Both Core and Cloud are supported by Monte Carlo.

Why connect dbt to Monte Carlo?

Monte Carlo offers multiple insights into your dbt environment and can show the run history of your dbt jobs:

Monte Carlo also associates dbt Models with the corresponding table that the transformations create in your environment. In the screenshot below you can see how Monte Carlo displays information for a model:

Here we can see vital information on a dbt Model such as the Name and Path to that model. Clicking the View Model in the upper right corner will also display the SQL that defines it.

We can also see when the model was last run, and if the Model runs resulted in any errors, failures, warnings or successes. We can also see if there have been any tests of the model that have passed or failed.

The graph shows the execution time of the model, and can be toggled to show the execution of test runs as well.

Below the graph is a breakdown of the job runs that affected this model as well as any logs that are associated with it:

dbt Failures As Monte Carlo Incidents

Additionally, Monte Carlo allows you to generate incidents based on dbt model errors and test errors . See below video for more details on how to enable that feature and what it looks like live!

Setting Up dbt

The following guide explains how to set up the integration.If you are using dbt Core,Β this guide will help you through set up. We are able to build our dbt Core integration into most existing CI/CD pipelines with pycarlo as well as Airflow.

If you are using dbt Cloud, this guide will help you through set up.