Airflow

With the Monte Carlo Airflow Integration, you can be alerted of failures, quickly determine what Airflow DAG potentially caused data-level incidents, and control Airflow with Rules and Circuit Breakers. Root cause and time to resolution can happen faster than ever when you can get visibility between Airflow and your Data Warehouse in a single pane of glass.

Setup and Getting Started

Create the Airflow integration at https://getmontecarlo.com/settings/integrations
Set up Airflow Lineage with the Airflow in Lineage annotation setup documentation.
Set up Airflow Incident and Tasks with the Airflow Incidents and Task Observability callback setup documentation (strongly recommended)
Use Circuit Breakers to control Airflow based on Monte Carlo rules with the Circuit breakers setup documentation (optional)

Airflow DAGs and Tasks in Monte Carlo Lineage

Understand which Airflow Task creates the lineage between two tables in your Data Warehouse. Click into the Airflow icon to see recent DAG & Task Runs for this lineage edge.

To setup Airflow Lineage, follow the Airflow in Lineage setup documentation.

Airflow DAG and Task Observability - Incidents

Use Monte Carlo as a single place to track and route Data Incidents by generating incidents from DAG failures. Check the status and duration of recent Airflow DAGs & Tasks relevant to the table in the Incident.

To setup DAG & Task Run Tracking, follow the Airflow DAG & Task Observability setup documentation. It's strongly recommended to also set up the Airflow in Lineage integration in order to properly correlate DAGs with Tables.

Monte Carlo Notifications of Airflow Failure Incidents

Monte Carlo allows you to surface Airflow failures and errors as Monte Carlo incidents. Among other things, this will enable you to:

Route and receive notifications similar to other Monte Carlo incidents
Analyze the downstream impact of those incidents
Create holistic incident reporting and tooling for all data issues

To set this up:

Create an audience that includes an "Other Notification"

Select Airflow Job Failures as the incident type
Currently, the notification within the Audience must be scoped to All affected data and All importance scores.

Monte Carlo Airflow Operators for Rules & Circuit Breakers

Run Monte Carlo rules or "circuit break” pipelines when data does not meet a set of quality or integrity thresholds. This can be useful for multiple purposes including, but definitely not limited to, checking if data does not meet your requirements between transformation steps, or after ETL/ELT jobs execute, but before BI dashboards are updated.

To setup Airflow Operators or Circuit Breakers, follow the Circuit breakers (beta) setup documentation.

FAQs

Why don't I have to give my Airflow credentials, as with other integrations?

Monte Carlo observes Airflow through existing query log integrations and when callbacks from Airflow report status to Monte Carlo. Monte Carlo does not reach out to gather Airflow information

Are multiple airflow instances supported?

Yes! See here for information on how to support lineage for multiple airflow environments https://docs.getmontecarlo.com/docs/airflow-in-lineage#multiple-airflow-environments

Which Airflow providers are supported?

All providers are supported, including MWAA (AWS), Cloud Composer (GCP) and Astronomer, as well as self-managed installations.

Is something missing? Request more functionality for the Monte Carlo Airflow Integration here.