Airflow

With the Monte Carlo Airflow Integration, you can be alerted of failures, quickly determine what Airflow DAG potentially caused data-level issues, and control Airflow with Rules and Circuit Breakers. Incident investigation and resolution can happen faster than ever when you can get visibility between Airflow and your Data Warehouse in a single pane of glass.

Setup and Getting Started

1. Create the Airflow integration at https://getmontecarlo.com/settings/integrations
2. Set up Airflow Lineage with the Airflow in Lineage annotation setup documentation.
3. Set up Airflow Alerts and Tasks with the Airflow alerts and Task Observability callback setup documentation (strongly recommended)
4. Use Circuit Breakers to control Airflow based on Monte Carlo rules with the Circuit breakers setup documentation (optional)

Airflow DAGs and Tasks in Monte Carlo Lineage

Understand which Airflow Task creates the lineage between two tables in your Data Warehouse. Click into the Airflow icon to see recent DAG & Task Runs for this lineage edge.

To setup Airflow Lineage, follow the Airflow in Lineage setup documentation.

Airflow DAG and Task Observability - Alerts

Use Monte Carlo as a single pane of glass for all data quality context for each table, including Airflow DAG/task runs. Check the status and duration of recent Airflow DAGs & Tasks relevant to a table of interest.

To setup DAG & Task Run Tracking, follow the Airflow DAG & Task Observability setup documentation. It's strongly recommended to also set up the Airflow in Lineage integration in order to properly correlate DAGs with Tables.

Monte Carlo Notifications of Airflow Failure Alerts

Monte Carlo allows you to surface Airflow failures and errors as Monte Carlo alerts. Among other things, this will enable you to:

• Route and receive notifications similar to other Monte Carlo alerts
• Analyze the downstream impact of those alerts
• Create holistic incident reporting and tooling for all data issues

To set this up:

• Create an audience that includes an "Other Notification"

• Select Airflow Job Failures as the alert type
• Under "Affected Data", select "Databases, schemas, tables, jobs, and tags" and add the Airflow DAGs of interest to include in this audience. You can also use tags to include all DAGs with the tag to the audience.

If you have a domain that already selected the desired Airflow DAGs, you can also select that domain as "Affected Data" to include alerts for those DAGs under the audience.

If "All" is selected as "affected data" for an audience, alerts for all Airflow DAGs will be included in this audience.

Airflow DAG tags

Monte Carlo allows you to leverage tags for Airflow DAG to ensure that teams only see the relevant incidents and jobs they need. You can use DAG tags to include only the tagged DAGs in an audience, you can also use tags to include only the tagged DAGs in a domain. Newly added DAGs under the tag are automatically included in the given audience or domain.

Tags from Airflow can be automatically imported into Monte Carlo. The tags will be collected on every DAG callback triggered. Ensure that the airflow-mcd package you are running is version 0.3.6 or above to import the tags.

Monte Carlo Airflow Operators for Rules & Circuit Breakers

Run Monte Carlo rules or "circuit break” pipelines when data does not meet a set of quality or integrity thresholds. This can be useful for multiple purposes including, but definitely not limited to, checking if data does not meet your requirements between transformation steps, or after ETL/ELT jobs execute, but before BI dashboards are updated.

To setup Airflow Operators or Circuit Breakers, follow the Circuit breakers setup documentation.

FAQs

Why don't I have to give my Airflow credentials, as with other integrations?

Monte Carlo observes Airflow through existing query log integrations and when callbacks from Airflow report status to Monte Carlo. Monte Carlo does not reach out to gather Airflow information

Are multiple airflow instances supported?

Yes! See here for information on how to support lineage for multiple airflow environments https://docs.getmontecarlo.com/docs/airflow-in-lineage#multiple-airflow-environments

Which Airflow providers are supported?

All providers are supported, including MWAA (AWS), Cloud Composer (GCP) and Astronomer, as well as self-managed installations.

How long does it take for Airflow data to show up in Monte Carlo?

DAG and Task run data will be immediately available in the Asset Page and Incidents. However, Airflow data in Lineage and the Catalog may have delays of up to 24 hours due to batch processing.

Does Monte Carlo support alerting or observability for SLA misses?

Currently, Monte Carlo does not support alerting or observability for SLA misses. If you are interested in having this functionality, please let your Monte Carlo representative know, or contact [email protected].

What airflow-mcd version supports tags importing?

Version 0.3.6is the version that supports tags importing.

Is something missing? Request more functionality for the Monte Carlo Airflow Integration here.

How long do Airflow jobs stay visible on the Assets page if they don’t have new runs?

Jobs will remain listed on the Assets page as long as they have at least one run in the past seven days. If a job has had no runs for more than seven days, it will still be stored in our database—but it won’t appear on the Assets page until a new run occurs. Once you trigger a new run, the job and all of its past runs will automatically reappear.