Azure Data Factory
Overview
Monte Carlo integrates with Azure Data Factory (ADF) to provide data lineage, run history, and pipeline failure alerts from your factories. Monte Carlo collects your pipelines, activities, and their runs, and links each pipeline to the tables it reads from and writes to. You can see which ADF pipeline produced a table, how its recent runs went, and whether a failed run explains an anomaly downstream.
When an ADF pipeline fails, Monte Carlo can raise it as an alert, so you get centralized incident triage, notification routing, and reporting across all your data and system issues. You can optionally configure an Azure alert that calls a Monte Carlo webhook, so failures are detected immediately rather than on the next hourly collection.
Feature Support
| Category | Capability | Support |
|---|---|---|
| Pipelines | Pipeline failure alerts | ✅ |
| Pipelines | Webhook triggered collection | ✅ |
| Pipelines | Pipeline run history | ✅ |
| Pipelines | Pipeline tags | ✅ |
| Lineage | Table-to-Table lineage | ✅ |
| Lineage | Activity-to-Activity lineage | ❌ |
| Lineage | Column-level lineage | ❌ |
What Monte Carlo Ingests
Monte Carlo collects the following from each configured factory:
Pipelines
Each ADF pipeline is collected as a job asset. Pipeline annotations are imported as tags, so you can use them in job monitors and domains.
Activities
The activities within each pipeline are collected as the pipeline's tasks. Copy activities whose input and output datasets are tables are used to derive lineage.
Pipeline Runs and Activity Runs
For each pipeline run and activity run, Monte Carlo captures:
- status (succeeded, failed, cancelled, in progress, queued, and so on)
- start and end times, and duration
- error message, failure type, and error code (if failure)

Prerequisites
Before adding the integration, you'll need:
- An Azure Data Factory instance
- An Azure service principal (App Registration) with the Reader role on the factory (see below)
Installation
Step 1: Create an Azure Service Principal with Reader access to your factory
- In the Azure Portal, create a new App Registration.

- Copy the Tenant ID and Client ID from your App Registration.

- Under Manage → Certificates & secrets, click New client secret. Copy the secret's Value. It can only be viewed once.

- In the Azure Portal, open your factory and copy the Resource group name and Subscription ID.

- On the same page, select Access control (IAM) and add a new role assignment.

- Choose the Reader role.

- Select the App Registration you created and save the role assignment.

Step 2: Add the ADF Integration in Monte Carlo
-
In Monte Carlo, navigate to Settings → Integrations.
-
Click Create and select Azure Data Factory from the Orchestration section.
-
Fill in the form:
Field Value Integration name A name for the integration in Monte Carlo. Tenant ID The Tenant ID of your App Registration (Step 1). Client ID The Client ID of your App Registration (Step 1). Client secret The client secret value you created in Step 1. Subscription ID The Azure subscription that contains the factory. Resource group name The resource group that contains the factory. Factory name The name of your Azure Data Factory.

-
Click Add. Monte Carlo verifies that the credentials can access the ADF REST APIs.
-
On success, click Continue to save the integration.
Once the integration is saved, Monte Carlo begins collecting your ADF metadata on an hourly schedule. Pipelines and runs typically appear within an hour of the first successful collection.
Step 3: Confirm data is flowing
- In Monte Carlo, navigate to Assets and search for one of your ADF pipelines.
- Open the asset and verify you can see:
- Runs — recent pipeline runs with status and duration
- Activities — the activities within the pipeline
- Open the Lineage tab on a table that one of your pipelines writes to, and look for the ADF pipeline between the upstream and downstream tables.
If your pipelines don't appear, go to Settings → Integrations, open the integration's menu, and click Test to run a validation test. If any check fails, the test shows steps to resolve it.
Lineage may take up to 24 hours to appear due to batch processing. Run history and asset metadata are typically available within an hour.
Pipeline Failure Alerts
When an ADF pipeline run fails, Monte Carlo raises a Job run failed event. A job monitor decides which pipeline failures alert and who is notified. This lets you:
- Route and receive notifications like other Monte Carlo alerts
- Analyze the downstream impact of a failure
- Report on pipeline failures alongside all your other data issues
Step 1: Create a job monitor
Create a job monitor on your ADF integration. A job monitor can cover all ADF pipelines, pipelines with a given tag, or pipelines you pick, and sends alerts to the audiences you choose. See Job monitors for setup steps.
Without a webhook, failures are picked up on the next hourly collection, so alerts may be delayed up to an hour.
Step 2: Configure the webhook (optional)
You can configure an Azure alert that calls a Monte Carlo webhook when a pipeline fails. The webhook triggers an immediate collection, so the failure reaches Monte Carlo within minutes rather than on the next hourly cycle.
Get your webhook URL
- In Monte Carlo, go to Settings → Integrations, find your ADF integration, open its menu, and click Manage webhook. The webhook URL is also shown in the Webhook configuration pane on the integration's details page.

- Copy the webhook URL.

The webhook URL contains a unique ID that identifies your integration. Treat it like a credential and share it only with the Azure alert that uses it.
Create an Azure action group
- In the Azure Portal, open Monitor → Alerts and create a new Action group.

- Fill in the Basics tab, then on the Actions tab select Webhook:
- Paste the Monte Carlo webhook URL.
- Select Yes for Enable the common alert schema. Monte Carlo requires the common alert schema.
- Review and create the action group.

Create an Azure alert rule
- In Monitor → Alerts, create a new Alert rule.

- On the Scope tab, select your Azure Data Factory.

- On the Condition tab, set the following and keep the defaults for everything else:
- Signal name: "Failed pipeline run metrics"
- Dimension name: "Pipeline"
- Dimension values: select all pipelines
- Check Include all future values

- On the Actions tab, select the action group you created.
- On the Details tab, fill in the required fields, then review and create the alert rule.
Once configured, any ADF pipeline failure triggers the Azure alert, which calls the Monte Carlo webhook and starts an immediate collection.
Lineage
Monte Carlo derives table-to-table lineage from Copy activities whose input and output datasets are tables, and attributes each edge to the ADF pipeline that ran it. Click the ADF icon on a lineage edge to see recent runs of that pipeline and activity.

Key points:
- A referenced table must be monitored by Monte Carlo for the edge to land on it.
- Table-level only. Column-level lineage is not produced.
- Lineage can take up to 24 hours to appear due to batching.
Lineage for Script and Stored Procedure activities
When a pipeline runs custom SQL, for example through a Script activity or a Stored Procedure activity, Monte Carlo can't attribute the resulting lineage to ADF on its own. Tag those queries with the ADF pipeline and activity name so Monte Carlo can link them. See Additional Support for ADF in Lineage for the tag format.
FAQs
Q: How often does Monte Carlo collect ADF data?
Collections run hourly by default. If you configure the webhook, pipeline failures trigger an immediate collection in addition to the scheduled hourly cycle.
Q: Are multiple factories supported?
Yes. Create a separate integration for each factory at Settings → Integrations. If pipelines or activities in different factories share names, add the mc_integration_name tag to tagged queries. See Multiple ADF Integrations.
Q: How long does it take for ADF data to show up in Monte Carlo?
Pipelines and runs are typically available on the asset page within an hour. Lineage may take up to 24 hours due to batch processing.
Q: A pipeline failed, but I didn't get an alert. What should I check?
- No job monitor covers the pipeline. A pipeline failure alerts only when an active job monitor covers it.
- The webhook isn't set up. Without the webhook, failures are picked up on the next hourly collection. The job monitor page warns you when an integration has Webhook not set up or No recent webhook events.
Q: Do ADF pipelines disappear from the Assets page if they haven't run recently?
No. A pipeline stays on the Assets page after it stops running, along with its past runs.
Updated 18 days ago
