Overview

Monte Carlo integrates with Databricks Workflows (Lakeflow Jobs) to provide data lineage, run history, and job failure alerts for your Databricks jobs. Monte Carlo collects your jobs, their tasks, and their runs from Databricks system tables, and links each job to the tables it reads from and writes to. You can see which job produced a table, how its recent runs went, the queries it ran, and the code each task runs.

When a Databricks job fails, Monte Carlo can raise it as an alert, so you get centralized incident triage, notification routing, and reporting across all your data and system issues. You can optionally configure a Databricks webhook notification so failures are detected immediately rather than on the next hourly collection.

The Databricks Workflows integration is layered onto your existing Monte Carlo Databricks integration, reusing the same service principal and SQL Warehouse. No new connection is required.

Feature Support

What Monte Carlo Ingests

Monte Carlo collects the following from your Databricks workspace:

Jobs

Each Databricks job is collected as a job asset, with its name, creator, run-as identity, description, and tags. Job tags are imported as key/value pairs, so you can use them in job monitors and domains.

Tasks

The tasks within each job are collected along with their dependencies, which Monte Carlo uses to draw the job's task graph on the Tasks tab.

Runs

For each job run and task run, Monte Carlo captures:

• result state (succeeded, failed, cancelled, and so on) and termination code
• start and end times
• how the run was triggered, and the run type
• run parameters

Queries a job ran

A Databricks job's page has a Queries tab. It lists the SQL statements the job's tasks ran, newest first, so you can see exactly what a job did to your tables.

The list includes statements that touch no table, such as CREATE CATALOG and CREATE SCHEMA. Statements that can contain credentials, such as CREATE CONNECTION, CREATE CREDENTIAL, and CREATE SECRET, aren't shown.

Task source code

Each task on a Databricks job's Tasks tab has a View source button. It shows the code the task runs, so you can check what a task does without leaving Monte Carlo.

What you see depends on the task type:

Prerequisites

• An existing Monte Carlo Databricks integration. See Databricks for setup steps 1 through 6.
• The service principal permissions below.

Required permissions

System tables — SELECT on the job system tables. These are part of the main Databricks setup, so most integrations already have them:

GRANT SELECT ON system.lakeflow.jobs TO <monte_carlo_service_principal>;
GRANT SELECT ON system.lakeflow.job_tasks TO <monte_carlo_service_principal>;
GRANT SELECT ON system.lakeflow.job_run_timeline TO <monte_carlo_service_principal>;
GRANT SELECT ON system.lakeflow.job_task_run_timeline TO <monte_carlo_service_principal>;

Lineage also uses system.access.table_lineage, which the main setup grants.

Can View on jobs — required for webhook failure alerts and task source code. Monte Carlo calls the Databricks Jobs API to fetch details of a failed run and the source of each task. To show task source, the service principal also needs read access to the notebooks, files, and saved queries the job's tasks use. Step 3 below grants Can View together with the webhook notification.

Setup

If your Databricks integration has the system table grants above, Monte Carlo collects your jobs automatically on an hourly schedule. No additional setup is required for run history, tags, queries, and lineage.

To confirm data is flowing:

1. In Monte Carlo, navigate to Assets and search for one of your Databricks jobs.
2. Open the asset and verify you can see:
   • Runs — recent job runs with status and duration
   • Tasks — the tasks in the job and the task graph
3. Open the Lineage tab on a table that one of your jobs writes to, and look for the Databricks job between the upstream and downstream tables.

Job Failure Alerts

When a Databricks job run fails, Monte Carlo raises a Job run failed event. A job monitor decides which job failures alert and who is notified. More info on how alerts work in Monte Carlo here.

Without a webhook, failures are picked up on the next hourly collection. To surface failures within minutes, add a webhook notification in Databricks (steps 1–3), then create a job monitor (step 4).

Step 1: Create a webhook integration key in Monte Carlo

Create via UI

1. On the Integrations page, open your Databricks integration.

2. In the Integration keys pane, under Select key scope, choose Webhook events. Keys with the Push ingestion scope are rejected by the webhook.

3. Click Generate integration key, and save the key ID and secret. You'll use them as the username and password of the webhook destination in Databricks.

   

Create via CLI

Alternatively, you can use the Monte Carlo CLI to create a Databricks webhook integration key:

% montecarlo integrations create-databricks-webhook-key --help
Usage: montecarlo integrations create-databricks-webhook-key
           [OPTIONS]

  Create an integration key for a Databricks webhook

Options:
  --integration-name TEXT  Name of associated Databricks metastore integration
                           (required if you have more than one)
  --option-file FILE       Read configuration from FILE.
  --help                   Show this message and exit.

Step 2: Add a webhook notification destination in Databricks

Copy the endpoint URL from the Webhook configuration pane on your Databricks integration's page in Monte Carlo. The pane also shows when Monte Carlo last received a webhook event.

In Databricks, go to Settings → Notifications and click Manage to create a new notification destination.

In the Create a new destination screen:

1. Select type Webhook and give the destination a name.
2. Under URL, paste the webhook endpoint URL from the Webhook configuration pane.
3. Under Username, enter the integration key ID from step 1.
4. Under Password, enter the integration key secret from step 1.
5. Click Create.

You can use the destination's Test feature to check that it's configured correctly:

For more details, see the Databricks documentation on notification destinations.

Step 3: Add notifications and permissions to Databricks jobs

For each job you want failures sent to Monte Carlo for, add the webhook destination as a Failure notification and grant the Monte Carlo service principal Can View on the job. You can do this in the Databricks UI one job at a time, or for all jobs at once with a script.

Via script (recommended)

1. Get the ID of the notification destination you created in step 2: in Databricks, go to Settings → Notifications and click the copy button next to Edit destination settings.

2. Get the Application ID of the Monte Carlo service principal at https://<your-databricks-workspace>/settings/workspace/identity-and-access/service-principals.

3. Run the enable_monte_carlo_databricks_job_incidents.py script with those two values. It configures all jobs by default; use --databricks-job-name to limit it to specific jobs.

Usage: enable_monte_carlo_databricks_job_incidents.py [OPTIONS]

  Enable Monte Carlo incidents for Databricks jobs

Options:
  --mcd-notification-id TEXT      UUID of the existing Databricks Notification
                                  pointing to the MC Webhook endpoint.
                                  [required]
  --mcd-service-principal-name TEXT
                                  Application ID of the existing Monte Carlo
                                  service principal in Databricks.  [required]
  --databricks-job-name TEXT      Databricks Job Name to enable MC incidents
                                  for. Can be used multiple times. If not
                                  specified, enable MC incidents for all jobs.
  --help                          Show this message and exit.

Example:

python enable_monte_carlo_databricks_job_incidents.py --mcd-notification-id '<mcd-notification-uuid>' --mcd-service-principal-name '<mcd-service-principal-application-id>' --databricks-job-name <job 1> --databricks-job-name <job 2>

Via Databricks UI

1. On the job's page in Databricks, under Job notifications, click Edit notifications. Add a notification with the webhook destination from step 2, check Failure, and save.

2. In the job's Permissions section, click Edit permissions and grant the Monte Carlo service principal Can View.

Step 4: Create a job monitor

Webhook events only bring failures into Monte Carlo. To decide which job failures alert and who is notified, create a job monitor on the Databricks integration. A job monitor can cover all Databricks jobs, jobs with a given tag, or jobs you pick, and sends alerts to the audiences you choose. See Job monitors for setup steps.

Network Access Control

You can restrict access to the Databricks webhook and metadata endpoints by IP address using Network Access Control Lists. See Network Access Control for setup and configuration details.

Lineage

Monte Carlo derives job lineage from Databricks' system.access.table_lineage system table, which records the tables each job reads and writes. Each edge is attributed to the job that produced it.

Click the Databricks icon on a lineage edge to see recent runs of that job.

Key points:

• Lineage requires Unity Catalog.
• A referenced table must be monitored by Monte Carlo for the edge to land on it.
• Job lineage is table-level. Edges between a job and its tables don't include columns.
• Task-to-task lineage comes from each task's dependencies and appears as the task graph on the job's Tasks tab.
• Lineage can take up to 24 hours to appear due to batching.

FAQs

Q: How often does Monte Carlo collect Databricks job data?

Collections run hourly. If you configure the webhook, job failures trigger an immediate collection in addition to the scheduled cycle.

Q: A job failed, but I didn't get an alert. What should I check?

1. No job monitor covers the job. A job failure alerts only when an active job monitor covers it.
2. The job has no failure notification. Without the webhook notification on the job, failures are picked up on the next hourly collection. The Webhook configuration pane on your Databricks integration shows when Monte Carlo last received a webhook event.
3. The service principal lacks Can View on the job. Monte Carlo needs Can View to fetch details of the failed run.

Q: Why don't I see the source code for a task?

The service principal needs Can View on the job and read access to the notebooks, files, and saved queries the task uses.

Q: Do Databricks jobs disappear from the Assets page if they haven't run recently?

No. A job stays on the Assets page after it stops running, along with its past runs.

For other Databricks issues, see Troubleshooting & FAQ: Databricks.