---
updatedAt: 2026-09-25T16:20:21.000Z
agentTools:
projectIndex: https://docs.getmontecarlo.com/llms.txt
---
# Databricks Workflows
Databricks Workflows is Databricks' managed orchestration services, which enables users to easily create and manage multitask workflows for ETL, analytics and machine learning pipelines.
Integrating MC with Databricks Workflows allows you to quickly determine which Databricks workflow potentially caused an anomaly downstream, accelerating your time to resolution. You will also be able to manage your workflow failure alerts along with all other data quality alerts in Monte Carlo, so you have centralized incident triage, notification routing, and data quality reporting across all your data and system issues.
The following docs will walk through
1. [Databricks Workflows on Lineage and as Assets](https://docs.getmontecarlo.com/docs/databricks-workflow#1-databricks-workflows-on-lineage-and-as-assets)
2. [Databricks Workflows Failures as MC Alerts](https://docs.getmontecarlo.com/docs/databricks-workflow#2-databricks-workflows-failures-as-mc-alerts)
# 1) Databricks Workflows on Lineage and as Assets
## Databricks Job on Lineage
Understand which Databricks Workflows jobs creates the lineage between two tables in your Databricks environment.
Click into the Databricks icon to see recent job runs for this lineage edge.
## Databricks Jobs As Assets
Use Monte Carlo as a single pane of glass for all data quality context for each table, including Databricks Job runs. Check the status and duration of recent Databricks job runs relevant to a table of interest.
## Setup and Getting Started
If you have set up the MC-Databricks integration after 10/1/2024 following the latest documentation or if you already get the described features above, there should be no additional configuration needed and you can proceed to [part 2](https://docs.getmontecarlo.com/docs/databricks-workflow#2-databricks-workflows-failures-as-mc-alerts). If not, you need to ensure the following two conditions:
i. Have a working MC-Databricks connection, following instructions step 1 through [step 6 here](https://docs.getmontecarlo.com/docs/overview-databricks#step-6-add-the-databricks-integration-in-monte-carlo), and
ii. Make sure permissions to job-related system tables below are granted to the Service Principal. These are also covered in [the setup guide](https://docs.getmontecarlo.com/docs/overview-databricks#system-tables):
```sql SQL
GRANT SELECT ON system.lakeflow.jobs TO ;
GRANT SELECT ON system.lakeflow.job_tasks TO ;
GRANT SELECT ON system.lakeflow.job_run_timeline TO ;
GRANT SELECT ON system.lakeflow.job_task_run_timeline TO ;
```
Then Monte Carlo will automatically start collecting Databricks Workflow Job information.
# 2. Databricks Workflows Failures as MC Alerts
Additional setup is required to receive Databricks Workflows failures in MC.
## Databricks Job Alerts in MC
Receive alerts on Databricks Workflows failures in Monte Carlo, so you can centrally manage and report on such issues along with all your other data incidents. More info on how alerts work in MC [here](https://docs.getmontecarlo.com/docs/incidents)
## Setup
### 1. Create Webhook Key and Secret in Monte Carlo
**Create via UI**
1. On the [Integrations](https://getmontecarlo.com/settings/integrations) page, open your Databricks integration.
2. In the **Integration keys** pane, under **Select key scope**, choose **Webhook events**. Keys with the **Push ingestion** scope are rejected by the webhook.
3. Click **Generate integration key**, and save the key ID and secret. You'll use them as the username and password of the webhook destination in Databricks.
**Create via CLI**
Alternatively, you can use the Monte Carlo CLI to create a Databricks webhook integration key:
```
% montecarlo integrations create-databricks-webhook-key --help
Usage: montecarlo integrations create-databricks-webhook-key
[OPTIONS]
Create an integration key for a Databricks webhook
Options:
--integration-name TEXT Name of associated Databricks metastore integration
(required if you have more than one)
--option-file FILE Read configuration from FILE.
--help Show this message and exit.
```
### 2. Add Webhook notification in Databricks
Note: This must be done by a workspace admin in Databricks.
Create a notification destination pointing to the Monte Carlo Databricks webhook endpoint. Copy the endpoint URL from the **Webhook configuration** pane on your Databricks integration's page in Monte Carlo. The pane also shows when Monte Carlo last received a webhook event.
Go to Databricks UI, under **Settings**, navigate to tab **Notifications**, click on "Manage" to create a new notification destination.
In the **Create a new destination** screen:
1. Select type **Webhook** and give the destination a name.
2. Under **URL**, paste the webhook endpoint URL from the **Webhook configuration** pane.
3. Under **Username**, enter the integration key ID from step 1.
4. Under **Password**, enter the integration key secret from step 1.
5. Click **Create**.
You can test that the notification is configured correctly using the "test" feature of the notification:
For more details on creating notification destination in Databricks, see [Databricks Documentation ](https://docs.databricks.com/en/admin/workspace-settings/notification-destinations.html).
### 3. Add Notifications and Permissions to Databricks Jobs
Next you will set up the notification and permission for each Databricks job you want alerts routed to MC for. You can do it via the Databricks UI for each job one by one, or you can use a script provided below to configure all jobs in bulk. **We recommend configuring for all jobs in bulk via script** given i) it is fast to set up, ii) you can still enable or disable alert generation for each job in MC settings per step 4 below.
**via Databricks UI**
For each job you want to send failures to Monte Carlo for, go to that job's page in Databricks UI. On the right-hand-side panel under **Job Notifications**, click on "Edit notifications". Add a new notification, add the Webhook destination you created in the previous step, and check **Failure** then save the notification.
As part of the failure alerting workflow, Monte Carlo will use the Databricks Jobs API to gather more information to provide context for the alert, which requires the **Can View** permission for those jobs. To grant that permission, go back to the job's page in Databricks and navigate to the **Permissions** section on the right hand side, click on "edit permission".
In the **Permission Settings** screen, grant your Monte Carlo service principal the **Can View** permission.
\--
**via Script (bulk option available)\[Recommended]**
Get the UUID of the Monte Carlo notification destination you created before via Databricks UI -> **Settings** -> **Notifications**. Click on the copy button next to "Edit destination settings" to copy destination ID as the UUID.
Then get the Application ID of the Service Principal used to allow Monte Carlo Access. You can find this at `https:///settings/workspace/identity-and-access/service-principals`
You will use the notification destination UUID and the Application ID for MC Service Principal in the script located [here](https://github.com/monte-carlo-data/mcd-public-resources/blob/main/scripts/databricks/enable_monte_carlo_databricks_job_incidents.py)
You can choose to enable alerts for all jobs, or specify which jobs to enable alerts for by using the `--databricks-job-name` option.
```
Usage: enable_monte_carlo_databricks_job_incidents.py [OPTIONS]
Enable Monte Carlo incidents for Databricks jobs
Options:
--mcd-notification-id TEXT UUID of the existing Databricks Notification
pointing to the MC Webhook endpoint.
[required]
--mcd-service-principal-name TEXT
Application ID of the existing Monte Carlo
service principal in Databricks. [required]
--databricks-job-name TEXT Databricks Job Name to enable MC incidents
for. Can be used multiple times. If not
specified, enable MC incidents for all jobs.
--help Show this message and exit.
```
Example:
```
python enable_monte_carlo_databricks_job_incidents.py --mcd-notification-id '' --mcd-service-principal-name '' --databricks-job-name --databricks-job-name
```
### 4. Enable Alerts for desired jobs in Monte Carlo
On the same connection where you created the Webhook Key, under the three dot menu, select the `Configure Jobs` option.
Here you can choose which jobs to generate alerts for. By default, alerts for jobs are disabled. To bulk enable/disable jobs, click on the checkboxes on the left for all jobs you want to alert on, then click on the "Alerts" button highlighted in blue, then you can either enable or disable alerts for all selected jobs.
### 5. Set up audience
Finally, make sure to set up an audience in MC to route your Databricks workflow failures. You can do that by going to MC UI, **Settings** -> **Notifications**, then either create a new audience or edit an existing audience to include your Databricks workflow failures.
You can send failures for all Databricks Workflows to an audience by selecting **Databricks job failures** in **Alert Type** and **All** for **Affected Data**.
Or you can route only a subset of Databricks jobs for an audience by selecting **Databases, schemas, tables, jobs, and tags** for **Affected Data**, then adding the jobs you want to include.
Or use tags to include all databricks workflows jobs under a certain tag, i.e. team: data-platform. Tags from Databricks are imported automatically into MC.
For more information on MC Audiences, see [Audiences](https://docs.getmontecarlo.com/docs/notifications-v2#audiences)
Now you should start receiving Databricks workflow failure alerts in the desired audience!
## Network Access Control
You can restrict access to the Databricks webhook and metadata endpoints by IP address using Network Access Control Lists. See [Network Access Control](https://docs.getmontecarlo.com/docs/network-access-control) for setup and configuration details.