Databricks Workflow (Beta)

Databricks Workflows is Databricks' managed orchestration services, which enables users to easily create and manage multitask workflows for ETL, analytics and machine learning pipelines.

Integrating MC with Databricks Workflow allows you to quickly determine which Databricks workflow potentially caused an anomaly downstream, accelerating your time to resolution. You will also be able to manage your workflow failure alerts along with all other data quality alerts in Monte Carlo, so you have centralized incident triage, notification routing, and data quality reporting across all your data and system issues.

1. Databricks Workflow on Lineage and as Assets

Databricks Job on Lineage

Understand which Databricks Workflow jobs creates the lineage between two tables in your Databricks environment.

Click into the Databricks icon to see recent job runs for this lineage edge.

Databricks Jobs As Assets

Use Monte Carlo as a single pane of glass for all data quality context for each table, including Databricks Job runs. Check the status and duration of recent Databricks job runs relevant to a table of interest.

Setup and Getting Started

If you have set up the MC-Databricks integration after 10/1/2024 following the latest documentation, there should be no additional configuration needed. If not, you need to ensure the following two conditions:
i. Have a working MC-Databricks connection, following instructions step 1 through step 6 here, and
ii. Make sure permissions to job-related system tables below are granted to the Service Principal. These are also covered in docs here:

GRANT SELECT ON system.lakeflow.jobs TO <monte_carlo_service_principal>;
GRANT SELECT ON system.lakeflow.job_tasks TO <monte_carlo_service_principal>;
GRANT SELECT ON system.lakeflow.job_run_timeline TO <monte_carlo_service_principal>;
GRANT SELECT ON system.lakeflow.job_task_run_timeline TO <monte_carlo_service_principal>;

Then Monte Carlo will automatically start collecting Databricks Workflow Job information.

2. Databricks Workflow Alerts in MC

Coming soon!