Databricks SQL Warehouse

📘

Prerequisites

To complete this guide, you will need admin rights in your Databricks workspace.

Monte Carlo can leverage a Databricks SQL Warehouse for running monitors (e.g. custom SQL, field health, etc.)

Create Databricks SQL Warehouse

1. Follow these steps to create a SQL Warehouse. The sizing of that warehouse depends on the type and number of monitors that you wish to use within Monte Carlo.
2. Save the Warehouse ID.

Create an API key

1. Follow these steps to generate an API key in your workspace with no specified lifetime. Monte Carlo recommends you create a service account associated with the token first.
2. Save the generated token.

Add a Databricks SQL Warehouse connection to MC

Monte Carlo Settings-Integrations UI

If you're setting up an entirely new Databricks integration, including a SQL warehouse is a recommended part of the main integration workflow.

To add a SQL Warehouse to an existing Databricks integration, start creating a new Databricks connection in the Settings-Integration page, and then select "Add to existing integration"

CLI

Provide connection details for the Databricks SQL Warehouse using Monte Carlo's CLI:

1. Follow this guide to install and configure the CLI. Requires >= 0.39.2.
2. Use the command add-databricks-sql-warehouse to set up Databricks SQL Warehouse connectivity. For reference and command options see the docs here. Below is an example:

montecarlo integrations add-databricks-sql-warehouse --databricks-workspace-url dbc-12345678-abcd.cloud.databricks.com --databricks-warehouse-id 1234567890abcdef --databricks-token -1 --name data-lake