Databricks - backup (contains cluster instructions)

This document walks through the steps to monitor a Databricks environment with Monte Carlo. The order of operations is important and it is strongly recommended to adhere to the documented sequence. These steps need to be repeated for each Databricks Workspace that you would like to observe with Monte Carlo.

Please note the Table of Contents to the right for the full outline of steps.

0. Pre-requisites

Upgrade your data collector. As of August 2023, v14887 or higher has the latest generation of Databricks collection.

1. Create a Personal Access Token or Service Principal

Creating a Personal Access Token is the simplest option to connect to Databricks. Databricks recommends using a Service Principal for API access though but it requires the Databricks CLI in order to create a Token.

Option 1: Creating a Service Principal (Recommended)

This option only is available if you are using Unity Catalog as Service Principals are a Unity Catalog feature.

As a Databricks account admin, login to the Databricks Account Console, click on User Management, and the Service Principals tab.
Click Add service principal, enter a Name for the service principal, and click Add.
Ensure that the Service Principal has Allow cluster creation (if using Cluster-based connection), Databricks SQL access, and Workspace access Entitlements.
Follow the Databricks documentation for creating a Service Principal Token (requires Databricks APIs) and save that Token.

Option 2: Creating a Personal Access Token (Not recommended)

You must be an Admin in the Databricks Workspace (admin access is required to generate the resources listed in step 4).
In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the drop down.
On the Access tokens tab, click Generate new token.
Enter a comment (monte-carlo-metadata-collection) that helps you to identify this token in the future and create a token with no lifetime by leaving the Lifetime (days) box empty. Click Generate.
Copy/save the displayed Token, and then click Done.

2. Grant permissions

Configure permissions depending on the type of metastore. If the workspace has catalogs in UC and not UC, both need to be configured. If you want to restrict access to specific schemas, check this out: Setting up dataset/schema controls.

Unity Catalog

Grant permissions to the Service Principal to read the Catalogs. This will cascade to all schemas within. In UC, the SELECT privilege implies USE:

GRANT SELECT ON CATALOG <CATALOG> TO <monte_carlo_service_principal>;

For details, see Databricks documentation.

See Troubleshooting & FAQ: Databricks if the command returns a "Privilege SELECT is not applicable" error.

Hive metastore

Grant permissions to the Service Principal to read the Catalogs - the most common scenario is that only one catalog is present, hive_metastore. Permissions will cascade to all schemas within. In the Hive mestastore, the SELECT privilege requires USAGE:

GRANT USAGE, READ_METADATA, SELECT ON CATALOG <CATALOG> TO <monte_carlo_service_principal>;

For details, see Databricks documentation.

3. Create a SQL Warehouse or Cluster for Metadata Collection

Adding the Metadata Connection allows Monte Carlo to gather metadata on a periodic basis. Monte Carlo supports metadata collection through SQL Warehouse, All-purpose Clusters and Job Clusters. If using Job Clusters, you must build the connection with an All-purpose Cluster (as specified below) and then optionally swap to a Job Cluster.

A SQL Warehouse [BETA] is recommended because it has better cost and availability, is quicker to spin up and terminate and requires less infrastructure to work. If you prefer to use an All-Purpose Cluster, or do not have access to SQL Warehouses in Databricks, please use Option 2.

🚧
Serverless SQL Warehouses and external Hive metastores
A Serverless SQL Warehouse does not work with an external Hive metastore. It works if Glue is the external metastore though.

Option 1: Creating a SQL Warehouse (Recommended)

Follow these steps to create a SQL Warehouse.
a. For environments with 10,000 tables or fewer Monte Carlo recommends starting with a 2X-Small. The Serverless type is recommended for cost efficiency reasons, but Pro is also supported. Classic is not supported. Most Metadata operations are executed in the driver, so a 2X-Small should be enough, but a larger SQL Warehouse might be required depending on the number and size of Delta tables. Please reach out to your account representative for help right-sizing!
Save the Warehouse ID.
Start the SQL Warehouse.

Option 2: Creating a Cluster (Not recommended)

Follow these steps to create an all-purpose cluster in your workspace.
a. For environments with 10,000 tables or fewer Monte Carlo recommends using an i3.xlarge node type. Most Metadata operations are executed in the driver, so it is recommended to disable autoscaling and initially only use 1 Worker node. More workers might be required depending on the number and size of Delta tables. Please reach out to your account representative for help right-sizing!
b. A Databricks runtime version with Spark >= 3.0 is required.
c. Please add spark.databricks.isv.product MonteCarlo+ObservabilityPlatform to the cluster config.
d. Avoid using "Single User" access mode, as it will prevent the metadata collection job to work with any other user in case credentials are updated in the future.
Navigate to the Cluster page, and copy/save the Cluster ID, which can be found at the end of the URL: https://<databricks-instance>/#/setting/clusters/<cluster-id>. For more details, see the Databricks documentation.
Start the cluster.
Optional: Change to a Job Cluster
Because we are only running the metadata collection job through the metadata connection, we can switch the cluster used on the job to a Job cluster. Migrating the Databricks Metadata Job to a Job Cluster

4. Create a SQL Warehouse or Cluster for a Query Engine

A SQL Warehouse [BETA] is recommended as the method for Monte Carlo to execute queries on Databricks because it has better cost and availability, and additionally is quicker to spin up for running queries. If you prefer to use an All-Purpose Cluster, or do not have access to SQL Warehouses in Databricks, please follow the Spark guide.

📘
Can I use the same SQL Warehouse / Cluster for both connections?
While it is possible to use the same cluster or SQL warehouse for both metadata and
query engine connections, because of the differences in requirements, we recommend using a different SQL Warehouse / Cluster for each connection.
Metadata collection is composed of many small queries that do not process data, only metadata, so the SQL Warehouse / Cluster can be small with few workers - one should be enough for most cases.
The SQL Warehouse / Cluster used as query engine needs to scale with the number and frequency of monitors and the data size.

Option 1: Creating a SQL Warehouse (Recommended)

Follow these steps to create a SQL Warehouse.
a. The size of the SQL Warehouse depends on the number of monitors and data size, Monte Carlo recommends starting with 2X-Small and scaling afterwards as needed. The Serverless type is recommended for cost efficiency reasons, but Pro is also supported.
Save the Warehouse ID.
Start the SQL Warehouse.

Option 2: Creating a Cluster (Not recommended)

Differently than the metatadata collection, a job cluster is not supported for the SQL engine connection.

Follow these steps to create an all-purpose cluster in your workspace.
a. The size of the All-pupose Cluster depends on the number of monitors and data size, Monte Carlo recommends starting with the i3.2xlarge node type for the driver and worker and scaling afterwards as needed.
b. A Databricks runtime version with Spark >= 3.0 is required.
c. Please add spark.databricks.isv.product MonteCarlo+ObservabilityPlatform to the cluster config.
d. Avoid using "Single User" access mode, as it will prevent the metadata collection job to work with any other user in case credentials are updated in the future.
Navigate to the Cluster page, and copy/save the Cluster ID, which can be found at the end of the URL: https://<databricks-instance>/#/setting/clusters/<cluster-id>. For more details, see the Databricks documentation.
Start the cluster.

5. Verify data access

Confirm that this SQL Warehouse or cluster has access to the catalogs, schemas, and tables that need to be monitored.

To check this, you can run the following commands in the Databricks SQL editor (for SQL Warehouse) or a notebook attached to your new all-purpose cluster. If all of the commands work and show the objects you expect, this cluster is configured correctly for the current user. Note this does not guarantee though the the job will work with the token configured in Monte Carlo if the token is from a different user. If it doesn't show the expected objects, this may be an issue with permissions or the settings on the cluster. Ensure that the SQL Warehouse / cluster is connecting to the correct metastore.

SHOW CATALOGS

SHOW SCHEMAS IN <CATALOG>

SHOW TABLES IN <CATALOG.SCHEMA>

DESCRIBE EXTENDED <CATALOG.SCHEMA.TABLE>

6. [Unity Catalog Only] Enable System Tables

Monte Carlo uses system tables to collect and present lineage. For any and all unity catalog environments, enable them and provide access to Monte Carlo.

Follow the instructions at https://docs.databricks.com/administration-guide/system-tables/index.html#enable-system-tables to enable the system tables within Databricks. Execute the instructions for each workspace that has any Unity Catalog.
Grant access for the Monte Carlo service principal to access the tables: GRANT SELECT ON system.access.table_lineage TO <monte_carlo_user>

7. Add the Connections in Monte Carlo

This step uses the Monte Carlo UI to add the Connections. Please ensure the the SQL Warehouses or Clusters are turned on in order to add the connections.

🚧
[Cluster] This creates resources in your Databricks workspace.
This automates the creation of a secret, scope, directory, notebook and job to enable metadata collection in your workspace. If you wish to create these resources manually instead, please reach out to your account representative.
None of these components are necessary if a SQL Warehouse is used.

To add the Connections, navigate to the Integrations page in Monte Carlo. If this page is not visible to you, please reach out to your account representative.
Under the Data Lake and Warehouses section, click the Create button and Databricks.
Use the Create Databricks metadata collection and querying connections button.
Under Warehouse Name, enter the name of the connection that you would like to see in Monte Carlo for this Databricks Workspace.
Under Workspace URL, enter the full URL of your Workspace, i.e. https://${instance_id}.cloud.databricks.com". Be sure to enter the https://.
Under Workspace ID, enter the Workspace ID of your Databricks Workspace. If there is o= in your Databricks Workspace URL, for example, https://<databricks-instance>/?o=6280049833385130, the number after o= is the Databricks Workspace ID. Here the workspace ID is 6280049833385130. If there is no o= in the deployment URL, the workspace ID is 0.
Under Personal Access or Service Principal Token, enter the Service Principal or Personal Access Token you created in Step 1.
For Metadata Collection Jobs, enter the SQL Warehouse ID (Step 2.1.2) or Cluster ID (Step 2.2.2).
Under Query Engine, select the integration type that matches what you set up in Step 3 and enter the SQL Warehouse ID (Step 3.1.2) or Cluster ID (Step 3.2.2)..
Enter the SQL Warehouse ID or Cluster ID.
Click Create and validate that the connection was created successfully.

Recommended: Check the Metadata Job
When the metadata connection is added, Monte Carlo will immediately start running the metadata job. Because of the way the job is constructed, we will try to gather metadata about all of the tables in the environment. Oftentimes, the permissions on the cluster prevent the metadata job from collecting information from certain tables. Its worthwhile to look at the job logs for the metadata job to see if there are any issues in collection.

Conclusion

You have connected all necessary integration points to get end-to-end observability for Databricks!

Advanced Options

In general, using this Databricks connection type will be sufficient. If your Databricks environment is connecting to an external metastore (Glue or Hive), and you wish to connect Monte Carlo directly to that metastore, we can still gather freshness and volume information on Delta Tables in the Databricks environment. Ask your Monte Carlo representative for more details.

Updated 6 days ago

Did this page help you?