Databricks Integration Upgrade

With the availability of SQL query history, a series of features for Databricks integration are added:

• Importance scores: estimates the importance (0 to 1) of assets based on various query history data (see details here).
• Usage stats: usage information like reads/day, writes/day, users etc are now available on the "assets" page, "general information" tab.
• Performance dashboard and monitors: the dashboard helps identify the slowest SQL queries and enables investigation on performance issues. Users can also use performance monitors to detect slow running SQL queries.
• Query logs: SQL query logs are now shown on the assets page
• Query Change RCA insights: associated SQL query changes are presented for a volume / freshness / field alert to help uncover query related root causes.

Limitations: note these features are available for assets on Unity Catalog and are constrained to SQL queries.

In order to enable these features, read permission needs to be granted for system.query.history table per docs here.