The is in set and is not in set operators now allow users to define a set using 3 possible methods:

• From a list: manually enter the values
• From a field: select a table and field, and the set with be populated with the distinct values in that field. The selected field will be referenced each time the monitor is run, so the set values may change.
• From a query: write SQL to define the values in the set. The query will run each time the monitor is run, so the set values may change.

Previously, "From a list" was the only way to define a set. These new options make it easy generate large sets and keep them automatically updated as your data evolves. They are ideal for referential integrity checks and scenarios where you have large numbers of allowed values.

dbt and Airflow alerts will now be raised on ALL tables in Monte Carlo – whether the table is muted/monitored or not.

The alerts can be configured on a Job-level basis for each integration.

dbt

To configure which dbt Jobs will raise alerts, go to Settings -> Integrations -> dbt integration -> Edit and find the below configuration.

Learn more at dbt Failures As Monte Carlo Alerts

Airflow

To configure which Airlow Jobs will raise alerts there are two options:

1. Go to Settings -> Integrations -> Airflow integration -> Configure Jobs and find the below configuration:

   

2. Under Assets, search and navigate to the Airflow Job that you want to configure alerts for. On the top right, toggle "Generates alerts".

   

Learn more at Airflow Alerts and Task Observability

dbt snapshot and seed errors are now available in MC as alerts, alongside model and test alerts. Users can go to settings -> dbt integrations -> edit, to configure the option to send those alerts. Make sure to add the new alert types in the relevant audiences to receive notifications. (docs)

With the availability of SQL query history, a series of features for Databricks integration are added:

• Performance dashboard and monitors: the dashboard helps identify the slowest SQL queries and enables investigation on performance issues. Users can also use performance monitors to detect slow running SQL queries.
• Importance scores: estimates the importance (0 to 1) of assets based on various query history data (see details here).
• Usage stats: usage information like reads/day, writes/day, users etc are now available on the "assets" page, "general information" tab.
• Query logs: SQL query logs are now shown on the assets page
• Query Change RCA insights: associated SQL query changes are presented for a volume / freshness / field alert to help uncover query related root causes.

Limitations: note these features are available for assets on Unity Catalog and are constrained to SQL queries. If you are using customer-managed keys in Databricks, these features also will not be supported.

Setup required: In order to enable these features, read permission needs to be granted to the service principal for system table system.query.history. This is also described in docs here.

GRANT SELECT ON system.query.history TO <monte_carlo_service_principal>;

For many businesses, incoming data looks very different during major holidays. Maybe there is more or less data than normal, or the profile of that data is much different. For example, a financial technology firm may see less new data than normal on July 4, because markets are closed in the United States. Or an e-commerce company based in the United States may see more new data than normal on Black Friday, because it's one of the busiest shopping days of the year.

You can now easily use exclusion windows to Manage Holidays in order to:

• Remove the unusual holiday data from influencing machine learning thresholds
• Avoid receiving unwanted alerts during holidays

The available holidays include the 11 Federal Holidays within the United States, plus a few additional unofficial holidays that often influence data (e.g. Black Friday).

Users with an Editor role can now create their own Data Products and edit any Data Products they have previously created and are the owner for.

Users with Account Owner and Domain Manager roles continue to have permissions to create and edit all Data Products that have been created in the account.

For more on Managed Roles and Permissions, refer to https://docs.getmontecarlo.com/docs/authorization#managed-roles-and-groups

These operators make it easy to express a condition like "Alert me if the value returned by the SQL is not between 5 and 5,000." Previously, this would have needed multiple SQL Rules to express.

MC-Dremio integration is now available. Customers can run SQL monitor, comparison rules, and schema change detections on Dremio with MC. Both Dremio cloud and Dremio software are supported. Docs here https://docs.getmontecarlo.com/docs/dremio

dbt Job and model info are now overlaid on lineage. Job info is shown on lineage edges, model last run status and timestamp are shown on nodes.

Table and column tags are now automatically imported from Unity Catalog to Monte Carlo. Table tags will show up on Assets page, General Information tab, column tags will show up on Field Lineage tab. Make sure your data collector is updated to the latest version.