We've added a rule under the Usage UI to let you select what tables to monitor based on tags. Under a Schema in the Usage UI, you can select Monitor tables where "tag" is one of: and specify a multi-select list of tags. This accompanies the rest of the rules we support, for a full list, see our documentation.
We've added two new rules to help you select which tables to monitor. Under the Schema level in the Usage UI there are two new rule types that can be added as include/exclude conditions:
Tables of a certain type
This rule type can be used to make sure a certain type of asset is included or excluded from monitoring. Supported types depend on your warehouse and integration with Monte Carlo.
Last Activity was within 30 days
This rule can be used to ensure that only recenly active tables are monitored by Monte Carlo. "Recent Activity" is defined as any Read or Write activity.
Users of Data Explorer can now drill-down into a field profile metric in Data Explorer to see how that metric has trended over time. This helps a data analyst to validate an issue reported by a business partner. For example, if someone in marketing ops is reporting they’re suddenly seeing a bunch of nulls in a key field, it’s now much easier to come in and validate that spike in nulls without ever writing SQL or setting up a monitor.
We've made key improvements to the automatic threshold option in SQL Rules, giving users more accuracy and more control in detecting anomalies. The automatic threshold now supports:
When configuring a SQL Rule with an automatic threshold, users can now select between Low, Medium, or High sensitivity.
For customers that have enabled the ServiceNow integration, the state of incidents from ServiceNow can now be sync'd back to incident status in Monte Carlo. This reduces toil of keeping multiple systems up to date.
Users can indicate the mapping of ServiceNow state → incident status in Settings > Integrations. See our ServiceNow docs for more details.
Users can now more easily allocate query cost from custom monitors back to the teams that set them up. This was a functionality previously available via API only, which can now more easily be done in UI.
Users can:
To learn more, check out our documentation.
Last month, we released 35+ new Field Metrics. At the time, only manual thresholds could be used on those new metrics. Today, we’ve released support for ML thresholds for many of those new metrics!
Specifically, we now support ML thresholds for the following:
In addition, you can now select the following as standalone metrics with ML thresholds (previously, they were all grouped together):
To see the full list of supported Field Metrics and available threshold types, see our Available Metrics documentation.
Data Explorer is a new tab on Assets that makes it easy to profile the contents of a table or view. This is helpful when investigating a data quality issue reported by a business partner, when considering which monitors to create, or when getting familiar with the contents of a table.
The experience is interactive and no-code, making it approachable for less technical roles. Users can point and click to adjust the time range of data and filter down for particular segments. In the future, we'd like to make it easy to compare multiple segments of data side-by-side.
Currently, Data Explorer is available for just a subset of customers and for the Snowflake integration only. Your Monte Carlo Agent must be version 16624 or higher. To learn more, check out our documentation.
Filters, Row count, and Segments sections of Data Explorer. Adjust the slider in Row count and click on values in Segments to filter the rest of the data in the dashboard.
Field profile and Sample rows sections of Data Explorer.
Users can now manage their freshness thresholds - whether automatic or manual - all from Assets > Freshness. To see this drawer, click View Details on the Freshness widget in Assets. User can then click Edit and choose between:
Freshness thresholds set through Assets > Freshness will follow the notification routing and grouping logic of out-of-the-box freshness monitors, regardless of whether “Automatic” or “Explicit” thresholds are selected.
These changes are geared to give users one place to manage the freshness thresholds for a table, instead of being fragmented across different experiences for out-of-the-box freshness and Freshness Rules.
We will continue investing in easy and scalable ways to deploy manual thresholds. But we don’t think that the old experience for Freshness Rules is the right path. As a result, we’re removing the ability to create the old-style Freshness Rules through the UI. Existing Freshness Rules will continue to function, and users can continue to create them through Monitors as Code. More details to come.
To learn more, see our documentation for Freshness monitoring.
We’re refreshing charts across the app so they all have 1) a clearly defined metric, 2) a clear threshold, 3) a legend, 4) properly scaled and labeled axis, 5) clearly marked anomalies.
New volume charts show just that – clear metrics (toggle between row count, or change in row count), clear thresholds, an interactive legend, better axis scaling, and clear anomalies. This improves usability and interpretability, so that understanding an anomaly is simple.
