Instead of re-creating the same SQL Rule for multiple tables, you can now create a single SQL Rule and iterate through multiple tables, fields, where clauses, etc. using variables. To get started, simply add a variable in the "Advanced Options" when defining your SQL query. You can add multiple variables and define multiple values for each variable. Monte Carlo will iterate through all combinations and test for breaches accordingly.

To improve the timeliness of anomaly detections, we reduced the detection delay from 3 hours to under 1.5 hours. This improvement to freshness, unchanged size and volume monitors will result in faster notifications when data issues occur.

We now support Looker within our field level Lineage product. With this feature, you can trace exactly how a field is used by Looker Views, Explores and Dashboards.

Email notifications for SQL rule incidents including SLIs will now contain the rule name in the subject line

If you want to test a SQL Rule or run it after shipping a pipeline change, you can trigger the rule to run directly from the Monte Carlo App. Visit the SQL Rule detail page and click the Run button in the upper right corner of the page.

What's new

• SQL Rules breach data profiling: for SQL rule breaches with sufficient number of breached rows, we now show overview of fields and types of the anomalous records, distribution metrics such as percentiles, and distribution of the most common field values. This feature is available for customers with data collector of version 2939 or later.
• Granular incident type selection: schema changes are now broken down to fields added, fields removed, field type change, and volume anomalies are now broken down to unchanged size, bytes/rows added, bytes/rows removed, and abnormal small size change. The granular types are available via incident feed filter as well as notification route filters.
• Volume SLI revamp and custom sampling: Revamped the volume SLI setup workflow to be more user-intuitive. Enabled volume metadata sampling at custom defined times. Currently supports SLIs based on total table size comparisons; SLIs based on table growth comparisons will be coming soon.
• Dynamic table tags in notifications: users can now add keys of table tags to pass the values of the tags to incident notifications.
• Monitor creation via catalog page: users can now directly create custom monitors from the catalog page of a table.
• In-app notifications: released in-app notifications for product change logs and data collector upgrade reminders.

Improvements and fixes

• Catalog search improvement: for catalog search, brought back the dropdown list of search matches that was previously removed; added grouping of search matches by categories, i.e. View, Table, Field.
• Warehouse credentials storage: customer warehouse credentials are now stored in HashiCorp Vault, an industry-standard secrets management system.
• SSO Disablement: SSO can now be disabled via the UI under Settings.
• Notification filter enhancement: users can now multi-select SQL rules and SLIs in notification filter settings without the selection list closing on every click.

What's next

• Field lineage integration with Looker: see field level dependencies with Looker dashboards.
• SQL rules variables: users will be able to set up custom SQL rules for multiple tables/fields at once, streamlining the creation and management of large number of SQL rules.
• Volume SLI based on table growth comparisons: the next version of release will enable customers to set up volume SLIs against volume growth measurements in addition to the existing total volume measurements.

:** Quickly check if upstream tables/fields had incidents to determine if the root cause started upstream of the current table + change the look-back range for when upstream incident occurred

• Query change Root Cause Analysis: RCA helper indicates if the write query on the table changed when the incident occurred to help quickly pinpoint the root cause

Improvements and fixes

• Granular SQL Rule and SLI filtering options: Released new granular filtering options to the Incident feed and Notification rules to improve relevance of Incidents
• Improved Incident Owner functionality: Assigning owners to incidents is now significantly easier with dropdown typeahead search, email notifications and the ability to remove owners
• Private Slack Channel Support: Send notifications to private Slack channels by entering the Private Channel ID in the Notification setup flow
• Field lineage improvements: Added support for parsing additional SQL clauses including GROUP BY, ORDER BY, UPDATE FROM, MERGE INTO, UPDATE () to our Field Lineage parsing engine
• Notes support for Monitors as code: Add notes to SQL Rule monitors specified in Monitors as Code
• Time window filtering to Incident IQ graphs: Change the look-back window of graphs in Incident IQ to help with investigations
• Recent Incidents in Monitor Detail page: When reviewing the details of a custom monitor, view all recent incidents generated by said monitor

What's next

• Airflow integration: View airflow task logs from within Monte Carlo to support investigations and orchestration context
• Field lineage integration with Looker: See field level dependencies with Looker dashboards

What's new

• Slack-UI Bidirectional Status Sync: When users update incident status in the UI, the corresponding incident status in Slack will also be updated accordingly.
• Airflow Integration Beta: beta version of the Airflow integration is available for customers who store their Airflow logs in s3. This feature helps users troubleshoot data incidents by exploring Airflow DAG and task failures directly from Incident IQ. Please reach out to your customer success manager to set up the integration.
• Incident Feedback: users are now able to provide feedback on the helpfulness of each incident by clicking on the emojis next to incident cards, or on top of the incident IQ page.

Improvements and fixes

• Field lineage in Incident IQ: for field health and dimension tracking incidents, we now show in the Incident IQ page the corresponding field level lineage in addition to the table lineage.
• Affected Reports Usability: the list of reports in the Affected Reports module is now filterable by report type and searchable via report name.

What's next

• Domains-based Access Controls: Define user groups to assign read only or write access per domain
• Field lineage integration with Looker: See field level dependencies with Looker dashboards
• Automated Root Cause Analysis: A suite of RCA tools to identify high correlations between certain field values and field anomalies, build statistical profile for anomalous rows, and trace upstream incidents using lineage.

What's new

• Segmented field health: Users can now set up field health monitor segmented by a specified field so anomalies can be detected for each segment.  
• SQL rules and SLIs misconfiguration warnings: in insight report, snowflake data share, and the monitors page in UI, we now show warnings if any SQL rules or SLIs are breaching 80% of the times they run, and if there is not enough data to detect anomalies for SQL rules with ML thresholds.
• Looker lineage liquid templates: added capability to parse and interpret liquid templates in Looker.
• Incident Status Tracker: released a visual tracker for incident status updates on incident feed page. Note that the tracker can be filtered by domains but not by other filters on the incident feed page.
• Freshness detector sensitivity tuning: For the automated freshness and unchanged size detectors, customers can now set a minimum number of hours that must pass since last update before an alert can be issued. Users can access t his feature by going to the freshness or size module in Catalog page.

Improvements and fixes

• Sampling/reproducing feature via Slack: Users can now go into sampling queries and reproducing queries for dimension tracking and field health incidents directly from Slack.
• Rules breach incident titles in email: rule breach alerts via email now clearly specify in their titles whether the rule breaches are freshness SLIs, volume SLIs, or SQL rules. 
• CLI improvement: Monte Carlo's CLI now supports Python 3.7, 3.8, 3.9, 3.10, and the current 3.11 alpha. In addition, Users can now generate help text via montecarlo help to retrieve documentation on all commands, subcommands and options.
• Incident dashboard improvement:  In incident dashboard, users can now click on the summary stats or the chart bars to drill in to the data behind the dashboard.

What's next

• Domains-based Access Controls: Define user groups to assign read only or write access per domain
• Airflow integration: View airflow task logs from within Monte Carlo to support investigations and orchestration context
• Field lineage integration with Tableau: See field level dependencies with Tableau dashboards and worksheets.

 

• Tableau field level lineage: Expanded field level lineage coverage all the way to Tableau workbooks to better understand field-level relationships across the warehouse and BI layer
• Enhanced support for seasonality and irregular table update patterns: Released multiple improvements to our anomaly detection models to reduce incidents generated by expected weekend usage patterns, irregularly updated tables, and more.
• Automatic Freshness and Volume detector status: We now show the detector status for our automatic Freshness and Volume detectors in the table detail view and soon the catalog view
• Snowflake streams support: We now support Snowflake streams in lineage and catalog to help improve the e2e coverage of your data stack in Monte Carlo

Improvements and fixes

• Impact radius: We now group data warehouse queries by user to help you more quickly parse the queries and determine the impact of the incident
• Show SQL Rules in recent incident module in Catalog: We now show SQL Rule incidents for a specific table within the Catalog Recent Incidents module
• Incident Status tracker improvements: We now breakdown the incident status tracker by status Type and show when there are no new incidents for the current week
• PyCarlo SDK improvements: Improved error handling including configurable automatic retries on typically transient errors (see release notes here)

What's next

• Domains-based Access Controls: Define user groups to assign read only or write access per domain
• Airflow integration: View airflow task logs from within Monte Carlo to support investigations and orchestration context
• Field lineage integration with Looker: See field level dependencies with Looker dashboards