Despite last week being short with the US holiday, the team shipped some great updates across the board.

What's new

• Manage warehouse and BI connections from settings: You can now add, update or remove warehouse and BI connections all from the Settings > Integrations section in the app - you can also continue make these configuration changes (and more!) via the CLI
• Add custom messages to notifications: Add custom messages including @ mentions, # tags, etc to notifications sent to Slack, email, Pagerduty, Opsgenie, etc. - these messages allow you to add context when those receiving the notifications may lack context

Improvements and fixes*

• Improved handling S3 Events metadata collection: Improved metadata collection within S3 Events integrations to improve reliability of the schema within MC
• Monitors as code UI handling: The recent release of monitors as code introduced potential conflicts with monitors configured in the MC app - with this change, any monitors configured via the YAML config will be clearly identified as so in the app

What's next

• Domains (previously named workspaces): To help larger teams manage complex datasets, we're building a domain model which will allow users to create database and schema filtered groups that users can configure to filter the UI by team, focus area, etc.
• Field level lineage: We're working on a way to expose field level lineage in the UI to help with both field-specific investigations and field deprecation

It's a short this week with Memorial Day in the US and Canada.

What's new

• Monte Carlo CLI overhaul: Dramatically improved usability of the CLI by adding support for deploying MC collectors across regions, enabling you to skip onboarding steps, testing telnet connections, BI tool onboarding and many usability improvements (see docs here)

Improvements and fixes*

• Append custom text to Incident Notifications: Add context to your MC incident Slack and email notifications by appending custom messages and/or @ mentions
• Add support for EU Opsgenie instance: Opsgenie customers with hosted instance in the EU can now update their URL
• Improved performance of sample SQL queries: When investigating field health anomalies, we reduced the time it takes to access sample queries and data

What's next

• Domains (previously named workspaces): To help larger teams manage complex datasets, we're building a domain model which will allow users to create database and schema filtered groups that users can configure to filter the UI by team, focus area, etc.
• Field level lineage: We're working on a way to expose field level lineage in the UI to help with both field-specific investigations and field deprecation

Another exciting week of product feature improvements and new releases!

What's new

• Dataset and table filters apply to SQL Rule Notifications: With this change customers no longer need to select individual SQL Rules and SLIs to filter which incident notifications they receive and instead can now use dataset/table/tag filters to automatically filter for incident notifications
• Download pipeline graph as CSV or PNG: Easily download a CSV export of the pipelines lineage graph to generate your own version of the lineage graph or download a PNG to easily share pipeline graphs with coworkers

Improvements and Fixes:

• Custom monitor WHERE clause support: Resolved an issue with Field Health and Dimension Tracking monitors with WHERE clauses and added WHERE clause filter to Field Health and Dimension tracking catalog view
• Sample query performance improvement: Improved performance of accessing sample queries and sample records for Field Health anomalies - click on the new debug icon when available

What's next

• Field level lineage: We're working on a way to expose field level lineage in the UI to help with both field-specific investigations and field deprecation
• Workspaces: To help larger teams manage complex datasets, we're building a workspaces model which will allow users to create database and schema filtered groups that users can configure to filter the UI by team, focus area, etc.

We have a double feature this week due to our hackathon the week before last

What's new

• SQL Rule snoozing: Easily snooze SQL rule notifications during incident investigations or while developing pipelines to reduce notification fatigue
• Monitors as code: To help data engineering teams manage Monte Carlo monitors, we released YAML-based config that allows you to deploy and maintain monitors as part of your CI/CD process (Reach out to us if you're interested in testing out the functionality!)

*Improvements and Fixes

• CLI Athena and Glue Connection Support: Improved connection status feedback when setting up Athena and Glue data lake integrations via the CLI
• Rule and SLI result Insights report: Added a new report to the Dashboard Insights which shows a 90 day history of custom rules and Freshness/Volume SLIs
• Infinite scroll and performance improvements on Incidents tab: The Incidents tab now loads significantly faster and infinite scroll means you no longer have to click to load the next set of results

What's next

• Field level lineage: We're working on a way to expose field level lineage in the UI to help with both field-specific investigations and field deprecation

The MC engineering team is running a hackathon this week so we'll have fewer updates to share next week.

What's new

• Filter notifications with table tags: To support more advanced notification filters, you can now use table tags to filter which notifications go to which Slack channels, emails, etc.

• Data Lake throughput anomaly detection: To provide the same observability Monte Carlo offers around volume changes in data warehouses, we shipped support for throughput monitoring in Data Lakes which will alert you to abnormal volume changes and unchanged size anomalies

• More UI usability improvements:

  • Select only to filter by a single monitor type in the Custom Monitors table
  • Update table name formatting across the application using the following standard format [data warehouse name] database_name:schema_name.table_name
  • Open incidents in new tab from Pipelines graph incidents
  • Edit date time using text input in addition to UI date selector selector component
  • Add search filter to field selector when enabling Field Health monitoring
  • Select all fields in the Field Health monitor flow by default

Improvements and Fixes

• Improved handling of low distinctness numeric field health monitors: We now provide better support for catching numerical anomalies in tables with very low distinct values
• Faster anomaly detection training period: To reduce the training time for anomaly detection models, we now pull 2 weeks of historical data when a monitor is deployed
• Select Data Collector Region during warehouse setup: When onboarding a new data warehouse, you will now be able to select your data collector region in the app

What's next

• SQL Rule Snooze: To minimize unwanted notifications during an investigation, we'll soon support the ability to snooze SQL Rule notifications for preset time period
• Workspaces: To support large organizations with multiple teams, we're working on a workspace model that will allow you to create collections of projects and schemas that will filter what the user sees in the UI
• Monitors as code: A highly requested feature from our roadmap sessions was to configure Monte Carlo as a part of the code release process - this integration will support YAML based configs to deploy custom monitors within Monte Carlo

We're focusing on shipping many smaller fixes and UI improvements that we hope will significantly increase the quality of life for our customers. More updates to come next week!

What's new

• Table tag Notification filters: When creating notifications, you can now filter notifications by table tag -- A common use case here is to send notifications for tables with a key_asset: true tag to a high priority Slack channel, email address, etc.

• UI Usability Improvements:

  • Tooltips throughout the UI have been updated to make them significantly easier to read with a darker background and larger font size
  • Custom Monitor last run status is now displayed in the Custom Monitors table
  • SQL Rule clone option makes it easier to duplicate SQL Rules without having to copy paste the SQL and all the corresponding settings
  • Edit single steps in custom monitor wizards without having to go through the entire wizard each time

• Multi-region support: We now support deploying the MC collector in different AWS regions - with this change, our CLI now also supports deploying and launching stacks

Improvements and Fixes

• Unchanged size anomaly detector: Previously, only anomalies would only be triggered if the volume of new records in a table went to 0 -- with this improvement, we now can notify you when the volume of new records approaches 0 records, but does not necessarily hit 0 new records

What's next

• SQL Rule Snooze: To minimize unwanted notifications during an investigation, we'll soon support the ability to snooze SQL Rule notifications for preset time period
• Workspaces: To support large organizations with multiple teams, we're working on a workspace model that will allow you to create collections of projects and schemas that will filter what the user sees in the UI
• Monitors as code: A highly requested feature from our roadmap sessions was to configure Monte Carlo as a part of the code release process - this integration will support YAML based configs to deploy custom monitors within Monte Carlo

Another week of new product features focused on usability and extensibility of Monte Carlo.

What's new

• Custom monitor table filters: Added search based filters and sorting to the custom monitors table
• Data lake throughput monitoring: For Data Lake customers, we now monitor for throughput changes which looks at the changes in object store size - visit the catalog for a data lake table to view the throughput history graph

What's next

• SQL Rule Snooze: To minimize unwanted notifications during an investigation, we'll soon support the ability to snooze SQL Rule notifications for preset time period
• Workspaces: To support large organizations with multiple teams, we're working on a workspace model that will allow you to create collections of projects and schemas that will filter what the user sees in the UI
• Monitors as code: A highly requested feature from our roadmap sessions was to configure Monte Carlo as a part of the code release process - this integration will support YAML based configs to deploy custom monitors within Monte Carlo

Another week of new product features focused on usability and extensibility of Monte Carlo.

What's new

• Incident overlay + filters on Pipelines: To make the pipelines lineage graph easier to use, we now show recent incidents on tables in the graph and allow you to filter by node and lineage depth
• Past incidents and custom monitors modules: In the catalog view, we now show both past incidents related to the table and active custom monitors

Improvements and Fixes

• Removed 10 minute rule breach notification timeout: Rule breach notifications will now be sent immediately after being triggered
• Moved Insights reports to Dashboard tab: To make it easier to find and download the Insights reports, we moved them to the Dashboard tab

What's next

• Workspaces: To support large organizations with multiple teams, we're working on a workspace model that will allow you to create collections of projects and schemas that will filter what the user sees in the UI
• Monitors as code: A highly requested feature from our roadmap sessions was to configure Monte Carlo as a part of the code release process - this integration will support YAML based configs to deploy custom monitors within Monte Carlo

The pushed out a lot of great changes last week. As always, we've love to hear your feedback!

What's new

• Opsgenie integration: We now support sending Incident notifications to Opsgenie automatically - visit Settings > Notifications to configure the integration
• Incident IQ Timeline: To build a rich post-mortem of what happened to an incident, we added comments and incident changes to the timeline including status update, owner changes, severity level changes and more
• Incident IQ Past Incidents: Easily access past incidents on a table to spot patterns and see what steps were taken previously to resolve the issue

Improvements and fixes*

• Monitors tab layout change: Updated the layout to make it easier to sort and filter the list of monitors in the Custom Monitors table

What's next

• Lineage filters and incident overlay: To make the lineage graph easier to use we're adding filters and overlaying recent incidents to help with determining if an upstream issue may be impacting a downstream table or report
• Workspaces: To support large organizations with multiple teams, we're working on a workspace model that will allow you to create collections of projects and schemas that will filter what the user sees in the UI
• Monitors as code: A highly requested feature from our roadmap sessions was to configure Monte Carlo as a part of the code release process - this integration will support YAML based configs to deploy custom monitors within Monte Carlo

It's a short week in the US but the new features keep on coming!

What's new

• Follow-on Slack Notifications: Reduce the chance of missing ballooning incidents where tables are added silently with follow-on Slack notifications each time 10 tables are added to an incident
• Deteriorating Queries Insight report: Visit Settings > Insights to check out a new report which shows which queries are seeing consistent increases in run times over the course of a month

Improvements and fixes*

• Search filter for Users: Added a simple search filter to the Settings > User table to help find specific users that are part of a big team

What's next

• Workspaces: To support large organizations with multiple teams, we're working on a workspace model that will allow you to create collections of projects and schemas that will filter what the user sees in the UI
• Monitors as code: A highly requested feature from our roadmap sessions was to configure Monte Carlo as a part of the code release process - this integration will support YAML based configs to deploy custom monitors within Monte Carlo