Reducing noise from monitors

When a monitor breaches, it creates an Alert that triggers a notification based on the Audiences configured in the monitor. If a monitor breaches repeatedly, this can result in repetitive notifications. Monte Carlo offers a few ways to control how breaches are grouped and when notifications are sent.

Reduce noisy notifications

When a monitor breaches repeatedly, Monte Carlo can group subsequent breaches into an existing open alert rather than firing a new alert and notification. This prevents duplicate notifications while an issue is already being investigated.

An alert is considered open when its status is No Status, Acknowledged, or Work in Progress. Once the status is set to Fixed, Expected, or No Action Needed, the next breach will create a new alert.

Table Monitors

Select one option in the Send notifications section of monitor creation:

  • Group breaches from the same schema for 5 hours (default) β€” breaches are grouped into the same alert for up to 5 hours, regardless of alert status
  • Group new breaches from the same schema into the currently open alert β€” breaches are grouped as long as the alert remains open and unresolved

In either case, breaches are grouped by alert type within the same schema β€” freshness, volume, and time since last row count change alerts group together; while schema change alerts are grouped separately.

Metric, Multi-table Metric & Comparison Monitors

Select one option in the Send notifications section of monitor creation:

  • Create a new alert every time the monitor breaches (default -- no grouping)
  • Group new breaches into the currently open alert β€” breaches are grouped as long as the alert remains open and unresolved

Custom SQL & Validation Monitors


SQL and Validation monitors may remain in a breached state for an extended period while the threshold stays violated. In addition to grouping by status, these monitors offer the following notification frequency controls:

  • Send another notification every [ X ] runs of the monitor
  • Send another notification only if the value or count of breached rows changes
  • Notify every time (no grouping or noise reduction)
  • Group new breaches into the currently open alert β€” breaches are grouped as long as the alert remains open and unresolved

The following illustrates how the first three options behave across 25 runs of a Custom SQL monitor configured to breach when row count exceeds 0, with Option 1 set to re-notify every 4 consecutive breached runs.

In Monitors as Code

All noise reduction settings are optional. When none are configured, each monitor type applies its default behavior as outlined above. To configure noise reduction via Monitors as Code, use the following fields:

  • event_rollup_count (Int: minimum value of 2)β€” re-notifies after every N consecutive breached runs. Applies to Custom SQL & Validation monitors.
  • event_rollup_until_changed (boolean)β€” re-notifies only when the value or count of breached rows changes when set to true . Applies to Custom SQL & Validation monitors.
  • alert_grouping β€” groups subsequent breaches into the currently open alert rather than creating a new one. Applies to Table, Metric, Multi-table Metric, Comparison, Custom SQL & Validation monitors.
alert_grouping:
  mode: group_into_open_alert

Visit the Monitors as Code documentation for full details, and per-monitor-type references.

Additional notes

  • Monte Carlo will not group more than 100 successive breaches of a monitor. Once 100 successive breaches are grouped, the next breach will generate a new alert.
  • These settings determine whether events are grouped into a single alert, not whether notifications are sent to each audience. When you add a new audience to an existing monitor, that audience will receive a notification on the next event regardless of where the monitor is in its rollup cycle for existing audiences.