Notifications

Monte Carlo's framework for Notifications makes it easy to send the right alerts to a broad variety of different recipients, including Slack, Microsoft Teams, Email, PagerDuty, Webex, OpsGenie, Jira, ServiceNow, and webhooks.

Audiences

To send notifications, first create an audience. An audience is a named group of one or more recipients. For example, an audience could include just a single Slack channel. Or it could include two Slack channels, five email addresses, a Microsoft Teams channel, and a PagerDuty routing key.

Audiences can either be created from within Settings > Notifications, or from the Send Notifications section of monitor creation. They can be easily edited to add, remove, or change recipients.

Adding notifications to an audience

Once an audience is created, you can add it to any existing monitors, by setting their Audiences or Failure audiences properties.

To receive notifications about events related to jobs (Airflow, DBT models, etc.), you can add a notification rule. Click the + button in the Job notifications column in the Notifications tab to add a Job notification for that audience.

Routing alerts by triage priority

By default an audience on a monitor is notified about every alert that monitor raises, as soon as it fires. You can instead narrow an audience so it is only notified about alerts at certain triage priorities.

In the Send alerts to section of monitor creation, each audience has four toggles: High, Medium, Low and Not triaged. A row with all four on is unconditional — that audience is notified immediately, exactly as before. Turn any toggle off and the audience becomes conditional — it waits for the Triage Agent to score the alert, and is notified only if the score is one it covers.

A typical setup routes urgent alerts to an on-call channel and everything else to a team channel:

Things worth knowing before you narrow an audience:

• Unconditional audiences are notified immediately. Conditional ones are held until the alert has a priority, so they arrive later — usually within a minute or two.
• Not triaged is its own priority, not an error. An alert lands there when it never gets a score: automatic triage is off, triage could not run, or the wait expired. Some audience must cover Not triaged, or those alerts would reach nobody. Monte Carlo will not let you save a monitor that leaves it uncovered.
• Leaving a scored priority uncovered is allowed. If you route only High, you are choosing that Medium and Low alerts notify nobody on that monitor. You will see a warning, not an error.
• Failure audiences are never conditional. A monitor that fails to run produces no alert to triage, so those notifications always go out immediately.

Narrowing an audience only affects the monitors you narrow it on. The same audience can be unconditional on one monitor and High-only on another.

Priorities can change after the alert is sent

The Triage Agent re-assesses an alert when it changes, which can move its priority after it has already notified someone. Notifications follow the priority upwards only:

• If an alert is re-assessed into a higher priority, audiences for the new, higher priority are notified.
• If it is re-assessed into a lower priority, nobody new is notified. Someone has already been paged, so Monte Carlo does not page a second, wider group afterwards.
• Either way the original notification is updated in place, so the priority shown on it always reflects the latest score.

Receive monitor lifecycle notifications

You can be notified when monitors are enabled, disabled, or deleted by enabling theReceive notifications when a monitor is enabled, disabled, or deleted setting on any Slack, MS Teams, or email recipient channel when creating or editing an audience. From then on, any monitor with that audience applied will send monitor lifecycle events to that channel.

Viewing all your audiences

You can view all your audiences, their monitors and job notification settings, in Settings > Notifications.
Click on any of the numbers in the Monitors or Job notifications columns to see details of the associated monitors and job notification configurations.

GraphQL API

To create and manage new notifications, you should use the Audiences API.

Here's a simplified guide:

Create or Update Audiences:

• Use createOrUpdateAudience to define an audience by specifying its label and notification settings.
• Delete an audience with deleteMonitorLabel.
• Manage individual recipients within an audience using createOrUpdateAudienceNotificationSetting and deleteAudienceNotificationSetting.

Configure Audience Routing Rules:

• Use createOrUpdateAudienceRoutingRule to link a notification rule to an audience. This operation's parameters align with those in the rules field of createOrUpdateNotificationSetting.
• Remove a rule with deleteAudienceRoutingRule.