PagerDuty

Integrating PagerDuty with Monte Carlo provides robust alerting and incident management capabilities, ensuring all data issues are promptly addressed and resolved. With real-time notifications, PagerDuty alerts on-call responders immediately when data issues arise, helping to prevent Data Downtime. The integration allows data teams to act quickly, assess the downstream business impact, and resolve issues efficiently. By leveraging PagerDuty's alerting system, teams can maintain the reliability and integrity of their data pipelines, ensuring continuous Data + AI observability and proactive incident management.

Integration Walkthrough

In PagerDuty

1. Go to Services. There are two ways to add an integration to a service:
   1. If you are adding your integration to an existing service: Click the name of the service you want to add the integration to. Then, select the Integrations tab, click the link to add a new integration, and select Monte Carlo from the integrations' selector.
   2. If you are creating a new service for your integration: Please read the PagerDuty documentation in Services and Integrations and follow the steps outlined in the Create a New Service section. Select Monte Carlo as the integration to configure in step 5 of that guide.
2. In the service, go to the Integrations tab. Open the Monte Carlo integration, and you will see a field on the right side with the Integration Key. Copy this value and keep it saved in a safe place, as it will be used when you configure the integration with Monte Carlo in the next section.

In Monte Carlo

1. From the Settings tab, click on Integrations on the left side of the screen.
2. In the Notifications and collaboration section, click Create and select PagerDuty.
3. You will need to enter:
   • Display name: This gives a name to identify the integration when using it in audiences. For example, you can use the name of the service defined in PagerDuty.
   • Integration Key: This is the Integration Key associated with the integration in the PagerDuty service (you can get it from PageDuty by opening the Integrations tab inside the PagerDuty service). It is a 32-character text.
4. Enable Sync status between Monte Carlo and PagerDuty (bidirectional) if you want to keep them synchronized. When this is enabled, you will need to configure a Webhook in PagerDuty (see the section below).
5. Click Create.
6. Then, you will be able to select the configured PagerDuty service within an audience (in Settings -> Notifications), to send notifications to PagerDuty for all notifications associated with that audience.

Sync Status between Monte Carlo and PagerDuty

When enabled, if the PagerDuty incident is marked as:

• Acknowledged: It will assign the status Acknowledged in Monte Carlo (if no other status is set).
• Resolved: It will assign the status Fixed (if no other resolved status is set).

On the other side, if the status is changed from Monte Carlo to:

• Acknowledged / Work in Progress: It will tell PagerDuty that the incident was acknowledged (by the Monte Carlo Integration).
• Fixed / Expected / No Action Needed / False Positive: It will tell PagerDuty that the incident was resolved (by the Monte Carlo Integration). After the incident is marked as Resolved in PagerDuty, it cannot be returned to Acknowledged (even if the status is changed in Monte Carlo)

To configure it in Monte Carlo

1. Sync status must be enabled in the PagerDuty integration (in Settings -> Integrations, in the Notifications and collaboration section).
2. Edit the PagerDuty integration, and copy the Webhook URL; you will need it to configure the Webhook in PagerDuty, which enables syncing statuses.
   • When you are defining a new PagerDuty integration, the Webhook URL is shown after you click Create.

To configure it in PagerDuty

1. Go to Services, select the service, and open the Integrations tab.
2. Go to the bottom, and click on the option to add webhooks to the service.
3. Click New Webhook.
4. Paste the Webhook URL obtained from Monte Carlo, and type a description (e.g., Monte Carlo).
5. Keep selected all event subscriptions.
6. Click Add Webhook at the bottom of the page.
7. (Optional) After the Webhook Subscription is created, PagerDuty will share a secret to verify webhook payloads. Copy the secret and configure it in Monte Carlo (in the PagerDuty service integration) to validate the authenticity of events received through the webhook.

Send High Urgency notifications to PagerDuty

At the audience level, you can associate a PagerDuty service and configure it to send high urgency notifications:

1. Go to Settings -> Notifications.
2. In the audience, open the three-dot menu, and select Edit.
3. Add the PagerDuty recipient, or edit an existing one.
4. Select Send High Urgency for dynamic notifications.
5. Click Update Recipient.

When high urgency notifications are selected, Monte Carlo will send alerts with severity error to PagerDuty. It will create High Urgency notifications if the service is configured with Dynamic notifications in PagerDuty (more info here). If the PagerDuty service is configured to send low/high urgency notifications explicitly, this setting will not override that configuration.

Monitor tags

Alerts carry the monitor tags of the monitor that raised them, under custom_details.monitor_tags in the PagerDuty event.

• Tags are rendered as a single comma-separated string. Each tag appears as name=value, or just name when the tag has no value.
• Tags are sent exactly as they appear in Monte Carlo — no reformatting or normalization.
• Every monitor type that supports tags is included. See Monitor tags for which monitors can be tagged and how to apply tags.
• When an incident groups several monitors that share a tag, that tag is sent once.
• The monitor_tags key is omitted entirely when the monitor has no tags — worth knowing if you write Event Orchestration rules that test for the field.

For example, a monitor tagged team / analytics and tier / 1:

"custom_details": {
  "body": "...",
  "monitor_tags": "team=analytics, tier=1"
}

custom_details carries other fields alongside monitor_tags, including body, incident_type, priority, and domains.

📘

Verifying with a test notification

A test notification sends a sample event that isn't tied to any monitor, so it won't include monitor tags. To confirm your tags are arriving, trigger or wait for a real alert from a tagged monitor and inspect that event in PagerDuty.