PagerDuty

Integrating PagerDuty with Monte Carlo provides robust alerting and incident management capabilities, ensuring all data issues are promptly addressed and resolved. With real-time notifications, PagerDuty alerts on-call responders immediately when data issues arise, helping to prevent Data Downtime. The integration allows data teams to act quickly, assess the downstream business impact, and resolve issues efficiently. By leveraging PagerDuty's alerting system, teams can maintain the reliability and integrity of their data pipelines, ensuring continuous data observability and proactive incident management.

Integration Walkthrough

In PagerDuty

1. Go to Services. There are two ways to add an integration to a service:
   1. If you are adding your integration to an existing service: Click the name of the service you want to add the integration to. Then, select the Integrations tab, click the link to add a new integration, and select Monte Carlo from the integrations' selector.
   2. If you are creating a new service for your integration: Please read the PagerDuty documentation in Services and Integrations and follow the steps outlined in the Create a New Service section. Select Monte Carlo as the integration to configure in step 5 of that guide.
2. In the service, go to the Integrations tab. Open the Monte Carlo integration, and you will see a field on the right side with the Integration Key. Copy this value and keep it saved in a safe place, as it will be used when you configure the integration with Monte Carlo in the next section.

In Monte Carlo

1. From the Settings tab, click on Integrations on the left side of the screen.
2. In the Notifications and collaboration section, click Create and select PagerDuty.
3. You will need to enter:
   • Display name: This gives a name to identify the integration when using it in audiences. For example, you can use the name of the service defined in PagerDuty.
   • Integration Key: This is the Integration Key associated with the integration in the PagerDuty service (you can get it from PageDuty by opening the Integrations tab inside the PagerDuty service). It is a 32-character text.
4. Enable Sync status between Monte Carlo and PagerDuty (bidirectional) if you want to keep them synchronized. When this is enabled, you will need to configure a Webhook in PagerDuty (see the section below).
5. Click Create.
6. Then, you will be able to select the configured PagerDuty service within an audience (in Settings -> Notifications), to send notifications to PagerDuty for all notifications associated with that audience.

Sync Status between Monte Carlo and PagerDuty

When enabled, if the PagerDuty incident is marked as:

• Acknowledged: It will assign the status Acknowledged in Monte Carlo (if no other status is set).
• Resolved: It will assign the status Fixed (if no other resolved status is set).

On the other side, if the status is changed from Monte Carlo to:

• Acknowledged / Work in Progress: It will tell PagerDuty that the incident was acknowledged (by the Monte Carlo Integration).
• Fixed / Expected / No Action Needed / False Positive: It will tell PagerDuty that the incident was resolved (by the Monte Carlo Integration). After the incident is marked as Resolved in PagerDuty, it cannot be returned to Acknowledged (even if the status is changed in Monte Carlo)

To configure it in Monte Carlo

1. Sync status must be enabled in the PagerDuty integration (in Settings -> Integrations, in the Notifications and collaboration section).
2. Edit the PagerDuty integration, and copy the Webhook URL; you will need it to configure the Webhook in PagerDuty, which enables syncing statuses.
   • When you are defining a new PagerDuty integration, the Webhook URL is shown after you click Create.

To configure it in PagerDuty

1. Go to Services, select the service, and open the Integrations tab.
2. Go to the bottom, and click on the option to add webhooks to the service.
3. Click New Webhook.
4. Paste the Webhook URL obtained from Monte Carlo, and type a description (e.g., Monte Carlo).
5. Keep selected all event subscriptions.
6. Click Add Webhook at the bottom of the page.
7. (Optional) After the Webhook Subscription is created, PagerDuty will share a secret to verify webhook payloads. Copy the secret and configure it in Monte Carlo (in the PagerDuty service integration) to validate the authenticity of events received through the webhook.

Send High Urgency notifications to PagerDuty

At the audience level, you can associate a PagerDuty service and configure it to send high urgency notifications:

1. Go to Settings -> Notifications.
2. In the audience, open the three-dot menu, and select Edit.
3. Add the PagerDuty recipient, or edit an existing one.
4. Select Send High Urgency for dynamic notifications.
5. Click Update Recipient.

When high urgency notifications are selected, Monte Carlo will send alerts with severity error to PagerDuty. It will create High Urgency notifications if the service is configured with Dynamic notifications in PagerDuty (more info here). If the PagerDuty service is configured to send low/high urgency notifications explicitly, this setting will not override that configuration.