On this page

PagerDuty

The Integrator sends each alert to PagerDuty as an Events API v2 event. One monitor = one incident: later alerts are merged into it, and the monitor’s recovery resolves it. You need the integration key of an Events API V2 integration on a PagerDuty service.

How Mugnsoft uses PagerDuty
API PagerDuty Events API v2 — https://events.pagerduty.com/v2/enqueue
Key The integration key, sent as routing_key
Incident key dedup_key = <probe>:<monitor>, for example probe-paris:url_web-shop_probe-paris. Every alert of the monitor lands in the same incident.
Non-OK status trigger event: opens the incident, or is de-duplicated into the open one
Severity CRITICAL → critical; MAJOR, ERROR, TIMEOUT, EXCEPTION, CONFIG → error; MINOR → warning
Back to OK resolve event: resolves the incident — with On change, and also with On fail alone
Summary The alert headline: [CRITICAL] web-shop · HTTP 503 · probe-paris
Details Monitor, type, status, target, probe, check time and message as custom details; the probe as source; a Locate in 3D view link

Step 1: create the integration in PagerDuty

You need the Manager role (or above) on the service.

  1. In PagerDuty, go to Services → Service Directory and open the service that should receive Mugnsoft incidents, or create one (+ New Service) with the escalation policy of the team on call.
  2. Open the Integrations tab and click Add another integration (or Add an integration).
  3. Search for and select Events API V2, then click Add.
  4. Expand the new integration with the chevron on its right and copy the Integration Key (32 characters).

Official guide: PagerDuty — Services and Integrations. API reference: Events API v2.

The key lets anyone open incidents on the service: treat it as a secret. Mugnsoft stores it encrypted.

Step 2: configure Mugnsoft

Default for every monitor of an Integrator — Integrator → General → Notifications → PagerDuty: switch On fail and/or On change on, and paste the key in Default API key.

Per monitor — monitor → Actions → PagerDuty card: switch the trigger(s) on. Leave API Key empty to use the Integrator’s default, or paste the key of another service to page another team.

Save both forms with the disk icon.

Recommended triggers:

Goal On fail On change
Page on a lasting failure, resolve automatically on (rule: e.g. error, after 2, for 1) off
Mirror every status change in one incident off on

With On fail alone, the incident is opened when the rule is met and resolved when the monitor is back to OK; short failures that never meet the rule open nothing. Trigger events that carry the key of an open incident are de-duplicated into it: repeated ON FAIL alerts do not page again.

Test it

From Mugnsoft — in the Integrator’s Notifications block, click ⚡ on the PagerDuty card. The Integrator opens a test incident ([TEST] Mugnsoft notification test) with the key shown in the field, or its saved default, and resolves it at once. This pages whoever is on call: warn them, or test on a service with a test escalation policy.

PagerDuty queues events before checking that the key exists, so a well-formed but wrong key is also accepted. Check that the test incident appears in the service’s incident log: if it does not, the key belongs to another service or account.

By hand, from the Integrator’s host:


# Open a test incident
curl -s -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{"routing_key":"YOUR_INTEGRATION_KEY","event_action":"trigger","dedup_key":"mugnsoft-test","payload":{"summary":"Mugnsoft test incident","source":"mugnsoft","severity":"info"}}'

# Resolve it
curl -s -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{"routing_key":"YOUR_INTEGRATION_KEY","event_action":"resolve","dedup_key":"mugnsoft-test"}'

Each call should answer {"status":"success","message":"Event processed",...} (HTTP 202).

Then run the end-to-end test.

Troubleshooting

The Integrator logs the PagerDuty error (sendMsgPagerDuty - [monitor] - all 3 attempts failed: <error>).

Symptom Cause and action
events API v2 returned HTTP 400: Invalid routing key Malformed or truncated key. Copy the Integration Key of the service’s Events API V2 integration again and run the test.
The test succeeds but no incident appears Wrong but well-formed key (another service or account, or an old Events API V1 key), the integration or service is disabled, or the service is in maintenance mode.
HTTP 429 Rate limit: an ON FAIL rule for 0 checks on many monitors sends an event at every check. Use for N checks.
Incidents stopped arriving after an upgrade to 4.3.0 The key belongs to an Events API V1 integration. Create an Events API V2 integration and replace the key (see the warning at the top).
Incidents never resolve The recovery was held by Debounce status-change alerts on the probe and the monitor failed again before it was stable.
Several incidents for one outage One incident per monitor and probe by design: a site checked from three probes opens three incidents. Group them with PagerDuty’s alert grouping if needed.
An incident resolved in PagerDuty comes back The monitor is still failing: the next trigger event opens a new incident with the same key.

More in Troubleshooting.

Translations

SIGNL4