Troubleshooting
A missing alert produces no error in the UI. Work through the checklist, then confirm with the Integrator log and a test monitor.
Checklist: no alert received
- The monitor’s Actions tab. Is the trigger (On fail / On change) on for this channel? Does the card show a warning (switched off on integrator, Email notifications are unavailable, empty destination with no default)? See What the monitor form tells you.
- The Integrator. Right-click it → edit server → Notifications: is the same trigger on for the channel? Is it the Integrator the monitor’s probe actually uses (probe → Integrators & notification resilience)?
- The rule. ON FAIL needs after x checks + 1 failed checks in a row at or above the chosen level, and stops after for x checks alerts. A monitor at MAJOR never fires a critical or worse rule. See ON FAIL and ON CHANGE.
- Debounce. With Debounce status-change alerts on the probe, a short change is never notified.
- The components are up. The probe and the Integrator are running and the probe reaches the Integrator: the check appears on the Events page.
- The provider. Click ⚡ on the channel’s card in the Integrator’s Notifications block: the answer names the problem. Or test the destination on its own: Email, Slack, Teams, PagerDuty, SIGNL4, script.
Read the Integrator log
The log is log/integrator.log in the Integrator’s install folder. Set Log Level to debug in the Integrator’s settings (Application Logs) while you investigate, and back afterwards.
| Line | What it tells you |
|---|---|
handleSetSetting - notification flags: emailOnF=true emailOnSC=false slackOnF=… smtpEnabled=true |
The Integrator switches, logged each time its settings are saved. |
handleNotifications - [<monitor>] emailIntFlag=true smtpEnabled=true emailRecipient=… emailMonFlag=true alertingFlag=false |
Email decision for one check. alertingFlag=false: the ON FAIL rule is not met for this check. emailRecipient= empty: no destination and no default. |
sendMsgSlack - [<monitor>] Success: alert sent (also sendAlertEmail, sendMsgTeams, sendMsgPagerDuty) |
Delivered. |
… - attempt 1/3 failed: <error>, retrying in 1s |
The provider refused or was unreachable; the error is the provider’s. |
… - all 3 attempts failed: <error> |
Given up for now; the notification goes to the retry buffer. |
runCommand - … Rejected invalid script name / Script not found / timeout occured |
Custom script problems: see Custom script. |
The retry buffer
A notification that failed 3 times is saved as a file in the Integrator’s notif_buffer folder and retried every 30 seconds for one hour; a newer alert for the same monitor and channel replaces it. Files that stay there mean the destination is still refusing: read the error in the log. After one hour the notification is dropped. Scripts are never buffered.
End-to-end test with a failing monitor
A throw-away monitor that always fails proves the whole chain: probe → Integrator → provider.
- Add a URL monitor named
notif-test, on a probe routed through the Integrator you want to test, with the URLhttps://notif-test.invalid/(the.invaliddomain never resolves) and a 1-minute schedule. - In its Actions tab, set the rule to error, after 1 check, for 1 check, and switch On fail and On change on for the channel under test. Leave the destination empty to test the Integrator’s default, or fill it in.
- Save. Check that the channel card shows no warning.
- Within about two minutes you receive one On fail alert, headed
[ERROR] notif-test, on the second failed check, then nothing more (for 1 check). No On change alert comes at first: a new monitor’s first status is not a change. - Delete the monitor once done.
If nothing arrives, follow the checklist above with the monitor name notif-test.
Too many alerts
| Symptom | Cause and action |
|---|---|
| One alert per check while a monitor is down | ON FAIL with for 0 checks (unlimited). Set for N checks. |
| Every alert arrives twice | The probe lists two Integrators and both have the channel on, or ON FAIL and ON CHANGE both fired on the same check (two different alerts). |
| Alerts for a flapping monitor | Switch on Debounce status-change alerts on the probe, raise after x checks, or review the monitor’s thresholds. |
| Alerts during planned work | Declare a downtime. |
Version notes
The Integrator default destinations and the delivery of a channel left blank on a monitor need the probe and the Integrator in version 4.2.0 or later. With an older probe, a monitor whose destination is empty never asks the Integrator to send: fill the destination on the monitor, or upgrade the probe.
Version 4.3.0 brings: the ⚡ test buttons, PagerDuty on Events API v2 (an Events API V1 key must be replaced, see PagerDuty) and its resolve with On fail alone, script arguments and MNS_* variables on the Integrator, the 30-second script timeout fallback, and a new monitor waiting for after x checks + 1 checks before its first ON FAIL alert. Each needs the component concerned upgraded (Webserver for the buttons; Integrator, probes and Sentinel agents for delivery).