How it works

Three components take part: the Integrator stores each check in InfluxDB, the prediction service learns each monitor’s behaviour from that history, and the Webserver asks for forecasts, turns them into a predicted status and decides when to alert.

The components

 Probe ──check──▶ Integrator ──▶ InfluxDB ◀──reads── Prediction service
                     ▲                                   ▲
                     │ predictive alerts                 │ signed HTTPS (forecast, train)
                     └──────────────── Webserver ────────┘

A forecast, step by step

1. The Webserver reads the monitor definitions on each probe and keeps those with prediction enabled (refreshed every 10 minutes, or at once with Run predictions now).
2. On the monitor’s own schedule (a 1-minute monitor every minute), it asks the service for a forecast of that monitor.
3. The service reads the latest checks of that monitor from InfluxDB, leaving out failed checks (ERROR, TIMEOUT, UNKNOWN, CONFIG, DOWNTIME): a timeout says nothing about response time.
4. The model forecasts the next checks: 5 by default, or the monitor’s own prediction count.
5. The service also answers the breach risk: for each of the monitor’s thresholds, the probability that one of those next checks reaches it. The highest level whose risk passes the monitor’s own cut is the predicted status (OK, MINOR, MAJOR or CRITICAL). Where the monitor’s history does not announce its breaches, there is no predicted status and no forecast alert: see Breach risk.
6. The service also scores the latest check: it compares it with what the model expected for it, from the checks before it. Three standard deviations or more of the model’s usual error is an anomaly.

The result is stored and shown in the Forecasts table. The monitor’s own status, its events, reports and SLA are never changed by a forecast.

The model

Each monitor on each probe has its own model, because a monitor behaves differently from every location.

Why a median and not a neural network

Earlier versions used a neural network (convolution + LSTM layers). It was compared with simple statistical forecasts on the real response-time history of monitors of every type (API, URL, DB, DNS, ping, TCP, WebSocket):

On these monitors the neural network was never more accurate than the median. On a wider set of 65 monitors, the median beat repeating the last check on 98 % of them (error 0.73), and detected 94 % of abnormal spikes for 2.8 % false alarms.

Response times are mostly noise around a level that changes now and then: there is little pattern for a neural network to learn, and the median is the forecast that reacts least to noise.

The neural network is still available: set PREDICTION_ENGINE=lstm on the prediction service (see Configuration). Each model keeps the engine it was trained with until it is retrained. An LSTM model has no breach risk: its predicted status compares the forecast values with the thresholds.

Breach risk

A flat forecast hardly ever crosses a threshold before the checks themselves do. On real history, comparing the forecast values with the thresholds warned of about 1 % of breaches before they started. So the predicted status comes from a breach risk instead.

At training, the service goes over the monitor’s history at every moment the monitor was still under its lowest threshold, and learns how the chance of a breach within the next checks depends on:

• where the recent median and the last check stand against the threshold,
• the highest of the last 5 checks,
• how often the threshold was reached over the last 60 and 600 checks,
• how much the checks vary.

It then replays this on the most recent 30 % of the history, which it did not learn from, and keeps forecast alerts on only if, there:

That risk level becomes the monitor’s cut: a level is forecast when the risk of reaching it passes the cut. The Forecasts table shows the risk; hover it to see how often warnings were right on this monitor’s history.

What to expect

Measured on the 65 monitors of the benchmark, each with its threshold at its own 95th percentile, on history kept apart from training:

On the other monitors, breaches came without warning signs in the response times: a sudden outage, a dependency, the network. Forecast alerts stay off for them and the Forecasts table says why; anomaly detection, which reacts to the check itself, still works.

A forecast breach is a risk, not a certainty: about one warning in two is followed by a breach within the forecast horizon.

Training

Nothing needs to be run by hand. The Webserver asks the service to train:

The service trains one model at a time, in a separate process, and replaces a model only when the new one is complete: forecasts continue with the old model meanwhile. The previous model is kept in the service’s previous folder.

With Manual training only, the Webserver never asks for training on its own; use Train now.

Predictive alerts

When Send predictive alerts is on, the Webserver watches each monitor’s predicted status and anomaly flag:

• An alert is sent when a new state is seen on two consecutive forecasts. One noisy forecast alerts no one.
• A recovery is sent when the state goes back to OK, under the same rule.
• Forecast alerts and anomaly alerts are tracked separately. Alert on chooses which ones are sent, and each monitor’s Forecast alerts setting (tab Prediction of the monitor) can override it for forecast alerts (see Configuration).
• Forecast alerts are only sent for monitors whose breach risk proved reliable on their history. The message gives the risk and how often such warnings were right on that monitor. When a monitor’s forecast alerts pause (thresholds changed), an open forecast alert is withdrawn.
• Alerts use the monitor’s on status change channels (email, Slack, Teams, PagerDuty, script), delivered by the Integrators. A monitor with no such channel gets no predictive alert.
• They are clearly marked: the monitor is named <monitor> (forecast), and the message starts with Forecast: or Anomaly:. They have their own PagerDuty incident, separate from the monitor’s.
• The Integrator only notifies: a predictive alert never reaches InfluxDB, the ITSM connectors or the Events page.

Security

High availability

In a Webserver cluster, only the active Webserver runs predictions, training requests and alerts. The signing key is shared by all nodes, so a failover needs no change on the prediction service.

Translations

• fr: Fonctionnement