How it works

Three components take part: the Integrator stores each check in InfluxDB, the prediction service learns each monitor’s behaviour from that history, and the Webserver asks for forecasts, turns them into a predicted status and decides when to alert.

The components

 Probe ──check──▶ Integrator ──▶ InfluxDB ◀──reads── Prediction service
                     ▲                                   ▲
                     │ predictive alerts                 │ signed HTTPS (forecast, train)
                     └──────────────── Webserver ────────┘

A forecast, step by step

1. The Webserver reads the monitor definitions on each probe and keeps those with prediction enabled (refreshed every 10 minutes, or at once with Run predictions now).
2. On the monitor’s own schedule (a 1-minute monitor every minute), it asks the service for a forecast of that monitor.
3. The service reads the latest checks of that monitor from InfluxDB, leaving out failed checks (ERROR, TIMEOUT, UNKNOWN, CONFIG, DOWNTIME): a timeout says nothing about response time.
4. The model forecasts the next checks: 5 by default, or the monitor’s own prediction count (at most 50), with an 80 % and a 95 % range around them (see Forecast ranges).
5. The service also answers the breach risk: for each of the monitor’s thresholds, the probability that one of those next checks reaches it. The highest level whose risk passes the monitor’s own cut is the predicted status (OK, MINOR, MAJOR or CRITICAL). Where the monitor’s history does not announce its breaches, there is no predicted status and no forecast alert: see Breach risk.
6. The service also scores the latest check: it compares it with what the model expected for it, from the checks before it. Three standard deviations or more of the model’s usual error is an anomaly.
7. The answer says when the newest check was taken. If it is older than three of the monitor’s schedules (15 minutes at least), the forecast is not current: it gets no predicted status and neither raises nor clears an alert (see Data freshness).

The result is stored and shown in the Forecasts table and, with a chart, in the monitor’s Prediction tab. The monitor’s own status, its events, reports and SLA are never changed by a forecast.

The model

Each monitor on each probe has its own model, because a monitor behaves differently from every location.

Why a median and not a neural network

Earlier versions used a neural network (convolution + LSTM layers). It was compared with simple statistical forecasts on the real response-time history of monitors of every type (API, URL, DB, DNS, ping, TCP, WebSocket):

On these monitors the neural network was never more accurate than the median. On a wider set of 65 monitors, the median beat repeating the last check on 98 % of them (error 0.73).

Re-measured in November 2026 on another set, 81 monitors of a demonstration installation (60 days of checks, each statistic fitted on the first half of a monitor’s history and judged on the second half only): error 0.73 of repeating the last check (median over monitors), better on 97.5 % of them; the average of the last 60 checks scored 0.90.

Response times are mostly noise around a level that changes now and then: there is little pattern for a neural network to learn, and the median is the forecast that reacts least to noise.

What anomaly detection catches

An anomaly is a check three standard deviations or more away from what the model expected. In the November 2026 measurement, 3.4 % of ordinary checks were flagged (median over monitors), and a single spike of +50 % of the monitor’s level was flagged 45 % of the time: large spikes are caught, moderate ones often are not. An anomaly alert needs two consecutive flagged forecasts, so a single slow check alerts no one.

The neural network is still available: set PREDICTION_ENGINE=lstm on the prediction service (see Configuration). Each model keeps the engine it was trained with until it is retrained. An LSTM model has no breach risk and no forecast ranges: its predicted status compares the forecast values with the thresholds.

Forecast ranges

The median is a single value. How far the checks may stray from it is shown by two ranges:

They come from the median’s own errors on the monitor (up to 2,000 recent checks), measured separately for each step ahead, so they widen further out and are not symmetric: response times jump up far more than they drop, so the upper bound is usually much further from the forecast than the lower one.

At training, the service also checks the ranges honestly: computed on the older 70 % of those errors, how often did the newest 30 % fall inside? That share is shown in the monitor’s Prediction tab (the 80 % range held 81 % of recent checks). When it is under 70 %, the tab warns that the monitor’s behaviour changed and the range is too narrow until the next training.

In the November 2026 measurement on 81 monitors, the 80 % range held 77 % of later checks (median; 64 % or less on one monitor in ten) and the 95 % range 91 %. Read them as a guide to the usual spread, not as a guarantee: a monitor whose behaviour shifts makes them too narrow until it is retrained (weekly by default).

Accuracy you can check

The monitor’s Prediction tab gives two measures, both against the simplest possible forecast, the next check will be like the last one:

Data freshness

A forecast is only as current as the checks it is made from. If the newest check the service found is older than three of the monitor’s schedules (15 minutes at least), the Webserver marks the forecast not current: No new check since …. It then has no predicted status, raises no alert and does not clear one. This happens when the monitor stopped, the probe is down, or the Integrator no longer writes to InfluxDB.

Breach risk

A flat forecast hardly ever crosses a threshold before the checks themselves do. On real history, comparing the forecast values with the thresholds warned of about 1 % of breaches before they started. So the predicted status comes from a breach risk instead.

At training, the service goes over the monitor’s history at every moment the monitor was still under its lowest threshold, and learns how the chance of a breach within the next checks depends on:

• where the recent median and the last check stand against the threshold,
• the highest of the last 5 checks,
• how often the threshold was reached over the last 60 and 600 checks,
• how much the checks vary.

It then replays this on the most recent 30 % of the history, which it did not learn from, and keeps forecast alerts on only if, there:

The risk level is chosen where warnings were right at least 60 % of the time (the floor plus 10 points), on one half of the replay, and counted on the other half, then the other way round. Choosing and judging on the same checks flatters the result: on test series, a level picked at the 50 % floor turned out right only 33 % to 40 % of the time on later checks. The precision and the share of breaches warned of that the Forecasts table shows are the cross-checked ones.

That risk level becomes the monitor’s cut: a level is forecast when the risk of reaching it passes the cut. The Forecasts table shows the risk; hover it to see how often warnings were right on this monitor’s history.

What to expect

Measured with service 1.4 on the 65 monitors of the benchmark, each with its threshold at its own 95th percentile, on history kept apart from training. Service 1.5 judges more strictly (above), so expect forecast alerts on fewer monitors, with precision figures that hold up better. The 1.5 figures have not been re-measured on long real histories: the November 2026 demonstration installation had at most a few weeks of checks per monitor, too few for any breach risk to be judged.

On the other monitors, breaches came without warning signs in the response times: a sudden outage, a dependency, the network. Forecast alerts stay off for them and the Forecasts table says why; anomaly detection, which reacts to the check itself, still works.

A forecast breach is a risk, not a certainty: about one warning in two is followed by a breach within the forecast horizon.

Training

Nothing needs to be run by hand. The Webserver asks the service to train:

The service trains one model at a time, in a separate process, and replaces a model only when the new one is complete: forecasts continue with the old model meanwhile. The previous model is kept in the service’s previous folder.

With Manual training only, the Webserver never asks for training on its own; use Train now.

Predictive alerts

When Send predictive alerts is on, the Webserver watches each monitor’s predicted status and anomaly flag:

• An alert is sent when a new state is seen on two consecutive forecasts. One noisy forecast alerts no one.
• A recovery is sent when the state goes back to OK, under the same rule.
• Forecast alerts and anomaly alerts are tracked separately. Alert on chooses which ones are sent, and each monitor’s Forecast alerts setting (tab Prediction of the monitor) can override it for forecast alerts (see Configuration).
• Forecast alerts are only sent for monitors whose breach risk proved reliable on their history. The message gives the risk and how often such warnings were right on that monitor. When a monitor’s forecast alerts pause (thresholds changed), an open forecast alert is withdrawn.
• Alerts use the monitor’s on status change channels (email, Slack, Teams, PagerDuty, script), delivered by the Integrators. A monitor with no such channel gets no predictive alert.
• They are clearly marked: the monitor is named <monitor> (forecast), and the message starts with Forecast: or Anomaly:. They have their own PagerDuty incident, separate from the monitor’s.
• The Integrator only notifies: a predictive alert never reaches InfluxDB, the ITSM connectors or the Events page.

Security

High availability

In a Webserver cluster, only the active Webserver runs predictions, training requests and alerts. The signing key is shared by all nodes, so a failover needs no change on the prediction service.

Translations

• fr: Fonctionnement