Configuration
Predictive analytics is configured in three places: the Webserver’s settings (once), each monitor (one switch), and the prediction service (its environment).
Webserver settings
Settings > Webserver settings window > Predictive Analytics (administrators only). This tab has its own Save prediction settings button: the window’s main Save button does not include it.
| Setting | Description | Default |
|---|---|---|
| Prediction API URL | Address of the prediction service, https://host:port. Empty: predictive analytics is off. |
(empty) |
| Default checks to forecast | Checks forecast for a monitor that sets none, and the window of its breach risk. At most 20. | 5 |
| TLS certificate fingerprint (SHA-256) | Pins the service’s certificate: only this one is accepted, self-signed or not. Fetch certificate reads it from the service. Empty: the certificate must be trusted by the Webserver host. | (empty) |
| Signing key (public) | The key to install on the service (PREDICTION_JWT_PUBLIC_KEY). Copy copies it, Rotate key replaces it (see below). |
created on first use |
| Retrain each model | Every day, every week, every two weeks, every month, or never. | every week |
| Manual training only | The Webserver never asks for training on its own; use Train now in the Forecasts table. | off |
| Send predictive alerts | Send forecast and anomaly alerts through the Integrators. | off |
| Alert on | Forecast breach and anomaly, Forecast breach or Anomaly. | both |
| Legacy settings, for an older prediction service | The legacy API key, only for a service still checking PREDICTION_API_KEY (requests are signed in any case), and an InfluxDB connection supplied by this webserver, only for a service started with PREDICTION_ALLOW_CLIENT_BACKEND=true. Leave them empty with a current service. |
(empty) |
Test connection checks the saved settings step by step: the URL is set, the service answers, it accepts this Webserver’s signature, and it reads InfluxDB. A failed step says what to fix. It also shows the service’s version, its number of models and its training queue. An edit not saved yet is flagged Unsaved changes: save it before testing.
The Forecasts table lists every monitor with prediction enabled, one line per probe. Filter monitors narrows it, and it refreshes on its own while models wait for training. When a monitor raises no forecast alert, the reason is written under its forecast.
Monitor settings
In the monitor’s details, tab Prediction (URL, API, TCP, UDP, Ping, DNS, Database, System and WebSocket monitors):
| Setting | Description |
|---|---|
| Enable | Forecast this monitor on every probe it runs on. |
| Checks to forecast | How many checks ahead to forecast for this monitor, up to 20, and the window of its breach risk. Empty: the default. |
| Custom model name | Optional name of the model on the service. Empty: <type>_<monitor key>, for example url_shop_probe1. Set it only to reuse a model trained by hand. |
| Forecast alerts | This monitor’s forecast alerts: Default, On or Off (see below). |
The other settings apply once Enable is on: they are dimmed while it is off. Next to them, the Prediction status box shows, for each probe, the model, the latest forecast with its breach risk, and whether forecast alerts are sent. Administrators open the Predictive Analytics settings from it.
The predicted status is the risk of reaching the monitor’s own minor / major / critical thresholds (see Breach risk): a monitor without thresholds has no predicted status and no forecast alert. Changing a threshold or the prediction count retrains the monitor’s model, in seconds. Predictive alerts use its on status change notification channels.
Forecast alerts per monitor
The Forecast alerts setting of a monitor:
| Value | Forecast alerts for this monitor |
|---|---|
| Default | Follow Send predictive alerts and Alert on. |
| On | Sent, even when Send predictive alerts is off or Alert on is Anomaly. Use it to alert on a few critical monitors only. |
| Off | Never sent. The forecast and its risk stay visible in the Forecasts table. |
It covers forecast (breach risk) alerts only: anomaly alerts always follow the global settings, and a forecast alert is still only sent when the monitor’s breach risk is reliable. The change is picked up within 10 minutes (at once with Run predictions now). Turning a monitor’s forecast alerts off withdraws an open forecast alert; turning them on alerts on the current level once two consecutive forecasts confirm it. The Forecast alerts column of the Forecasts table shows the result for each monitor and probe, Default (on) or Default (off) for the current global settings.
Like the other prediction settings, it is part of the monitor’s CSV export, and an import restores it.
Monitors created by a Sentinel agent are configured on the agent.
Prediction service
The service is configured with environment variables, in a .env file next to api.exe, or with --env-file for the container.
| Variable | Description | Default |
|---|---|---|
INFLUXDB_URL |
InfluxDB the Integrator writes to. Required. | |
INFLUXDB_TOKEN |
InfluxDB token (read access is enough). | |
INFLUXDB_ORG, INFLUXDB_BUCKET |
InfluxDB v2 organisation and bucket. | |
INFLUXDB_DATABASE |
InfluxDB v3 database. | |
PREDICTION_TSDB |
influxdb (v2, Flux) or influxdb3 (v3, SQL). |
influxdb3 if only a database is set, else influxdb |
PREDICTION_JWT_PUBLIC_KEY |
File holding the Webserver’s public signing key (several keys allowed). Re-read when it changes. | |
PREDICTION_REQUIRE_JWT |
true: refuse the legacy API key. |
false |
PREDICTION_API_KEY |
Legacy shared key. Leave empty once every Webserver signs its requests. | |
PREDICTION_SSL_KEY, PREDICTION_SSL_CERT |
TLS key and certificate. Without them the service answers plain HTTP: keep it on localhost then. | private.key, certificate.crt |
API_HOST, API_PORT |
Listening address and port. | 0.0.0.0, 8000 |
PREDICTION_MODEL_DIR |
Folder of the models and of the training state. In the container: the /models volume. |
current folder |
PREDICTION_LOOKBACK_DAYS |
History searched for the latest checks of a forecast. | 30 |
PREDICTION_MODEL_CACHE |
Models kept in memory. | 32 |
PREDICTION_ENGINE |
Model for new trainings: median (fast, see How it works) or lstm (neural network, minutes per monitor). Existing models keep their engine until retrained. |
median |
PREDICTION_RISK_MIN_PRECISION |
Share of right warnings a monitor’s breach risk must reach on its history for forecast alerts to stay on. Higher: fewer monitors, fewer false warnings. Applies from the next training. | 0.5 |
PREDICTION_RISK_MIN_LIFT |
How much better than chance the breach risk must rank breaches. Applies from the next training. | 2 |
PREDICTION_TRAIN_TIMEOUT |
Seconds before a training job is stopped. | 3600 |
PREDICTION_TRAIN_QUEUE_MAX |
Queued jobs before new requests are refused. | 100 |
PREDICTION_DEBUG |
true adds a diagnostic endpoint, POST /debug/series. |
false |
PREDICTION_ALLOW_CLIENT_BACKEND |
true lets callers send their own InfluxDB connection and query. Unsafe: anyone with access to the service could make it connect anywhere. |
false |
Rotate the signing key
- Rotate key in the Predictive Analytics tab. From now on, requests are signed with the new key.
- The text box now holds both the new and the previous public key. Copy it to the service’s key file: the service accepts either key until you change the file again.
- Test connection shows Connected again. Until step 2 is done, it reports that the service refuses this Webserver’s signature.
In a Webserver cluster the key is shared: rotate it once, from any node.
API
Webserver (ADMIN, on the API port, administrator token):
| Method | Path | Description |
|---|---|---|
GET |
/predictions |
Every stored forecast, with model state and alert state. ?monitor=<key> for one monitor. |
GET |
/prediction/:type/:key |
One monitor’s forecast, for example /prediction/url/shop_probe1. |
POST |
/runPredictions |
Refresh the monitor list and forecast every enabled monitor now. |
POST |
/trainPrediction |
Train one monitor now: {"type":"url","key":"shop_probe1"}. |
GET / POST |
/predictionSettings / /updatePredictionSettings |
Read (secrets masked) or save the settings. |
GET |
/predictionHealth |
Same check as Test connection. |
GET |
/predictionPublicKey |
The public signing key(s) for the service. |
POST |
/predictionRotateKey |
Rotate the signing key. |
POST |
/predictionFetchCert |
Read a service’s certificate: {"url":"https://host:8000"}. |
Prediction service (signed requests only, except /health):
| Method | Path | Description |
|---|---|---|
GET |
/health |
Version, models, training queue, authentication modes. No credentials needed, no data returned. |
POST |
/predict |
Forecast one monitor. |
POST |
/train |
Queue a training job. |
GET |
/models, /models/{name} |
Model states: untrained, queued, training, ready, failed. |
GET |
/jobs/{id} |
One training job. |
DELETE |
/models/{name} |
Retire a model (moved to previous). |