Configuration

Predictive analytics is configured in three places: the Webserver’s settings (once), each monitor (one switch), and the prediction service (its environment).

Webserver settings

Settings > Webserver settings window > Predictive Analytics (administrators only). This tab has its own Save prediction settings button: the window’s main Save button does not include it.

Setting Description Default
Prediction API URL Address of the prediction service, https://host:port. Empty: predictive analytics is off. (empty)
Default checks to forecast Checks forecast for a monitor that sets none, and the window of its breach risk. At most 20. 5
TLS certificate fingerprint (SHA-256) Pins the service’s certificate: only this one is accepted, self-signed or not. Fetch certificate reads it from the service. Empty: the certificate must be trusted by the Webserver host. (empty)
Signing key (public) The key to install on the service (PREDICTION_JWT_PUBLIC_KEY). Copy copies it, Rotate key replaces it (see below). created on first use
Retrain each model Every day, every week, every two weeks, every month, or never. every week
Manual training only The Webserver never asks for training on its own; use Train now in the Forecasts table. off
Send predictive alerts Send forecast and anomaly alerts through the Integrators. off
Alert on Forecast breach and anomaly, Forecast breach or Anomaly. both
Legacy settings, for an older prediction service The legacy API key, only for a service still checking PREDICTION_API_KEY (requests are signed in any case), and an InfluxDB connection supplied by this webserver, only for a service started with PREDICTION_ALLOW_CLIENT_BACKEND=true. Leave them empty with a current service. (empty)

Test connection checks the saved settings step by step: the URL is set, the service answers, it accepts this Webserver’s signature, and it reads InfluxDB. A failed step says what to fix. It also shows the service’s version, its number of models and its training queue. An edit not saved yet is flagged Unsaved changes: save it before testing.

The Forecasts table lists every monitor with prediction enabled, one line per probe. Filter monitors narrows it, and it refreshes on its own while models wait for training. When a monitor raises no forecast alert, the reason is written under its forecast.

Monitor settings

In the monitor’s details, tab Prediction (URL, API, TCP, UDP, Ping, DNS, Database, System and WebSocket monitors):

Setting Description
Enable Forecast this monitor on every probe it runs on.
Checks to forecast How many checks ahead to forecast for this monitor, up to 20, and the window of its breach risk. Empty: the default.
Custom model name Optional name of the model on the service. Empty: <type>_<monitor key>, for example url_shop_probe1. Set it only to reuse a model trained by hand.
Forecast alerts This monitor’s forecast alerts: Default, On or Off (see below).

The other settings apply once Enable is on: they are dimmed while it is off. Next to them, the Prediction status box shows, for each probe, the model, the latest forecast with its breach risk, and whether forecast alerts are sent. Administrators open the Predictive Analytics settings from it.

The predicted status is the risk of reaching the monitor’s own minor / major / critical thresholds (see Breach risk): a monitor without thresholds has no predicted status and no forecast alert. Changing a threshold or the prediction count retrains the monitor’s model, in seconds. Predictive alerts use its on status change notification channels.

Forecast alerts per monitor

The Forecast alerts setting of a monitor:

Value Forecast alerts for this monitor
Default Follow Send predictive alerts and Alert on.
On Sent, even when Send predictive alerts is off or Alert on is Anomaly. Use it to alert on a few critical monitors only.
Off Never sent. The forecast and its risk stay visible in the Forecasts table.

It covers forecast (breach risk) alerts only: anomaly alerts always follow the global settings, and a forecast alert is still only sent when the monitor’s breach risk is reliable. The change is picked up within 10 minutes (at once with Run predictions now). Turning a monitor’s forecast alerts off withdraws an open forecast alert; turning them on alerts on the current level once two consecutive forecasts confirm it. The Forecast alerts column of the Forecasts table shows the result for each monitor and probe, Default (on) or Default (off) for the current global settings.

Like the other prediction settings, it is part of the monitor’s CSV export, and an import restores it.

Monitors created by a Sentinel agent are configured on the agent.

Prediction service

The service is configured with environment variables, in a .env file next to api.exe, or with --env-file for the container.

Variable Description Default
INFLUXDB_URL InfluxDB the Integrator writes to. Required.
INFLUXDB_TOKEN InfluxDB token (read access is enough).
INFLUXDB_ORG, INFLUXDB_BUCKET InfluxDB v2 organisation and bucket.
INFLUXDB_DATABASE InfluxDB v3 database.
PREDICTION_TSDB influxdb (v2, Flux) or influxdb3 (v3, SQL). influxdb3 if only a database is set, else influxdb
PREDICTION_JWT_PUBLIC_KEY File holding the Webserver’s public signing key (several keys allowed). Re-read when it changes.
PREDICTION_REQUIRE_JWT true: refuse the legacy API key. false
PREDICTION_API_KEY Legacy shared key. Leave empty once every Webserver signs its requests.
PREDICTION_SSL_KEY, PREDICTION_SSL_CERT TLS key and certificate. Without them the service answers plain HTTP: keep it on localhost then. private.key, certificate.crt
API_HOST, API_PORT Listening address and port. 0.0.0.0, 8000
PREDICTION_MODEL_DIR Folder of the models and of the training state. In the container: the /models volume. current folder
PREDICTION_LOOKBACK_DAYS History searched for the latest checks of a forecast. 30
PREDICTION_MODEL_CACHE Models kept in memory. 32
PREDICTION_ENGINE Model for new trainings: median (fast, see How it works) or lstm (neural network, minutes per monitor). Existing models keep their engine until retrained. median
PREDICTION_RISK_MIN_PRECISION Share of right warnings a monitor’s breach risk must reach on its history for forecast alerts to stay on. Higher: fewer monitors, fewer false warnings. Applies from the next training. 0.5
PREDICTION_RISK_MIN_LIFT How much better than chance the breach risk must rank breaches. Applies from the next training. 2
PREDICTION_TRAIN_TIMEOUT Seconds before a training job is stopped. 3600
PREDICTION_TRAIN_QUEUE_MAX Queued jobs before new requests are refused. 100
PREDICTION_DEBUG true adds a diagnostic endpoint, POST /debug/series. false
PREDICTION_ALLOW_CLIENT_BACKEND true lets callers send their own InfluxDB connection and query. Unsafe: anyone with access to the service could make it connect anywhere. false

Rotate the signing key

  1. Rotate key in the Predictive Analytics tab. From now on, requests are signed with the new key.
  2. The text box now holds both the new and the previous public key. Copy it to the service’s key file: the service accepts either key until you change the file again.
  3. Test connection shows Connected again. Until step 2 is done, it reports that the service refuses this Webserver’s signature.

In a Webserver cluster the key is shared: rotate it once, from any node.

API

Webserver (ADMIN, on the API port, administrator token):

Method Path Description
GET /predictions Every stored forecast, with model state and alert state. ?monitor=<key> for one monitor.
GET /prediction/:type/:key One monitor’s forecast, for example /prediction/url/shop_probe1.
POST /runPredictions Refresh the monitor list and forecast every enabled monitor now.
POST /trainPrediction Train one monitor now: {"type":"url","key":"shop_probe1"}.
GET / POST /predictionSettings / /updatePredictionSettings Read (secrets masked) or save the settings.
GET /predictionHealth Same check as Test connection.
GET /predictionPublicKey The public signing key(s) for the service.
POST /predictionRotateKey Rotate the signing key.
POST /predictionFetchCert Read a service’s certificate: {"url":"https://host:8000"}.

Prediction service (signed requests only, except /health):

Method Path Description
GET /health Version, models, training queue, authentication modes. No credentials needed, no data returned.
POST /predict Forecast one monitor.
POST /train Queue a training job.
GET /models, /models/{name} Model states: untrained, queued, training, ready, failed.
GET /jobs/{id} One training job.
DELETE /models/{name} Retire a model (moved to previous).

Translations