Configuration

Predictive analytics is configured in three places: the Webserver’s settings (once), each monitor (one switch), and the prediction service (its environment).

Webserver settings

Settings > Webserver settings window > Predictive Analytics (administrators only). This tab has its own Save prediction settings button: the window’s main Save button does not include it.

Setting Description Default
Prediction API URL Address of the prediction service, https://host:port. Empty: predictive analytics is off. (empty)
Default checks to forecast Checks forecast for a monitor that sets none, and the window of its breach risk. At most 20. 5
Service certificate SHA-256 fingerprint that pins the service’s certificate: only this one is accepted, self-signed or not. Fetch certificate reads it from the service. Empty: the certificate must be trusted by the Webserver host. The badge says which applies: Pinned or System trust. (empty)
Webserver signing key Its fingerprint is shown; Copy public key copies the key to install on the service (PREDICTION_JWT_PUBLIC_KEY), Show key displays it, Rotate key replaces it (see below). created on first use
Retrain each model Every day, every week, every two weeks, every month, or never. every week
Manual training only The Webserver never asks for training on its own; use Train now in the Forecasts table. off
Send predictive alerts Send forecast and anomaly alerts through the Integrators. off
Alert on Forecast breach and anomaly, Forecast breach or Anomaly. both
Legacy settings, for an older prediction service The legacy API key, only for a service still checking PREDICTION_API_KEY (requests are signed in any case), and an InfluxDB connection supplied by this webserver, only for a service started with PREDICTION_ALLOW_CLIENT_BACKEND=true. Leave them empty with a current service. (empty)

The card at the top of the tab shows the four setup steps: the URL is set, the service answers, it accepts this Webserver’s signature, and it reads InfluxDB. Its colour gives the overall state (green: forecasts run; orange: a step to fix; red: unreachable), and the first failing step’s fix is written under it as Next step. Test connection checks the saved settings again and shows the service’s version, its number of models, its training queue and its latency. An edit not saved yet is flagged Unsaved changes next to Save prediction settings: save it before testing.

The Forecasts table lists every monitor with prediction enabled, one line per probe. Chips above it count the monitors forecasting, waiting for a model, in error, without recent checks, with an anomaly and with forecast alerts. Filter monitors narrows it, and it refreshes on its own while models wait for training. When a monitor raises no forecast alert, the reason is written under its forecast.

Monitor settings

In the monitor’s details, tab Prediction (URL, API, TCP, UDP, Ping, DNS, Database, System and WebSocket monitors):

Setting Description
Enable Forecast this monitor on every probe it runs on.
Checks to forecast How many checks ahead to forecast for this monitor, up to 20, and the window of its breach risk. Empty: the default.
Custom model name Optional name of the model on the service. Empty: <type>_<monitor key>, for example url_shop_probe1. Set it only to reuse a model trained by hand.
Forecast alerts This monitor’s forecast alerts: Default, On or Off (see below).

The other settings apply once Enable is on: they are dimmed while it is off. Next to them, the Prediction status box shows, for each probe, the model, the latest forecast with its breach risk, and whether forecast alerts are sent, then a chart of the latest checks with the forecast, its 80 % and 95 % ranges, the thresholds and any anomaly, and how accurate the forecast has been on this monitor (see How it works). Administrators open the Predictive Analytics settings from it.

The predicted status is the risk of reaching the monitor’s own minor / major / critical thresholds (see Breach risk): a monitor without thresholds has no predicted status and no forecast alert. Changing a threshold or the prediction count retrains the monitor’s model, in seconds. Predictive alerts use its on status change notification channels.

Forecast alerts per monitor

The Forecast alerts setting of a monitor:

Value Forecast alerts for this monitor
Default Follow Send predictive alerts and Alert on.
On Sent, even when Send predictive alerts is off or Alert on is Anomaly. Use it to alert on a few critical monitors only.
Off Never sent. The forecast and its risk stay visible in the Forecasts table.

It covers forecast (breach risk) alerts only: anomaly alerts always follow the global settings, and a forecast alert is still only sent when the monitor’s breach risk is reliable. The change is picked up within 10 minutes (at once with Run predictions now). Turning a monitor’s forecast alerts off withdraws an open forecast alert; turning them on alerts on the current level once two consecutive forecasts confirm it. The Forecast alerts column of the Forecasts table shows the result for each monitor and probe, Default (on) or Default (off) for the current global settings.

Like the other prediction settings, it is part of the monitor’s CSV export, and an import restores it.

Monitors created by a Sentinel agent are configured on the agent.

Prediction service

The service is configured with environment variables, in a .env file next to api.exe, or with --env-file for the container.

Variable Description Default
INFLUXDB_URL InfluxDB the Integrator writes to. Required.
INFLUXDB_TOKEN InfluxDB token (read access is enough).
INFLUXDB_ORG, INFLUXDB_BUCKET InfluxDB v2 organisation and bucket.
INFLUXDB_DATABASE InfluxDB v3 database.
PREDICTION_TSDB influxdb (v2, Flux) or influxdb3 (v3, SQL). influxdb3 if only a database is set, else influxdb
PREDICTION_JWT_PUBLIC_KEY File holding the Webserver’s public signing key (several keys allowed). Re-read when it changes.
PREDICTION_REQUIRE_JWT true: refuse the legacy API key. false
PREDICTION_API_KEY Legacy shared key. Leave empty once every Webserver signs its requests.
PREDICTION_SSL_KEY, PREDICTION_SSL_CERT TLS key and certificate. Without them the service answers plain HTTP: keep it on localhost then. private.key, certificate.crt
API_HOST, API_PORT Listening address and port. 0.0.0.0, 8000
PREDICTION_MODEL_DIR Folder of the models and of the training state. In the container: the /models volume. current folder
PREDICTION_LOOKBACK_DAYS History searched for the latest checks of a forecast. 30
PREDICTION_MODEL_CACHE Models kept in memory. 32
PREDICTION_ENGINE Model for new trainings: median (fast, see How it works) or lstm (neural network, minutes per monitor). Existing models keep their engine until retrained. median
PREDICTION_RISK_MIN_PRECISION Share of right warnings a monitor’s breach risk must reach on its history for forecast alerts to stay on. Higher: fewer monitors, fewer false warnings. Applies from the next training. 0.5
PREDICTION_RISK_MIN_LIFT How much better than chance the breach risk must rank breaches. Applies from the next training. 2
PREDICTION_TRAIN_TIMEOUT Seconds before a training job is stopped. 3600
PREDICTION_TRAIN_QUEUE_MAX Queued jobs before new requests are refused. 100
PREDICTION_MAX_CONNECTIONS Open connections accepted before the service answers 503. Idle keep-alive connections count: below about 16, the Webserver’s 8 parallel forecasts are refused now and then. 64
PREDICTION_SAVE_PREDICTIONS true writes each forecast to <model>_predictions.txt in the model folder (a file write per forecast, read by nothing). false
PREDICTION_DEBUG true adds a diagnostic endpoint, POST /debug/series. false
PREDICTION_ALLOW_CLIENT_BACKEND true lets callers send their own InfluxDB connection and query. Unsafe: anyone with access to the service could make it connect anywhere. false

InfluxDB 3 Core

InfluxDB 3 Core refuses any query that would read more than --query-file-limit Parquet files: 432 by default, about 72 hours of data. From service 1.5 the history is read newest first, in time slices sized to stay under that limit, so training over 280 days and forecasts work without raising it. A service before 1.5 fails there with Query would scan 432 Parquet files, exceeding the file limit: upgrade it.

Rotate the signing key

  1. Rotate key in the Predictive Analytics tab. From now on, requests are signed with the new key.
  2. The text box now holds both the new and the previous public key. Copy it to the service’s key file: the service accepts either key until you change the file again.
  3. Test connection shows Connected again. Until step 2 is done, it reports that the service refuses this Webserver’s signature.

In a Webserver cluster the key is shared: rotate it once, from any node.

API

Webserver (ADMIN, on the API port, administrator token):

Method Path Description
GET /predictions Every stored forecast, with model state and alert state. ?monitor=<key> for one monitor.
GET /prediction/:type/:key One monitor’s forecast, for example /prediction/url/shop_probe1.
POST /runPredictions Refresh the monitor list and forecast every enabled monitor now.
POST /trainPrediction Train one monitor now: {"type":"url","key":"shop_probe1"}.
GET / POST /predictionSettings / /updatePredictionSettings Read (secrets masked) or save the settings.
GET /predictionHealth Same check as Test connection.
GET /predictionPublicKey The public signing key(s) for the service.
POST /predictionRotateKey Rotate the signing key.
POST /predictionFetchCert Read a service’s certificate: {"url":"https://host:8000"}.

Prediction service (signed requests only, except /health):

Method Path Description
GET /health Version, models, training queue, authentication modes. No credentials needed, no data returned.
POST /predict Forecast one monitor.
POST /train Queue a training job.
GET /models, /models/{name} Model states: untrained, queued, training, ready, failed.
GET /jobs/{id} One training job.
DELETE /models/{name} Retire a model (moved to previous).

Translations