Troubleshooting
Start with Test connection in the Predictive Analytics tab, then read the Forecasts table: most problems show there with their cause.
Connection
| Status line says | Cause | Fix |
|---|---|---|
| Not configured | No Prediction API URL. | Enter the service address and save. |
| Unreachable: … connection refused or timeout | The service is not running, or a firewall blocks the port. | Start the service; check curl -k https://<host>:<port>/health from the Webserver host. |
| Unreachable: … does not match the pinned fingerprint | The service’s certificate changed (renewed, reinstalled). | Fetch certificate, check the new fingerprint against the service, save. |
| Unreachable: … certificate signed by unknown authority | No fingerprint is pinned and the certificate is not trusted by the Webserver host. | Pin it with Fetch certificate, or install a trusted certificate on the service. |
| Reachable, but it refuses this webserver’s signature | The service does not have this Webserver’s public key: never copied, or the key was rotated. | Copy Signing key (public) into the service’s PREDICTION_JWT_PUBLIC_KEY file. |
| Reachable … status 500: JWT public key not configured or unreadable | PREDICTION_JWT_PUBLIC_KEY points to a missing file. |
Fix the path; the service reads the file when it appears. |
| no InfluxDB configured on the service | The service has no INFLUXDB_URL. |
Set the InfluxDB variables on the service and restart it. |
Forecasts table
| You see | Cause | Fix |
|---|---|---|
| No monitor has prediction enabled yet | No monitor has Enable switched on, or the list is not refreshed yet. | Enable prediction on a monitor, then Run predictions now (the list is otherwise refreshed every 10 minutes). |
| Model Queued for long | The service trains one model at a time; others wait their turn. | Wait. The service’s /health shows the queue length. |
| Model Failed, hover shows The training query returned no data | InfluxDB holds nothing for this monitor: the Integrator does not write to InfluxDB, or not to the database / bucket the service reads. | Check the Integrator’s InfluxDB output and the service’s INFLUXDB_* variables. Training is retried after 24 hours, or use Train now. |
| Model Failed, hover shows Not enough history to train: N usable checks, at least 138 needed | Too little history (a new monitor, or one that mostly fails). | Wait for more successful checks; training is retried every 24 hours. |
| Waiting for a model | Normal until the first training ends. | |
| Forecast UNKNOWN with status 400 … Insufficient data | Fewer recent successful checks than the model needs (60). | The monitor has been failing or paused; forecasts resume with enough successful checks. |
| Forecast reads No forecast alert: … | The monitor’s history does not announce its breaches well enough for a warning to be trusted; the end of the line says why (no threshold, not enough history, too few past breaches, no warning signs, too often wrong). See Breach risk. | Usually nothing to fix: these breaches come without warning, and anomaly alerts still work. Without thresholds, set minor / major / critical thresholds on the monitor. |
| Forecast reads Forecast alerts paused: retraining | The thresholds or the prediction count changed since the model was trained. | Nothing: the model retrains at the next forecast. With Manual training only, use Train now. |
| Forecast always OK | The breach risk stays under the monitor’s cut. | Normal for a healthy monitor. The hover shows how often warnings were right on its history. |
| Forecast flat or far off | Very little or very irregular history, or a recent change of behaviour. | Train now after the change; check the model’s error (column Trained). |
| A monitor is missing from the table | Its probe did not answer the last refresh, or the monitor type is not forecast (EUM, SNMP, WMI, log, directory). | Check the probe. A probe that does not answer keeps its previous monitors in the list. |
Alerts
| Problem | Cause | Fix |
|---|---|---|
| No predictive alert at all | Send predictive alerts is off, or the monitor has no channel switched on for status change. | Switch both on. Predictive alerts never use the on fail channels. |
| Alert expected but not sent | A new state must hold for two consecutive forecasts; or the Integrator has that channel disabled. | Wait one more forecast; check the channel in the Integrator’s settings. |
| Only forecast (or only anomaly) alerts | Alert on limits the kind. | Choose Forecast breach and anomaly. |
| The Webserver logs sendPredNotice … buffering for retry | The Integrator is unreachable or older than this feature (answers 404). | Start or upgrade the Integrator. Buffered alerts are retried for one hour. |
Logs
| Component | Where | Look for |
|---|---|---|
| Webserver | log/webserver.log |
runPrediction, predMonitors, queuePredTraining, runPredictionTrainingScan, sendPredNotice |
| Prediction service | its console, journalctl -u mugnsoft-prediction, or docker logs prediction |
training <model>, trained <model>, training … failed, Invalid token |
| Integrator | log/integrator.log |
data2integratorHandler - [prediction], forecast_ |