On this page

Auto-thresholds

In this section you will learn how the Sentinel Agent computes a baseline from a metric’s own history, when that baseline is refreshed, and how to trigger it on demand.

A fixed threshold has to be right for every host and every process, which is rarely possible. An auto-threshold instead learns what is normal for each individual metric and raises an alert when the metric leaves its own usual range.

How the baseline is computed

Over a configurable lookback window, the Sentinel Agent computes the average and the standard deviation of the metric, then places the three levels at increasing distances from the average:


minor    = 1 + average + (minor factor    x standard deviation)
major    = 1 + average + (major factor    x standard deviation)
critical = 1 + average + (critical factor x standard deviation)

The three factors default to 2, 4 and 6. A metric that stays close to its average therefore gets tight thresholds, while a naturally noisy metric gets wide ones.

The lookback window and the three factors are configured per agent, separately for system metrics and for process metrics:

Setting Applies to Default
auto threshold range lookback window 1 day
minor / major / critical factor multiplier on the standard deviation 2 / 4 / 6

Available lookback windows are 1 day, 1 week, 2 weeks, 4 weeks and 8 weeks. A longer window produces more stable thresholds that tolerate daily and weekly cycles, at the cost of reacting more slowly to a genuine change in the workload.

Percentage metrics never reach 100%

For system cpu, memory, filesystem and cpu iowait, and for process cpu, a baseline can compute above 100%, which would make the level unreachable. When that happens the three levels are redistributed proportionally in the space between the average and 99%, so critical is always attainable but never impossible.

When the baseline is refreshed

The baseline is recomputed automatically once a day, at 2 AM. Until the first run completes, the agent has no baseline to use.

Note:

While a metric has no baseline yet, the agent falls back to that metric’s fixed threshold values. Auto-threshold being enabled does not mean fixed values are ignored: they are what governs the metric during the whole period before the first baseline exists.

This is why the fixed values matter even on a fully automatic setup. If they are left empty the metric simply reports OK until the baseline is built.

Computing the baseline on demand

Waiting for the nightly run is inconvenient on a freshly registered agent. The compute threshold action, available from the component list, recomputes system and process baselines immediately using the same logic as the nightly job.

  1. Hover the Sentinel Agent in the component list and click the compute threshold icon

The action returns as soon as the computation is triggered. Reopen the system or process report to see the new values in the threshold pills.

Note:

Trigger this action once the agent has been collecting for a while, not immediately after registering it. A baseline is only as good as the history it is derived from: computed over a handful of samples it describes a few minutes of activity rather than a normal working day.

When no baseline is produced

The agent deliberately refuses to build a baseline it cannot trust, and leaves the fixed threshold in place instead. This happens in two cases.

Not enough samples. Fewer than 30 usable samples in the lookback window is not a population worth averaging. This mainly affects a newly registered agent, or a process that has just been added to the monitored list.

A perfectly flat metric. When a metric never varies, its standard deviation is zero and the formula collapses to average + 1, expressed in the metric’s own unit. For a process whose swap usage is always 0 MB that would produce a critical threshold of 1 MB, alerting on the very first megabyte ever swapped. A metric with no variation carries no information about what an abnormal value would be, so no baseline is derived from it.

Note:

A metric that is always flat, such as swap on a host that never swaps, will therefore never obtain an auto-threshold. If you want to be alerted the moment it moves at all, give it an explicit fixed threshold. An auto-threshold cannot express that intent.

Choosing between fixed and automatic

Use a fixed threshold when Use an auto-threshold when
the limit is imposed from outside, such as a filesystem filling up or a licence cap normal usage differs from host to host or from process to process
you must alert on any movement at all, such as any swap or any dropped packet you care about a departure from the usual behaviour rather than an absolute value
the metric is flat by nature the metric fluctuates during normal operation

The two are not exclusive. Enabling an auto-threshold while keeping sensible fixed values gives you a safety net that covers the period before the first baseline is built, and any period where the agent declines to produce one.

See also

Translations

Tuning thresholds