On this page

System, process, log & folder monitoring

The Sentinel Agent (Discovery Agent) watches a host from four angles: the whole system, individual processes, log files, and directories. Each is defined as a set of rules with severity thresholds, evaluated on a schedule, and turned into a Minor / Major / Critical status.

All four monitor kinds are stored as JSON in the agent and run as independent cron jobs. You define them two ways:

  • Sentinel Agent Settings page — edit the monitors interactively in the agent’s Web UI.
  • Component Config Builder — the guided wizard’s Custom Sentinel Agent flow walks you through log monitors, folder monitors, and the rest, then writes a ready-to-run discovery_agent.json.

When a rule breaches its threshold the agent raises an alert at the rule’s severity and routes it through the configured notification channels.

The four monitor kinds

Kind Watches Defined by
Global system Host-wide metrics: CPU, memory, load, swap, disk, network, sockets Per-metric Minor/Major/Critical thresholds
Process Named/discovered processes: CPU, memory, threads, FDs, traffic Per-process thresholds
Log file One file or glob of files: pattern matches, staleness, throughput A path + a list of rules
Directory One folder: file count, total size A path + a list of rules

Schedules use the labels 1MIN, 5MIN, 10MIN, 15MIN, 30MIN, 60MIN. Each monitor runs as an independent job; if a scan takes longer than its interval, the overlapping tick is skipped rather than run concurrently, so a slow disk or a large directory can never stack up overlapping scans.

Global system monitoring

The agent samples host-wide metrics and compares each to its thresholds. Covered metrics include CPU, memory, load average (1/5/15 min), swap (in/out rate and usage), filesystem usage and inodes, disk I/O (read/write rate, queue, utilisation), network traffic in/out, TCP sockets, CPU I/O wait, and database connections.

Each metric has Minor / Major / Critical thresholds, set either as fixed values or computed automatically from a historical baseline (mean + standard deviation). Network traffic thresholds additionally carry an operator (> or <).

For the full field reference, see System Metric Thresholds and Threshold Configuration.

Process monitoring

Processes are monitored either by manual declaration (a process list) or by port scan (the process bound to a port). For each monitored process the agent tracks CPU, memory, thread count, file-descriptor count, and network traffic, each with its own severity thresholds.

For the full field reference, see Per-Process Metric Thresholds.

Log file monitoring

A log monitor watches one file — or a glob pattern such as /var/log/app-*.log — and evaluates a list of rules against the new lines appended since the last tick. The agent tracks a byte offset per file, so each scan reads only what was added; glob patterns pick up newly created files automatically.

Log rotation is handled automatically. Whether the file is rotated by rename-and-recreate (a new file takes the old name) or by copy-and-truncate (the file is emptied in place), the agent detects it and resumes from the start of the new file, so lines are neither silently skipped nor double-counted. Byte offsets are persisted, so a scan also resumes cleanly across an agent restart.

Defining a log monitor

Field Description
path File path, or a glob pattern (*, ?, […]) for multiple files
schedule Scan interval (1MIN … 60MIN)
alertOnMissing on (default) alerts when the file is absent; off stays silent
rules List of pattern rules (below)

Each rule matches lines and alerts on how many matched:

Field Description
name Rule label (shown in the alert)
pattern Regular expression to match on each line
excludePattern Lines also matching this are skipped
operator >, >=, <, <=, =, or first
threshold Match count the operator compares against
severity minor, major, or critical
captureLines Include the first N matching lines in the alert
joinLines Join N consecutive lines before matching (multi-line entries)
enabled off disables the rule without deleting it

The first operator fires on the first occurrence regardless of count — useful for “alert the moment this ever appears” patterns like FATAL or panic.

File-level checks

Independent of the regex rules, a log monitor can also watch the file’s behaviour:

Check Field Fires when
Staleness stalenessMinutes + stalenessSeverity The file has not changed for N minutes (a silent app)
Throughput burst throughputBurstKB + throughputBurstSeverity More than N KB written in one interval (a log storm)
Throughput silence throughputSilenceKB + throughputSilenceSeverity Fewer than N KB written in one interval (a stalled writer)

Set any of these to 0 to disable it.

Large files are read safely

New lines are streamed through the rules one at a time rather than buffered, so memory stays flat regardless of how large the file is or how many rules it carries. Each scan also caps how many new lines it reads per tick (default 100,000, tunable via logMaxLinesPerTick): if a file produces more than that between ticks, the first batch is evaluated, the offset jumps to end-of-file, and a warning is logged — the agent never tries to load an unbounded backlog at once.

Directory (folder) monitoring

A directory monitor scans one folder and evaluates rules on its contents. Use it to catch a spool that stops draining, a directory that fills up, or a drop-folder that never receives files.

Defining a directory monitor

Field Description
path Directory to scan
schedule Scan interval (1MIN … 60MIN)
recursive on walks sub-directories; off counts the top level only
alertOnMissing on alerts when the directory is absent, at alertOnMissingSeverity
alertOnMissingSeverity Severity for a missing directory (minor/major/critical, default critical)
rules List of content rules (below)

Each rule measures one property of the folder:

Field Description
name Rule label
metric fileCount (number of files) or totalSizeMB (total size in MB)
operator >, >=, <, <=, or =
threshold Value the operator compares against
severity minor, major, or critical
enabled off disables the rule

Test a monitor before you save

On the Sentinel Agent Settings page, each log-file and folder monitor has a ⚡ Test button. It runs that monitor’s rules against the real file or folder on the agent host right now and shows the outcome inline — per-rule match counts and status, captured sample lines, folder file-count / size, and staleness — so you can confirm a pattern or threshold behaves as intended without waiting for the next scheduled scan. The test reads from the start of the file (capped for responsiveness) and persists nothing. Throughput burst/silence checks are interval-based and are not evaluated by a one-shot test.

How a status is produced

For every kind, the flow is the same on each tick:

  1. The agent collects the value (a metric sample, a match count, a file size, …).
  2. It applies the rule’s operator and threshold.
  3. If breached, the result carries the rule’s severity (MINOR, MAJOR, or CRITICAL); otherwise OK.
  4. Results are stored for the dashboards and, when non-OK, drive alerts through the agent’s notification channels (email, Slack, Teams, PagerDuty) subject to the alert-timing settings.

See also

Translations

Configuration Reference