Sentinel Agent

The Sentinel Agent monitors the host it runs on, discovers services, collects system and process metrics, and generates threshold-based alerts.

Overview

Service Management

CLI Commands

# Install as service and register with the Webserver
discovery_agent install <webserver_ip>:<port>

# Re-register with the Webserver
discovery_agent register <webserver_ip>:<port>

# Migrate to a new Webserver
discovery_agent migrate <webserver_ip>:<port> <jwt_token>

# Start / Stop / Restart
discovery_agent start
discovery_agent stop
discovery_agent restart

# Run in foreground
discovery_agent run

# Remove the service
discovery_agent uninstall

Configuration

Core Settings

{
  "name": "agent1",
  "port": "8070",
  "webserver": "192.168.1.100:8050",
  "integratorEndPoint": "192.168.1.100:8052",
  "location": "Datacenter-1",
  "sched": "5MIN",
  "processes": "nginx,mysql,java",
  "ports": "80,443,3306",
  "logLevel": "info"
}

Scheduling Options

Directory Structure

<install_dir>/
├── discovery_agent(.exe)     # Executable
├── discovery_agent.json      # Service configuration
├── config/
│   ├── sec/                  # RSA keys
│   └── ssl/                  # TLS certificates
├── data/
│   ├── process_list.txt      # Monitored process names
│   ├── port_list.txt         # Monitored network ports
│   ├── process_thresholds.json   # Per-process threshold config
│   ├── system_thresholds.json    # System threshold config
│   ├── discover_metrics.json     # Latest collected metrics
│   └── alerts.json           # Generated alerts
├── dbs/                      # embedded key-value databases
│   ├── discovery.db          # Main discovery data
│   ├── metrics.db            # Historical metrics
│   ├── process.db            # Process data
│   ├── system.db             # System data
│   └── backup/               # Automated backups
└── log/                      # Rotated log files

Discovery Mechanisms

Process Discovery

The Sentinel Agent monitors processes by name. Configure which processes to watch:

Method 1: Configuration file

{
  "processes": "nginx,mysqld,java,python3"
}

Method 2: Process list file (data/process_list.txt)

nginx:web,production
mysqld:database,production
java:application
python3.12:scripts

Format: processname[.exe]:tag1,tag2 (colon-separated, with optional tags)

Port Discovery

Monitor which processes are listening on specific ports:

Configuration file:

{
  "ports": "80,443,3306,5432,8080"
}

Port list file (data/port_list.txt):

80
443
3306
5432

The agent maps port listeners to process names via TCP connection enumeration.

Network Traffic Discovery

Uses packet capture (pcap) via the gopacket library for live traffic analysis:

• Captures packets on all network interfaces
• Duration configurable via discoCapInterval
• Aggregates traffic by process ID (PID)
• Calculates bandwidth in kbits/s (in/out)
• Generates network topology graph in Cytoscape format for visualization

The Web UI renders that graph with link colours driven by traffic severity and label colours driven by protocol family — see Reading the discovery graph.

Packet capture is a runtime dependency

Bandwidth figures come from captured packets, so the host needs a pcap runtime and the privilege to open it. This is a runtime requirement only — nothing needs to be installed to build or upgrade the agent.

ln -s /usr/lib64/libpcap.so.1 /usr/lib64/libpcap.so.0.8

What you lose when capture is unavailable

The failure is silent and the agent stays up. It logs that it is proceeding without capturing packets, and keeps collecting everything else:

• Kept — the full connection inventory (local/remote IP and port, PID, process name, direction, interface). The topology map is intact.
• Kept — protocol labels, but inferred from port number and process name only, with no wire evidence. Everything lands in the guessed confidence tier, which is why the Unidentified bucket in the Proto filter grows on a host with no capture.
• Lost — every kbits/s value. Per-process Traffic In / Traffic Out stay at 0, map link usage stays at 0, and traffic thresholds can never fire.

Metrics that do not depend on packet capture are unaffected: the per-interface netBytesRate and netTxDrops come from OS counters, as do CPU, memory, disk and filesystem metrics.

Capture status is reported, so zero is never ambiguous

A bandwidth of 0 looks the same whether a link is genuinely idle or whether no driver was there to measure it. Every collection run therefore reports a capture status alongside the metrics, and the Web UI surfaces it:

• an amber No packet capture badge in the discovery-graph toolbar, whose tooltip repeats the agent’s own reason (no pcap runtime, permission denied, no usable interface);
• a line at the top of the Proto filter dropdown, explaining the Unidentified bucket where you meet it;
• edge tooltips read “not measured — no packet capture” instead of 0.000 kbits/s, and their bandwidth sparkline is suppressed rather than drawn flat from a history of zeros;
• in the process report, the Traffic In and Traffic Out threshold pills and their two chart panels carry a warning icon. No other metric in that report is affected.

When several agents are merged into one map (level 2), the badge means at least one contributing agent could not capture, and the wording says so.

A genuine 0.000 is still displayed when capture is working — that one is a measurement.

Capture behaviour settings

Both flags are read at the start of every collection run, so a change takes effect on the next cycle — no restart. An absent or unreadable key falls back to the default above, so configurations written before these keys existed keep their previous behaviour.

Delegated Monitors (URL / API / Port checks)

Beyond host metrics, a Sentinel Agent can define synthetic checks — urls, apis, pings, nslookups, tcps, dbs, and syss — directly in its configuration. The agent does not run these checks itself; it delegates them to one or more Monitor probes named in probeEndPoint.

{
  "appName": "Shop",
  "probeEndPoint": "probe1,probe-win2",
  "ports": "80,443,3306",
  "urls": [{ "name": "shop-home", "url": "https://shop.example.com/" }],
  "apis": [
    {
      "name": "checkout-api",
      "url": "https://api.example.com/health",
      "typeApi": "GET"
    }
  ]
}

This lets you describe a host’s full service surface (system health and its endpoints) in one place, while the actual URL/API execution runs on a probe that has network reach to the targets.

Collected Metrics

System Metrics

Per-Process Metrics

Metric Enable/Disable

Each metric can be individually enabled or disabled:

{
  "discoCPUGMonEnabled": "true",
  "discoMEMGMonEnabled": "true",
  "discoLoadAvg5minGMonEnabled": "true",
  "discoCPUPMonEnabled": "true",
  "discoMEMPMonEnabled": "true",
  "discoSWAPPMonEnabled": "false"
}

Threshold Configuration

Fixed Thresholds

Set explicit threshold values per metric:

{
  "discoCPUGFixedThreshCri": "95",
  "discoCPUGFixedThreshMaj": "85",
  "discoCPUGFixedThreshMin": "75"
}

Auto Thresholds

Let the agent calculate thresholds based on historical data:

{
  "discoCPUPAutoThreshold": "true",
  "discoMEMPAutoThreshold": "true"
}

Auto-thresholds use statistical analysis (standard deviation) of historical metrics to dynamically set threshold values.

When auto-threshold is enabled for a metric, leave the corresponding *FixedThresh* fields empty (""). The web UI displays these fields as ∞: while the baseline is being computed (warm-up period), no alert fires for that metric.

Traffic Thresholds

Per-process network traffic thresholds with directional operators:

{
  "trafficThresholds": {
    "nginx": {
      "operator": ">",
      "critical": 10000,
      "major": 5000,
      "minor": 1000
    }
  }
}

Operators: > (alert when above) or < (alert when below).

Alerting

Alert Status Levels

Notification Channels

Identical to other components:

Alert Timing

Alert Routing

Alerts can be routed in two ways:

1. Via Integrator (recommended) – Sentinel Agent pushes metrics to the Integrator, which handles alerting centrally
2. Local alerting – Sentinel Agent sends alerts directly via configured channels (when no Integrator is configured)

Concurrency is controlled by semaphores: 20 concurrent goroutines for Integrator communication, 10 for local alerting.

Execution Flow

Cron trigger (every N minutes)
  |
  v
execDiscovery()
  |
  v
runDiscoveryMetrics()
  ├── CollectSystemMetrics()    --> CPU, memory, load, FS, sockets
  ├── CollectProcessMetrics()   --> per-process CPU, memory, I/O, traffic
  ├── CollectNetworkMetrics()   --> pcap traffic capture, connections
  └── GenerateCytoscapeData()   --> network topology graph
  |
  v
Write data/discover_metrics.json + data/alerts.json
  |
  v
discoveryDataFile()
  ├── processMetricsKV()    --> evaluate per-process thresholds
  ├── systemMetricsKV()     --> evaluate system thresholds
  └── Route alerts:
       ├── goDataSent2Integrator()   (if Integrator configured)
       └── goHandleAlerting()         (local fallback)

Background Cron Jobs

Per-item metrics and devices that go away

Disk and network metrics are stored per item — one series per drive, mount point, or interface. The agent creates a series the first time it sees an item, so a USB drive mounted once, or a network adapter the OS has since renamed, leaves a series behind that no longer receives samples.

Two mechanisms keep those out of the way:

• Reports drop them. For auto-discovered items (disks, network interfaces), a series that produced no sample inside the requested time window is left out of the report entirely, so the UI no longer draws an empty “no data” panel for a drive that does not exist. An item that is merely idle still writes a sample every cycle and is unaffected.
• Retention removes them. The nightly purge deletes any item whose newest sample predates the retention window, together with its change and status entries.

Retention covers every metric

The metrics purge walks the whole metrics store rather than a fixed list of metric names, because the metric set differs by platform (no load average or I/O wait on Windows) and grows with each release.

REST API Endpoints

Authentication

Discovery

Data Access

Settings

Health

Backup & Recovery

Certificates

Logs

Translations

• fr: Discovery Agent