Sentinel Agent

The Sentinel Agent monitors the host it runs on, discovers services, collects system and process metrics, and generates threshold-based alerts.

Overview

Service Management

CLI Commands

# Install as service and register with the Webserver
discovery_agent install <webserver_ip>:<port>

# Re-register with the Webserver
discovery_agent register <webserver_ip>:<port>

# Migrate to a new Webserver
discovery_agent migrate <webserver_ip>:<port> <jwt_token>

# Start / Stop / Restart
discovery_agent start
discovery_agent stop
discovery_agent restart

# Run in foreground
discovery_agent run

# Remove the service
discovery_agent uninstall

Configuration

Core Settings

{
  "name": "agent1",
  "port": "8070",
  "webserver": "192.168.1.100:8050",
  "integratorEndPoint": "192.168.1.100:8052",
  "location": "Datacenter-1",
  "sched": "5MIN",
  "processes": "nginx,mysql,java",
  "ports": "80,443,3306",
  "logLevel": "info"
}

Scheduling Options

Directory Structure

<install_dir>/
├── discovery_agent(.exe)     # Executable
├── discovery_agent.json      # Service configuration
├── config/
│   ├── sec/                  # RSA keys
│   └── ssl/                  # TLS certificates
├── data/
│   ├── process_list.txt      # Monitored process names
│   ├── port_list.txt         # Monitored network ports
│   ├── process_thresholds.json   # Per-process threshold config
│   ├── system_thresholds.json    # System threshold config
│   ├── discover_metrics.json     # Latest collected metrics
│   └── alerts.json           # Generated alerts
├── dbs/                      # embedded key-value databases
│   ├── discovery.db          # Main discovery data
│   ├── metrics.db            # Historical metrics
│   ├── process.db            # Process data
│   ├── system.db             # System data
│   └── backup/               # Automated backups
└── log/                      # Rotated log files

Discovery Mechanisms

Process Discovery

The Sentinel Agent monitors processes by name. Configure which processes to watch:

Method 1: Configuration file

{
  "processes": "nginx,mysqld,java,python3"
}

Method 2: Process list file (data/process_list.txt)

nginx:web,production
mysqld:database,production
java:application
python3.12:scripts

Format: processname[.exe]:tag1,tag2 (colon-separated, with optional tags)

Port Discovery

Monitor which processes are listening on specific ports:

Configuration file:

{
  "ports": "80,443,3306,5432,8080"
}

Port list file (data/port_list.txt):

80
443
3306
5432

The agent maps port listeners to process names via TCP connection enumeration.

Network Traffic Discovery

Uses packet capture (pcap) via the gopacket library for live traffic analysis:

• Captures packets on all network interfaces
• Duration configurable via discoCapInterval
• Aggregates traffic by process ID (PID)
• Calculates bandwidth in kbits/s (in/out)
• Generates network topology graph in Cytoscape format for visualization

Delegated Monitors (URL / API / Port checks)

Beyond host metrics, a Sentinel Agent can define synthetic checks — urls, apis, pings, nslookups, tcps, dbs, and syss — directly in its configuration. The agent does not run these checks itself; it delegates them to one or more Monitor probes named in probeEndPoint.

{
  "appName": "Shop",
  "probeEndPoint": "probe1,probe-win2",
  "ports": "80,443,3306",
  "urls": [{ "name": "shop-home", "url": "https://shop.example.com/" }],
  "apis": [
    {
      "name": "checkout-api",
      "url": "https://api.example.com/health",
      "typeApi": "GET"
    }
  ]
}

This lets you describe a host’s full service surface (system health and its endpoints) in one place, while the actual URL/API execution runs on a probe that has network reach to the targets.

Collected Metrics

System Metrics

Per-Process Metrics

Metric Enable/Disable

Each metric can be individually enabled or disabled:

{
  "discoCPUGMonEnabled": "true",
  "discoMEMGMonEnabled": "true",
  "discoLoadAvg5minGMonEnabled": "true",
  "discoCPUPMonEnabled": "true",
  "discoMEMPMonEnabled": "true",
  "discoSWAPPMonEnabled": "false"
}

Threshold Configuration

Fixed Thresholds

Set explicit threshold values per metric:

{
  "discoCPUGFixedThreshCri": "95",
  "discoCPUGFixedThreshMaj": "85",
  "discoCPUGFixedThreshMin": "75"
}

Auto Thresholds

Let the agent calculate thresholds based on historical data:

{
  "discoCPUPAutoThreshold": "true",
  "discoMEMPAutoThreshold": "true"
}

Auto-thresholds use statistical analysis (standard deviation) of historical metrics to dynamically set threshold values.

When auto-threshold is enabled for a metric, leave the corresponding *FixedThresh* fields empty (""). The web UI displays these fields as ∞: while the baseline is being computed (warm-up period), no alert fires for that metric.

Traffic Thresholds

Per-process network traffic thresholds with directional operators:

{
  "trafficThresholds": {
    "nginx": {
      "operator": ">",
      "critical": 10000,
      "major": 5000,
      "minor": 1000
    }
  }
}

Operators: > (alert when above) or < (alert when below).

Alerting

Alert Status Levels

Notification Channels

Identical to other components:

Alert Timing

Alert Routing

Alerts can be routed in two ways:

1. Via Integrator (recommended) – Sentinel Agent pushes metrics to the Integrator, which handles alerting centrally
2. Local alerting – Sentinel Agent sends alerts directly via configured channels (when no Integrator is configured)

Concurrency is controlled by semaphores: 20 concurrent goroutines for Integrator communication, 10 for local alerting.

Execution Flow

Cron trigger (every N minutes)
  |
  v
execDiscovery()
  |
  v
runDiscoveryMetrics()
  ├── CollectSystemMetrics()    --> CPU, memory, load, FS, sockets
  ├── CollectProcessMetrics()   --> per-process CPU, memory, I/O, traffic
  ├── CollectNetworkMetrics()   --> pcap traffic capture, connections
  └── GenerateCytoscapeData()   --> network topology graph
  |
  v
Write data/discover_metrics.json + data/alerts.json
  |
  v
discoveryDataFile()
  ├── processMetricsKV()    --> evaluate per-process thresholds
  ├── systemMetricsKV()     --> evaluate system thresholds
  └── Route alerts:
       ├── goDataSent2Integrator()   (if Integrator configured)
       └── goHandleAlerting()         (local fallback)

Background Cron Jobs

REST API Endpoints

Authentication

Discovery

Data Access

Settings

Health

Backup & Recovery

Certificates

Logs

Translations

• fr: Discovery Agent