Integrator

The Integrator is a high-throughput data forwarding engine that receives monitoring data and dispatches it to third-party systems and notification channels.

Overview

Service Management

CLI Commands

# Install as service and register with the Webserver
integrator install <webserver_ip>:<port>

# Re-register with the Webserver
integrator register <webserver_ip>:<port>

# Migrate to a new Webserver
integrator migrate <webserver_ip>:<port> <jwt_token>

# Start / Stop / Restart
integrator start
integrator stop
integrator restart

# Run in foreground (Docker or debugging)
integrator run

# Remove the service
integrator uninstall

# Display help
integrator help

Configuration

Core Settings

{
  "name": "integrator1",
  "port": "8052",
  "logLevel": "info",
  "nbworker": "4",
  "queuesize": "10000"
}

Directory Structure

<install_dir>/
├── integrator(.exe)         # Executable
├── integrator.json          # Service configuration
├── config/
│   ├── sec/                 # RSA keys for JWT signing
│   └── ssl/                 # TLS certificates
│       ├── certificates/
│       └── private/
├── dbs/                     # embedded key-value databases
│   ├── integrator.db        # Main KV store (settings, users, JWT)
│   ├── event.db             # Incoming monitor data cache (for pull integrations)
│   ├── metrics.db           # System metrics (CPU, memory, uptime)
│   └── backup/              # Automated backups
├── log/                     # Rotated log files
├── data/                    # CSV/JSON data exports
├── notif_buffer/            # Disk-backed retry queue for failed notifications
└── scripts/                 # Custom alert scripts

Worker Pool Architecture

The Integrator uses a producer-consumer pattern for high-throughput data processing:

HTTP Handler (producer)                   Workers (consumers)
     |                                         |
     |-- parse JSON, create Task -->           |
     |-- enqueue with 5s timeout -->  [Queue]  |
     |-- return HTTP 200 immediately           |
     |                                    +--> Worker 1 --> process + forward
     |                                    +--> Worker 2 --> process + forward
     |                                    +--> Worker 3 --> process + forward
     |                                    +--> Worker 4 --> process + forward

Key characteristics:

• Non-blocking: HTTP handlers return immediately after queueing
• Configurable workers: Set via nbworker (default: 4)
• Queue capacity: Set via queuesize (default: 10,000 tasks)
• Graceful reconfiguration: Workers drain the queue before restarting with new settings

Cross-Agent Correlation Engine

The Integrator includes an optional correlation engine that performs root-cause analysis. When a monitor breaches, the engine looks back over a configurable window for an earlier breach of an upstream monitor type that can be shown to concern the same host. If it finds one, it attaches a root-cause reference to the event instead of letting every symptom travel as an unrelated alert.

Example

Without correlation, a single failing host produces three independent alerts:

22:48  disco/agent       on probe-prod-01  → CRITICAL  (agent unreachable)
22:50  url/shop-frontend on probe-prod-02  → ERROR     (connection refused)
22:50  api/checkout-api  on probe-prod-02  → ERROR     (connection refused)

With correlation enabled, the downstream failures are linked back to the earliest breach — here across two different probes, because the Sentinel Agent that broke first runs on the very host the url and api checks target:

[RCA] url_shop-frontend_probe-prod-02 — [Correlated: disco/agent (declared) breached 2m ago — root cause]
[RCA] api_checkout-api_probe-prod-02  — [Correlated: disco/agent (declared) breached 2m ago — root cause]

The Webserver Events page then shows this as 3 events → 1 incident.

Configuration

These can be set two ways:

1. integrator.json — edit the file in the install directory (persists across restarts).
2. Live from the Webserver UI — go to Settings → Integrator → [your integrator]. The Webserver pushes the values to the Integrator’s /setsetting endpoint and the engine reconfigures immediately, no restart required.

Causal model

The engine uses a fixed dependency graph defining which monitor type can be the upstream cause of another:

The layering follows what a check actually depends on, from the network up: a url failing can be explained by anything below it, while a ping failing can only be explained by the host itself being in trouble. The monitor types eum, sys, snmp, wmi and app are not in the model and are never correlated — app is an aggregate rollup downstream of everything, and the others are not hooked into the window.

Host linkage — how a candidate becomes a cause

Being upstream and in-window is not enough. The engine keeps a candidate only when it can establish that the two monitors concern the same host, and it records how it established that in the event’s rca.link field. Ranked by confidence:

Links with rca.targetMatch = true (declared, address, target) are treated as confirmed. A probe link is flagged as unverified everywhere it is displayed — a question-mark icon in the Events table and on the incident chip — so an operator can tell a real dependency from a coincidence of scheduling.

Service linkage — did the cause serve the port under test?

Host linkage asks “is this the same machine?". For a Sentinel missing cause — a monitored process that disappeared — that question alone is too coarse: on a host running a dozen monitored processes, any one of them vanishing would adopt every tcp and api failure on that box as its symptom, whether or not it had anything to do with the port being tested.

Port evidence adds the narrower question: “is this the same service?"

How the two sides get their ports. The probe reports the port its check connects to as hostport (proto/port, e.g. tcp/5432) — taken from the explicit port for tcp, udp and db checks, and from the URL (or 443/80 by scheme) for url and api. The Sentinel Agent remembers, for every monitored process, the set of ports it was last observed listening on while it was still alive, and sends that set (procports) with the missing event. A missing process has no sockets left to inspect, so a remembered value is the only kind of port evidence that can exist for it.

The verdict then lands in rca.portEvidence:

Ranking. Unproven, missing is no longer top-tier: “a process vanished somewhere on this host” is weaker evidence than “this host is saturated”, which at least applies to every service on the box. A missing cause therefore only keeps tier 0 while portEvidence is matched; unknown and indirect drop it to the default tier, where a cpu, mem or fs breach in the same window wins the tie.

Mixed fleets degrade, never break. An older probe sends no hostport, an older Sentinel sends no procports; both land in unknown, which is hedged and demoted but never rejected. An older Integrator ignores the new fields entirely. The one behaviour change on an un-upgraded estate is the demotion of missing from tier 0 — which is the intended correction.

Onset — could the cause have started it?

Host linkage answers “could this candidate explain the failure”. Onset answers “could it have started it”, and it is the one thing the look-back window alone cannot show.

A failing monitor re-reports on every tick, and the engine re-evaluates each of those reports. Without an onset rule, a monitor that has been failing for days adopts whatever fresh breach happens to be in the window at that moment:

11 API monitors on probe-win1, ERROR since 3d 3h   ← all provisioned by the Sentinel legion-vi7
discovery-win1-missing-Notepad.exe, CRITICAL 17m   ← same Sentinel, unrelated process
→ all 11 wrongly grouped under "disco/missing"

Declared ownership made that Sentinel breach eligible for every monitor it provisioned — correctly, they do watch that host — and nothing asked whether those monitors were already broken before it happened. Something already broken cannot be a symptom of something that started later.

Two independent tests, and either one disqualifies a candidate:

A probe that sends no since disables the second test for that event rather than dropping the correlation — the same “reject only on positive evidence” principle host linkage follows. Rejections are logged at DEBUG level:

[RCA] api_influxdb3_health — dropped disco/missing: already failing at 2026-08-02T18:03:41+02:00, candidate breached 2026-08-02T18:10:41+02:00

The rca payload

Correlated events carry an rca object alongside the usual result fields. The Webserver reads it to build the Events page:

Incidents on the Events page

When at least one enabled Integrator has correlation on, the Webserver Events page grows an RCA column and an incident strip:

• Each event is shown as a root (something points at it, with its symptom count), a symptom (naming its cause and how many minutes earlier it broke), or independent.
• The strip headlines the ratio that is the whole point of the feature — 24 events → 9 incidents, with the folded-away symptom count.
• Clicking an incident chip, or an RCA cell in the table, isolates that incident and re-sorts it oldest-first so the root cause reads at the top followed by what it took down. Clicking again clears.
• A root cause that is not in the current view (filtered out by tags, or handled by another Integrator) still forms an incident and is flagged with an eye-slash icon.
• A grouping that is not fully confirmed — one unverified host link, or any symptom whose port evidence is indirect or unknown — carries a question-mark icon on the chip and on the row. It is all-or-nothing: one hedged symptom hedges the whole incident.

For how to read and drive that column day to day — filters, the incident strip, isolating an incident — see Events page.

How it works

non-OK result arrives (url / api / tcp / db / ping / nslookup / disco)
        │
        ▼
recordRecentEvent()  ──►  in-memory sliding window
        │                  (capped at 500 entries, RWMutex-protected)
        ▼
correlateEvent()
        │  1. look up upstream types in the causal model
        │  2. scan the window for in-window, different-monitor,
        │     upstream-type breaches
        │  3. qualify each candidate by host linkage
        │     (declared → address → target → probe)
        │     then, for a port-scoped Sentinel cause, by port evidence
        │     (matched → service link | disjoint → reject or hedge)
        │  4. drop candidates this monitor was already failing before
        │     (window precedence + reported onset)
        │  5. pick the EARLIEST qualifying match
        ▼
attach `rca` to the event  +  log "[RCA] … [Correlated: …]" at INFO level

Only non-OK statuses are recorded. A cron job (purgeOldRecentEvents, every minute) removes entries older than correlationWindowMinutes; the window is also hard-capped at 500 entries.

Recommended window

Set the window to roughly 2× the longest monitor interval so an upstream breach is recorded before downstream monitors fire. With cross-probe links in play, take the longest interval across all probes involved, not just one.

Ticketing: one incident instead of one per monitor

With ServiceNow enabled, serviceNowIncidentScope decides how many incidents an outage is worth:

root needs the correlation engine enabled: with it off nothing is ever a symptom, so every event still opens its own incident. See ServiceNow integration.

Current limitations

Supported Integrations

Push Integrations (Integrator sends data outbound)

InfluxDB (v1.x / v2.x / v3.x)

{
  "influxDBEnabled": "true",
  "influxDBVersion": "2.x",
  "influxDBServer": "influxdb.example.com",
  "influxDBPort": "8086",
  "influxDBOrg": "myorg",
  "influxDBBucket": "mugnsoft",
  "influxDBToken": "your-token",
  "influxDBSSL": "true"
}

Data is sent in line protocol format:

eumResponseTime,name=MyScenario,type=eum,status=NORMAL,location=Paris responseTime=125.5,statusInt=0 1234567890000000000

Features: connection pooling, retry logic (2 retries, 5s wait), SSL support.

Splunk (HTTP Event Collector)

{
  "splunkEnabled": "true",
  "splunkCollectorServer": "splunk.example.com",
  "splunkCollectorPort": "8088",
  "splunkAuthorizationToken": "your-hec-token",
  "splunkIndex": "mugnsoft",
  "splunkSSL": "true"
}

Data is sent as JSON events to the Splunk HEC endpoint.

Elasticsearch

{
  "elasticEnabled": "true",
  "elasticServer": "elastic.example.com",
  "elasticPort": "9200",
  "elasticUser": "elastic",
  "elasticPwd": "password",
  "elasticSSL": "true"
}

Data is sent via the Bulk API (_bulk endpoint) as JSON documents.

Kafka

{
  "kafkaEnabled": "true",
  "kafkaBrokers": "broker1:9092,broker2:9092",
  "kafkaTopic": "mugnsoft-metrics",
  "kafkaTLS": "true",
  "kafkaSASLMechanism": "SCRAM-SHA256",
  "kafkaSASLUser": "user",
  "kafkaSASLPwd": "password"
}

Supports: TLS with custom CA/cert, SASL authentication (PLAIN, SCRAM-SHA256, SCRAM-SHA512), connection pooling.

Canopsis

{
  "canopsisEnabled": "true",
  "canopsisServer": "canopsis.example.com",
  "canopsisPort": "8082",
  "canopsisUser": "root",
  "canopsisPwd": "password",
  "canopsisSSL": "false"
}

ServiceNow

{
  "serviceNowEnabled": "true",
  "serviceNowServer": "dev12345.service-now.com",
  "serviceNowPort": "443",
  "serviceNowUser": "mugnsoft.integration",
  "serviceNowPwd": "password",
  "serviceNowSSL": "true",
  "serviceNowResolvedState": "6",
  "serviceNowCloseCode": "Resolved by caller"
}

Unlike the other push targets, ServiceNow follows the full incident lifecycle: it opens an incident when a monitor breaches and auto-resolves it when the monitor recovers. Incidents are correlated through the ServiceNow correlation_id field, so resolve works even across Integrator restarts. Actions are taken only on a confirmed status change, so repeated non-OK polls never open duplicate incidents. Auto-resolve respects ITSM ownership: an incident a human has already taken (assigned / past New) is left open with a recovery note rather than being closed automatically.

GLPI

{
  "glpiEnabled": "true",
  "glpiServer": "glpi.example.com",
  "glpiPort": "443",
  "glpiUser": "mugnsoft.integration",
  "glpiPwd": "password",
  "glpiAppToken": "app-token",
  "glpiSSL": "true",
  "glpiSolvedStatus": "5"
}

Like ServiceNow, GLPI follows the full ticket lifecycle: it opens a ticket when a monitor breaches and auto-resolves it when the monitor recovers. GLPI has no native correlation field, so the open ticket is located through an embedded key stamped into the ticket title — resolve therefore works even across Integrator restarts. Actions are taken only on a confirmed status change, so repeated non-OK polls never open duplicate tickets. Auto-resolve respects ITSM ownership: a ticket a human has already taken (assigned / processing / pending) is left open with a recovery follow-up rather than being closed automatically. Authentication is HTTP Basic against the GLPI REST API, with an optional App-Token when the GLPI API client requires one.

Jira (Cloud)

{
  "jiraEnabled": "true",
  "jiraServer": "your-domain.atlassian.net",
  "jiraPort": "443",
  "jiraUser": "you@example.com",
  "jiraPwd": "api-token",
  "jiraSSL": "true",
  "jiraProjectKey": "OPS",
  "jiraIssueType": "Bug",
  "jiraResolveTransition": "Done"
}

Like ServiceNow, Jira follows the full issue lifecycle: it opens an issue when a monitor breaches and auto-resolves it when the monitor recovers. Jira has no native correlation field, so the open issue is located through a sanitized label (mugnsoft-<type>-<name>-<probe>) matched server-side with an exact JQL query — resolve therefore works even across Integrator restarts. Jira has no direct status-set, so resolve is a workflow transition looked up by name (jiraResolveTransition, default Done). Auto-resolve respects ITSM ownership: an issue a human has already taken (assigned / In Progress) is left open with a recovery comment rather than being transitioned. Authentication is HTTP Basic with the account email and an API token (Jira Cloud, REST API v3, ADF descriptions).

Pull Integrations (Third-party queries the Integrator)

Zabbix

{
  "zabbixEnabled": "true",
  "zabbixServer": "zabbix.example.com",
  "zabbixPort": "443",
  "zabbixAuthType": "token",
  "zabbixToken": "your-api-token",
  "zabbixVersion": "6.x",
  "zabbixSSL": "true"
}

Zabbix retrieves data by querying the Integrator’s REST API:

GET /integrator/{bucket}/allV

Data is cached in the Integrator’s embedded key-value store with configurable TTL.

Data Export

Data Reception Endpoints

The Integrator exposes type-specific endpoints for receiving monitoring data:

Incoming Data Payload

Each payload contains:

{
  "name": "My Monitor",
  "shortname": "mymon",
  "probe": "probe1",
  "monType": "eum",
  "timestampEpoch": 1234567890,
  "hostname": "target-host",
  "status": "NORMAL",
  "statusInt": "0",
  "location": "Paris",
  "value": 125.5,
  "transactions": { "Login": 45.2, "Search": 80.3 },
  "dnsLookup": "5.2",
  "tcpConnTime": "12.1",
  "tlsHandshake": "35.4",
  "serverTime": "52.8",
  "responseTime": "125.5",
  "emailR": "ops@example.com",
  "emailOnF": true,
  "emailOnSC": true,
  "resSC": "status_has_changed",
  "alerting": "true"
}

Alerting

The Integrator handles centralized alerting when monitors are linked to it.

Notification Channels

Alert Triggers

Status Colors

Notification Buffer (Reliable Delivery)

If a notification cannot be delivered — the SMTP server is down, Slack returns an error, a webhook times out — the Integrator does not drop it. The failed notification is written to disk and retried by a background worker.

Each buffered item records the channel, monitor identity, status, message, recipient, creation time, and attempt count. The flush worker is started at boot (initNotifBuffer) and stopped gracefully on shutdown. This guarantees that a transient outage in a notification channel never causes a lost alert, as long as the channel recovers within the retry budget.

REST API

Authentication

Management

Data Access

Users

System

Connection Pooling

The Integrator maintains connection pools for outbound integrations:

Error Handling

• Retry logic: 2 retries with 5-second wait between attempts
• Non-blocking: Failed deliveries are logged but don’t block task processing
• Status verification: Checks HTTP status codes (200/204 for success)
• Graceful shutdown: Workers finish current tasks before stopping

See also

• Integrator Configuration — complete settings reference
• Integrator HTTP API — full API documentation
• Integration guides — step-by-step setup for Splunk, InfluxDB, Elastic, Zabbix, and Grafana
• Platform Overview — how the Integrator fits into the architecture
• Data Flow — monitoring and integration data flow diagrams

Translations

• fr: Integrator