Container monitoring (Kubernetes & Docker)

A URL monitor tells you an application is slow. A workload monitor tells you why: the Deployment is running 1 of 3 replicas, a pod is in CrashLoopBackOff, the container was OOM-killed ten minutes ago. Mugnsoft reads this straight from the Kubernetes apiserver or the Docker Engine API, from a probe, with nothing installed in the cluster.

How it is modelled

Container monitoring follows the same device with children pattern as SNMP and WMI devices.

A workload is identified by what survives a deployment, never by a pod name or a container ID:

Before you start

Kubernetes: a read-only ServiceAccount

The probe only ever calls get and list. A ClusterRole, rather than a namespaced Role, lets you leave Namespace empty and see every namespace.

apiVersion: v1
kind: Namespace
metadata: { name: mugnsoft }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: mugnsoft-probe, namespace: mugnsoft }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: mugnsoft-probe }
rules:
  - apiGroups: [""]
    resources: ["pods", "nodes", "namespaces", "events"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets"]
    verbs: ["get", "list"]
  - apiGroups: ["batch"]
    resources: ["cronjobs", "jobs"]
    verbs: ["get", "list"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: mugnsoft-probe }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: mugnsoft-probe }
subjects:
  - { kind: ServiceAccount, name: mugnsoft-probe, namespace: mugnsoft }
---
# Long-lived token. Since Kubernetes 1.24 a ServiceAccount no longer gets one automatically.
apiVersion: v1
kind: Secret
metadata:
  name: mugnsoft-probe-token
  namespace: mugnsoft
  annotations:
    kubernetes.io/service-account.name: mugnsoft-probe
type: kubernetes.io/service-account-token

Apply it, then read back the three values the form needs:

kubectl apply -f mugnsoft-probe.yaml

# Endpoint
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'; echo
# Token (one line, no "Bearer " prefix)
kubectl -n mugnsoft get secret mugnsoft-probe-token -o jsonpath='{.data.token}' | base64 -d; echo
# CA certificate (PEM)
kubectl -n mugnsoft get secret mugnsoft-probe-token -o jsonpath='{.data.ca\.crt}' | base64 -d

Docker: socket or TLS endpoint

tcp:// follows the Docker convention: it means TLS when a CA or client certificate is set or the port is 2376, plain HTTP otherwise. Use an unencrypted port only on a trusted private network. It is unauthenticated root access to the host.

Or run the probe inside the cluster

A probe installed in the cluster with the Helm chart needs none of the above: it uses its own ServiceAccount, with the read-only role the chart creates. Set Auth mode to inCluster and leave Endpoint empty. See Install a probe on Kubernetes.

Adding a container host

Open Container hosts and click Add. The form has four tabs.

Settings

Connection

Workloads

Click Discover & Test (on the Settings or Workloads tab). It posts the form as typed, not the saved record, so it doubles as the connection test:

OK via probe probe-01
4 workload(s) visible

  default/deployment/web            2/2 ready
  kube-system/deployment/coredns    2/2 ready
  kube-system/daemonset/kube-proxy  3/3 ready
  default/statefulset/pg            1/1 ready

An ERROR line names the reason: unreachable endpoint, rejected token, missing RBAC, unknown CA. An empty list under OK means the scope matched nothing.

The list is grouped by namespace or compose project, with standalone containers last; a group’s checkbox ticks every workload in it.

Ticking a workload is the adoption decision: its monitor is created enabled on every selected probe when you save.

Actions

The same notification settings as an SNMP device, applied to every workload of the host:

Saving the host pushes these settings to all its workloads, so one change reaches every workload monitor. Without an Integrator the probe sends the notifications itself; with one, the Integrator does. Email needs the SMTP setting to be validated first.

Discovery, adoption and drift

• What is offered. Kubernetes: Deployments, StatefulSets, DaemonSets and CronJobs. Jobs are not offered, because CronJobs create and delete them all the time; the CronJob itself is. Docker: containers and compose services. docker compose run one-off containers are ignored.
• Auto-adopt. Every 10 minutes the Webserver re-discovers each host that has Auto-adopt on:
  • a new workload is proposed, never adopted: its monitor is created disabled, up to the adoption cap;
  • a workload that vanished is flagged “gone” and its monitor disabled. It is never deleted, its history stays queryable, and it keeps your adoption decision if it comes back.
• Partial answers never retire anything. If discovery was cut short (a kind the token may not list, a timeout, a very large cluster), nothing is marked gone.
• Deleting a container host keeps its workload monitors and their history. Retiring them is an explicit action on each monitor.

How a workload is graded

The severity ladder

ERROR ranks above CRITICAL, and that is deliberate: “I could not look” must never read as “I looked and it is fine”, and must stay distinct from “I looked and it is bad”.

The rules

Metrics

Every check records seven groups, charted one panel per group in the workload report. Every key is written on every check, set to zero where the runtime has no equivalent, so a Docker container still reports the pod-phase counters as 0 rather than leaving holes in the series.

The response time charted for a workload is the API latency of the check, not the application’s latency.

Where the results show up

The container host’s own status combines four independent signals, because they fail independently: the endpoint is reachable, the API answers healthily (/readyz or /_ping), the worst node condition (Kubernetes only), and the worst workload. An expired token therefore reads as an API problem rather than as the cluster being down. On nodes, memory, disk or PID pressure is MINOR, a NotReady node MAJOR, no ready node at all CRITICAL, and a cordoned node never raises severity, so draining for maintenance does not page anyone.

Limitations

Troubleshooting

See also

• Monitor types — the catalogue of every type
• Monitor Component Deep Dive — storage, engine and endpoints
• Integrator — outputs, ITSM and the causal model, including workload
• SNMP & WMI evaluation — the other device-with-children types
• How thresholds work — severity model and status computation

Translations

• fr: Supervision des conteneurs (Kubernetes et Docker)