Operations

What to do on a running cluster: read its state, stop a node for maintenance, replace one, recover when nodes are lost, or go back to a standalone Webserver.

Cluster state

In the UI: edit the Webserver component; the High availability cluster panel lists the nodes, their role, whether they are up and whether they are in sync.

From the command line, on any node:


webserver ha status

For a monitoring tool or a load balancer: GET /ha/health answers 200 on the active node and 503 on a standby.

Planned maintenance

Stop the service on the node. If it was the active one it hands the role over first. Start it again when done: it catches up on its own.

Work on one node at a time, and wait for the panel to show Healthy before the next one: with two nodes stopped the cluster has no majority and answers 503.

Replacing a node

Remove the old node from the cluster, on a node that is still running:


webserver ha remove <NodeID>

Install the new node with its own HA block (same key, its own NodeID and Bind) and start it: it copies the data and joins.
If its address changed, update Peers on the other nodes. They keep working without it, but it is what they use to find the cluster when they start.

A Webserver replaced by a new machine needs a new license; a witness does not. See When a new license is needed.

When the cluster is down

The other nodes are only stopped

Start them. The cluster resumes on its own as soon as two of the three nodes reach each other.

The other nodes are lost for good

Make the surviving Webserver run alone. Stop its service, then:


webserver ha recover confirm

Start the service again: the node is the active Webserver with everything it held. This creates a new cluster of one. To rebuild a full cluster, delete the ha folder of each replacement node and start it: it copies the data and joins.

Only do this when the other nodes cannot come back. Changes that only they held are lost, and an old node that reappears is refused until its ha folder is deleted.

Going back to a standalone Webserver

Remove the HA block from webserver.json and start the service. The node keeps its data and runs standalone; its cluster state is moved to ha.disabled-<date>.

With a cluster license this is allowed on the first server listed in the license. On the other servers use webserver ha recover instead, which gives the same result under the cluster license.