Lock down the master-to-node channel

Every command the master sends to a worker node is already signed. Two optional switches, both off by default, make the master stricter about which worker certificate and which reply signatures it accepts. They only affect master-to-worker traffic, so on a single server they change nothing.

Before you start

• Sign in as an Owner.
• Every worker is online and runs the same Hyperion version as the master. See Check the health of your nodes and Update a node.
• Each worker row on Nodes shows TLS pin on file and Response auth on file under Connection. If a row shows No TLS pin or No response auth, fix that first with the steps in “If it fails”.
• A step takes effect within about 30 seconds of saving.

Steps

Before you turn a step on

While Pin each worker’s TLS certificate or Require signed replies is on, the master refuses a worker that has no pin or no signing key on file. A refused worker is cut off from the panel: its sites drop out of every list until its pin or key is on file again. The sites themselves keep serving.

Confirm readiness first. The page names the nodes that are not ready (Do not turn this on yet: followed by labels) and shows counts such as 2 of 3 nodes have a TLS pin on file. Turn a step on only when the count is complete. To undo, clear the box and click Save; the change applies within about 30 seconds.

1. Open Settings in the sidebar, then Cluster under the Cluster group.

2. In Master ↔ node channel, find Channel hardening. Its badge reads off, step 1 on, both steps on or out of order. Take the steps in order.

3. Step 0, each node verifies the master’s certificate. This is not a switch on this page. It is the verify_tls key under [enrollment] in each node’s /etc/hyperion/agent.toml. The node installer sets it: true when the master’s certificate verified from the new machine, false with the reason when it did not. With the key absent, the agent verifies whenever the master’s URL is https:// with a DNS hostname. A failed check is never retried unverified. The node stops sending heartbeats and logs the fix. To see the decision on the node:

   sudo journalctl -u hyperion-agent -g 'TLS policy'

4. Step 1, pin each worker’s TLS certificate. Check the readiness line under the box, N of M nodes have a TLS pin on file. Then check the master’s log shows no pin-mismatch warnings:

   sudo journalctl -u hyperion-web -g 'SPKI pin'

   When every node has a pin and the log is clean, tick Pin each worker’s TLS certificate.

5. Step 2, require signed replies. Wait until the line under Require signed replies reads M of M nodes have a response-signing key on file. Then tick it. Turn it on after step 1. The badge reads out of order if step 2 is on without step 1, and the page shows a warning.

6. Click Save. The page shows “Channel hardening saved — the agent restarts in about 5 s”.

What each step does

• Pinning. The master records the certificate pin each worker reports on its heartbeat. With pinning on, it refuses any call to a worker whose certificate does not match. A pin is recorded once and is not replaced silently: a worker that later reports a different pin is refused and logged.
• Signed replies. A node that has published a response-signing key signs every reply. The master discards a reply with a bad signature whether or not the step is on. The step adds refusing an unsigned reply from a node known to sign, and refusing a node with no key on file.

Recover a node whose keys changed on purpose

After you reinstall a worker’s agent or rotate its keys, the master refuses it because its pin or key no longer matches. If the change was deliberate:

Before you clear pinned crypto

This makes the master forget the TLS pin and the response-signing key it has on file for the node, and trust whatever the node presents at its next heartbeat, within about a minute. Use it only after you deliberately reinstalled the agent or rotated its keys. If the node’s keys changed and you did not expect it, leave them and find out why first.

Nothing is sent to the node; only master-side state changes. Confirm the node is the machine you expect: sudo hctl info on it prints its hostname and enrollment. The old pin cannot be restored; the node’s next report becomes the new one.

1. Open Nodes, click the worker’s row, and under Connection click Clear pinned crypto…. The button shows only while a pin or key is on file.
2. Type the node’s label in the dialog and confirm Clear pinned crypto.
3. The page shows “Pinned crypto cleared for <node-id>. The TLS pin and response-signing key chips stay off until the node’s next heartbeat”. Wait for the next heartbeat.

From a shell on the master: sudo hctl node reset-crypto <node-id>.

The other direction is also pinned. A worker pins the master’s remote-call public key the first time it sees it. After you deliberately rotate the master’s key, run this on each worker, and the new key is pinned within about a minute:

sudo hctl agent repin

Check that it worked

• The Channel hardening badge reads both steps on.
• On Nodes, every worker shows TLS pin on file and Response auth on file, and no row lists no TLS pin on file or no response-signing key.
• Click Test connection on each worker. It shows reachable.
• After clearing pinned crypto, the chips return once the node has sent a heartbeat.

If it fails

• A row shows No TLS pin: restart hyperion-agent on that node and wait one heartbeat.
• A row shows No response auth: run Update on the node, then wait one heartbeat. See Update a node.
• After you save, a node’s sites vanish from the lists: the node has no pin or key on file, or reports a different one. The page names it under Refused right now:. Clear the box and click Save to restore the lists, then fix the node.
• A node that went stale right after you changed the master’s certificate: run sudo journalctl -u hyperion-agent -g 'TLS policy' on the node. It prints the fix.
• “Node … not found (already removed?)”: the node is not enrolled any more.

Related

• Add a worker node