A node shows as unreachable or its answers are rejected
What you see
Section titled “What you see”On Nodes, on Service health or after a click that targets a worker, the panel shows one of:
- “node <label> unreachable: …” followed by a reason.
- “node <label> response failed authentication: …”.
- “node <label> has no TLS certificate pin on file while pinning is enforced”.
- “envelope too old” or “envelope in future”, with advice about the clocks.
- “target node … is not enrolled”.
- “target node … has no public_ip on record — cannot reach”.
- A notice “Incomplete view.” on a cluster-wide page, naming a node whose sites are missing from the list.
“unreachable” and its reason
Section titled ““unreachable” and its reason”The master could not open a connection to the worker. The reason text says where:
| Reason shown | Meaning and check |
|---|---|
| DNS lookup failed for the worker’s hostname | The master cannot resolve the worker’s address. Check the DNS record and the master’s resolver. |
| TCP connect refused (agent down or firewall blocking 9443) | On the worker, run sudo systemctl status hyperion-agent. Then check that port 9443 is open to the master, see Ports. |
| Timed out waiting for the worker to respond | The worker is overloaded or a firewall drops packets. Check the worker’s load and the route. |
| TLS handshake failed (agent’s cert is not valid yet) | The worker’s clock is behind. Check its time. |
| Connection reset by the worker mid-handshake | Restart hyperion-agent on the worker and retry. |
| Worker TLS certificate not trusted | The master could not verify the worker’s certificate. |
| TLS certificate pin mismatch — the worker’s cert no longer matches the SPKI pin it reported. Restart the worker’s agent so it re-reports its pin, or turn off “Pin each worker’s TLS certificate” in Settings → Cluster → Channel hardening. | The worker’s certificate changed after the master pinned it. Follow the text. |
On the worker:
sudo systemctl status hyperion-agentsudo journalctl -u hyperion-agent -n 50 --no-pagerClocks disagree
Section titled “Clocks disagree”“envelope too old” and “envelope in future” mean the master and the node disagree about the time. The node refuses every request until they agree. On both servers:
timedatectl statussudo timedatectl set-ntp true“response failed authentication”
Section titled ““response failed authentication””The worker answered, but the master could not verify the answer’s signature, or the worker answered unsigned although it has published a signing key. This is not downtime. Do not retry in a loop. Check the node’s page on Nodes for the Response auth on file line and check that the agent on the worker is the one you enrolled and is up to date, see Update Hyperion. If the node was reinstalled and keeps its id, make the master forget the keys it pinned with sudo hctl node reset-crypto <node-id> (CLI reference); the node pins new keys on its next heartbeat. To enroll a node afresh, see Add a node.
“no TLS certificate pin on file while pinning is enforced”
Section titled ““no TLS certificate pin on file while pinning is enforced””Pin each worker’s TLS certificate is on and the master holds no pin for this node, so it refuses to connect. A node reports its pin on a heartbeat, which can take about a minute after the node’s agent restarts. If it still shows nothing, restart hyperion-agent on the worker.
“is not enrolled” or “has no public_ip on record”
Section titled ““is not enrolled” or “has no public_ip on record””The node id is unknown to the master, or the node has no address on record. Check Nodes and, for a missing address, the advertise_addr under [remote_rpc] in Configuration.
“Incomplete view.”
Section titled ““Incomplete view.””A page that fans out across nodes could not read one of them. The sites of that node are missing from the list, not deleted. Fix the node as above and reload.
Fix the cause the table or the text names, then reload the page.
Check that it is fixed
Section titled “Check that it is fixed”Service health lists the node’s services again and the error is gone from Nodes. From the master:
sudo hctl node list