Recover from a failed update

What an update leaves behind depends on where it stopped. The script protects you in three ways: everything before the services stop changes nothing, a failure after they stop puts the previous programs back and starts them, and the database migrations are tried on a copy before the real database is touched.

Before you start

Root access to the server over SSH.
The panel may be offline. Everything here works without it.

Steps

Read the last lines of the update output. If you started it from the panel, read the log instead:
Terminal window
```
sudo tail -n 80 /var/log/hyperion/node-update.log
```
Check the services.
Terminal window
```
sudo hyperion status
```

Match what you see.

The output says	State of the server	Do
An error before “Stopping hyperion-web”, such as `git fetch failed`, `cargo build failed`, or a local-changes message, and “nothing was stopped or changed”	Untouched	Fix the cause and run `sudo hyperion update` again.
“Update failed before anything was installed, starting the services again”	Old version running	Fix the cause and run again.
“Update failed after the new binaries were installed, restoring the previous ones”, then “Previous version restored and started”	Old version running	Fix the cause and run again.
“DB migration dry-run FAILED” and “Not starting the new version; your data is untouched”	Old version restored	The new release cannot use your database. Do not retry in a loop. Keep the output and report it.
“Health check FAILED after update” and “Rolled back to … the panel is serving again” (only when `--safe` was used or ticked)	Old version running	Read the journal lines printed above it, fix the cause, run again.
“hyperion-web is NOT running” or “hyperion-agent is NOT running” without a rollback	New version installed, not healthy	Go to step 4.
“Could NOT restore the previous binaries, services left stopped”	Services stopped	Go to step 4, then restore by hand (step 5).

Read why a service does not start.

sudo journalctl -u hyperion-agent -n 50 --no-pager
sudo journalctl -u hyperion-web -n 50 --no-pager

To put the previous programs back by hand, use the copy the update took before it installed anything:
Terminal window
```
ls /var/lib/hyperion/update-snapshot
cat /var/lib/hyperion/update-snapshot/VERSION
```
This replaces /usr/sbin/hyperion-agent, /usr/sbin/hyperion-web and /usr/bin/hctl with the copies from before the update. It restores programs only, not data, and the copy is overwritten by the next update. The update keeps migrations additive, so an older program normally starts against the newer database. Check that VERSION shows the version you expect. To undo, run the update again.
Terminal window
```
sudo install -m 0755 /var/lib/hyperion/update-snapshot/hyperion-agent /usr/sbin/hyperion-agent
sudo install -m 0755 /var/lib/hyperion/update-snapshot/hyperion-web /usr/sbin/hyperion-web
sudo install -m 0755 /var/lib/hyperion/update-snapshot/hctl /usr/bin/hctl
sudo systemctl restart hyperion-agent hyperion-web
```
On a worker, which has no hyperion-web, skip that line and restart only hyperion-agent.
Run the update again when the cause is fixed:
Terminal window
```
sudo hyperion update
```
Add --safe so a failed health check restores the previous programs by itself.

What the panel shows

While the panel is offline, its address serves a static page. During an update it is titled Hyperion is updating and lists the steps: fetch, download or build, stop, install, refresh configuration, check database migrations, start. If the update stopped without restoring the panel, it reads The update didn’t finish and names the step. It does not recover by itself. Run sudo hyperion update on the server. If the page is The control panel isn’t answering with no update running, the web service is down for another reason: use step 4.

On Nodes, a worker update that was cut short, for example by a reboot, shows interrupted. Run it again. It first repairs half-installed packages.

Check that it worked

sudo hyperion status
hctl info

Expected: both services active (running), and hctl info prints the agent’s host name, version and hosting count. The panel loads and Nodes shows no row in an error state.

If it fails

hctl info fails but the unit is active: the agent is restarting in a loop, usually from a configuration or schema error. Read the agent journal from step 4.
The update says another update is running: a node update is one systemd unit. Check systemctl status hyperion-node-update.service and wait for it to finish.
Jobs that were running when the services stopped show failed: run those actions again from the panel.