Update Hyperion

An update replaces the Hyperion programs on one server, refreshes its configuration and restarts its services. The script update.sh works on exactly one server, the one it runs on. It does not know about the other nodes, so on a cluster you choose the order.

Before you start

• Root access over SSH to the master.
• No backup, import or move you care about is running. The update waits for running jobs by default (see below), so nothing is cut off, but you can check Background jobs first.
• Nothing edited by hand in /opt/hyperion. The update stops with “Working tree at /opt/hyperion has local changes” and changes nothing.

Steps

On a single server

1. Run:

   sudo hyperion update

2. The script fetches the source, downloads the pre-built release and checks it against SHA256SUMS. The panel keeps serving during these steps. Then it waits for running jobs, stops the services, installs the programs, refreshes configuration, checks the database migrations on a copy of the database, starts the services and checks that they answer.

3. Browsers that have the panel open show an “update in progress” banner, then an Hyperion is updating page that lists the steps and returns to the panel by itself.

On a cluster

Update the master first, over SSH. The panel has no Hyperion update button for the master, and the panel is stopped partway through its own update.

1. On the master, run sudo hyperion update.
2. Open Nodes and open a worker’s row. Check that its details load. Every worker is still on the old version at this point, and the new master has to drive it.
3. Update the workers one at a time, as Update a node describes, either by SSH with sudo hyperion update, or from the master. In the worker’s row, under Update, choose what to run and click Start update. All three options are ticked by default. Untick System packages to update only Hyperion.
   • System packages runs apt upgrade. It never removes a package.
   • Hyperion installs the new release.
   • Safe update (ticked by default) copies the running programs aside first and puts them back if the node is not healthy afterwards. It restores programs, not data.
4. Wait between workers. The row shows TLS pin on file and Response auth on file once the updated node has sent a heartbeat. The first heartbeat takes up to about a minute after the restart. Then update the next worker.
5. The status under the form shows running, then done or failed with the exit code, and the end of the log. The log is /var/log/hyperion/node-update.log on the worker.

The update runs as a separate systemd unit, hyperion-node-update.service, so it survives the agent restarting. The master row has only Install system updates, which upgrades OS packages and does not touch Hyperion.

From a shell, hctl node update --hyperion --safe --follow does the same for the server you run it on, and hctl node update-status shows the last run.

Note

While nodes run different versions, moving or copying a site between nodes is refused with “agent version mismatch”, naming the three versions. Plan moves outside the rollout.

Automatic updates

Under Settings, Panel, Updates, tick Install new releases automatically and click Save. It is off by default. When on, each node checks for a new release hourly and installs it in safe mode. System packages are never upgraded automatically. On a cluster the setting is copied to every node.

The dashboard shows an Update available notice when a newer release exists.

Options of hyperion update

Option	Effect
--no-wait	Do not wait for running jobs. Whatever is running is interrupted.
--wait-timeout=SECONDS	Give up, changing nothing, if jobs are still running after this long.
--safe	Restore the previous programs if the health check after the start fails.
--from-source	Compile locally instead of downloading.
--release=TAG	Install that release and its source.
--ref=REF	Use another branch, tag or commit of the source.
--repair	Also delete orphan rows left by failed site creations. It does not remove files, vhosts, databases or users.
--no-build	Refresh configuration only, installing no programs.

By default the update waits until no job or backup is running. Entries whose last sign of life is older than an hour (jobs) or six hours (backups) are treated as dead and ignored.

Check that it worked

The script ends with a box that reads Hyperion update with the old and new commit, and both services active. Then:

hyperion version

Expected: the new commit. On Nodes, the Agent version of each row matches the master, and no row is flagged as differing.

If it fails

• “Working tree at /opt/hyperion has local changes”: nothing was changed. Remove or commit the edits and run again.
• “git fetch failed”: nothing was changed. Fix the network and run again.
• “cargo not found and no usable pre-built release”: the script wanted to compile, which happens when the published release is older than the source it fetched, and this server has no Rust toolchain. Nothing was changed. Run again later, once the release has been republished.
• “Jobs were still running after N s”: only with --wait-timeout. Nothing was changed.
• Anything after the services stopped: see Recover from a failed update.