Copy backups off-site

Off-site copies are what is left when a node and its disk are lost. Backups always land on the node’s disk first. As soon as one finishes, the node that made it pushes the archive and its database dump to each off-site target in use. Two kinds of target exist: one FTP, FTPS or SFTP server, and any number of S3-compatible buckets.

Before you start

• To edit the FTP / SFTP target you are an Owner. To add, edit, test or delete S3 targets and to copy existing backups off-site you hold the Manage off-site backup targets permission, which Administrators have.
• The server or bucket is reachable from every node, not only from the master. Settings are copied to every node, and each node pushes its own sites’ backups.
• For S3, every node has the AWS CLI (apt install awscli), and age (apt install age) if you encrypt. Neither is installed by the Hyperion installers.
• With What is kept set to snapshots only, no archives are made and nothing reaches these targets.

Steps

Set up the FTP, FTPS or SFTP target

1. Open Settings in the sidebar, then Backups, and find FTP / SFTP target.
2. Tick Copy every backup to this server.
3. Choose Protocol: FTP, FTPS or SFTP. Plain FTP sends the password and every backup unencrypted.
4. Fill in Host (for example backup.example.com), Port (21 for FTP and FTPS, 22 for SFTP), Username and Password. Once a password is stored the field shows Stored — change it; leave it closed to keep the stored password.
5. Set Base path (for example /hyperion-backups). Each site gets a folder under it named after the site’s system user.
6. Set Keep the newest (0 to 1000, default 0). It is the number of archives kept per site on this server. At 0 nothing is ever pruned there. Above 0, older copies of that site are deleted after each of its backups lands, never the copy that triggered the cleanup.
7. Click Save. Saving restarts the agent, which takes about 5 seconds. The page then tests the connection by itself.
8. Click Test connection at any time. It connects, logs in and lists the base path, using the target as saved.

The account needs permission to write, list and delete files on the server. Deleting is used by Keep the newest and by the delete buttons.

Add an S3 target

1. In Settings → Backups, find S3 target.
2. Fill in Name (short and unique, for example backups-eu), Region (default us-east-1), Endpoint (for example https://s3.example.com), Bucket, Access key ID and Secret key.
3. Paste an age public key (age1…) into Encryption key (age public key). Left blank, copies are sent unencrypted.
4. Set Keep the newest (0 to 1000, default 7), the number of backups kept per site in the bucket. 0 keeps everything.
5. Leave Send backups to this target ticked and click Add target. The other nodes get the target within a minute.

To change a target, click Edit in its row, change the fields and click Save changes. Leave Secret key blank to keep the stored one. The secret is kept in a mode 0600 file under /etc/hyperion/secrets/ and never shown again.

With an age key, each file is encrypted on the node and streamed to the bucket, so no encrypted copy is written to local disk. Only the public key is stored. Keep the private key somewhere safe and off the servers: it is the only way to read these copies.

In the Off-site targets table, Test on an S3 row asks the bucket’s address for an answer. It proves the endpoint and bucket name, not the keys.

Send a site’s backups to one S3 target only

1. Open the site, open the Backups tab and expand Schedule and off-site destination… (Administrators and Owners).
2. In S3 off-site copy, choose the target. Targets that cannot take a backup are disabled and say why (disabled in Settings → Backups → Off-site targets, missing its secret key).
3. Click Save destination.

A pinned site goes to that target only, and gets no S3 copy while that target cannot take it. The card then warns Off-site copies are not happening for this site. Changing the pin affects new backups; earlier copies stay where they are. The FTP / SFTP target is not affected by the pin.

Copy backups made before a target existed

Backups made before a target was set up are not sent retroactively.

1. In Settings → Backups, find Maintenance.
2. Click Copy existing backups off-site and confirm Start copying.

This runs as a background job on the master and every node. It sends backups with no off-site copy recorded, oldest first, at most 200 per node per run. Run it again to continue. It never deletes anything. Backups whose archive was already pruned are reported as gone.

The second button, Copy off-site & delete local, also deletes local copies and is described in Set how long backups are kept.

Check that it worked

1. On the site’s Backups tab, the newest row’s Off-site column shows off-site ✓ for an FTP copy (listed back from the server at the right size) or uploaded for an S3 copy. Hover over it for the S3 target names, the FTP file, or the error. off-site FAILED is a failed push.

2. Open Off-site copies at the bottom of the tab. It reads the FTP server itself and shows the number of copies. It lists Taken, Archive (with also on this site’s node or only off-site), Size and the database dump size. The badge reads no target when no FTP target exists, never pushed when the folder is missing, and empty when the folder exists but holds no files.

3. For S3, list the bucket with your own credentials:

   read -r AWS_ACCESS_KEY_ID && read -rs AWS_SECRET_ACCESS_KEY && export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
   aws --endpoint-url https://s3.example.com s3 ls s3://<bucket>/<system-user>/

   Expected: for each backup a <domain>-<timestamp>.tar.gz, a .sql (sites with a database) and a .manifest.json, each with .age appended when encrypted.

Restore from an off-site copy

• FTP copy: use Restore from off-site, Files only or the Off-site copies card. See Restore a site from a backup.

• S3 copy: the panel cannot list or restore these, because the decryption key is not on the server. Restore by hand:

  1. Download the archive and dump:

     aws --endpoint-url https://s3.example.com s3 cp s3://<bucket>/<system-user>/<domain>-<timestamp>.tar.gz.age .
     aws --endpoint-url https://s3.example.com s3 cp s3://<bucket>/<system-user>/<domain>-<timestamp>.sql.age .

  2. Decrypt each with your private key:

     age -d -i <private-key-file> -o <domain>-<timestamp>.tar.gz <domain>-<timestamp>.tar.gz.age
     age -d -i <private-key-file> -o <domain>-<timestamp>.sql <domain>-<timestamp>.sql.age

  3. Put both files, with the same name stem, in /var/lib/hyperion/backups/incoming/ on the site’s node.

  4. Open the site’s Backups tab, Other ways to restore, From an archive already on the server, enter the archive’s path and restore. The .sql beside it is restored with it.

If it fails

• "<name>" is missing its secret key — backups of this site are NOT being copied off-site until that is fixed. (or "<name>" is disabled in Settings → Backups → Off-site targets): edit the target and paste the secret key again, or tick Send backups to this target, or pick another destination for the site.
• An S3 push fails because a tool is missing: the audit log entry for the push tells you to run apt install awscli (or apt install age when the target has an age key) on that node. Install the package there.
• unreachable after Test on an S3 row: check the endpoint and your network; the probe does not use the keys.
• The S3 target was not saved: …: the message names the field the agent refused, for example the name or endpoint.
• The FTP test fails, or a row shows off-site FAILED: the message in the Test result or the row’s hover text names the curl error. Check host, port, protocol, credentials and that the account may list and write under Base path.
• Nothing is copied off-site: losing a server loses its sites and their backups.: no target is in use. Set one up above.