How to Backup Your Enhance Backups?

Note: This guide is now outdated as Enhance v12 has been released. For the updated method follow this new guide:
How to Backup Your Enhance v12 Backups (Full System Restore)

In a previous tutorial, we covered How to Set Up Your Enhance Server Backup VPS. Still maintaining the older Enhance BTRFS backup system? This article keeps its backup and recovery steps available. For the original setup, see our guide on How to Install the Backup Role on an Enhance Server to get started.

Today, We’ll talk about a scenario no one wants to face but should absolutely prepare for—what if your backup server crashes? Sounds grim, right? Well, don’t worry. With the magic of BTRFS and a little bit of scripting, we’ve got you covered. In this guide, you’ll learn how to backup your Enhance backup server—because yes, even your backups need backups.

Why Backup Your Backup Server?

Your Enhance Backup server is a fortress safeguarding your critical data. But what if the fortress itself is under siege? Hard drive failure, hardware issues, or even accidental deletion can render your precious backups inaccessible. That’s why backing up your backup server is not just smart—it’s essential.

Requirements

Before diving into the script, make sure your setup meets these requirements:

BTRFS Filesystem: Both the source and destination /backups directories must reside on a BTRFS filesystem.
SSH Access: This standalone BTRFS script needs passwordless SSH access between the source (your Enhance backup server) and the destination server. This is separate from cPFence v4+, where only MultiRun and remote WordPress backup/restore need passwordless inter-server SSH.
Administrative Privileges: You must have sufficient privileges to manage BTRFS subvolumes on both servers.

What Does the Script Do?

This script is a Swiss Army knife for your Enhance Backup server. It takes care of the following tasks automatically:

Synchronizes the Live State (backup-subvolume):
- Ensures the current live state of your backups is always up-to-date on the destination server.
Transfers Snapshots (snapshot-*):
- Sends all snapshots to the destination server using btrfs send and btrfs receive.
Deletes Outdated Snapshots:
- Cleans up old snapshots on the destination server that no longer exist on the source server to save space and keep things tidy.
Restore Mode:
- Designed for those “oh no!” moments when your Enhance Backup server fails. With restore_mode=on, the script ensures all transferred subvolumes and snapshots are set to writable (ro=false) on the destination, making them ready for immediate use.
Troubleshooting and Tips:
- Offers practical commands for fixing read-only issues and managing snapshots manually if things don’t go as planned.
Error Logging :
- The script also includes logging, so you’ll always know if the process succeeded or encountered errors.

Using the Script

Step 1: Download The Script & Set It Up

Download the script from the following link:

https://gist.github.com/cPFence/6496978d877a6b3ab57602451db0ec93

Want to back up only specific sites? This modified version of the script allows you to back up only specific sites instead of everything.

https://gist.github.com/cPFence/56c523f81f2ae7cd5dad3a31e7146815

Review the script, save it as backup_script.sh on your backup server, and make it executable. Set SOURCE_BACKUPS, DEST_SERVER_IP and DEST_BACKUPS for your environment; use your own address instead of the example in the download. For the selective version, set TARGET_SITES to the intended site IDs.

Step 2: Use Screen

When running the script manually, we highly recommend using screen to prevent connection issues:

screen -S enhance-backup sudo ./backup_script.sh

If you get disconnected, you can reattach using: `screen -r enhance-backup`

Step 3: Automate with Cron

Set up a cron job to run the script regularly. For example, to back up your server daily:

0 2 * * * /path/to/backup_script.sh

Check /var/log/btrfs-backup.log and /var/log/btrfs-backup-errors.log after each run. Scheduling a job does not confirm that every snapshot was copied.

Important: Make sure you set up the cron job at a time when your Enhance backup jobs are not running!!.

Keep an independent copy: The script deletes destination snapshots that are missing from the source and replaces the destination’s live backup subvolume. A synced copy does not preserve every snapshot you previously had.

What can be improved?

Here’s what’s on our to-do list:

Incremental Backups for Subvolumes: Currently, the script does not utilize incremental backups for subvolumes, which would significantly enhance efficiency. This feature is on our radar, and we hope to address it soon—time permitting. Nevertheless, you will be amazed at how fast this performs; yes, it’s the power of btrfs.
Restore Functionality: This script has been thoroughly tested to back up our backups, and we even tested restoring on a test cluster. However, we have not (yet) had the heart—or the guts—to intentionally destroy a production backup server and test the restore process. Rehearse recovery on a disposable test cluster with a separate recovery copy; do not destroy a production backup server to test it.

Disclaimer:

THIS SCRIPT IS PROVIDED “AS IS” WITHOUT ANY WARRANTY. USE AT YOUR OWN RISK.

Ensure you test this script in a non-production environment before deployment.

How to Restore Your Backups in Case of Disaster?

Restoring your Enhance Backup server in the event of a disaster is straightforward and requires just two steps:

Prepare the Script on the Other Server: Prepare it on the server holding the recovery copy. Set SOURCE_BACKUPS to that copy, and change DEST_SERVER_IP and DEST_BACKUPS to the replacement Enhance backup server. Both backup directories must still use BTRFS. Check the direction carefully: restore mode does not swap these settings for you.
Enable Restore Mode: Set restore_mode=on before running sudo ./backup_script.sh. This ensures that all transferred subvolumes and snapshots are automatically set to writable (ro=false) during the restoration process. This step is crucial because the Enhance control panel requires writable subvolumes and snapshots for normal operations.

After the transfer, inspect both logs, check the restored backups in Enhance, and test a website restore on a disposable target before relying on them.

Of course, this process restores your backups, not the root system of the server itself. To recover the entire server, including its operating system and configurations, you need a separate snapshot backup of the root system handy.

Troubleshooting & Tips

The script includes a handy troubleshooting section for common issues:

Fixing Read-Only Subvolumes: Sometimes if this script gets halted or stopped unexpectedly, some btrfs snapshots are left in read-only status. This can cause issues with Enhance backup/restore functionalities. To fix this, you can manually search and set them back to writable. Refer to the commented troubleshooting section in the script to find the command to use for this.
Cleaning Up Snapshots: In the troubleshooting section, you will find a handy command in case you need to delete all snapshots on the backup server. This is useful if you are moving to a new server or want to start fresh for some reason. Use this with extreme caution.

Final Thoughts

Backing up your Enhance Backup server might sound like overkill, but as they say: “Hope for the best, prepare for the worst.” This script gives you the tools to be ready for anything. If you have questions, suggestions, or hilarious tales of server mishaps, drop them in the comments. We’d love to hear from you. Our companion guide takes things up a notch by showing you how to create full bare-metal snapshots. Keep your recovery plan handy, and start safeguarding your backups today!

Add protection alongside your backups

Use cPFence v4+ for your Enhance servers while keeping independent, tested recovery copies.

Start free trial