How to Backup Your Enhance Backups?

Disclaimer

Please note that this guide is provided as-is for experienced server admins only, and we do not offer support for the backup process. Proceed at your own risk.

In a previous tutorial, we covered How to Set Up Your Enhance Server Backup VPS. Not using the Enhance BTRFS backup system yet? Well, you’re missing out! It’s fast, efficient, and reliable. Curious? Check out our guide on How to Install the Backup Role on an Enhance Server to get started.

Today, We’ll talk about a scenario no one wants to face but should absolutely prepare for—what if your backup server crashes? Sounds grim, right? Well, don’t worry. With the magic of BTRFS and a little bit of scripting, we’ve got you covered. In this guide, you’ll learn how to backup your Enhance backup server—because yes, even your backups need backups.

Why Backup Your Backup Server?

Your Enhance Backup server is a fortress safeguarding your critical data. But what if the fortress itself is under siege? Hard drive failure, hardware issues, or even accidental deletion can render your precious backups inaccessible. That’s why backing up your backup server is not just smart—it’s essential.

Requirements

Before diving into the script, make sure your setup meets these requirements:

• BTRFS Filesystem: Both the source and destination /backups directories must reside on a BTRFS filesystem.
• SSH Access: You need passwordless SSH access between the source (your Enhance Backup server) and the destination server.
• Administrative Privileges: You must have sufficient privileges to manage BTRFS subvolumes on both servers.

What Does the Script Do?

This script is a Swiss Army knife for your Enhance Backup server. It takes care of the following tasks automatically:

1. Synchronizes the Live State (backup-subvolume):
   • Ensures the current live state of your backups is always up-to-date on the destination server.
2. Transfers Snapshots (snapshot-*):
   • Sends all snapshots to the destination server using btrfs send and btrfs receive.
3. Deletes Outdated Snapshots:
   • Cleans up old snapshots on the destination server that no longer exist on the source server to save space and keep things tidy.
4. Restore Mode:
   • Designed for those “oh no!” moments when your Enhance Backup server fails. With restore_mode=on, the script ensures all transferred subvolumes and snapshots are set to writable (ro=false) on the destination, making them ready for immediate use.
5. Troubleshooting and Tips:
   • Offers practical commands for fixing read-only issues and managing snapshots manually if things don’t go as planned.
6. Error Logging :
   • The script also includes logging, so you’ll always know if the process succeeded or encountered errors.

Using the Script

Step 1: Download The Script & Set It Up

Download the script from the following link:

https://gist.github.com/cPFence/6496978d877a6b3ab57602451db0ec93

Copy the script to your backup server, update the variables to match your environment, and you’re good to go.

Step 2: Use Screen

When running the script manually, we highly recommend using screen to prevent connection issues:

screen -S enhance-backup
sudo ./backup_script.sh

If you get disconnected, you can reattach using: `screen -r enhance-backup`

Step 3: Automate with Cron

Set up a cron job to run the script regularly. For example, to back up your server daily:

0 2 * * * /path/to/backup_script.sh

This ensures your backups stay consistent without manual intervention.

Important: Make sure you set up the cron job at a time when your Enhance backup jobs are not running!!.

What can be improved?

Here’s what’s on our to-do list:

• Incremental Backups for Subvolumes: Currently, the script does not utilize incremental backups for subvolumes, which would significantly enhance efficiency. This feature is on our radar, and we hope to address it soon—time permitting. Nevertheless, you will be amazed at how fast this performs; yes, it’s the power of btrfs.
• Restore Functionality: This script has been thoroughly tested to back up our backups, and we even tested restoring on a test cluster. However, we have not (yet) had the heart—or the guts—to intentionally destroy a production backup server and test the restore process. If you’re feeling adventurous, give it a try and let us know how it goes!

Disclaimer:

THIS SCRIPT IS PROVIDED “AS IS” WITHOUT ANY WARRANTY. USE AT YOUR OWN RISK.

Ensure you test this script in a non-production environment before deployment.

How to Restore Your Backups in Case of Disaster?

Restoring your Enhance Backup server in the event of a disaster is straightforward and requires just two steps:

1. Run the Script on the Other Server: Instead of running the script on your main Enhance Backup server, run it directly on the backup server. This reverses the process, transferring data from the backup server (destination) back to the main Enhance Backup server (source).
2. Enable Restore Mode: Set restore_mode=on in the script. This ensures that all transferred subvolumes and snapshots are automatically set to writable (ro=false) during the restoration process. This step is crucial because the Enhance control panel requires writable subvolumes and snapshots for normal operations.

By following these two steps, your backups will be restored efficiently.

Of course, this process restores your backups, not the root system of the server itself. To recover the entire server, including its operating system and configurations, you need a separate snapshot backup of the root system handy.

Troubleshooting & Tips

The script includes a handy troubleshooting section for common issues:

• Fixing Read-Only Subvolumes: Sometimes if this script gets halted or stopped unexpectedly, some btrfs snapshots are left in read-only status. This can cause issues with Enhance backup/restore functionalities. To fix this, you can manually search and set them back to writable. Refer to the commented troubleshooting section in the script to find the command to use for this.
• Cleaning Up Snapshots: In the troubleshooting section, you will find a handy command in case you need to delete all snapshots on the backup server. This is useful if you are moving to a new server or want to start fresh for some reason. Use this with extreme caution.

Final Thoughts

Backing up your Enhance Backup server might sound like overkill, but as they say: “Hope for the best, prepare for the worst.” This script gives you the tools to be ready for anything. If you have questions, suggestions, or hilarious tales of server mishaps, drop them in the comments. We’d love to hear from you. Our next guide will take things up a notch by showing you how to create full bare-metal snapshots. Stay tuned, and start safeguarding your backups today!