I have used Borg backup for a long time to backup my servers and some desktop PCs to a central location. These backups are encrypted by Borg and stored on external USB disks. I finally got around to getting that backup data stored in the cloud so I didn't have to worry about switching those USB disks around any more. In this example I am using wasabi S3 storage, but you can use anything that is supported by rclone like Hetzner storageboxes or Microsoft OneDrive.

Thanks to Rclone I can just transfer the Borg data to a compatible cloud service almost the same way that I would rsync it to my external USB disks. To save space on the 'backup' server (which is actually doing other stuff as well) I still use an external USB disk as the main storage device before transferring the data to the cloud. But this can obviously be any internal disk too.

Backup Diagram

The diagram above shows that the devices backup to the Borg repository on the server. Then the server runs Rclone to sync that data to the cloud. On my server I am using Wasabi as the storage provider.

Both Rclone and Borg provide a mount command. This makes it very easy if you just want to recover a few files from a backup. You can rclone mount the cloud storage which will give you access to the Borg repository, then you can borg mount the repository somewhere and go to grab a copy of whatever you need. Note: There seems to be a bug in the current version of Borg (1.1.17) which means you get the error "Transport endpoint is not connected" if you try to use borg mount without the --foreground option. Hopefully this will be fixed in Borg 1.2.

Update: this bug still seems to exist in borg 1.2 - it might actually be something to do with Python llfuse. So you may still need to use the --foreground option of borg mount if you get the error

borg.locking.NotMyLock: Failed to release the lock /repo/lock.exclusive (was/is locked, but not by me).

I have found it also helps to not remove the lock.exclusive directory, just empty it. This prevents the borg error Found malformed lock and the rclone error Dir.Rename error: can't copy directory.

Another thing you can try if you are encountering errors like destination already exists or file does not exist is to enable full caching in rclone by adding --vfs-cache-mode full to your rclone mount option.

Backing up data

You should see the Borg backup documentation to find out how it works and what options you have when creating your repository. For my example I will create an encrypted repository

Creating the Borg repository

The following command will create an encrypted repository and ask you to provide a passphrase.

$ borg init --encryption=repokey /backups/borgrepo

It is now ready for creating backups.

Set up Rclone

The next thing that needs initializing is your rclone.conf. This can be done with the following command.

$ rclone config

The options you choose depend on which cloud service you will be using. If you are using an Amazon S3 compatible provider then the instructions here can be used as an example. I set up a remote called wasabi and created a bucket called borgrepo on the Wasabi website. This can be referenced by rclone as wasabi:borgrepo.

Make a backup

Now everything is initialized a backup can be created. I will backup the directory called Documents into an archive called testbackup with todays date appended:

$ borg create /backups/borgrepo::testbackup-`date --iso-8601` ./Documents
Enter passphrase for key /backups/borgrepo: 
Creating archive at "/backups/borgrepo::testbackup-2021-08-11"

You will be asked for the passphrase you provided when initializing the repository, but won't see any other output unless you give the -v or --stats options. Once the backup is completed you can check the archive exists with the command

$ borg list /backups/borgrepo
Enter passphrase for key /backups/borgrepo: 
testbackup-2021-08-11                Wed, 2021-08-11 00:57:48 [71252445bad44f445cdaafcba10a7a2f6aae652645da20dc684596dcac827dd7]

Sync it to the cloud

Since Rclone is already configured, sending data to it is as simple as proving a source and destination. I will use the sync option to make sure that any file removed from the local borgrepo will also be removed from the cloud too

$ rclone sync /backups/borgrepo wasabi:borgrepo

How long this takes will depend on how much data you have and how fast your connection is. My upload speed isn't that great so I limited the speed rclone uses and the number of simultaneous transfers with the following options:

$ rclone --bwlimit 1125K --transfers 2 sync /backups/borgrepo wasabi:borgrepo

Again you won't see much happening unless you provide the -v option to rclone. If you don't get any error messages though, your Documents should now be encrypted and backed up in the cloud.

Restoring data

If you want to restore all the data from the cloud then it would probably be best using rclone copy to transfer the entire Borg repository to a local disk first. You can then use borg extract to extract everything from an archive.

But if you just want to restore a few files, or don't want to wait for the initial data transfer before you get started on restoration, then you can use the mount options of Rclone and Borg.

If you are doing this on a new PC you will first need to configure rclone the same way as above. Once that is done you can create some paths to mount your backup and start restoring data. This will need to be done in 3 separate terminals or screen/tmux windows. Terminal 1:

$ mkdir -p /restore/borgrepo /restore/testbackup
$ rclone mount --vfs-cache-mode full wasabi:borgrepo /restore/borgrepo

This will mount your cloud storage and enable caching to prevent some errors, then in terminal 2:

$ borg list /restore/borgrepo
Enter passphrase for key /restore/borgrepo:
Warning: The repository at location /restore/borgrepo was previously located at /backups/borgrepo
Do you want to continue? [yN] y
testbackup-2021-08-11                Wed, 2021-08-11 00:57:48 [71252445bad44f445cdaafcba10a7a2f6aae652645da20dc684596dcac827dd7]

We can see the backup we made earlier is there, so:

$ borg mount --foreground /restore/borgrepo /restore/testbackup
Enter passphrase for key /restore/borgrepo:

Because of the --foreground option this command won't exit. So in another terminal you can access your files:

$ ls -l /restore/testbackup
drwxr-xr-x 1 root root 0 Aug 11 00:57 testbackup-2021-08-11/

You can navigate this directory just like you can any mounted file system and copy the files you need from it to the new system. Once you are done use the commands borg umount /restore/testbackup and then umount /restore/borgrepo to unmount your repository and your cloud storage.

Previous Post Next Post