zfs-autobackup as a DR tool?

[rsysadmin-com]

Hi all!

First things first: @psy0rz - thanks for such a useful tool.

I landed on zfs_autobackup after testing other ZFS-related tools to take snapshots and, so far, it has been the one that ticks most the boxes, but not all. Perhaps, and most likely, due to my lack of knowledge...

So, I have created the following instances for a PoC:

• zfs-node1 - as primary node with PRODPOOL/dataset
• zfs-backup - as backup server with BACKUPS/dataset
• zfs-node2 - as secondary node with PRODPOOL/dataset

The backup server steers how snapshots are taken and distributed, and the workflow was easy to implement:

job1: zfs-node1 <--pull-- zfs-backup
job2: zfs-backup <--pull-- zfs-node2

This works relatively fine when zfs-node1 is the primary node and snapshots flow from zfs-node1 to zfs-node2, via zfs-backup.
The zero-trust option described in the documentation to directly pull-push is nice but nothing remains on the backup server, so it's a no-go.

Now, when I reverse the polarity, a.k.a. zfs-node2 becomes primary and zfs-node1 becomes secondary, the backup server has some issues when taking snapshots:

root@zfs-backup:~/zfs_test# zfs-autobackup -v -F --strip-path=1 --ssh-source zfs-node2 prodpool1 BACKUPS
  zfs-autobackup v3.1.3 - (c)2021 E.H.Eefting ([email protected])

  Selecting dataset property : autobackup:prodpool1
  Snapshot format            : prodpool1-%Y%m%d%H%M%S
  Hold name                  : zfs_autobackup:prodpool1

  #### Source settings
  [Source] Datasets on: zfs-node2
  [Source] Keep the last 10 snapshots.
  [Source] Keep every 1 day, delete after 1 week.
  [Source] Keep every 1 week, delete after 1 month.
  [Source] Keep every 1 month, delete after 1 year.

  #### Selecting
  [Source] PRODPOOL: Selected
  [Source] PRODPOOL/dataset: Selected
  [Source] PRODPOOL/dataset/logs: Selected

  #### Snapshotting
  [Source] PRODPOOL: No changes since prodpool1-20230822151253
  [Source] PRODPOOL/dataset: No changes since prodpool1-20230822151331
  [Source] Creating snapshots prodpool1-20230822151351 in pool PRODPOOL

  #### Target settings
  [Target] Datasets are local
  [Target] Keep the last 10 snapshots.
  [Target] Keep every 1 day, delete after 1 week.
  [Target] Keep every 1 week, delete after 1 month.
  [Target] Keep every 1 month, delete after 1 year.
  [Target] Receive datasets under: BACKUPS

  #### Synchronising
  [Source] PRODPOOL: sending to BACKUPS
! [Target] BACKUPS: Cant find common snapshot with source.
! [Source] PRODPOOL: FAILED: You probably need to delete the target dataset to fix this.
  [Source] PRODPOOL/dataset: sending to BACKUPS/dataset
! [Target] BACKUPS/dataset: Cant find common snapshot with source.
! [Source] PRODPOOL/dataset: FAILED: You probably need to delete the target dataset to fix this.
  [Source] PRODPOOL/dataset/logs: sending to BACKUPS/dataset/logs
! [Target] BACKUPS/dataset/logs: Cant find common snapshot with source.
! [Source] PRODPOOL/dataset/logs: FAILED: You probably need to delete the target dataset to fix this.
! 3 dataset(s) failed!
root@zfs-backup:~/zfs_test#

With this, snapshots are created on zfs-node2 but are not pulled by zfs-backup.

Once I purge the existing snapshots on the targets (both zfs-backup and zfs-node1, in this case), everything works as expected.

Now... the idea behind all this setup is to have a way to implement a mechanism to have PRODPOOL/dataset from the primary node, no matter which one it is, synchronized to the max with the secondary one, so when a DR happens, I don't need to wait forever for ZFS to transfer terabytes of information, thus making the DR very tedious and also creating an economical disaster...

So, some questions for you, zfs-autobackup gurus out there:

1. Am I looking at this approach correctly?
2. Am I overseeing something obvious?
3. Are there better ways to accomplish this, within the realm of ZFS?

TIA!

[psy0rz]

if you want to be able to "reverse" the dataflow, make sure you use the same backupname everywhere.

so dont use prodpool1 and prodpool2 as a name. just make it prodpool

[rsysadmin-com]

Yep... that is exactly what I'm doing since the beginning and it works quite well, I must say.

Once again, thanks for creating the tool! It's a life saver! :-)

[psy0rz]

ah so you solved the problem? :)

[rsysadmin-com]

Sorf of... It works perfectly well in my PoC.
My automation with Ansible perfectly shuffles things around and I can smoothly switch the active/standby nodes.
Active pushes backups to the backup server and the standby pulls them from the backup server... all good so far.

I will now implement this in the development environment where larger amounts of data are used. If that goes well too, next stop would be production :-)

[psy0rz]

nice :)