OpenZFS on Debian+systemd, run a script before zfs-import

[g-a-c]

I have a Debian 12 system with several ZFS datasets. Some are encryption roots with passphrases. Entering the passphrases on reboot is obviously inconvenient so I'd like to switch this up to pull from a secrets manager (in this case 1Password, but I don't think that's necessarily relevant) with a service account and store the keys in a tmpfs so they aren't persisted on disk.

I have my 1Password systemd service unit that pulls the passphrase down into a file working great. But I'm having trouble with the ordering of it.

I thought this would be as simple as

1password.service:

[Unit]
Before=zfs-import.target

[Install]
RequiredBy=zfs-import.target

but things don't always run in the right order so after a reboot I find myself on a box where the key files are present in tmpfs but zfs get keystatus shows unavailable because the [email protected] has failed saying the file doesn't exist.

Has anyone done something similar and got any tips for reliably ordering something to start up and execute before OpenZFS tries importing the pool/datasets? Thanks!

[amano-kenji]

Perhaps, 1password.service doesn't actually wait until tmpfs is mounted? If 1password.service was not written by you, write your own version of it.

[g-a-c]

It was written by me - just a really simple service unit. Currently it's just a oneshot service with a few ExecStart commands (for ZFS and some other stuff that is not required immediately on boot) . It doesn't specifically wait for a tmpfs mount, its only dependency is network-online.target for obvious reasons (being able to reach 1Password).

But the fact that the key files never fail to be present makes me think that this unit is working OK, there's just something about the ordering that I'm missing, so the ZFS keys are present eventually, just not soon enough to be used for the mounting process.

[rincebrain]

I mean, if you want a specific step to happen after that service runs, then make it happen before that service, not the broader target.

[g-a-c]

Maybe I'm just not understanding the systemd terminology. I think the docs refer to .target units as "synchronisation points", and when I did a systemd-analyze plot it looked like zfs-import.target was the point where all key loading and mounting and stuff was kicked off from.

So to my mind, making my service a dependency from zfs-import.target is the future-proof approach because then if I add more encrypted datasets they would just be handled transparently because of zfs-import.target. It's possible that I'm not understanding systemd correctly though.

[kleemann67]

Hi guys,
I have a similar problem with one of my backup-servers. After any reboot of the machine - I'don´t know why - systemd starts zfs-import-cache before all discs are up. In the past my predecessor solved that manually like this:

zpool status -x
zpool export izbackup4-pool1
zpool import izbackup4-pool1
zpool status
zpool status -x
zpool clear izbackup4-pool1
zpool status -x
zpool status -v

What I am looking for now is a smart and reliable solution to avoid this in other words to advice systemd to delay the start of zfs-import-cache until all discs are proven up and ready.

Can I configure this directly in the /etc/systemd directory or do I have to write a script to do that? Iam neither a systemd specialist nor a developer my superviser send my this. Maybee it give you an idea how to fix that.

https://www.baeldung.com/linux/systemd-conditional-service-start

https://www.baeldung.com/linux/systemd-postpone-script-boot

Many thanks in advance,

Uli
Linux Admin

[g-a-c]

Just adding some context for anyone who stumbles across this, as this has been ongoing for a while, sometimes working, sometimes not.

At some point, I realised the one thing I hadn't been checking was the overall system logfile (i.e. journalctl rather than journalctl -u <UNIT>) so there were many log entries that I hadn't seen. These were mostly related to ordering dependencies, and having a cycle. This cycle did not seem to expose itself with systemctl list-dependencies because I think this tool operates on a different level and showed the final results with the cycle resolved. Instead I used systemd-analyze dot --no-pager --order > test and this script to find the cycle.

I found that on my Debian system I was seeing several cyclic dependencies:

networking.service -> local-fs.target -> mnt-storage-containers-unifi.mount -> mnt-storage-containers.mount -> op.service -> network-online.target -> network.target -> networking.service

Essentially, my 1Password "daemon" was depending on having a valid network connection (network-online.target). This depended on local-fs.target, which depended on all filesystems including ZFS, which depended on 1Password.

So I think my answer here is to remove the systemd-level dependencies from 1Password to network-online.target and find a way that the service can start but then wait for a successful network connection in a Condition or an ExecStartPre meaning that there is no actual dependency on the network. This way I believe

• 1Password should start, and download the keys
• ZFS mounts will happen
• satisfying local-fs.target
• allowing network-online.target to start, and bootup to continue.

---

Something I've now found to work...

• my 1Password oneshot service now has an ExecStartPre command to verify connectivity, and does not Want or After on network-online.target anymore; this allows this service to run in parallel with reaching network-online.target
• it also has DefaultDependencies=No so that I can better control what it does depend on
• my ZFS datasets have an After on 1Password (so the ZFS key and mount wait until the keys are downloaded)

This has worked for several reboots in a row so seems to have broken the dependency cycle.