(multi-device) bcachefs mount fails on boot since bcachefs-tools 1.9.0 #308
Comments
|
Do you have the means to add |
Unfortunately I see no obvious way to do so with systemd's mount unit. :/ That said, I was surprised that bcachefs-tools seems to exit with a non-zero exit status without emitting an error message. Is this expected behavior? Any other ideas on how to approach the issue? I ran out of ideas… |
|
It looks like it already fails with the Can you try if it's better with #309 if you set |
|
FWIW, this also happens with 1.11.0
You mean |
It sure does, but how does a bcache command have an impact on a bcachefs mount? It would match my suspicion that this is related to my dual bcache / bcachefs setup (and the feeling that not many users are affected by this, since they probably have an exclusive bcachefs setup). In any case, here is the trace output (in reverse order, due to Unsuccessful mount using bcachefs-tools 1.11.0: Successful mount using bcachefs-tools 1.7.0: |
- 1.11.0 builds with sys-libs/musl-1.2.5 so 1.4.0 can be dropped - 1.9.4 is stable so <1.9.4, sans 1.7.0, can be dropped as well See-also: koverstreet/bcachefs-tools#308 Signed-off-by: Christopher Fore <[email protected]>
|
FWIW, I have the issue on a second system which does not use bcache, too. Therefore, the issue is unrelated to the existence of a bcache block device on the system. It know occurred to me that it could be the form of the used fstab entry, which is Is the the idiomatic format for a multi-device fstab entry with systemd? |
|
No, mount tries to resolve the UUID= line to a specific block device, which is not how we do things. Since we now have scanning in userspace we might be able to make that work now, but it'd need to be looked at. Try OLD_BCACHEFS_UUID=; that was the workaround we added awhile back. |
|
Thanks for the reply.
I replaced UUID with OLD_BCACHEFS_UUID in the fstab entry and upgrade bcachefs-tools from 1.7.0 to 1.11.0. However, the result was the same: the data\x2dscratch.mount unit fails and systemd presents me with a prompt to either enter the root password or to press Ctrl-D to continue. I usually enter the root password, afterwards I am presented with a shell, where I enter However, this time, I somehow simply hit Ctrl-D. To my surprise, the system booted normally and the bcachefs volume was successfully mounted. Here is the relevant part of the system's journa with BCACHEFS_LOG=trace enabled for the mount unit: Interestingly, we only find bcachefs-tools trace logs in the second mount attempt, but not in the first. It appears as the systemd mount unit doesn't invoke bcachefs-tools when initially executing the mount unit, but somehow on the second time. I am not sure what exactly is going on, and the initially mount failing without any further error messages doesn't help much. |
|
There is a similar report as mine at systemd/systemd#8234 (comment) However,, this issue report is about a regression after bcachefs-tools 1.7.0. The system boots fine with bcachefs-tools 1.7.0, regardless of whether UUID or OLD_BLK_UUID is used in the fstab entry. And it fails to boot as soon as any version later than 1.7.0 is installed (again, regardless of UUID or OLD_BLK_UUID).
Just to verify: you meant OLD_BLK_UUID here, right? Does it really matter if UUID or OLD_BLK_UUID is used, given that bcachefs-tools treat them equal at bcachefs-tools/src/commands/mount.rs Lines 268 to 269 in d27d4cf ? |
|
I managed to debug this issue further by hooking the mount.bcachefs invocation with a script that prints the used arguments and invokes the actual mount.bcachefs with strace. In short, setting Remember that since bcachefs-tools 1.9, the system fails to mount the bcachefs volume on start. However, after dropping into the recovery shell and just issuing The following is an excerpt of the log of the first mount failing using bcachefs-tools 1.13.0, sda5 is a part of a multi-device bcachefs volume: The same part, now succeeding on the second mount. It appears that /run/udev/data is not populated at the time mount.bcachefs is invoked the first time, but later is. But why does it work flawlessly with bcachefs-tools 1.7? Again, here is the excerpt of the related log using 1.7: That made me dig once again into the bachefs-tools source and its history. I found 761ef99 ("Add env. variable BCACHEFS_BLOCK_SCAN") and 1fb1898 ("mount: Leverage udev db"), which where commited between 1.7 and 1.9. Setting I suspect the issue got introduced with 1fb1898 ("mount: Leverage udev db"), even though the commit message states that there is a fallback to traversing block devices. I did not yet find the time to look into it. Maybe @tasleson can comment on this. |
Just returning a failure exit status makes it hard to tell what the actual issue was. Instead, the mount command now also provides some textual output. See koverstreet#308 for an example where mount does not provide any helpful output, even with BCACHEFS_LOG=trace enabled. Signed-off-by: Florian Schmaus <[email protected]>
My initial thoughts are you're missing the lastest fixes to util-linux which includes bcachefs corrections for probing. The reason the old code works and new code works with the env. variable However, this comment
Seems to indicate you have a race condition in that your block device isn't ready yet by the time the fstab gets processed. This is the exact behavior I would see during testing where I purposely prevented a device from showing up at boot. I'm also wondering if you have different blkid running in your initramfs, vs. normal runtime. I would suggest enabling udev debug output, see |
bcachefs-tools/src/commands/mount.rs Lines 202 to 220 in 4f9293b If we find no bcachefs devices in the udev db, the code does revert to walking all the block devices. However, it is using the udev db to get the list of block devices. If your block device isn't in the udev db, it wouldn't be probed. However, the old bcachefs mount was using the udev db to get block devices too.
You do have more than one device with a bcachefs signature, thus all we need it one of them to be in the udev db for the fallback to not run. |
|
There's a 'udev settle' command we use in some of our tests to wait for block devices to show up, do we need to be doing the equivalent of that here as well? |
I think there probably needs to be better support for bcachefs in systemd. When the fstab is being processed, systemd waits for the FS UUID to show up before it issues the mount. When you have a multi-disk setup as soon as one disk gets probed and is found to contain the FS UUID of interest, the mount gets issued. Systemd needs to wait until all the block devices for the FS are present before issuing the mount. A FS UUID which is stacked on md or lvm etc. doesn't become available until the block layer has all the devices and brings up the logical volume etc. I suppose we could modify bcachefs mount to wait when at least 1 disk of a mult-disk setup is found, but that might break other things. |
2.39.4
I would not assume that to be the case, then again, I am not sure how to verify that. Does blkid here refer to the filesystem's UUID? Why would that change between initramfs and normal runtime? My bcachefs setup just involves one NVMe and two HDDs. No mdraid nor lvm in between, on a Gentoo mostly-stable system (systemd is at 256.7).
How about running the fallback in every case? Just until there is better systemd support. I think we have enough indication to assume that the issue is caused by the bcachefs mount being issued before all block devices are in the udev db. |
|
BTW, |
Flowdalic
changed the title
blkid is a utility that retrieves the information about the block device including the FS UUID. A copy would exist in the initramfs as well as on your filesystem. It could be different if user space is newer than the last time the initramfs was generated. You could examine the contents of your iniitramfs to verify, but its not needed. Your copy on your FS doesn't have all the fixes for bcachefs, but I don't think those missing fixes are contributing to the issue you're seeing.
NVMe is likely available immediately, disks needs to spin up, thus the reason the mount fails.
That is an option, the env. variable was added to give users this as a workaround too. The env. variable should be documented. Please note that none of this addresses the root problem. You could fail to mount at boot even when doing the signature scan of every block device at boot as the udev db is used to locate all the block devices to walk. I'm assuming if you do a warm reboot/reset the mount works fine without |
|
Currently I work around the issue by using Note that I use OLD_BLK_UUID (not sure if this is required) and I still have BCACHEFS_BLOCK_SCAN=1 set for the systemd mount (not sure if this is still required). |
If systemd is ensuring that the devices are present before issuing the mount, then you shouldn't need Thanks for following up with this workaround. |
Starting with bcachefs-tools 1.9.0 my bcachefs mount fails on boot (i.e., with systemd mount unit), but suceeeds if I manually mount it.
I have the following fstab entry
which works fine with bcachefs-tools 1.7.0. However, as soon s I upgrade to 1.9.0 or 1.9.1, the mount fails on boot. Unfortunately, there seems to be no error message in the journal:
is all I got. Note that this is a system with a bcache block device and a bcachefs filesystem, both are separate (of course). Not sure if this matters.
Can reliable reproduce with 1.9.0 and 1.9.1, problem disappears always once I downgrade back to 1.7.0.
Downstream issue report: https://bugs.gentoo.org/938596
The text was updated successfully, but these errors were encountered: