install: Add bootc install finish to take over from "raw ostree" installs

[kfox1111]

alma linux image. when I boot from bios, kargs works. if I boot efi, its failing to show custom args.

ex:

  echo 'kargs = ["ipv6.disable=1"]' > /usr/lib/bootc/kargs.d/00-config.toml

I dont see the arguments making it into /boot/loader.1/entries/ostree-*

Something I'm missing?

[cgwalters]

What is your deployment path? bootc-image-builder? Anaconda? bootc install to-filesystem wrapped with something else?

[cgwalters]

Is this after an in-place update or is this a fresh install issue?

[kfox1111]

I've seen it both ways.

In bios mode today, I did an upgrade with the new image and it seemed to work. then reinstalled forcing the firmware over to efi. it installed without the args. Then I tweaked the image then in-place updated it and still no options applied.

[cgwalters]

The thing is with the default bootloader flow we have, bios and UEFI both invoke grub which both just read the bootloader configs from /boot so there should really be no possible difference between the two.

Can you please include the container OS version and confirm the bootloader path?

[kfox1111]

FROM quay.io/almalinuxorg/almalinux-bootc:9.4-20241023
RUN echo 'kargs = ["ipv6.disable=1"]' > /usr/lib/bootc/kargs.d/00-config.toml

When I boot, I see the config.toml

# cat  /usr/lib/bootc/kargs.d/00-config.toml
kargs = ["ipv6.disable=1"]

But the generated loader args are skipping the kargs.d file:

# more /boot/loader.1/entries/ostree-1.conf 
title AlmaLinux 9.20241023.0.4 (Seafoam Ocelot) (ostree:1)
version 1
options crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M root=UUID=43eb3e40-ba57-4699-a52b-97c31139cf1f rw ostree=/ostree/boot.1/default/cbeefe89fd291c49b81b228f857a42b65366392f437dbd0489b160b1cbb50dab/1
linux /ostree/default-cbeefe89fd291c49b81b228f857a42b65366392f437dbd0489b160b1cbb50dab/vmlinuz-5.14.0-427.40.1.el9_4.x86_64
initrd /ostree/default-cbeefe89fd291c49b81b228f857a42b65366392f437dbd0489b160b1cbb50dab/initramfs-5.14.0-427.40.1.el9_4.x86_64.img
aboot /ostree/deploy/default/deploy/bc3560ecaf50123a8d79b2dcdf73b74df1ec274b6c98a164a93c9196671c2b35.0/usr/lib/ostree-boot/aboot.img
abootcfg /ostree/deploy/default/deploy/bc3560ecaf50123a8d79b2dcdf73b74df1ec274b6c98a164a93c9196671c2b35.0/usr/lib/ostree-boot/aboot.cfg

[cgwalters]

We chatted in person about this and realized it's because Anaconda isn't using bootc install yet, but we're working on that!

[kfox1111]

So, fixing Anaconda won't be quick I think....

What workaround might we use until then? bootc upgrading didn't seem to apply the kernel args either. Is there some way we could get it to reset things so that bootc upgrade after install would apply the container kargs?

[cgwalters]

The most viable hack would be in %post but it'd be pretty ugly, we're actively working on switching anaconda to run through the bootc flow.

[kfox1111]

With RHEL 9.5 just being released though, it will probably be a long time before a new anaconda drops.

I'm curious what the workaround might be. Maybe just a rough idea on whats going on. Even bootc upgrade --apply doesn't apply the new image kargs. Does machine kargs (via kickstart) somehow have precedence over the container ones and prevents them from being applied? If so, how might those be looked at/reset?

[cgwalters]

I'm curious what the workaround might be.

I will do some investigation if we can do something like this:

%post --erroronfail
bootc install finish
%end

or so...this would allow us to avoid needing to respin ISOs, but it'd likely be somewhat hacky..

Even bootc upgrade --apply doesn't apply the new image kargs.

Yes, because we diff between the old and new state and we expect the old state to be valid.

So we have active investigation right now for handling automatically making Anaconda installs Do The Right Thing, then after that works we'll investigate something like this %post bootc install finish.

[kfox1111]

Thanks for the insight! That helped narrow things down. I was able to boot an image that didnt have any overrides, then bootc switch image to one with custom kargs in the image, and have that work. So I have a not horrible workaround for now.

[cgwalters]

PR in #915

[cgwalters]

So #915 is going to be really useful for being able to take e.g. the stock RHEL 9.5 (or even 9.4) ISO, add that kickstart fragment, and get the fixes.

But what we still need to do is fix things up so cargo culting that kickstart fragment is unnecessary. Things are actually substantially easier here now that we have merged in the ostree-rs-ext codebase - we can directly have the old ostree-ext code call into bootc (instead of forking). Combined with having rpm-ostree depend on bootc - what we need to do to complete this story is move ownership of the /usr/libexec/libostree/ext/ostree-container to bootc. Further work on that in #916