Stale UEFI boot entry left behind after reprovisioning

[bgilbert]

On UEFI systems, shim is invoked as the fallback bootloader on first boot and creates a UEFI boot entry. That boot entry encodes the partition GUID of the ESP, so each reprovision of the machine will create a new boot entry and leave the stale one in place. In principle this might eventually fill up flash.

On first boot, we currently randomize the disk's GUID and the filesystem UUIDs of / and /boot, but we do not randomize partition GUIDs. (And we can't randomize the ESP partition GUID then, since it'll break the boot entry just created by shim.) cosa's create_disk.sh does randomize partition GUIDs, however, so each OS release will have different ones. That seems like an awkward middle ground.

Perhaps we should hardcode the partition GUID for the ESP, allowing the existing boot entry to be reused after reprovisioning. That would undoubtedly cause trouble if two disks have FCOS installed, but there are already other reasons the OS won't boot successfully in that case. It would also double down on violating the GPT spec, which wants GUIDs to be globally unique.

xref https://bugzilla.redhat.com/show_bug.cgi?id=1997805
xref https://bugzilla.redhat.com/show_bug.cgi?id=1977983

[dustymabe]

@bgilbert - does this have overlap with #563 ?

[dustymabe]

We discussed this at the community meeting today.

13:08:27   dustymabe | #agreed we'll drop the shim fallback .CSV file
                     | (/boot/efi/EFI/fedora/*.CSV) OR pick up a new shim subpackage which
                     | does the same, which will prevent the creation of a new UEFI boot
                     | entry (variable stored in nvram/flash) every time FCOS is
                     | reprovisioned

[dustymabe]

I assume we need to make sure this applies to both the platform artifact images and the live ISO/PXE images (anywhere EFI is used to boot)?

[bgilbert]

I'm not sure yet about EFI PXE boot, but it does apply to the live ISO. However, dropping the CSV file from the ISO's EFI image causes the machine to bootloop.

[bgilbert]

And yes, #563 looks like shim initializing the boot variable and rebooting. It appears that shim < 15.3 (in some circumstances?) then reboots rather than falling through to GRUB, while ≥ 15.3 offers a menu with the choice to keep going.

[bgilbert]

@vathpela It looks like when the CSV file is removed, shim 15.4 boot-loops in the Boot Option Restoration screen.

[travier]

See openshift/assisted-installer#229 & openshift/assisted-installer#313

[travier]

See answer in openshift/assisted-installer#229 (comment)

[cgwalters]

We may need to offer something like coreos-installer install --reset-uefi or so?

[bgilbert]

It'd be good to avoid adding obscure flags to things if possible. I'm hopeful that #946 (comment) will solve the problem going forward. But it's possible that once we start bumping bootimages in existing OCP clusters, we'll uncover enough badly-behaved firmware that we'll need a workaround for existing deployments. We could probably have a limited-purpose boot variable scrubber that runs automatically as part of install, inside coreos-installer or as a separate unit.

[bgilbert]

Looking through the shim code, I don't think we want to drop the CSV file; I think we want to drop fallback.efi altogether and have shim chain directly into GRUB. shim explicitly supports doing this when fallback.efi isn't available.

RFC cosa PR at coreos/coreos-assembler#2421, though probably bootupd should handle it instead.

[dustymabe]

I haven't reviewed the PR, but reading this I thought of one thing that could be a show stopper for this

Looking through the shim code, I don't think we want to drop the CSV file; I think we want to drop fallback.efi altogether and have shim chain directly into GRUB.

I recall pjones saying during our meeting a few weeks back that fallback.efi functionality was going to me merged in to something else soon and so fallback.efi wouldn't exist (i.e. not easy to just drop it). Do you remember that? If so, was that considered here?

[bgilbert]

Ah, indeed:

16:44:04 <pjones> is it possible to just not include /boot/efi/EFI/fedora/*.CSV in your images?
16:44:18 <pjones> that's where fallback is getting the information from
16:44:25 <bgilbert> pjones: would that disable the behavior?  I actually haven't looked
16:44:48 <pjones> it should, yes.  removing fallback.efi would also do it currently, but that particular file will go away in the future.

If we were only working around this in cosa, I think it'd be okay to just add a test and then fix up cosa once shim is refactored. But I've been thinking we should handle this in bootupd instead, and I'm more hesitant to make code changes there if we expect the invariants to change later.

[bgilbert]

I chatted with pjones, and his advice is to delete both fallback.efi and the CSV, bearing in mind that fallback.efi may not be present.

[dustymabe]

Where does this stand? From coreos/coreos-assembler#2421 (comment) it looks like half of it is done but we still need to do some work in bootupd?

[bgilbert]

Yup, that's correct. We've removed fallback.efi from the ISO but not yet from the regular disk image.

[dustymabe]

Is there a tracker issue for the bootupd work (or more importantly, is the work on anyone's radar)?

[bgilbert]

I'd say this is the tracker issue. It's on my plate.

[bgilbert]

This isn't actively being worked on. Some old draft pieces of the work:

• 35coreos-ignition: randomize partition GUIDs on first boot fedora-coreos-config#1207
• mantle/kola: confirm absence of EFI boot entry from basic coreos-assembler#2426

I'll close those PRs for now, but they're there for future reference if anyone picks this back up.