Partitioning Support

[mripard]

Hi,

I've discussed this with @cgwalters and on Matrix before, but I've been trying to leverage containers to create bootable images for ARM, and more generally, any platform that doesn't have a standardized boot setup.

I wrote a blog post for the larger context here.

As a prototype, I wrote ocibootstrap, that now supports most ARM boards setup I came across, and actually support systems from five different vendors (ARM, Amlogic, NXP, Rockchip, TI).

The basic idea is that the container would hold everything the system needs to boot: kernel, bootloader, user-space, etc. plus some metadata that describes how the partitioning should work. ocibootstrap then uses those metadata to create a partition table and copies the container content into a disk image.

I believe it's something that should be useful for bootc too, and there's a lot of overlap anyway so it would at least make sense to share our effort.

In addition to defining partitions and their mount-points, the additional requirements I came across are:

• MBR Support
  • Ability to set the payload, and with an offset (Amlogic)
  • Ability to define partitions with a raw content, and at a given offset (Amlogic)
• GPT Support
  • Ability to change the maximum number of partitions (Allwinner)
  • Ability to define partitions with a raw content, and at a given offset (NXP, Rockchip)

ocibootstrap has a set of metadata and the code to deal with all this. It has a set of design goals that might or might not be compatible with bootc though:

• Fire-and-Forget thing: everything should be contained in the container, the tool itself should work by default, without any particular option.
• Cross-platform support: we need to be able to create an image for, say, an ARM platform from an x86 machine.

At the very least, I think it would be worthwhile to share our efforts on a common partitioning layout definition. Let me know what you think.

[ondrejbudai]

Heya!

There's definitely a ton of overlap between ocibootstrap and bootc-image-builder (bib) that I'm working on. BIB is already capable of doing cross-arch builds and distributed as a container image.

With regards to partitioning: bootc-image-builder just merged a new partitioning schema that should be much more expandable than the previous one. We now have full support for LVM, btrfs and swap partitions. The next goals is to support MBR vs. GPT and custom partition GUIDs/types in order to fix Raspberry Pi. What's also nice is that is same code is also used to produce package mode centos/RHEL images, so the code is well tested. :)

So I think that we can definitely collaborate on adding support for the extra features that other ARM boards need.

Also, reading the blueprint (our configuration format) from the container image itself is something that is also on our long-term plan.

[cgwalters]

bootc itself is pretty unopinionated on the partition layouts; the primary target we maintain here is bootc install to-filesystem which supports writing to an extant root filesystem.

However there is also bootc install to-disk which contains a very simple hardcoded partitioning default; but it's intended more as a "demo".

Also, reading the blueprint (our configuration format) from the container image itself is something that is also on our long-term plan.

Right. However, there is still a very important thing looming over this, which is whether an install is done by "dd raw disk image" or is "run a container image" (or is "run a live environment like an ISO which can run code to do an install"). In some cases the latter is very desirable...and takes it currently out of what bootc-image-builder is scoped to do today.

One thing I'd like to do to enable the latter is making it easier to "hook" or fully replace what bootc install to-disk does. That could be backed by whatever (from shell script in the image to partitioning pulled from manifest annotations or embedded blueprints/kickstarts/systemd-repart or whatever). But, that's in the end just a slightly more obvious way to ship a custom installer embedded as part of the container that ends up running bootc install to-filesystem as a middle or end phase.

(Quoting the blog post now)

We found that the partition layout is platform-specific, and that we expect to have a container for each platform or board. This means that we can encode the partition table along with the container.

In your conception of ocibootstrap, are updates thereafter done by e.g. "apt/dnf update", i.e. the container image is a one-time mechanism?

That would make it quite different from bootc today, but indeed we should ideally aim to share code and ideas I think.

I'm not too familiar with the details of some of the smaller ARM devices you mention in the blog; one thing we do depend on here is bootupd - see specifically coreos/bootupd#432 which touches on the generic boot loader bits. I'd also like to make that part more hookable/customizable.

Overall if my understanding of ocibootstrap being a "one time use" thing is correct, then that's by far the biggest axis that splits it versus bootc. In bootc (and bootupd) it's all about making sure that "day 1" and "day 2" are as symmetric as possible to support in-place transactional updates and rollbacks.

[mripard]

With regards to partitioning: bootc-image-builder just merged a new partitioning schema that should be much more expandable than the previous one. We now have full support for LVM, btrfs and swap partitions. The next goals is to support MBR vs. GPT and custom partition GUIDs/types in order to fix Raspberry Pi.

Note that, as far as partitioning is concerned, the RaspberryPi is pretty simple. So it's not really the end of the road, but merely the beginning :)

So I think that we can definitely collaborate on adding support for the extra features that other ARM boards need.

Also, reading the blueprint (our configuration format) from the container image itself is something that is also on our long-term plan.

We were discussing it with @alexlarsson by mail too, and the more I think about it, the more I think it would be better to use a JSON (or similar) format attached to a label. That way, we could define the format somewhere, publish a schema, and we could all use the same format making all these tools interoperable. I'm ready to start working on a first proposal to merge both bib and ocibootstrap requirements. Would you be interested?

Also, reading the blueprint (our configuration format) from the container image itself is something that is also on our long-term plan.

Right. However, there is still a very important thing looming over this, which is whether an install is done by "dd raw disk image" or is "run a container image" (or is "run a live environment like an ISO which can run code to do an install"). In some cases the latter is very desirable...and takes it currently out of what bootc-image-builder is scoped to do today.

The former is what I'm mostly interested in, and what ocibootstrap tries to address.

(Quoting the blog post now)

We found that the partition layout is platform-specific, and that we expect to have a container for each platform or board. This means that we can encode the partition table along with the container.

In your conception of ocibootstrap, are updates thereafter done by e.g. "apt/dnf update", i.e. the container image is a one-time mechanism?

At least, it's something I really want to support. There's platforms out there that don't have the resources (or use-cases) to support anything else, and I believe we should have a solution for them still.

That would make it quite different from bootc today, but indeed we should ideally aim to share code and ideas I think.

See my question above, but I really think the partitioning description should be fairly easy to share and would be useful, no matter if we want to use ostree or a package manager for example.

I'm not too familiar with the details of some of the smaller ARM devices you mention in the blog; one thing we do depend on here is bootupd - see specifically coreos/bootupd#432 which touches on the generic boot loader bits. I'd also like to make that part more hookable/customizable.

The main issue I can see with bootupd (but I don't really know its internal, just uses it since last month or so, so I might be wrong) is where the bootloader is coming from. My understanding is that it's from a package, and bootupd then runs during the post-install hook. I think it's working fine for cases like x86 where the bootloader is generic across the whole architecture. However, on ARM systems, the bootloader will be different from one board to the other, even if they share their SoC. So the challenge now comes from packaging and distributing all those images. It would create a pretty big maintenance burden, and I'm not sure it's something any distro will want to do.

Overall if my understanding of ocibootstrap being a "one time use" thing is correct, then that's by far the biggest axis that splits it versus bootc. In bootc (and bootupd) it's all about making sure that "day 1" and "day 2" are as symmetric as possible to support in-place transactional updates and rollbacks.

It's definitely a one time use so far.

[cgwalters]

The main issue I can see with bootupd (but I don't really know its internal, just uses it since last month or so, so I might be wrong) is where the bootloader is coming from. My understanding is that it's from a package, and bootupd then runs during the post-install hook. I think it's working fine for cases like x86 where the bootloader is generic across the whole architecture. However, on ARM systems, the bootloader will be different from one board to the other, even if they share their SoC. So the challenge now comes from packaging and distributing all those images. It would create a pretty big maintenance burden, and I'm not sure it's something any distro will want to do.

I want to make sure bootupd is cleanly extendable to these things too, it shouldn't require packages. IOW it should be able to install content added in a container build. See coreos/bootupd#766

[cgwalters]

We were discussing it with @alexlarsson by mail too, and the more I think about it, the more I think it would be better to use a JSON (or similar) format attached to a label. That way, we could define the format somewhere, publish a schema, and we could all use the same format making all these tools interoperable. I'm ready to start working on a first proposal to merge both bib and ocibootstrap requirements. Would you be interested?

It's clearly relevant and related. But I will continue to use "can configure LUKS for the rootfs" as a very simple differentiator between "toy" and "maybe usable for enterprise". AFAIK, your proposed schemas don't attempt to do LUKS? Neither do blueprints today. But kickstart, Ignition and systemd-repart do. Also of note is that cases like LUKS with tpm2 binding actually need code execution on the target.

In general personally like to see "us" building bridges and integration with these existing tools (all of them are quite relevant) as opposed to inventing a new schema.

All of these have various tradeoffs (you already ran into repart only doing GPT, Ignition is really designed only to run in the initramfs today, kickstart is...battle tested but very tied to Anaconda which is a huge project).

Also of note though is there's already a bootc install config; I think at this point I don't want to try growing it too much and I think we should invest in those bridges, but just noting it exists too.

[ondrejbudai]

We were discussing it with @alexlarsson by mail too, and the more I think about it, the more I think it would be better to use a JSON (or similar) format attached to a label. That way, we could define the format somewhere, publish a schema, and we could all use the same format making all these tools interoperable. I'm ready to start working on a first proposal to merge both bib and ocibootstrap requirements. Would you be interested?

I'm not convinced by labels. We were thinking about them too, but ultimately decided to just read files from container images if we need more metadata for disk image builds. People seem to be more used to putting files into containers, rather than using labels. Moreover, files can be shared across multiple Containerfiles, but I don't think that's possible with labels.

I'm happy to collaborate on requirements, but I'm afraid that we are not interested in implement support for yet another schema in bootc-image-builder, especially right after we just defined a new schema. You can review it here: https://osbuild.org/docs/user-guide/partitioning#using-disk-customizations

We were discussing it with @alexlarsson by mail too, and the more I think about it, the more I think it would be better to use a JSON (or similar) format attached to a label. That way, we could define the format somewhere, publish a schema, and we could all use the same format making all these tools interoperable. I'm ready to start working on a first proposal to merge both bib and ocibootstrap requirements. Would you be interested?

It's clearly relevant and related. But I will continue to use "can configure LUKS for the rootfs" as a very simple differentiator between "toy" and "maybe usable for enterprise". AFAIK, your proposed schemas don't attempt to do LUKS? Neither do blueprints today. But kickstart, Ignition and systemd-repart do. Also of note is that cases like LUKS with tpm2 binding actually need code execution on the target.

FTR, blueprints don't support LUKS because no one asked for them. LVM on the other hand is and always was a highly requested feature.

[cgwalters]

FTR, blueprints don't support LUKS because no one asked for them.

Okay, asked for in osbuild/bootc-image-builder#747 then