crosvm support

[g2p]

crosvm is a qemu alternative with a smaller attack surface, and sometimes better virtio support.

To build it, you need either a full checkout of the chromeos development tree, or a checkout containing at least:

• chromium.googlesource.com/chromiumos/platform/crosvm
• chromium.googlesource.com/chromiumos/third_party/adhd
• android.googlesource.com/platform/external/minijail

It builds as a standard Rust project, with cargo build.

Then you can test it with commands like these:

touch /tmp/disk.img
fallocate -zl1G /tmp/disk.img
RUST_LOG=debug ~/src/chromium.googlesource.com/chromiumos/platform/crosvm/target/debug/crosvm \
  run --disable-sandbox --disk /tmp/disk.img --socket $XDG_RUNTIME_DIR/crosvm.sock \
  ~/src/solo5/tests/test_blk/test_blk.virtio 

[g2p]

So the virtio image is multiboot-based (multiboot 1), and crosvm hardcodes some offsets based on x86_64 Linux conventions. So either Solo5's loader should match those conventions or crosvm should be updated to read a multiboot header before setting up registers.

[mato]

That makes sense to me. I'd try and fix crosvm to support multiboot, otherwise we'd have to maintain some kind of virtio-for-crosvm subtarget inside Solo5 which would be a pain.

[g2p]

crosvm matches the 64-bit BOOT PROTOCOL described in https://www.kernel.org/doc/Documentation/x86/boot.txt.

Solo5's multiboot1 header can't be used because multiboot1 is an x86_32 specification, and crosvm only emulates the CPU in x86_64 mode.
It looks like both ends will need patches for a successful boot.

[mato]

crosvm matches the 64-bit BOOT PROTOCOL described in [...]

Hmm, okay. I guess that makes sense for them since all the(ir) world is Linux. Though you mentioned that crosvm tries to boot our kernels anyway, is that because the Linux boot protocol has no magic number that lets you detect if the thing you're loading supports it?

Here's a thought. Would it be possible for us to build a virtio binary that supports both protocols in the same file? That would be the way to do it, since it doesn't introduce any kind of "sub-target" or compile-time option/setting.

[g2p]

I'll have to look into it. I think it's possible, since the protocols both use their own offsets (through a symbol in the mutiboot case) and headers.
For now my first idea was to chainload Linux->Solo5/Virtio through kexec.

[mato]

For now my first idea was to chainload Linux->Solo5/Virtio through kexec.

I think some people tried that ages ago and it didn't work. Our virtio code is minimal so tends to get unhappy if anything in the chain-loading process leaves the virtio "hardware" in a non-default state.

[g2p]

After the elf has been loaded in memory, crosvm will skip past the first 200 bytes which may contain some 32-bit initialization code, the freshly initialized cpu seems to slide past any zero instructions (I had trouble debugging that, but I expect that there's just no logic to handle them yet), then it lands on an entry point that follows Linux's startup_64 convention.

As a consequence, our hybrid elf image should put its multiboot header past the text segment.

[g2p]

Though the multiboot header's physical position within the elf should be quite early (first 8192 bytes), it should either be loaded within the first 200 bytes, loaded past the start of the .text section, or not loaded at all. I'll look at what I can do with linker scripts.

Edit: it needs to be loaded. Another option would be to insert a jump past the header.

[mato]

Off the top of my head, you could add a .text.linuxboot or somesuch with the Linux entry point into boot.S, then it should just be a case of something like:

    .text :
    {
        *(.text.linuxboot)
        *(.data.multiboot)
        *(.text)
        *(.text.*)
    }

in the linker script. Assuming .text.linuxboot is not so large as to push .data.multiboot beyond 8k, which it shouldn't be.

WDYT?

[g2p]

Yeah, I did that with a jmp and it worked.

[xiaobo55x]

Hi @g2p &@mato , Could you share more info about this issue? Will you try to boot the solo5 kernel from the crosvm?