AArch64

The AArch64 Port of OSv has been started, initially targeting the QEMU Mach-virt platform, running on the ARM Foundation Model v8.

Functional Status

The aarch64-next branch already can support the loader image and running of a simple "hello world" C application on top of OSv.

https://github.com/hw-claudio/osv_aarch64.git https://github.com/hw-claudio/osv_aarch64/tree/aarch64-next

mainline is a bit behind but we hope to fill the gap as soon as possible by posting the necessary patches.

Component Status

build system: the first pass mostly works through cross-compilation, but there are issues to proceed any further. The build system produces the final image on x86_64 by performing multiple passes where at each stage the image generated in the previous step is executed to generate the successive. This multiple stage build has not been designed with cross-compilation in mind, and this involves not only the compilation and linking of binaries, but a lot of scripts launched externally by the build system as well (see scripts/). One short term workaround solution identified is to use qemu tcg system emulation to run an AArch64 image in software on x86_64. The scripts still need to be reworked quite a bit. A better solution would be to avoid the multi-stage build completely, and just perform the tasks that currently require running OSv in a user space process as part of the normal build. However there seem to be issues related to zfs to make this work.

qemu-system-aarch64 tcg software system emulation: currently the AArch64 image runs on the Foundation Model but also successfully on the QEMU tcg software system emulation. This requires the very latest QEMU which fixes issues with pte attributes and the latest OSv where we fixed a GIC cpumask issue.

tests with available hardware (development boards): preliminary tests with the hardware currently available should be done. This is also an alternative short term route to get a full build working (with the real long term solution still remaining making the build fully cross-compilable, and avoiding unnecessary multistage builds).

external dependencies: mostly done, some surprises might come from the Java part (we are far from there). Currently they are taken from the Linaro toolchain and LEG Image (for OpenJDK8). https://github.com/huawei-erc/osv_external_aarch64 Avi is working on replacing these with Fedora packages, which seem to work for him but not for me at the moment.

devices support: NONE available at the moment. A PCI bus should be added to the Mach-Virt Platform (or some alternative solution should be found to make it possible to see a PCI bus from the OSv guest). On top of that, virtio devices should be added and tested.

smp support: currently only one CPU can boot. Having multiple cpus boot already requires many dependencies (possibly ACPI) and work. Running smp will trigger bugs and missing functionality which will need to be sorted out.

libc: We need to implement setjmp, longjmp and signal code.

hardware information passing from the host to the guest is not available yet, addresses, interrupts, etc are hard-coded based on the QEMU Mach-virt platform. Need: ACPI from QEMU. Work in progress for the ACPI guest support.

ELF64: initial relocations for get_init() are supported, plus basic relocs necessary to run simple applications. More work is foreseen to support more complex applications.

console: only basic pl011 UART output for the debug_console is available, no input.

backtrace: basic functionality now available.

power: currently stubbed, depends on the ACPI work.

exception handling: basic infrastructure there, but need work for sync exceptions to differentiate page_faults from other sync aborts.

MMU: done, but need to revisit for the sigfault detection speedup feature added to x64 (and stubbed on AArch64). Also there might be an issue with permissions of the page tables which triggers in QEMU sys emulation and not in the Foundation Model.

page fault and VMA handling: basic functionality now available.

interrupts: basic functionality now available.

GIC: more or less done, v2.

Generic Timers: basic functionality available, needs a patch for the host bootwrapper for the foundation models (CNTFRQ_EL0 needs to be set to the correct value in the Makefile => 100Mhz)

scheduling: basic support for task switching available (switch-to-first, switch-to)

signals: work started.

arch trace: nothing available.

sampler: sampler support missing for AArch64.

scripts: most scripts have not even been looked at for AArch64

tests: some tests build but most don't because of other missing components. No attempt to run any tests beside tst-hello.so.

memcpy optimizations: not a priority atm, still not implemented. Trivial memcpy and memcpy_backwards functions are used on AArch64.

Build instructions

These are some brief instructions about how to cross-compile OSv's loader.img (very incomplete still) on an X86-64 build host machine.

At the time of writing this, the available functionality is minimal: the loader image boots, gic is initialized, timers are initialized, etc, and a simple hello world application is started on top of OSv.

Crosscompiler Tools and Host Image from Linaro

You can find a 32bit (needs multilib 😟) crosscompiler from Linaro, in particular the package

gcc-linaro-aarch64-linux-gnu-4.8-2013.12_linux, which is not distro-specific 😃, and includes all tools needed for building.

http://releases.linaro.org/13.12/components/toolchain/binaries

For the host root filesystem for AArch64, a good option is the Linaro LEG Image

linaro-image-leg-java-genericarmv8-20131215-598.rootfs.tar.gz

http://releases.linaro.org/13.12/openembedded/aarch64/

Crosscompiler Tools for Ubuntu

For Ubuntu there are AArch64 crosscompilers available in the official repositories as well. Packages are named g++-4.8-aarch64-linux-gnu and gcc-4.8-aarch64-linux-gnu.

Preparing the AArch64 Host

You will need to have or build an AArch64 linux kernel for the Host, which will be run on top of the Foundation v8 Model. In addition to that, you will need the bootwrapper, which you can get from:

http://git.kernel.org/pub/scm/linux/kernel/git/cmarinas/boot-wrapper-aarch64.git

Use the foundation-v8.dts, and my suggestion is to use nfsroot to mount the root filesystem (the linaro LEG image). The boot wrapper takes as input the linux kernel Image and produces linux-system.axf, which will be the input for the Foundation Model.

Running ARMv8 Foundation Model

Start the Foundation model:

./Foundation_v8 --image=linux-system.axf --cores=1 --network=nat --network-nat-ports=1234=1234

The latter option will expose the 1234 port on the host side to the same port number in the guest running inside the model. You can add additional mappings as desired/needed.

If you are skipping the user space initialization via something like init=/bin/sh for speedup (edit the bootwrapper Makefile), in Foundation model you might need to run:

/sbin/udhcpc eth0

Preparing the guest: External dependencies

The external dependencies are still not bolted in as a git submodule. Therefore you will need to manually fetch:

https://github.com/huawei-erc/osv_external_aarch64

and for example symlink to that material from the external directory as aarch64. For example:

cd ~/osv/external
ln -s ~/osv_external_aarch64 aarch64

Preparing the guest: Environment Variables for make

In addition to the general requirements for building OSv (see README.md),

note that the simple build system recognizes the CROSS_PREFIX environment variable, and looks for the following build tools:

CXX=$(CROSS_PREFIX)g++
CC=$(CROSS_PREFIX)gcc
LD=$(CROSS_PREFIX)ld
STRIP=$(CROSS_PREFIX)strip
OBJCOPY=$(CROSS_PREFIX)objcopy
HOST_CXX=g++

At the beginning of the build process, look for this message:

build.mk:
build.mk: building arch=aarch64, override with ARCH env
build.mk:

If the message does not say arch=aarch64, the crosscompiler could not be found or run correctly. In this case, check the CROSS_PREFIX variable, or the compiler binary name if it's not canonical (do you need to add a symlink for example from g++-4.8.3 to g++ ?). As a last resort, you can manually override all the tools using environment variables for each one of them.

Jani Kokkonen <[email protected]>
Claudio Fontana <[email protected]>