-
Notifications
You must be signed in to change notification settings - Fork 4
Running a FreeBSD guest under bhyve on ARM systems
To run a FreeBSD guest under bhyve for ARM you will need both a FreeBSD host (can be a virtual machine) and a Linux or Windows machine (the FastModels emulator does not run under FreeBSD). It is recommended to not run the emulator inside a virtual machine since running a virtual machine (bhyve) under an emulator is highly demanding.
The build process has two steps: cross-compiling the FreeBSD kernels and running the emulator.
To cross-compile the FreeBSD host and guest kernels clone this repository and switch to the projects/bhyvearm repository:
root@freebsd:# git clone https://github.com/FreeBSD-UPB/freebsd
root@freebsd:# cd freebsd
root@freebsd:# git checkout projects/bhyvearm
Here you can find the helper script rebuild.sh. It can be used to build everything (world, host and guest).
The kernels will be build with small filesystems built directly into the binary. The specification of these filesystems is placed in an external repository (at https://github.com/FreeBSD-UPB/bhyvearm-utils). The rebuild.sh script expects this repository to be placed in the same directory as freebsd. The script can clone it for you if necessary. However, if you wish to clone it yourself, use the following command (the WORKDIR variable must point to the parent directory of the freebsd directory that contains the sources):
root@freebsd:# git clone https://github.com/FreeBSD-UPB/bhyvearm-utils ${WORKDIR}/bhyvearm-utils
Important: The repository will also have to be cloned on your Linux host in order to run the emulator.
To build everything, simply run freebsd/rebuild.sh -A. The script will clone the bhyvearm-utils repository if required and create a clean cross-compiled build of everything - world, guest and host - in ${WORKDIR}/freebsd-obj. Almost no output will be displayed by the script; if you wish to check the status of the build, view the files {world,guest,host}.out, depending on the build stage. You can use less +F <file> in order to read the output of the build.
Note: The host's filesystem expects a public SSH key to exist, named bhyvearm-utils/ramdisk/id_rsa_board.pub. If you wish, you can create the SSH key, use ssh-keygen -t rsa and copy the public key to the aforementioned location. If the host build fails with a message saying the key is missing, run rebuild -h after copying the key.
To run the The FastModels emulator (available at: https://developer.arm.com/products/system-design/fast-models) and the Fixed Virtual Platforms (available at https://developer.arm.com/products/system-design/fixed-virtual-platforms). You can retrieve an evaluation copy for both from ARM; to do so, please follow the steps on their site. For the Fixed Virtual Platforms please download the Cortex-A15 FVP variant.
As mentioned before, please also clone the bhyvearm-utils repository to your Linux host. After that, please install both FastModels and FixedVirtualPlatforms in the bhyvearm-utils/FastModels directory with their default names (ARM and FVP_VE_Cortex-A15x124 currently). Please also read the included README file.
Note 1: Since it is impossible to automatically retrieve information about the FreeBSD host, please update the REMOTE_WORKDIR (the work directory on the remote) and REMOTE_HOST (the name of the FreeBSD host in the ~/.ssh/config file; a string like <user>@<hostname> or <user>@<ip> should also work, since the name is used by scp) variables accordingly.
Note 2: If you wish to connect the the emulated host using SSH, you will need to create a tap/bridge pair using the add_bridge_64.sh script included in the FastModelPortfolio/ModelNetwork directory with the default names (the files/app.txt configuration file specifies that the tap interface should be named ARM<username>; this in by the run script before starting the emulator using a sed script). You will also need to set an IP address on the bridge.
The network.sh (automatically called by run_bhyve.sh), script on the emulated host assigns the address 10.0.4.42 to its interface, so assigning the 10.0.4.41 address to the bridge should work (you can use ip address add 10.0.4.41 dev armbr0 for this).
Note 3: The kernel image will be automatically copied by the run script at every run; however, the linux-system-semi.axf file must be manually copied (it does not change, so copying it just once should be enough). Please copy it to the bhyvearm-utils/FastModels/files directory (e.g., scp <user>@<hostname>:${REMOTE_WORKDIR}/bhyvearm-utils/boot-wrapper/linux-system-semi.axf bhyvearm-utils/FastModels/files).
After the script has been set up, run ./run to start the emulator and the FreeBSD guest. To run the guest, use the /run_bhyve.sh script. The script will also create a tap interface for virtio-net and a bridge with the IP address 10.0.4.86.
Note: The emulator may display multiple cpu0 GICV_EOIR written but there is no active preemption levelCPU0 has pending interrupts warning messages. This a known error.
To test the VirtIO network device, run the guest.sh script inside the guest. The script will assign the IP address 10.0.4.87 to its vtnet0 network device. You should now be able to ping the host's bridge interface using ping 10.0.4.86.
To test the VirtIO console device you will need to connect to the emulated host via SSH in order to obtain a second console. Please use the key you provided when building the host earlier.
On the host run nc -U /test_sock (the /test_sock socket is created by VirtIO console, according to the command line argument provided by /run_bhyve.sh)
On the guest run cu -l /dev/ttyV0.0 (the ttyV0.0 pseudo-terminal is created by the guest's VirtIO driver)
You should now be able to see messages sent from on console to the other.
To test the VirtIO block device you can mount /dev/vtbd0 and check the contents of the memory device. The VirtIO block device emulation will export the blk_disk file as a block device for the guest. For example, to read the hello file in the block device file, use:
# mount /dev/vtbd0 /mnt
# cat /mnt/hello