bhyve-scripts

A collection of shell scripts to run bhyve VMs on my FreeBSD desktop and server machines.

Environment

• Host Machines

  • FreeBSD 10.2-RELEASE

  • ZFS zvols as the virtual disks for bhyve VMs

  • Bridged tap devices for network access

• Guest Machines

  • Linux Guests: CentOS 6, 7, RHEL 7 and Ubuntu 14.04 LTS

  • BSD Guests: OpenBSD 5.8

ZFS zvols

These disk1 are thin-provisioned zvols.

$ zfs list -r zroot/bhyve
NAME                                    USED  AVAIL  REFER  MOUNTPOINT
zroot/bhyve                            107G   364G   144K  none
zroot/bhyve/centos6-vertica           49.7G   364G    96K  none
zroot/bhyve/centos6-vertica/disk1     49.7G   364G  41.2G  -
zroot/bhyve/centos7-docker1           38.2G   364G   144K  none
zroot/bhyve/centos7-docker1/disk1     38.2G   364G  32.3G  -
zroot/bhyve/openbsd                    969M   364G    96K  none
zroot/bhyve/openbsd/disk1              969M   364G   969M  -
zroot/bhyve/rhel7-ost-icehouse        7.55G   364G   144K  none
zroot/bhyve/rhel7-ost-icehouse/disk1  7.55G   364G  8.27G  -
zroot/bhyve/rhel7-ost-juno            5.20G   364G   144K  none
zroot/bhyve/rhel7-ost-juno/disk1      5.20G   364G  4.76G  -
zroot/bhyve/ubuntu-14.04              5.00G   364G   144K  none
zroot/bhyve/ubuntu-14.04/disk1        5.00G   364G  5.00G  -

$ sudo zfs create zroot/bhyve/centos7-docker1
$ sudo zfs create -V 48G -s zroot/bhyve/centos7-docker1/disk1

Prerequisites

(TODO) Install grub-bhyve etc.

Setup

The following script loads the kernel modules for bhyve and creates tap devices for the VMs.

$ sudo ./setup.sh

Installing Linux — Booting a VM from installer ISO Image

Edit device.map.

bhyve-scripts/centos7/device.map

(hd0) /dev/zvol/zroot/bhyve/centos7-docker1/disk1
(cd0) /home/tatsuya/installers/centos/CentOS-7.0-1406-x86_64-DVD.iso

Run grub-bhyve as root.

$ cd centos7
$ sudo grub-bhyve -m device.map -r cd0 -M 4096M centos7-docker1
grup> ls
(hd0) (hd0,msdos2) (hd0,msdos1) (cd0) (cd0,msdos2) (host) (lvm/centos-root) (lvm/centos-swap)
grub> ls (cd0)/
CentOS_BuildTag EFI/ EULA GPL images/ isolinux/ LiveOS/ Packages/
repodata/ RPM-GPG-KEY-CentOS-Testing-7 RPM-GPG-KEY-CentOS-7 TRANS.TBL
grup> ls (cd0)/isolinux
boot.cat boot.msg grub.conf initrd.img isolinux.bin isolinux.cfg
memtest splash.png TRANS.TBL upgrade.img vesamenu.c32 vmlinuz

grub> linux (cd0)/isolinux/vmlinuz
grub> initrd (cd0)/isolinux/initrd.img
grub> boot

NOTE:

For CentOS, it’s recommended to perform VNC-based graphical installation rather than console-based installation because graphical one gives you more customization options (e.g. custom disk layout) To do that, add the following options to linux command.

grub> linux (cd0)/isolinux/vmlinuz vnc headless ip=dhcp ksdevice=eth0 lang=en_US keymap=us

Once the installer started, it will give tell you VNC server address and port.

For example,

Please manually connect your vnc client to 10.0.0.117:1 to begin the install.

Run bhyve as root. Note that the VM has the installer DVD ISO attached.

$ sudo bhyve -c 4 -m 4096M -H -P -A -l com1,stdio \
    -s 0:0,hostbridge -s 1:0,lpc \
    -s 2:0,virtio-net,tap1 \
    -s 3,ahci-cd,/home/tatsuya/installers/centos/CentOS-7.0-1406-x86_64-DVD.iso \
    -s 4,virtio-blk,/dev/zvol/zroot/bhyve/centos7-docker1/disk1 \
    centos7-docker1

When installation is completed, press the "Restart" button on the installer screen. This will shutdown the VM and drop you into a shell prompt (instead of restarting). Proceed to next chapter to boot the VM from the disk image.

Booting a Linux VM from Disk Image

$ cd centos7

Edit grub.in. You might want to take a look at grub menu for kernel parameters.

Run the VM.

$ sudo ./run.sh

Installing OpenBSD — Booting a VM from installer ISO Image

Edit device.map.

bhyve-scripts/openbsd/device.map

(hd0) /dev/zvol/zroot/bhyve/openbsd/disk1
(cd0) /home/tatsuya/installers/openbsd/openbsd-5.7/install57.iso

Run grub-bhyve as root. Ensure to boot with the OpenBSD install kernel (bsd.rd).

$ cd openbsd
$ sudo grub-bhyve -m device.map -r cd0 -M 256M openbsd
grup> ls
(hd0) (cd0) (host)
grub> ls (cd0)/
5.7/ etc/ TRANS.TBL
grup> ls (cd0)/5.7
amd64/ TRANS.TBL
grub> ls (cd0)/5.7/amd64/
base57.tgz boot.catalog bsd bsd.mp bsd.rd cdboot cdbr comp57.tgz
game57.tgz INSTALL.amd64 man57.tgz SHA256 TRANS.TBL xbase57.tgz
xfont57.tgz xserv57.tgz xshare57.tgz

grub> kopenbsd -h com0 (cd0)/5.7/amd64/bsd.rd
grub> boot

NOTE: Example

Run bhyve as root. Note that the VM has the installer ISO attached. Also -W option seems required. It forces virtio to use single-vector MSI.

$ sudo bhyve -W -m 256M -H -P -A -l com1,stdio \
    -s 0:0,hostbridge \
    -s 1:0,lpc \
    -s 2:0,virtio-net,tap4 \
    -s 3,ahci-cd,/home/tatsuya/installers/openbsd/openbsd-5.7/install57.iso \
    -s 4,virtio-blk,/dev/zvol/zroot/bhyve/openbsd/disk1 \
    openbsd

IMPORTANT: Adjust the System Clock

Login to the VM and ensure the system clock is correct. By some reason it’s usually advance 3 days, so ntpd won’t be able to adjust the clock. Run ntpdate to fix it.2

[docker1] /home/tatsuya% sudo systemctl stop ntpd.service
[docker1] /home/tatsuya% sudo ntpdate -b pool.ntp.org
16 Mar 13:46:27 ntpdate[2046]: step time server 202.112.29.82 offset -277199.180289 sec
[docker1] /home/tatsuya% sudo systemctl start ntpd.service

NOTE: This might be fixed in 10.2-RELEASE (Reference)

When Linux Kernel in a VM is Upgraded

Edit grub.in so that it will boot from the latest Linux Kernel.

bhyve-scripts/centos7/grub.in

linux (hd0,msdos1)/vmlinuz-3.10.0-123.20.1.el7.x86_64 root=/dev/mapper/centos-root
initrd (hd0,msdos1)/initramfs-3.10.0-123.20.1.el7.x86_64.img
boot

License

bhyve-script is open-sourced under the MIT license. See the LICENSE file for details.