Save and Restore a virtual machine using bhyve

NOTE: The project is under development and may not be completed.

Setup - Clone, build, install

Clone project

root@host # git clone https://github.com/FreeBSD-UPB/freebsd
root@host # cd freebsd
root@host # git checkout projects/bhyve_save_restore 

Build and install world

root@host # make buildworld -j9
root@host # make installworld -j9
root@host # reboot

Build and install kernel

root@host # make buildkernel -j9
root@host # make installkernel -j9
root@host # reboot

Setup - Create a FreeBSD virtual machine using bhyve

To create a virtual machine using bhyve, you can use the tutorial from here.

Run the virtual machine

Run the previous created virtual machine using bhyve.

# load the OS
root@host # bhyveload -c stdio -m 512M -d guest.img guest_vm

# start the virtual machine
root@host # bhyve -c 1 -m 512M -H -A -P -s 0:0,hostbridge -s 1:0,lpc -s 3:0,virtio-net,tap0 -s 4:0,virtio-blk,guest.img -l com1,stdio guest_vm 
....
root@guest #

Snapshot the virtual machine

From another terminal, run the following command in order to make a checkpoint:

root@host # bhyvectl --checkpoint file.ckp --vm=guest_vm

Use sync to be sure that the info is written on the disk (since we use a copy-on-write mechanism and the checkpoint thread runs independently from the bhyverun).

root@host # sync

Three new files will be created after the checkpoint process is completed:

• file.ckp - this file contains the saved guest's virtual memory
• file.ckp.kern - in this file are saved the kernel related structures such as struct vm (the virtual machine's representation in bhyve), struct vhpet (the virtual HPET related to the guest) etc.
• file.ckp.meta - some metadata used in order to know the structure of the file.ckp.kern file and other things necessarily in the restore process

Power off the virtual machine:

root@guest # poweroff

Alternatively, the bhyvectl tool can be used to destroy a virtual machine:

root@guest # bhyvectl --destroy --vm=guest_vm 

Suspend virtual machine's state

There is also available an option to suspend the virtual machine, while saving its state in the checkpoint files. While the virtual machine is running, from another terminal, run the following command in order to suspend the virtual machine's state:

root@host # bhyvectl --suspend file.ckp --vm=guest_vm

The information will be saved in file.ckp, file.ckp.kern and file.ckp.meta and the guest will be powered off.

Restore the virtual machine from checkpoint

# load the OS
root@host # bhyveload -c stdio -m 512M -d guest.img guest_vm

# run bhyverun with the same parameters used in the "Run the virtual machine" + "-r file.ckp"

root@host # bhyve -c 1 -m 512M -H -A -P -s 0:0,hostbridge -s 1:0,lpc -s 3:0,virtio-net,tap0 -s 4:0,virtio-blk,guest.img -l com1,stdio -r file.ckp guest_vm
...
root@guest #