-
Notifications
You must be signed in to change notification settings - Fork 4
Debug bhyve
There are different strategy that can be used to debug bhyve while you are developing it. This document is a working in progress where we will try to cover as much as possible different ways to debug bhyve.
The FreeBSD HEAD with GENERIC kernel by default is compiled with DEBUG symbols.
[araujo@pipoca] /usr/src/sys/amd64/conf# pwd
/usr/src/sys/amd64/conf
[araujo@pipoca] /usr/src/sys/amd64/conf# vi GENERIC
On GENERIC file, we will see an option:
makeoptions DEBUG=-g
Also there is an option where we can debug the guest vm, to be able to do that we need to add a new device on kernel GENERIC.
device bvmdebug
We need build the user-land with debug symbols. The bhyve(8) itself is compiled with debug symbols by default:
[araujo@pipoca] /usr/src/usr.sbin/bhyve# pwd
/usr/src/usr.sbin/bhyve
[araujo@pipoca] /usr/src/usr.sbin/bhyve# cat Makefile|grep DEBUG
DEBUG_FLAGS= -g -O0
Also you can set DEBUG_FLAGS as a global value to build all world with debug symbols. You need to add inside file /etc/src.conf the following flag:
DEBUG_FLAGS=-g
[araujo@pipoca] /# cd /usr/src/
[araujo@pipoca] /usr/src# make buildworld -j<N_CPU>
[araujo@pipoca] /usr/src# make installworld
NOTE: After we rebuild everything, sometimes we want rebuild parts of the code without clean everything and start from scratch, we can build only the difference.
[araujo@pipoca] /usr/src# make buildworld -j<N_CPU> -DNO_CLEAN
NOTE: Also for bhyve(8) you can build it without rebuild all world.
[araujo@pipoca] /usr/src# cd /usr/src/usr.sbin/bhyve
[araujo@pipoca] /usr/src/usr.sbin/bhyve# make clean; make; make install
[araujo@pipoca] /usr/src# make buildkernel KERNCONF=GENERIC -j<N_CPU>
[araujo@pipoca] /usr/src# make installkernel KERNCONF=GENERIC
NOTE: Also for Kernel you can use the flag -DNO_CLEAN if you don't want to do a clean kernel build.
The last step is merge the changes of some configuration files and scripts.
[araujo@pipoca] /usr/src# mergemaster
[araujo@pipoca] /usr/src# reboot
First launch bhyve using gdb(4)
[araujo@pipoca] /usr/src# gdb /usr/sbin/bhyve
GNU gdb (GDB) 8.1 [GDB v8.1 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/bhyve...Reading symbols from /usr/lib/debug/usr/sbin/bhyve.debug...done.
done.
You will notice the following line: Reading symbols from /usr/sbin/bhyve...Reading symbols from /usr/lib/debug/usr/sbin/bhyve.debug...done. Now we can launch a vm.
(gdb) run -A -H -w -c 1 -m 2048 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm135A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 3,e1000,tap0,mac=00:a0:98:65:89:b6 -s 29,fbuf,vncserver,tcp=192.168.100.111:6035,w=800,h=600,, -s 30,xhci,tablet -s 4,ahci-hd,/dev/zvol/tank/freebsdUFS 135_FreeBSD12UFS
As soon as bhyve(8) crashes you will receive the following messages:
Thread 1 "mevent" received signal SIGBUS, Bus error.
__je_bitmap_unset (bitmap=<optimized out>, binfo=<optimized out>, bit=<optimized out>)
at /freenas-11-nightlies/freenas/_BE/os/contrib/jemalloc/include/jemalloc/internal/bitmap.h:248
248 /freenas-11-nightlies/freenas/_BE/os/contrib/jemalloc/include/jemalloc/internal/bitmap.h: No such file or directory.
(gdb) bt
#0 __je_bitmap_unset (bitmap=<optimized out>, binfo=<optimized out>, bit=<optimized out>)
at /freenas-11-nightlies/freenas/_BE/os/contrib/jemalloc/include/jemalloc/internal/bitmap.h:248
#1 arena_run_reg_dalloc (run=<optimized out>, ptr=<optimized out>) at jemalloc_arena.c:345
#2 arena_dalloc_bin_locked_impl (tsdn=<optimized out>, arena=<optimized out>, chunk=<optimized out>, ptr=<optimized out>,
bitselm=<optimized out>, junked=<optimized out>) at jemalloc_arena.c:2903
#3 0x0000000801733d47 in __je_tcache_bin_flush_small (tsd=<optimized out>, tcache=<optimized out>, tbin=0x80200d148,
binind=<optimized out>, rem=0) at jemalloc_tcache.c:134
#4 0x0000000801733b6f in __je_tcache_event_hard (tsd=0x3e7, tcache=0x80200d000) at jemalloc_tcache.c:45
#5 0x0000000801759621 in __free (ptr=0x88620eb40) at jemalloc_jemalloc.c:1935
#6 0x0000000000413eb9 in mevent_build (mfd=10, kev=0x7fffffffdff0)
at /freenas-11-nightlies/freenas/_BE/os/usr.sbin/bhyve/mevent.c:221
#7 0x0000000000413b7d in mevent_dispatch () at /freenas-11-nightlies/freenas/_BE/os/usr.sbin/bhyve/mevent.c:459
#8 0x000000000040b069 in main (argc=1, argv=0x7fffffffe9a8)
at /freenas-11-nightlies/freenas/_BE/os/usr.sbin/bhyve/bhyverun.c:1052
The first line gives us an idea where it breaks: Thread 1 "mevent" received signal SIGBUS, Bus error. The rest of lines show us where the problem happens, in this particular case the problem is exactly at line #6.
Some of bhyve(8) source files includes two macros DPRINTF(debug) and WPRINTF(warning), It is a good approach to add them on source files that you are working on as well as in points that it would help for debugging. Usually those source files has an int flag to enable or disable DPRINTF.
src: pci_nvme.c
79 static int nvme_debug = 0;
80 #define!DPRINTF(params) if (nvme_debug) printf params
Some examples of DPRINTF usage:
src: pci_nvme.c
363 !·······DPRINTF(("%s\r\n", __func__));
OR
485 !·······!·······DPRINTF(("%s sq %u size %u gaddr %p cqid %u\r\n", __func__,
486 !·······!······· qid, nsq->size, nsq->qbase, nsq->cqid));
First install devel/gdb
root# pkg install gdb
NOTE: Kernel must be compiled with debug symbols: makeoptions DEBUG=-g
Now you can launch kgdb as follow:
root# kgdb /boot/kernel/kernel /var/crash/vmcore.0
If you have kernel.debug and all symbols necessary, you will be able to start the backtrace as follow:
(kgdb) bt
You will see several lines followed by a # and a number, to go to that line you can do as follow:
(kgdb) up 14
#14 0xffffffff83b41760 in emulate_inout_port (vm=<optimized out>, vcpuid=<optimized out>,
vmexit=<optimized out>, retu=<optimized out>) at /z/freebsd-upb/sys/amd64/vmm/vmm_ioport.c:125
125 error = (*handler)(vm, vcpuid, vmexit->u.inout.in,
(kgdb)
Also you can list the frame as follow (ex.: we are listing the frame of line #16):
(kgdb) bt
...
#14 0xffffffff83b41760 in emulate_inout_port (vm=<optimized out>, vcpuid=<optimized out>,
vmexit=<optimized out>, retu=<optimized out>) at /z/freebsd-upb/sys/amd64/vmm/vmm_ioport.c:125
#15 vm_handle_inout (vm=0xfffffe00b5c00000, vcpuid=0, vmexit=0xfffffe00b5c002c0,
retu=0xfffffe00b5e225a7) at /z/freebsd-upb/sys/amd64/vmm/vmm_ioport.c:169
#16 0xffffffff83b38272 in vm_run (vm=0xfffffe00b5c00000, vmrun=0xfffff802836b7a00)
at /z/freebsd-upb/sys/amd64/vmm/vmm.c:1830
...
(kgdb) list *0xffffffff83b38272
0xffffffff83b38272 is in vm_run (/z/freebsd-upb/sys/amd64/vmm/vmm.c:1830).
1825 case VM_EXITCODE_INST_EMUL:
1826 error = vm_handle_inst_emul(vm, vcpuid, &retu);
1827 break;
1828 case VM_EXITCODE_INOUT:
1829 case VM_EXITCODE_INOUT_STR:
1830 error = vm_handle_inout(vm, vcpuid, vme, &retu);
1831 break;
1832 case VM_EXITCODE_MONITOR:
1833 case VM_EXITCODE_MWAIT:
1834 case VM_EXITCODE_VMINSN:
(kgdb)
For more information, here there are some additional resources: