Load ebpf module without host kernel header dependency

[rootfs]

Is your feature request related to a problem? Please describe.
Currently kepler dynamically compiles the ebpf program when started. This compilation requires kernel headers. If kernel headers are missing, the compilation will fail.

Describe the solution you'd like

1. A pre-compiled ebpf module targeting a range of kernel versions can go without the kernel headers dependency.
2. A pre-installed kernel source that matches the host kernel version so that kepler can use this pre-installed headers to compile the ebpf program.

[rootfs]

Package pre-compiled ebpf module

In bcc module https://github.com/sustainable-computing-io/kepler/blob/main/vendor/github.com/iovisor/gobpf/bcc/module.go#L83

func newModule(code string, cflags []string) *Module {
	cflagsC := make([]*C.char, len(defaultCflags)+len(cflags))
	defer func() {
		for _, cflag := range cflagsC {
			C.free(unsafe.Pointer(cflag))
		}
	}()
	for i, cflag := range cflags {
		cflagsC[i] = C.CString(cflag)
	}
	for i, cflag := range defaultCflags {
		cflagsC[len(cflags)+i] = C.CString(cflag)
	}
	cs := C.CString(code)
	defer C.free(unsafe.Pointer(cs))
	c := C.bpf_module_create_c_from_string(cs, 2, (**C.char)(&cflagsC[0]), C.int(len(cflagsC)), (C.bool)(true), nil)
	if c == nil {
		return nil
	}
	return &Module{
		p:              c,
		funcs:          make(map[string]int),
		kprobes:        make(map[string]int),
		uprobes:        make(map[string]int),
		tracepoints:    make(map[string]int),
		rawTracepoints: make(map[string]int),
		perfEvents:     make(map[string][]int),
	}
}

Similarly, an ebpf pre-compiler can call C.bpf_module_create_c_from_string and serialize pointer c so Kepler can load it without having to re-compile on the fly.

This pre-compiled module, however, has to have fixed cflags.

Pre-packaged Kernel Source

If the target kernel is known, kepler can be built into a container image that has the kernel headers pre-installed and BCC_KERNEL_SOURCE to point to that source during compilation.

[jichenjc]

do we want to build by our own or seems we should provide a tool to help create the module?

[rootfs]

for pre-compiled, yes, new tools are needed. The pre-installed kernel headers way will need kepler to compile against the pre-installed headers.

[marceloamaral]

This is related to BPF CO-RE (Compile Once – Run Everywhere)
https://nakryiko.com/posts/bpf-portability-and-co-re/

The cilium ebpf library has overcome this issue: cilium/ebpf#114.
It might be good to give a try instead of using gobpf

[rootfs]

@marceloamaral yes, eventually that's the solution to go if we can get perf event work

[tohojo]

I think the right thing to do here is to drop BCC and dynamic compilation entirely, and go for fully pre-compiled BPF programs with CO-RE. There's some discussion on how to achieve this from Go in this Cilium discussion: cilium/ebpf#548

(Side note: cool project! Hadn't heard about it before :) )

[rootfs]

For pre-compiled module, we can use the same compilation flags as bcc. This can be done by setting DEBUG_PREPROCESSOR flag in bpf_module_create_c_from_string, i.e. setting to 6 instead of 2.

Below is the clang command to generate the module

clang -cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -emit-llvm-uselists -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name main.c -mrelocation-model static -fno-jump-tables -mframe-pointer=none -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-cpu x86-64 -tune-cpu generic -mllvm -treat-scalable-fixed-error-as-warning -debug-info-kind=constructor -dwarf-version=5 -debugger-tuning=gdb -fcoverage-compilation-dir=/usr/src/kernels/4.18.0-425.3.1.el8.x86_64 -nostdsysteminc -nobuiltininc -resource-dir lib/clang/14.0.6 -isystem /virtual/lib/clang/include -include ./include/linux/kconfig.h -include /virtual/include/bcc/bpf.h -include /virtual/include/bcc/bpf_workaround.h -include /virtual/include/bcc/helpers.h -isystem /virtual/include -I /home/hchen/src/github.com/sustainable-computing-io/kepler -D __BPF_TRACING__ -I arch/x86/include/ -I /lib/modules/4.18.0-425.3.1.el8.x86_64/build/arch/x86/include/generated -I include -I /lib/modules/4.18.0-425.3.1.el8.x86_64/build/include -I arch/x86/include/uapi -I /lib/modules/4.18.0-425.3.1.el8.x86_64/build/arch/x86/include/generated/uapi -I include/uapi -I /lib/modules/4.18.0-425.3.1.el8.x86_64/build/include/generated/uapi -D __KERNEL__ -D KBUILD_MODNAME="bcc" -D MAP_SIZE=10240 -D NUM_CPUS=12 -D SET_GROUP_ID -D NUMCPUS=12 -O2 -Wno-deprecated-declarations -Wno-gnu-variable-sized-type-not-at-end -Wno-pragma-once-outside-header -Wno-address-of-packed-member -Wno-unknown-warning-option -Wno-unused-value -Wno-pointer-sign -fdebug-compilation-dir=/usr/src/kernels/4.18.0-425.3.1.el8.x86_64 -ferror-limit 19 -fgnuc-version=4.2.1 -vectorize-loops -vectorize-slp -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o main.bc -x c /virtual/main.c

Note, some of the headers are from bcc and bcc-devel.

If the module can be compiled, we can load it into kepler without having to compile from source
https://github.com/iovisor/gobpf/blob/master/elf/elf.go#L497

[tohojo]

For pre-compiled module, we can use the same compilation flags as bcc. This can be done by setting DEBUG_PREPROCESSOR flag in bpf_module_create_c_from_string, i.e. setting to 6 instead of 2.

That seems a bit long. I'd suggest modelling the precompiled BPF modules on the libbpf-tools in BCC instead: https://github.com/iovisor/bcc/tree/master/libbpf-tools/

There is also the bpf-examples repo you can borrow code from: https://github.com/xdp-project/bpf-examples

In both cases, it should be possible to get rid of the BCC header dependencies completely, and just have a relatively simple Makefile rule that builds the BPF objects...

[rootfs]

@tohojo thanks! that's a good pointer!

[tohojo]

You're welcome :)

[marceloamaral]

Using libbpf might be a more general solution that will work in different environments. We just need to make sure that we will compile the bpf code in the system that has BTF.

Here is a code with example to load a pre-compiled libbpf code in golang:
https://github.com/lizrice/libbpfgo-beginners

Cilium has ported to golang many libbpf functions and can also be a source of inspiration.

[tohojo]

I believe most Go people prefer to use the native (Cilium) go-bpf library over the libbpf bindings, partly because the FFI linking to libbpf for libbpfgo is a bit of a hassle (not sure of the details, not a Go person, really). There are some features that are supported by libbpf that are not in the go-bpf library, but if you don't need those go-bpf may be easier to use and integrate...

[rootfs]

sounds good. The bare minimum is to remove the BCC macros from bpf code to stay framework neural, we can then choose which library we can use next.

[sunya-ch]

Update:

I can now load pre-compiled eBPF program (.bpf.o) to the go program using only libbpf library based on Makefile in libbpfgo-beginners.

fmt.Printf("%d: %s, %d, %d, %d\n", ct.PID, ct.Command, ct.ProcessRunTime, ct.CPUCycles, ct.CacheMisses)

~/kepler-bpf$ sudo ./kepler
2740463: containerd-shim, 0, 0, 0
34: migration/3, 0, 0, 0
1546592: kworker/4:0, 1019980987, 261876359, 361474
2504916: kube-rbac-proxy, 0, 0, 0
2718138: xarchiver, 106289, 136329490, 182528
2498486: containerd, 0, 0, 0
2501372: containerd-shim, 0, 0, 0
14: ksoftirqd/0, 0, 0, 0
2500175: containerd-shim, 0, 0, 0
2653615: panel-8-pulseau, 999898582, 658179945, 1604033

Will merge to Kepler code and push PR soon.

[stleerh]

Shouldn't this be closed since it was fixed by PR 733?