support for systemd-nspawn and runc

[InformatiQ]

there is no support for systemd-nspawn and runc containers

[charles-dyfis-net]

FWIW, one can filter for processes in such containers today using cgroup-based filtering.

[mskarbek]

+1

[leoluk]

+1 - the CoreOS rkt container engine uses systemd-nspawn

[jmcarp]

Any plans to support parsing runc container metadata?

[luca3m]

We have an old branch that parses runc. By the way I'm not pretty sure what's the usecase of runc as it is, usually is embedded into higher level runtimes like docker. Can you explain your usecase for it?

[jmcarp]

@luca3m: my team is looking into using sysdig to monitor containers managed by cloudfoundry, which uses https://github.com/cloudfoundry/guardian to set up runc containers.

[payalc5]

@jmcarp I am on the product team at Sysdig and we are looking at supporting runc in the near future.

[jmcarp]

Thanks @payalc5! Do you have a rough estimate of when runc support might land?

[blues-man]

+1

As CRI-O is becoming the standard interface to talk with OCI runtimes like runc

[holgrrr]

What is the current status of runc support?
I was expecting this to be working out of the box but had significant issues in combination with Podman on CentOS 8.1. 2020 seemed like "near future" in Mid 2017 ;)
Or am I doing something fundamentally wrong?

Container for testing

[holgrrr@c8host0 ~]$ podman run --name sysdigtesting -ti --runtime /usr/bin/runc --rm alpine sh
/ # hostname
5eb2ffaf38a9

Container visible with podman ps

[holgrrr@c8host0 ~]$ podman ps
CONTAINER ID  IMAGE                            COMMAND  CREATED         STATUS             PORTS  NAMES
5eb2ffaf38a9  docker.io/library/alpine:latest  sh       16 seconds ago  Up 16 seconds ago         sysdigtesting

Container sort of visible with lscontainers chisels (not with the csysdig container view)

[root@c8host0 ~]# sysdig -c lscontainers
container.type container.image container.name      container.id 
-------------- --------------- ------------------- ------------ 
docker                                             5eb2ffaf38a9 

Filtering by name given at podman run -does not collect any events (as expected, given the chisels output)

[root@c8host0 ~]# sysdig  container.name=sysdigtesting

Filtering based on the container.id does work to a certain level:

[root@c8host0 ~]# sysdig container.id=5eb2ffaf38a9 and evt.type=read
718 22:35:17.284176954 1 sh (3836) > read fd=0(<f>/dev/pts/0) size=1 
719 22:35:17.284189468 1 sh (3836) < read res=1 data=a 

BUT it seems to be missing at least some events (tested with evt.type=execve), probably as it fails assigning the collected events to the respective container (here starting "uptime" inside the container).

[root@c8host0 ~]# sysdig container.id=5eb2ffaf38a9 and evt.type=execve

-> collects nothing.

However if not combined with the container.id the collection itself works, but is sadly far from usable in a container environment.

[root@c8host0 ~]# sysdig evt.type=execve
4456 22:36:23.774057367 0 sh (3905) > execve filename=/usr/bin/uptime 
4553 22:36:23.775926964 0 uptime (3905) < execve res=0 exe=uptime args= tid=3905(uptime) pid=3905(uptime) ptid=3836(sh) cwd= fdlimit=1024 pgft_maj=0 pgft_min=15 vm_size=1572 vm_rss=4 vm_swap=0 comm=uptime cgroups=cpuset=/.cpu=/.cpuacct=/.io=/.memory=/user.slice/user-1000.slice/session-3.sc... env=HOSTNAME=5eb2ffaf38a9.SHLVL=1.HOME=/root.container=podman.TERM=xterm.PATH=/us... tty=34816 pgid=12 loginuid=0 

System Details:

[holgrrr@c8host0 ~]$ cat /etc/redhat-release 
CentOS Linux release 8.1.1911 (Core) 
[holgrrr@c8host0 ~]$ podman --version
podman version 1.6.4
[holgrrr@c8host0 ~]$ /usr/bin/runc --version
runc version spec: 1.0.1-dev
[root@c8host0 ~]# sysdig --version
sysdig version 0.26.5

FIY: currently running the traditional non-eBPF way, as I am affected by the issue #1611 After patching the fillers.h as described the eBPF driver compiles and is usable. However the behavior described above regarding runc containers remains the same when using the -B switch.

[hashkeks]

EDIT 2022-11-17: Added crun version. Using crun instead of runc does not help with this issue. Updated Fedora and utilities version.

Hello there,

I'm chiming in to ask if there has been progress with this issue. I am currently using Sysdig on two systems (Ubuntu 22.04 and Fedora 36 Cloud Edition) with Podman and it seems like Sysdig does not catch any activity on a running Podman container (using filters like container.id or container.name) or knows about it's existence at all (using lscontainers) - unlike @holgrrr and his small "success" of having sysdig -c lscontainers recognizing the container or having any output when filtering for the container ID.

Running a Docker container works just fine with Sysdig as expected.

The versions I'm currently using are:

# Ubuntu 22.04.1 LTS
 
sysdig version 0.27.1
 
podman version 3.4.4
 
runc version 1.1.0-0ubuntu1.1
spec: 1.0.2-dev

crun version 0.17
commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

and

# Fedora 36 Cloud Edition
 
sysdig version 0.30.2

podman version 4.3.0
 
runc version 1.1.4
spec: 1.0.2-dev
go: go1.19.2
libseccomp: 2.5.3

crun version 1.7
commit: 40d996ea8a827981895ce22886a9bac367f87264
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL

Thank you in advance

[therealbobo]

Hi @hashkeks! I took a quick look and it seems that the problem is more related to podman: if I start a runc/crun container with docker I can record event correctly but, doing the same thing with podman, results in no event collected. Maybe you could open an issue on https://github.com/falcosecurity/libs 😄 In the meantime I'll investigate 👀

[therealbobo]

Hi @InformatiQ! The runc runtime is supported by libs; so sysdig is capable to capture events 😄 Meanwhile the systemd-nspawn is not yet supported. If you are interested you could open an issue on https://github.com/falcosecurity/libs and start a discussion to bring the support on libs. 😄