Provide tooling for debugging images

[ryanemerson]

The image is based upon ubi-minimal in order to reduce it's footprint. Consequently, the image does not have many of the tools that developers are accustomed to when debugging issues. However, additional packages can be installed at image build time.

What packages do we require for the most common debugging tasks?

[rigazilla]

Not required but I would consider:
vi
more
tar (if I'm not wrong this is required to copy things to/from pod)
ps

[danberindei]

@rigazilla more, really? :))

[tristantarrant]

• Editor: I think nano would be the smallest thing we can package
• Pager: Not sure we need one (use the editor), but I'd go with less
• Sockets and files: lsof (it can list both sockets and file descriptors)
• Network configuration: ip which can list interface addresses and routes
• Process tool: ps (Although you can probably use lsof to get similar results)

I see the image already contains:

• grep for filtering
• coreutils contains a bunch of useful stuff like sort, uniq, tail, etc
• bsdtar is a tar replacement that has built-in support for gzip compression

Copying files from a container can be achieved using [docker|podman] cp ...

[ctron]

• find from findutils
• unzip (good for JAR files)

[tristantarrant]

We could also provide a little help page that is shown when logging into an interactive shell that tells users how to retrieve information (e.g. ip address show to list all network interfaces and addresses)

[ryanemerson]

Good idea! I think a help page will be very useful.

[tristantarrant]

Some recipes:
ps -fC java to get the PID of the java process
ss -t -a get all TCP sockets
ss -u -a get all UDP sockets
lsof |grep -v "IPv[46]" list all open files excluding network sockets

[belaban]

My wishlist:

• netstat
• nc (netcat)
• ping

[tristantarrant]

ss does what netstat does

[belaban]

Does it dump the routing table (netstat -nr), too? How about showing multicast groups (netstat -ng)?
If it does that, no problem, let's minimize what we're adding to an image.

[tristantarrant]

For that, we will use ip which can print interface, route and memberships

[rigazilla]

@rigazilla more, really? :))

more or less :)

[tristantarrant]

ip route Shows unicast routes
ip maddress shows multicast routes

[ctron]

mount

[ctron]

Standard Java tools like: jmap, jps, jstack, …

[ctron]

which

[tristantarrant]

-1 for which. The command builtin can do the same job:
command -v java will output /usr/bin/java

[tristantarrant]

I'm afraid the j* tools pull in java-1.8.0-openjdk-devel which is huge

[tristantarrant]

Not sure about mount either: wouldn't you mount additional volumes from outside the container ?

[ctron]

Not sure about mount either: wouldn't you mount additional volumes from outside the container ?

Yes, but you can inspect mount points inside the pod, like mounted volumes etc:

Example of mount

[jreimann@jreimann ~]$ oc rsh standard-authservice-5bff75468-8bfmw mount
overlay on / type overlay (rw,relatime,context="system_u:object_r:container_file_t:s0:c14,c22",lowerdir=/var/lib/containers/storage/overlay/l/3XN7VZCCBPSWQKPBVUAUGNKIRU:/var/lib/containers/storage/overlay/l/BKA2IZLZ5NHAHKP5SVBKN6JDBL:/var/lib/containers/storage/overlay/l/UEN3XK756EO7HR27BT4WJI3M2E:/var/lib/containers/storage/overlay/l/SRYAIDSM7PXVLCDFGTGVWRZ6QH:/var/lib/containers/storage/overlay/l/5JQOANVYWSDBSRLKL3YDDEZSDD:/var/lib/containers/storage/overlay/l/FVA4PY5AOANEHGBJSN423ZHCZ3:/var/lib/containers/storage/overlay/l/Y3BGL5BRKDCV3FCYZY5MBCNXHV:/var/lib/containers/storage/overlay/l/R6XSNCKYQFPIWR3XUE5W76GT54:/var/lib/containers/storage/overlay/l/JA5LFRFONQ7JDPPSQSKTZQMHY5,upperdir=/var/lib/containers/storage/overlay/a76d8beab207cf140c1e830aac5789c85beb45b9e95869bb51690a619f1c7f6c/diff,workdir=/var/lib/containers/storage/overlay/a76d8beab207cf140c1e830aac5789c85beb45b9e95869bb51690a619f1c7f6c/work)
proc on /proc type proc (rw,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,context="system_u:object_r:container_file_t:s0:c14,c22",size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,context="system_u:object_r:container_file_t:s0:c14,c22",gid=5,mode=620,ptmxmode=666)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,context="system_u:object_r:container_file_t:s0:c14,c22",mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,devices)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/rdma type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,rdma)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,freezer)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,context="system_u:object_r:container_file_t:s0:c14,c22",size=65536k)
tmpfs on /etc/resolv.conf type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755)
tmpfs on /etc/hostname type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /etc/passwd type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755)
/dev/vda3 on /dev/termination-log type xfs (rw,relatime,seclabel,attr2,inode64,prjquota)
/dev/vda3 on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,prjquota)
tmpfs on /run/secrets type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /run/secrets type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /opt/enmasse/cert type tmpfs (ro,relatime,seclabel)
/dev/vda3 on /opt/jboss/keycloak/providers type xfs (rw,relatime,seclabel,attr2,inode64,prjquota)
/dev/vda3 on /opt/jboss/keycloak/standalone/configuration type xfs (rw,relatime,seclabel,attr2,inode64,prjquota)
192.168.12.2:/exports/pvs/pv036 on /opt/jboss/keycloak/standalone/data type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.12.105,local_lock=none,addr=192.168.12.2)
tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime,seclabel)
proc on /proc/asound type proc (ro,relatime)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
proc on /proc/sysrq-trigger type proc (ro,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c14,c22")
tmpfs on /proc/kcore type tmpfs (rw,nosuid,context="system_u:object_r:container_file_t:s0:c14,c22",size=65536k,mode=755)
tmpfs on /proc/keys type tmpfs (rw,nosuid,context="system_u:object_r:container_file_t:s0:c14,c22",size=65536k,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,context="system_u:object_r:container_file_t:s0:c14,c22",size=65536k,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,context="system_u:object_r:container_file_t:s0:c14,c22",size=65536k,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c14,c22")
tmpfs on /sys/firmware type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c14,c22")

[ctron]

I'm afraid the j* tools pull in java-1.8.0-openjdk-devel which is huge

How would you inspect java processes, detect deadlocks, or do a memory dump without JMX?

[tristantarrant]

Not sure about mount either: wouldn't you mount additional volumes from outside the container ?

Yes, but you can inspect mount points inside the pod, like mounted volumes etc:
Example of mount

cat /proc/mounts
returns the same :)

[ctron]

Not sure about mount either: wouldn't you mount additional volumes from outside the container ?

Yes, but you can inspect mount points inside the pod, like mounted volumes etc:
Example of mount

cat /proc/mounts
returns the same :)

Just learned something new 😁 … yes, you are right!

[galderz]

You can check whether you can connect to an ip:port by using:

(: </dev/tcp/172.17.0.6/11222) &>/dev/null && echo "OPEN" || echo "CLOSED"

[galderz]

Hence, don't think you need ping

[galderz]

Also, we should remember that some environment issues with images on Kubernetes/OpenShift can be debugged by launching a pod that contains all the bells and whistles you need, e.g.:

kubectl run rheltoolbox --image=registry.access.redhat.com/rhel7/rhel-tools --restart=Never --attach -i --tty

This can be useful to do things like: if I have a pod in the same namespace where I'm running X, can I connect to X? What is the name resolved for Y? I've found this useful in the past.

[ryanemerson]

#34

For a start. More packages can be added later if required.

We could also provide a little help page that is shown when logging into an interactive shell that tells users how to retrieve information (e.g. ip address show to list all network interfaces and addresses)

This has proven to be a PIA with ubi-minimal and docker, so I'm shelving it for now in favour of a simple entry in the README.md.