Multiple copies of events appearing when uprobe distinct copies of an .so

[RobAtticus]

I am trying to use this library in a containerized environment. Each container on my node runs PostgreSQL and the function I am uprobing is in an .so that is loaded as part of PG's shared_preload_libraries. Each container contains its own copy of this library, but they are the same binary. If I probe one container's library -- using the container specific path -- all is well. However, when I start probing additional copies of the library in different paths, I start seeing the probes from all libraries multipled by the number of copies that have been probed. That is, if two containers have been uprobed, I see 2 copies of each event when I use the perf.Reader. If three containers have been uprobed, then I see 3, etc.

Has anyone seen this, or have a recommendation? Perhaps a UprobeOption needs to be set? I can't easily use PID in there because Postgres is multi-process, which would require me to probe an ever-changing set of PIDs.

[ti-mo]

Interesting! Sounds like a potential side effect on the kernel side? I doubt the lib has much control over this.

Maybe @mmat11 @olsajiri have any ideas?

[RobAtticus]

I think I have found the source of the problem but I am still doing more testing.

A colleague pointed out the kernel differentiates uprobes on inodes, so looking at the inodes of the libraries in different containers revealed that they had different devices but the same inode. Adding some code to only add a uprobe once per inode appears to have resolved the multiple copies. Like I said, I'm still doing some more testing to make sure I am getting all the records I expect in all cases, but this seems like it might be the issue.

[ti-mo]

That makes sense. When running multiple copies of an OCI image, the same base gets bind-mounted into each container, which should make all copies resolve to the same inode (as long as they haven't been modified from within the container).

Thanks for reporting back!

[RobAtticus]

I feel pretty confident to say that this was an issue with the inode value of each copy of the library being the same was causing multiple copies of records because we were essentially adding multiple uprobes to the same function. I hope this helps someone else in the future if they run into the same issue.

Some keywords to help people search: k8s, kubernetes, kube, docker, containers, multiple copies, duplicates, uprobe, inodes :)