Make it possible to dlopen libfluent-bit.so #8005
Conversation
tobim
requested review from
fujimotos,
niedbalski,
patrick-stephens and
celalettin1286
as code owners
The static copy of jemalloc that is included in the distributed fluent-bit binary packages prevents use loading the library at runtime as it is typically done in plugin systems. A slight change to the jemalloc configuration lifts this restriction. Signed-off-by: Tobias Mayer <[email protected]>
tobim
force-pushed
the
unbreak-dlopen-libfluent-bit.so
branch
from
9fae87a to
e2088ec
Compare
|
thanks for your contribution. Jemalloc exists to provide better performance in memory management, by disabling that flag will affect performance per the description on this comment: I think I would be easier if you built Fluent Bit without Jemalloc. Or have some perf comparison with and without that flag to understand potential side effects. |
|
Thanks for your reply. I kind of agree that An arguably better approach would be to only link
Vendored dependencies cause their own set of problems and it's possible that other people run into the same problem as me. I'm still hopeful that we can find an a resolution of the issue that is acceptable for upstream inclusion. |
The official fluent-bit package contains a bundeld copy of jemalloc that poses 2 problems: First, it is built in a way that makes it impossible to load `libfluent-bit.so` with `dlopen()`. And second, the way it initializes thread local variables creates problems with the jemalloc present in `libarrow.so`, specifically it causes a segfault when initializing the s3 filesystem. I'm trying to resolve this issue upstream at fluent/fluent-bit#8005, but until it is fixed we have to rely on this fallback.
The official fluent-bit package contains a bundeld copy of jemalloc that poses 2 problems: First, it is built in a way that makes it impossible to load `libfluent-bit.so` with `dlopen()`. And second, the way it initializes thread local variables creates problems with the jemalloc present in `libarrow.so`, specifically it causes a segfault when initializing the s3 filesystem. I'm trying to resolve this issue upstream at fluent/fluent-bit#8005, but until it is fixed we have to rely on this fallback.
|
Let's discuss this approach in #8011. |
The official fluent-bit package contains a bundeld copy of jemalloc that poses 2 problems: First, it is built in a way that makes it impossible to load `libfluent-bit.so` with `dlopen()`. And second, the way it initializes thread local variables creates problems with the jemalloc present in `libarrow.so`, specifically it causes a segfault when initializing the s3 filesystem. I'm trying to resolve this issue upstream at fluent/fluent-bit#8005, but until it is fixed we have to rely on this fallback. Signed-off-by: Tobias Mayer <[email protected]>
The static copy of jemalloc that is included in the distributed fluent-bit binary packages prevents use loading the library at runtime as it is typically done in plugin systems.
A slight change to the jemalloc configuration lifts this restriction.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.