Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dockerfiles: reduce debug information from the production container image. #8821

Conversation

anthisfan
Copy link

follow-up for #8807

By reducing debug information from the production container image, we aim to achieve a lightweight container image and enhance security.
I considered using the same fluent-bit binary for both production and debug containers, while including the debug information file only in the debug container, allowing for the continued use of debuggers such as gdb as before.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • [N/A] Example configuration file for the change
  • [N/A] Debug log output from testing the change
  • [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [N/A] Documentation required for this feature

Backporting

  • [N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@anthisfan anthisfan force-pushed the removing-debug-info-from-production-image branch from 4acffd6 to 8f0d861 Compare May 13, 2024 15:40
@patrick-stephens
Copy link
Contributor

Can you provide some details of the changes in the size, etc.?

@anthisfan anthisfan force-pushed the removing-debug-info-from-production-image branch 2 times, most recently from a85421e to 1e55cf1 Compare May 14, 2024 10:51
@anthisfan
Copy link
Author

anthisfan commented May 14, 2024

Can you provide some details of the changes in the size, etc.?

# Example of building with the mainline Dockerfile (production target)

$ git branch
* master
$ docker buildx build --target production --platform "linux/amd64"  -t fluent-bit:current-20240514 --load .
$ docker image ls fluent-bit:current-20240514
REPOSITORY   TAG                IMAGE ID       CREATED          SIZE
fluent-bit   current-20240514   3a8fe68841a3   18 seconds ago   99MB
$ id=$(docker create fluent-bit:current-20240514)
$ docker cp $id:/fluent-bit ./test
$ du -smh ./test/bin/fluent-bit 
60M     ./test/bin/fluent-bit

# Example of building with the modified Dockerfile from this change #

$ git branch
  master
* removing-debug-info-from-production-image
$ docker buildx build --target production --platform "linux/amd64"  -t fluent-bit:stripped-20240514 --load .
$ docker image ls fluent-bit:stripped-20240514
REPOSITORY   TAG                 IMAGE ID       CREATED          SIZE
fluent-bit   stripped-20240514   d3cb02620594   18 seconds ago   47.2MB
$ id=$(docker create fluent-bit:stripped-20240514)
$ docker cp $id:/fluent-bit ./test
$ du -smh ./test/bin/fluent-bit 
9.8M    ./test/bin/fluent-bit

Hello, @patrick-stephens

As an example for amd64, the commands above demonstrate the following effect.

Specifically, with the current production build, Fluent Bit is being built as a binary of around 60M.

By using the modified Dockerfile in this PR to build the container, we can reduce the binary size to around 9M. As a result, the container size is reduced from 99M to 47M.

(I've updated the commit message to adhere to the commit message rules)

@anthisfan anthisfan force-pushed the removing-debug-info-from-production-image branch from 1e55cf1 to 5f13ab3 Compare May 14, 2024 10:59
… image.

follow-up for fluent#8807
By reducing debug information from the production container image,
we aim to achieve a lightweight container image and enhance security.
The debug image still includes the debug information file,
allowing for easy debugging with tools like gdb.

Signed-off-by: anthis <[email protected]>
@anthisfan anthisfan force-pushed the removing-debug-info-from-production-image branch from 5f13ab3 to a687c62 Compare May 14, 2024 11:00
@anthisfan
Copy link
Author

Additional information:
Although it overlaps a bit with the explanation in the PR description, when building the debug images, the debug files are placed under /usr/lib/debug/fluent-bit/bin.
This is linked as the PATH for debug information to /fluent-bit/bin/fluent-bit, and these files are only available in the debug images.
I confirmed that debug information can be obtained using gdb or other tools, as shown below:

$ git branch
  master
* removing-debug-info-from-production-image
$ docker buildx build --target debug --platform "linux/amd64"  -t fluent-bit:stripped-debug-20240514 --load .
$ docker run -it fluent-bit:stripped-debug-20240514 gdb /fluent-bit/bin/fluent-bit
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /fluent-bit/bin/fluent-bit...
Reading symbols from /usr/lib/debug//fluent-bit/bin/fluent-bit.debug...
(gdb) 

Please let me know if there is anything else I can assist with.

@@ -29,6 +29,7 @@ ARG FLB_CHUNK_TRACE=On
ENV FLB_CHUNK_TRACE=${FLB_CHUNK_TRACE}

RUN mkdir -p /fluent-bit/bin /fluent-bit/etc /fluent-bit/log
RUN mkdir -p /usr/lib/debug/fluent-bit/bin
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for a separate layer, just add it above

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

61ef5d4
I made corrections and added new commits.
Also, I consolidated the objcopy processing into a single layer.

@patrick-stephens
Copy link
Contributor

Please resolve the failing checks as well.

@anthisfan
Copy link
Author

anthisfan commented May 14, 2024

Thanks for the review.I will rework it later :)

@patrick-stephens patrick-stephens changed the title Reducing debug information from the production container image. dockerfiles: reduce debug information from the production container image. May 14, 2024
@anthisfan
Copy link
Author

Thank you for the review. @patrick-stephens
(And thank you for updating the pull request title as well.)

@edsiper
Copy link
Member

edsiper commented May 15, 2024

will this change affect symbol resolution when we hit a crash ? e, .g: we have backtrace and we get a hint on where things got wrong

@anthisfan
Copy link
Author

anthisfan commented May 15, 2024

will this change affect symbol resolution when we hit a crash ? e, .g: we have backtrace and we get a hint on where things got wrong

Thank you for your feedback. @edsiper
I believe this will be a trade-off between weight reduction, security, and debugging capabilities.
Removing debug symbol information will reduce the amount of information available when a crash occurs.
If a core dump is available at the time of a crash, we can use the debug symbol files included in the debug image to obtain detailed information.
While I made this proposal with a focus on weight reduction and security, after some self-reflection, I have not seen (at least to my knowledge) any container environments that store debug symbols. Therefore, I am considering withdrawing this proposal as it may make debugging more difficult.
(Especially if there have been many cases in the past where investigations were conducted based on stack trace information from user-reported crashes, I believe this would be all the more true)
I would appreciate your thoughts on this matter.

@anthisfan anthisfan closed this May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants