Fluent bit 3.2.4 taking more CPU for rewrite_tag compared to older versions (1.9.9 or 2.2.2) with same config

[ShankarSrinivasM]

Bug Report

Describe the bug
I'm currently trying to upgrade Fluent bit on our embedded Linux device (which is very similar to Beagle bone black) from 1.9.9 to release 3.2.4. We noticed that with fluent-bit 3.2.4, the CPU utilization of fluent bit has increased compared to older version 1.9.9 using the same configurations.

After some deep dive, I noticed that the rewrite_tags plugin is the one that seems to be causing an issue. We are currently using rewrite_tags in multiple instances within our config for parsing differently formatted data, coming from same log file (say /var/log/messages). This has been working fine with fluent-bit 1.9.9 for a while now.

The fluent bit seems to be consuming CPU even when there is no data to be processed from the tail (I did this by making the fluent-bit tail a temp file that no one logs to).

I ran the same thing on the fluent bit 3.2.4 docker container and the results are the same. We do see some CPU usage when there is no data to be processed (see graphs attached). Also the CPU usage increases with increase in number of rewrite tags used.

Replacing rewrite_tags with something like grep plugin fixed the CPU utilization, however in our case we can not change it in all the places that we have used rewrite_tag unless we redirect our logs to separate log files and use grep plugin.

I understand that there is considerable changes/improvements made from 1.9.9, however I still don't see any reason why the fluent is consuming the CPU when there is no data to be processed.

Is this expected?

I have attached a sample config file that is very similar to how we have used to minimize complexity. Also attached are the CPU usage plots over first 5 mins (captured using top command, scanned every 500ms) on multiple Fluent bit versions with same attached configuration, captured using appropriate version docker containers.

To Reproduce

• Steps to reproduce the problem:
  • Download the docker container for fluent bit 3.2.4 from fluent-bit documentation here
  • Copy/Export and unzip the attached workdir.zip to the root directory of the docker container
  • Create messages file, if it does not exist using command touch /var/log/messages
  • Run fluent bit with the conf file inside the workdir as /fluent-bit/bin/fluent-bit -c /workdir/fluent-bit.conf &
  • Monitor/Capture CPU usage using top command

Expected behavior
CPU usage of fluent bit under no load conditions should be close to Zero

Screenshots

Your Environment

• Version used: 3.2.4

• Configuration:
  workdir.zip

• Environment name and version (e.g. Kubernetes? What version?): Fluent bit Docker Container 3.2.4-debug

• Server type and version:

• Operating System and version:

• Filters and plugins: