-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
in_kafka plugin stopped counting fluentbit_input_records_total #8212
Comments
Hi @ab0oo , Would you share your fluent-bit.conf ? |
Give me a day or two to get it sanitized and approved for release. Here's a snippet from the /api/v1/metrics JSON (since it's easier to grok when run through jq):
Interestingly, on the output side, there are correct record counts:
In my case, I'm pulling prometheus counters via the prometheus_exporter, using this pipeline:
|
Similar issue #8182 seems to be fixed by #8223 |
I pulled this current master: 323f343 which is current as of this writing, and can confirm that both input record and byte counts work for in_kafka: Assuming this change makes it to the next release, I'm happy as a clam. Thank you! |
Bug Report
Describe the bug
In the most recent version of Fluent-bit (2.2.0), we are seeing all zeros for input_records_total in the prometheus and API outputs.
To Reproduce
Running fluentbit with one or more Kafka inputs, poll the prometheus metrics server looking for input statistics:
Expected behavior
Prior to version 2.2.0, the _records_total counters would indicate the correct number of records processed per input event
Your Environment
Custom compiled code on RHEL8
n/a
Linux RHEL8.6, kernel 4.18.0-custom
kafka inputs
HTTP outputs ( we are flushing all data to BQL via HTTPS)
Additional context
We are currently using record count to verify all input/output numbers match up, to ensure no messages are dropped during a complex/convoluted chain to go from application to BQL.
The text was updated successfully, but these errors were encountered: