Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Counter should start from 0 #69

Open
mhoshino opened this issue Dec 4, 2018 · 6 comments
Open

Counter should start from 0 #69

mhoshino opened this issue Dec 4, 2018 · 6 comments

Comments

@mhoshino
Copy link

mhoshino commented Dec 4, 2018

Current implementation is not following the below rule

https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics

To avoid this, export 0 (or NaN, if 0 would be misleading) for any time series you know may exist in advance.

This is leading to situation's such as creating prometheus alert.
For example

rate(my_metric[1m]) > 0

Will not fire alert when my_metric=1 but instead my_metric > 2.
This is because my_metric does not start with 0 and start's with 1

@dkalashnik
Copy link

Usually that happen after fluentd restart. Maybe there would be a way to make some storage to persist metrics/values between reboots? Some sort of buffer on disk

@mhoshino
Copy link
Author

Hi @dkalashnik .
Do we really need to store this on buffer disk?
The problem I am seeing is that, it needs to capture a message at least 2 times every time for it to generate alert after restart happens on fluentd. If it can start from 0 after the restart, I am assuming it will be able to fire the alert on the 1st message

@mescanef
Copy link

@mhoshino basicly its about prometheus/prometheus#1673

@mhoshino
Copy link
Author

mhoshino commented Sep 3, 2019

It seems without this being resolved there counter for the prometheus ruby client does not start from 0

prometheus/client_ruby#141

@Kuckkuck
Copy link

Kuckkuck commented Feb 6, 2020

+1
We still need this for error counting and doing delta alerting, otherwise the first counter will never be catched and you cannot use this metric for alerting.

@yeplaa
Copy link

yeplaa commented May 3, 2022

@mhoshino @Kuckkuck have you found a solution to use fluentd-plugin-prometheus metrics from prometheus?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants