Still got the dead Puma workers' metrics

[macchiang]

Hi,
I encountered an issue that I got the dead and live Puma workers' metrics together. I leveraged puma_worker_killer gem in my project, so it will kill one of worker periodically. How to get the live Puma workers' metrics only? Any work around? thanks.

Current processes

root@7f54f74cb9-gk6jj:/# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  20060  3872 ?        Ss   Dec17   0:00 bash -c rails s -p 3000 -b '0.0.0.0'
root          10  0.0  1.2 1336288 344940 ?      Sl   Dec17   0:31 puma 4.3.1 (tcp://0.0.0.0:3000) [api]
root        1776  0.0  0.0  20308  4192 pts/3    Ss   Dec17   0:00 bash
root       13033  2.8  2.7 2022132 747392 ?      Sl   Dec17  12:49 puma: cluster worker 4: 10 [api]
root       26600  2.9  1.9 1682532 530292 ?      Sl   Dec17   2:46 puma: cluster worker 3: 10 [api]
root       26663  2.5  2.1 1682532 572832 ?      Sl   Dec17   2:24 puma: cluster worker 5: 10 [api]
root       26736  2.5  2.1 1683560 585588 ?      Sl   Dec17   2:24 puma: cluster worker 0: 10 [api]
root       26791  2.5  1.8 1683560 494260 ?      Sl   Dec17   2:22 puma: cluster worker 1: 10 [api]
root       26855  2.7  1.8 1616996 494264 ?      Sl   Dec17   2:30 puma: cluster worker 2: 10 [api]
root       30221  0.5  0.0  20308  4056 pts/6    Ss   01:30   0:00 bash
root       30229  0.0  0.0  38348  3412 pts/6    R+   01:30   0:00 ps aux

What I got from metrics

root@7f54f74cb9-gk6jj:/# curl http://127.0.0.1:3000/metrics
# HELP puma_backlog Multiprocess metric
# TYPE puma_backlog gauge
puma_backlog{index="0",pid="12860"} 0
puma_backlog{index="0",pid="12913"} 0
puma_backlog{index="0",pid="13033"} 0
puma_backlog{index="0",pid="24"} 0
puma_backlog{index="0",pid="26600"} 0
puma_backlog{index="0",pid="26663"} 0
puma_backlog{index="0",pid="26736"} 0
puma_backlog{index="0",pid="26791"} 0
puma_backlog{index="0",pid="26855"} 0
puma_backlog{index="0",pid="37"} 2
puma_backlog{index="0",pid="63"} 0
puma_backlog{index="0",pid="6446"} 0
puma_backlog{index="0",pid="76"} 0
puma_backlog{index="0",pid="89"} 0
puma_backlog{index="1",pid="12860"} 0
puma_backlog{index="1",pid="12913"} 0
puma_backlog{index="1",pid="13033"} 0
puma_backlog{index="1",pid="24"} 1
puma_backlog{index="1",pid="26600"} 0
puma_backlog{index="1",pid="26663"} 0
puma_backlog{index="1",pid="26736"} 0
puma_backlog{index="1",pid="26791"} 0
puma_backlog{index="1",pid="26855"} 0
puma_backlog{index="1",pid="37"} 2
puma_backlog{index="1",pid="63"} 0
puma_backlog{index="1",pid="6446"} 0
puma_backlog{index="1",pid="76"} 0
puma_backlog{index="1",pid="89"} 0
puma_backlog{index="2",pid="12860"} 0
puma_backlog{index="2",pid="12913"} 0
puma_backlog{index="2",pid="13033"} 0
puma_backlog{index="2",pid="24"} 0
puma_backlog{index="2",pid="26600"} 0
puma_backlog{index="2",pid="26663"} 1
puma_backlog{index="2",pid="26736"} 1
puma_backlog{index="2",pid="26791"} 0
puma_backlog{index="2",pid="26855"} 0
puma_backlog{index="2",pid="37"} 2
puma_backlog{index="2",pid="63"} 1
puma_backlog{index="2",pid="6446"} 0
puma_backlog{index="2",pid="76"} 0
puma_backlog{index="2",pid="89"} 0
puma_backlog{index="3",pid="12860"} 0
puma_backlog{index="3",pid="12913"} 0
puma_backlog{index="3",pid="13033"} 0
puma_backlog{index="3",pid="24"} 0
puma_backlog{index="3",pid="26600"} 0
puma_backlog{index="3",pid="26663"} 0
puma_backlog{index="3",pid="26736"} 0
puma_backlog{index="3",pid="26791"} 0
puma_backlog{index="3",pid="26855"} 0
puma_backlog{index="3",pid="37"} 0
puma_backlog{index="3",pid="63"} 1
puma_backlog{index="3",pid="6446"} 0
puma_backlog{index="3",pid="76"} 0
puma_backlog{index="3",pid="89"} 0
puma_backlog{index="4",pid="12860"} 0
puma_backlog{index="4",pid="12913"} 0
puma_backlog{index="4",pid="13033"} 0
puma_backlog{index="4",pid="24"} 0
puma_backlog{index="4",pid="26600"} 0
puma_backlog{index="4",pid="26663"} 0
puma_backlog{index="4",pid="26736"} 0
puma_backlog{index="4",pid="26791"} 0
puma_backlog{index="4",pid="26855"} 0
puma_backlog{index="4",pid="37"} 0
puma_backlog{index="4",pid="63"} 0
puma_backlog{index="4",pid="6446"} 2
puma_backlog{index="4",pid="76"} 0
puma_backlog{index="4",pid="89"} 0
puma_backlog{index="5",pid="12860"} 0
puma_backlog{index="5",pid="12913"} 0
puma_backlog{index="5",pid="13033"} 0
puma_backlog{index="5",pid="24"} 0
puma_backlog{index="5",pid="26600"} 0
puma_backlog{index="5",pid="26663"} 0
puma_backlog{index="5",pid="26736"} 1
puma_backlog{index="5",pid="26791"} 0
puma_backlog{index="5",pid="26855"} 0
puma_backlog{index="5",pid="37"} 0
puma_backlog{index="5",pid="63"} 0
puma_backlog{index="5",pid="6446"} 1
puma_backlog{index="5",pid="76"} 0
puma_backlog{index="5",pid="89"} 1
# HELP puma_booted_workers Multiprocess metric
# TYPE puma_booted_workers gauge
puma_booted_workers{pid="12860"} 6
puma_booted_workers{pid="12913"} 6
puma_booted_workers{pid="13033"} 6
puma_booted_workers{pid="24"} 6
puma_booted_workers{pid="26600"} 6
puma_booted_workers{pid="26663"} 6
puma_booted_workers{pid="26736"} 6
puma_booted_workers{pid="26791"} 6
puma_booted_workers{pid="26855"} 6
puma_booted_workers{pid="37"} 6
puma_booted_workers{pid="63"} 6
puma_booted_workers{pid="6446"} 6
puma_booted_workers{pid="76"} 6
puma_booted_workers{pid="89"} 6
# HELP puma_max_threads Multiprocess metric
# TYPE puma_max_threads gauge
puma_max_threads{index="0",pid="12860"} 5
puma_max_threads{index="0",pid="12913"} 5
puma_max_threads{index="0",pid="13033"} 5
puma_max_threads{index="0",pid="24"} 5
puma_max_threads{index="0",pid="26600"} 5
puma_max_threads{index="0",pid="26663"} 5
puma_max_threads{index="0",pid="26736"} 5
puma_max_threads{index="0",pid="26791"} 5
puma_max_threads{index="0",pid="26855"} 5
puma_max_threads{index="0",pid="37"} 5
puma_max_threads{index="0",pid="63"} 5
puma_max_threads{index="0",pid="6446"} 5
puma_max_threads{index="0",pid="76"} 5
puma_max_threads{index="0",pid="89"} 5
puma_max_threads{index="1",pid="12860"} 5
puma_max_threads{index="1",pid="12913"} 5
puma_max_threads{index="1",pid="13033"} 5
puma_max_threads{index="1",pid="24"} 5
puma_max_threads{index="1",pid="26600"} 5
puma_max_threads{index="1",pid="26663"} 5
puma_max_threads{index="1",pid="26736"} 5
puma_max_threads{index="1",pid="26791"} 5
puma_max_threads{index="1",pid="26855"} 5
puma_max_threads{index="1",pid="37"} 5
puma_max_threads{index="1",pid="63"} 5
puma_max_threads{index="1",pid="6446"} 5
puma_max_threads{index="1",pid="76"} 5
puma_max_threads{index="1",pid="89"} 5
puma_max_threads{index="2",pid="12860"} 5
puma_max_threads{index="2",pid="12913"} 5
puma_max_threads{index="2",pid="13033"} 5
puma_max_threads{index="2",pid="24"} 5
puma_max_threads{index="2",pid="26600"} 5
puma_max_threads{index="2",pid="26663"} 5
puma_max_threads{index="2",pid="26736"} 5
puma_max_threads{index="2",pid="26791"} 5
puma_max_threads{index="2",pid="26855"} 5
puma_max_threads{index="2",pid="37"} 5
puma_max_threads{index="2",pid="63"} 5
puma_max_threads{index="2",pid="6446"} 5
puma_max_threads{index="2",pid="76"} 5
puma_max_threads{index="2",pid="89"} 5
puma_max_threads{index="3",pid="12860"} 5
puma_max_threads{index="3",pid="12913"} 5
puma_max_threads{index="3",pid="13033"} 5
puma_max_threads{index="3",pid="24"} 5
puma_max_threads{index="3",pid="26600"} 5
puma_max_threads{index="3",pid="26663"} 5
puma_max_threads{index="3",pid="26736"} 5
puma_max_threads{index="3",pid="26791"} 5
puma_max_threads{index="3",pid="26855"} 5
puma_max_threads{index="3",pid="37"} 5
puma_max_threads{index="3",pid="63"} 5
puma_max_threads{index="3",pid="6446"} 5
puma_max_threads{index="3",pid="76"} 5
puma_max_threads{index="3",pid="89"} 5
puma_max_threads{index="4",pid="12860"} 5
puma_max_threads{index="4",pid="12913"} 5
puma_max_threads{index="4",pid="13033"} 5
puma_max_threads{index="4",pid="24"} 5
puma_max_threads{index="4",pid="26600"} 5
puma_max_threads{index="4",pid="26663"} 5
puma_max_threads{index="4",pid="26736"} 5
puma_max_threads{index="4",pid="26791"} 5
puma_max_threads{index="4",pid="26855"} 5
puma_max_threads{index="4",pid="37"} 5
puma_max_threads{index="4",pid="63"} 5
puma_max_threads{index="4",pid="6446"} 5
puma_max_threads{index="4",pid="76"} 5
puma_max_threads{index="4",pid="89"} 5
puma_max_threads{index="5",pid="12860"} 5
puma_max_threads{index="5",pid="12913"} 5
puma_max_threads{index="5",pid="13033"} 5
puma_max_threads{index="5",pid="24"} 5
puma_max_threads{index="5",pid="26600"} 5
puma_max_threads{index="5",pid="26663"} 5
puma_max_threads{index="5",pid="26736"} 5
puma_max_threads{index="5",pid="26791"} 5
puma_max_threads{index="5",pid="26855"} 5
puma_max_threads{index="5",pid="37"} 5
puma_max_threads{index="5",pid="63"} 5
puma_max_threads{index="5",pid="6446"} 5
puma_max_threads{index="5",pid="76"} 5
puma_max_threads{index="5",pid="89"} 5
# HELP puma_old_workers Multiprocess metric
# TYPE puma_old_workers gauge
puma_old_workers{pid="12860"} 0
puma_old_workers{pid="12913"} 0
puma_old_workers{pid="13033"} 0
puma_old_workers{pid="24"} 0
puma_old_workers{pid="26600"} 0
puma_old_workers{pid="26663"} 0
puma_old_workers{pid="26736"} 0
puma_old_workers{pid="26791"} 0
puma_old_workers{pid="26855"} 0
puma_old_workers{pid="37"} 0
puma_old_workers{pid="63"} 0
puma_old_workers{pid="6446"} 0
puma_old_workers{pid="76"} 0
puma_old_workers{pid="89"} 0
# HELP puma_pool_capacity Multiprocess metric
# TYPE puma_pool_capacity gauge
puma_pool_capacity{index="0",pid="12860"} 1
puma_pool_capacity{index="0",pid="12913"} 2
puma_pool_capacity{index="0",pid="13033"} 1
puma_pool_capacity{index="0",pid="24"} 0
puma_pool_capacity{index="0",pid="26600"} 5
puma_pool_capacity{index="0",pid="26663"} 2
puma_pool_capacity{index="0",pid="26736"} 1
puma_pool_capacity{index="0",pid="26791"} 2
puma_pool_capacity{index="0",pid="26855"} 3
puma_pool_capacity{index="0",pid="37"} 0
puma_pool_capacity{index="0",pid="63"} 2
puma_pool_capacity{index="0",pid="6446"} 1
puma_pool_capacity{index="0",pid="76"} 0
puma_pool_capacity{index="0",pid="89"} 0
puma_pool_capacity{index="1",pid="12860"} 1
puma_pool_capacity{index="1",pid="12913"} 0
puma_pool_capacity{index="1",pid="13033"} 1
puma_pool_capacity{index="1",pid="24"} 0
puma_pool_capacity{index="1",pid="26600"} 5
puma_pool_capacity{index="1",pid="26663"} 1
puma_pool_capacity{index="1",pid="26736"} 2
puma_pool_capacity{index="1",pid="26791"} 3
puma_pool_capacity{index="1",pid="26855"} 4
puma_pool_capacity{index="1",pid="37"} 0
puma_pool_capacity{index="1",pid="63"} 1
puma_pool_capacity{index="1",pid="6446"} 2
puma_pool_capacity{index="1",pid="76"} 2
puma_pool_capacity{index="1",pid="89"} 3
puma_pool_capacity{index="2",pid="12860"} 2
puma_pool_capacity{index="2",pid="12913"} 1
puma_pool_capacity{index="2",pid="13033"} 0
puma_pool_capacity{index="2",pid="24"} 0
puma_pool_capacity{index="2",pid="26600"} 5
puma_pool_capacity{index="2",pid="26663"} 0
puma_pool_capacity{index="2",pid="26736"} 0
puma_pool_capacity{index="2",pid="26791"} 0
puma_pool_capacity{index="2",pid="26855"} 5
puma_pool_capacity{index="2",pid="37"} 0
puma_pool_capacity{index="2",pid="63"} 0
puma_pool_capacity{index="2",pid="6446"} 2
puma_pool_capacity{index="2",pid="76"} 1
puma_pool_capacity{index="2",pid="89"} 0
puma_pool_capacity{index="3",pid="12860"} 4
puma_pool_capacity{index="3",pid="12913"} 2
puma_pool_capacity{index="3",pid="13033"} 1
puma_pool_capacity{index="3",pid="24"} 0
puma_pool_capacity{index="3",pid="26600"} 4
puma_pool_capacity{index="3",pid="26663"} 2
puma_pool_capacity{index="3",pid="26736"} 1
puma_pool_capacity{index="3",pid="26791"} 1
puma_pool_capacity{index="3",pid="26855"} 5
puma_pool_capacity{index="3",pid="37"} 2
puma_pool_capacity{index="3",pid="63"} 0
puma_pool_capacity{index="3",pid="6446"} 1
puma_pool_capacity{index="3",pid="76"} 5
puma_pool_capacity{index="3",pid="89"} 1
puma_pool_capacity{index="4",pid="12860"} 3
puma_pool_capacity{index="4",pid="12913"} 3
puma_pool_capacity{index="4",pid="13033"} 3
puma_pool_capacity{index="4",pid="24"} 3
puma_pool_capacity{index="4",pid="26600"} 5
puma_pool_capacity{index="4",pid="26663"} 2
puma_pool_capacity{index="4",pid="26736"} 4
puma_pool_capacity{index="4",pid="26791"} 4
puma_pool_capacity{index="4",pid="26855"} 5
puma_pool_capacity{index="4",pid="37"} 2
puma_pool_capacity{index="4",pid="63"} 0
puma_pool_capacity{index="4",pid="6446"} 0
puma_pool_capacity{index="4",pid="76"} 2
puma_pool_capacity{index="4",pid="89"} 1
puma_pool_capacity{index="5",pid="12860"} 1
puma_pool_capacity{index="5",pid="12913"} 1
puma_pool_capacity{index="5",pid="13033"} 1
puma_pool_capacity{index="5",pid="24"} 1
puma_pool_capacity{index="5",pid="26600"} 5
puma_pool_capacity{index="5",pid="26663"} 0
puma_pool_capacity{index="5",pid="26736"} 0
puma_pool_capacity{index="5",pid="26791"} 5
puma_pool_capacity{index="5",pid="26855"} 5
puma_pool_capacity{index="5",pid="37"} 1
puma_pool_capacity{index="5",pid="63"} 2
puma_pool_capacity{index="5",pid="6446"} 0
puma_pool_capacity{index="5",pid="76"} 1
puma_pool_capacity{index="5",pid="89"} 0
# HELP puma_running Multiprocess metric
# TYPE puma_running gauge
puma_running{index="0",pid="12860"} 5
puma_running{index="0",pid="12913"} 5
puma_running{index="0",pid="13033"} 5
puma_running{index="0",pid="24"} 5
puma_running{index="0",pid="26600"} 5
puma_running{index="0",pid="26663"} 5
puma_running{index="0",pid="26736"} 5
puma_running{index="0",pid="26791"} 5
puma_running{index="0",pid="26855"} 5
puma_running{index="0",pid="37"} 5
puma_running{index="0",pid="63"} 5
puma_running{index="0",pid="6446"} 5
puma_running{index="0",pid="76"} 5
puma_running{index="0",pid="89"} 5
puma_running{index="1",pid="12860"} 5
puma_running{index="1",pid="12913"} 5
puma_running{index="1",pid="13033"} 5
puma_running{index="1",pid="24"} 5
puma_running{index="1",pid="26600"} 5
puma_running{index="1",pid="26663"} 5
puma_running{index="1",pid="26736"} 5
puma_running{index="1",pid="26791"} 5
puma_running{index="1",pid="26855"} 5
puma_running{index="1",pid="37"} 5
puma_running{index="1",pid="63"} 5
puma_running{index="1",pid="6446"} 5
puma_running{index="1",pid="76"} 5
puma_running{index="1",pid="89"} 2
puma_running{index="2",pid="6446"} 5
puma_running{index="2",pid="76"} 5
puma_running{index="2",pid="89"} 5
puma_running{index="2",pid="12860"} 5
puma_running{index="2",pid="12913"} 5
puma_running{index="2",pid="13033"} 5
puma_running{index="2",pid="24"} 5
puma_running{index="2",pid="26600"} 5
puma_running{index="2",pid="26663"} 5
puma_running{index="2",pid="26736"} 5
puma_running{index="2",pid="26791"} 5
puma_running{index="2",pid="26855"} 5
puma_running{index="2",pid="37"} 5
puma_running{index="2",pid="63"} 5
puma_running{index="3",pid="12860"} 5
puma_running{index="3",pid="12913"} 5
puma_running{index="3",pid="13033"} 5
puma_running{index="3",pid="24"} 5
puma_running{index="3",pid="26600"} 5
puma_running{index="3",pid="26663"} 5
puma_running{index="3",pid="26736"} 5
puma_running{index="3",pid="26791"} 5
puma_running{index="3",pid="26855"} 5
puma_running{index="3",pid="37"} 5
puma_running{index="3",pid="63"} 5
puma_running{index="3",pid="6446"} 5
puma_running{index="3",pid="76"} 5
puma_running{index="3",pid="89"} 5
puma_running{index="4",pid="12860"} 5
puma_running{index="4",pid="12913"} 2
puma_running{index="4",pid="13033"} 5
puma_running{index="4",pid="24"} 5
puma_running{index="4",pid="26600"} 5
puma_running{index="4",pid="26663"} 5
puma_running{index="4",pid="26736"} 5
puma_running{index="4",pid="26791"} 5
puma_running{index="4",pid="26855"} 5
puma_running{index="4",pid="37"} 5
puma_running{index="4",pid="63"} 5
puma_running{index="4",pid="6446"} 5
puma_running{index="4",pid="76"} 5
puma_running{index="4",pid="89"} 5
puma_running{index="5",pid="12860"} 5
puma_running{index="5",pid="12913"} 5
puma_running{index="5",pid="13033"} 5
puma_running{index="5",pid="24"} 5
puma_running{index="5",pid="26600"} 5
puma_running{index="5",pid="26663"} 5
puma_running{index="5",pid="26736"} 5
puma_running{index="5",pid="26791"} 5
puma_running{index="5",pid="26855"} 5
puma_running{index="5",pid="37"} 5
puma_running{index="5",pid="63"} 3
puma_running{index="5",pid="6446"} 5
puma_running{index="5",pid="76"} 5
puma_running{index="5",pid="89"} 5
# HELP puma_workers Multiprocess metric
# TYPE puma_workers gauge
puma_workers{pid="12860"} 6
puma_workers{pid="12913"} 6
puma_workers{pid="13033"} 6
puma_workers{pid="24"} 6
puma_workers{pid="26600"} 6
puma_workers{pid="26663"} 6
puma_workers{pid="26736"} 6
puma_workers{pid="26791"} 6
puma_workers{pid="26855"} 6
puma_workers{pid="37"} 6
puma_workers{pid="63"} 6
puma_workers{pid="6446"} 6
puma_workers{pid="76"} 6
puma_workers{pid="89"} 6

[Envek]

Good point! We should somehow unset metrics for not existing processes. Or, better, not to include pid label of worker process in metrics at all (it only creates redundant time-series IMO).

@dsalahutdinov, do you use pid label for any reason?

[Envek]

Ah, now I can see that pid label is being generated by prometheus-client with direct file store.

To fix it I will have to enhance Yabeda itself and Prometheus adapter for it to do metric aggregation.

Related issues: yabeda-rb/yabeda#8 and yabeda-rb/yabeda-prometheus#10

That will take some time

[Envek]

Please try yabeda-puma-plugin 0.6.0 with prometheus-client 2.1.0 or newer with direct file store set up. I believe that it should fix this.