in_node_exporter_metrics: implement processes metrics #7880

cosmo0920 · 2023-08-30T09:15:59Z

To monitor for statuses of processes and threads, we need to implement processes metrics on in_node_exporter_metrics plugin.

Fixes #7866

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change

$ bin/fluent-bit -i node_exporter_metrics -p metrics=processes -o stdout

Debug log output from testing the change

Fluent Bit v2.1.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/08/31 11:02:00] [ info] Configuration:
[2023/08/31 11:02:00] [ info]  flush time     | 1.000000 seconds
[2023/08/31 11:02:00] [ info]  grace          | 5 seconds
[2023/08/31 11:02:00] [ info]  daemon         | 0
[2023/08/31 11:02:00] [ info] ___________
[2023/08/31 11:02:00] [ info]  inputs:
[2023/08/31 11:02:00] [ info]      node_exporter_metrics
[2023/08/31 11:02:00] [ info] ___________
[2023/08/31 11:02:00] [ info]  filters:
[2023/08/31 11:02:00] [ info] ___________
[2023/08/31 11:02:00] [ info]  outputs:
[2023/08/31 11:02:00] [ info]      stdout.0
[2023/08/31 11:02:00] [ info] ___________
[2023/08/31 11:02:00] [ info]  collectors:
[2023/08/31 11:02:00] [ info] [fluent bit] version=2.1.9, commit=eb36c698a2, pid=606478
[2023/08/31 11:02:00] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/08/31 11:02:00] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/08/31 11:02:00] [ info] [cmetrics] version=0.6.3
[2023/08/31 11:02:01] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.procfs = /proc
[2023/08/31 11:02:00] [ info] [ctraces ] version=0.3.1
[2023/08/31 11:02:01] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.sysfs  = /sys
[2023/08/31 11:02:01] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] initializing
[2023/08/31 11:02:01] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] enabled metrics processes
[2023/08/31 11:02:01] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] storage_strategy='memory' (memory only)
[2023/08/31 11:02:01] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] [thread init] initialization OK
[2023/08/31 11:02:01] [ info] [output:stdout:stdout.0] worker #0 started
[2023/08/31 11:02:01] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] thread instance initialized
[2023/08/31 11:02:01] [debug] [node_exporter_metrics:node_exporter_metrics.0] created event channels: read=30 write=31
[2023/08/31 11:02:01] [debug] [stdout:stdout.0] created event channels: read=34 write=35
[2023/08/31 11:02:01] [ info] [sp] stream processor started
[2023/08/31 11:02:06] [debug] [input chunk] update output instances with new chunk size diff=1538, records=0, input=node_exporter_metrics.0
[2023/08/31 11:02:06] [debug] [task] created task=0x8447660 id=0 OK
2023-08-31T02:02:05.589311697Z node_processes_threads = 1684
2023-08-31T02:02:05.589311697Z node_processes_max_threads = 513122
2023-08-31T02:02:05.589311697Z node_processes_threads_state{thread_state="R"} = 1
2023-08-31T02:02:05.589311697Z node_processes_threads_state{thread_state="S"} = 1573
2023-08-31T02:02:05.589311697Z node_processes_threads_state{thread_state="D"} = 0
2023-08-31T02:02:05.589311697Z node_processes_threads_state{thread_state="Z"} = 3
2023-08-31T02:02:05.589311697Z node_processes_threads_state{thread_state="T"} = 0
2023-08-31T02:02:05.589311697Z node_processes_threads_state{thread_state="I"} = 107
2023-08-31T02:02:05.589311697Z node_processes_state{state="R"} = 0
2023-08-31T02:02:05.589311697Z node_processes_state{state="S"} = 369
2023-08-31T02:02:05.589311697Z node_processes_state{state="D"} = 0
2023-08-31T02:02:05.589311697Z node_processes_state{state="Z"} = 3
2023-08-31T02:02:05.589311697Z node_processes_state{state="T"} = 0
2023-08-31T02:02:05.589311697Z node_processes_state{state="I"} = 107
2023-08-31T02:02:05.589311697Z node_processes_pids = 479
2023-08-31T02:02:05.589311697Z node_processes_max_processeses = 4194304
[2023/08/31 11:02:06] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/08/31 11:02:06] [debug] [out flush] cb_destroy coro_id=0
[2023/08/31 11:02:06] [debug] [task] destroy task=0x8447660 (task_id=0)
^C[2023/08/31 11:02:07] [engine] caught signal (SIGINT)
[2023/08/31 11:02:07] [ warn] [engine] service will shutdown in max 5 seconds
[2023/08/31 11:02:07] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] thread pause instance
[2023/08/31 11:02:07] [ info] [engine] service has stopped (0 pending tasks)
[2023/08/31 11:02:07] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] thread pause instance
[2023/08/31 11:02:07] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/08/31 11:02:07] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] thread exit instance
[2023/08/31 11:02:07] [ info] [output:stdout:stdout.0] thread worker #0 stopped

Attached Valgrind output that shows no leaks or memory corruption was found

==606478== 
==606478== HEAP SUMMARY:
==606478==     in use at exit: 0 bytes in 0 blocks
==606478==   total heap usage: 188,120 allocs, 188,120 frees, 23,613,784 bytes allocated
==606478== 
==606478== All heap blocks were freed -- no leaks are possible
==606478== 
==606478== For lists of detected and suppressed errors, rerun with: -s
==606478== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

fluent/fluent-bit-docs#1184

Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

cosmo0920 · 2023-08-30T09:17:37Z

Note: This PR might be able to cause conflict for merging #7876.
So, I'm still mark as a draft for now.

Signed-off-by: Hiroshi Hatake <[email protected]>

edsiper · 2023-09-01T14:27:57Z

@cosmo0920 has this been tested in CentOS 7 ?

cosmo0920 · 2023-09-02T04:40:26Z

This is not tested yet. But it is not enabled by default.

cosmo0920 · 2023-09-04T03:51:44Z

It works on Docker M1 macOS!

[root@9a4610912d2f /]# uname -a
Linux 9a4610912d2f 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

[root@9a4610912d2f /]# /opt/fluent-bit/bin/fluent-bit -i node_exporter_metrics -p metrics=processes -o stdout -v
Fluent Bit v2.1.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/04 03:50:38] [ info] Configuration:
[2023/09/04 03:50:38] [ info]  flush time     | 1.000000 seconds
[2023/09/04 03:50:38] [ info]  grace          | 5 seconds
[2023/09/04 03:50:38] [ info]  daemon         | 0
[2023/09/04 03:50:38] [ info] ___________
[2023/09/04 03:50:38] [ info]  inputs:
[2023/09/04 03:50:38] [ info]      node_exporter_metrics
[2023/09/04 03:50:38] [ info] ___________
[2023/09/04 03:50:38] [ info]  filters:
[2023/09/04 03:50:38] [ info] ___________
[2023/09/04 03:50:38] [ info]  outputs:
[2023/09/04 03:50:38] [ info]      stdout.0
[2023/09/04 03:50:38] [ info] ___________
[2023/09/04 03:50:38] [ info]  collectors:
[2023/09/04 03:50:38] [ info] [fluent bit] version=2.1.9, commit=, pid=62
[2023/09/04 03:50:38] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2023/09/04 03:50:38] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/04 03:50:38] [ info] [cmetrics] version=0.6.3
[2023/09/04 03:50:38] [ info] [ctraces ] version=0.3.1
[2023/09/04 03:50:38] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] initializing
[2023/09/04 03:50:38] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] storage_strategy='memory' (memory only)
[2023/09/04 03:50:38] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.procfs = /proc
[2023/09/04 03:50:38] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.sysfs  = /sys
[2023/09/04 03:50:38] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] enabled metrics processes
[2023/09/04 03:50:38] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] [thread init] initialization OK
[2023/09/04 03:50:38] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] thread instance initialized
[2023/09/04 03:50:38] [debug] [node_exporter_metrics:node_exporter_metrics.0] created event channels: read=30 write=31
[2023/09/04 03:50:38] [debug] [stdout:stdout.0] created event channels: read=34 write=35
[2023/09/04 03:50:38] [ info] [sp] stream processor started
[2023/09/04 03:50:38] [ info] [output:stdout:stdout.0] worker #0 started
[2023/09/04 03:50:42] [debug] [input chunk] update output instances with new chunk size diff=1538, records=0, input=node_exporter_metrics.0
[2023/09/04 03:50:43] [debug] [task] created task=0xffffad23c500 id=0 OK
[2023/09/04 03:50:43] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
2023-09-04T03:50:42.336229550Z node_processes_threads = 6
2023-09-04T03:50:42.336229550Z node_processes_max_threads = 30516
2023-09-04T03:50:42.336229550Z node_processes_threads_state{thread_state="R"} = 1
2023-09-04T03:50:42.336229550Z node_processes_threads_state{thread_state="S"} = 5
2023-09-04T03:50:42.336229550Z node_processes_threads_state{thread_state="D"} = 0
2023-09-04T03:50:42.336229550Z node_processes_threads_state{thread_state="Z"} = 0
2023-09-04T03:50:42.336229550Z node_processes_threads_state{thread_state="T"} = 0
2023-09-04T03:50:42.336229550Z node_processes_threads_state{thread_state="I"} = 0
2023-09-04T03:50:42.336229550Z node_processes_state{state="R"} = 0
2023-09-04T03:50:42.336229550Z node_processes_state{state="S"} = 2
2023-09-04T03:50:42.336229550Z node_processes_state{state="D"} = 0
2023-09-04T03:50:42.336229550Z node_processes_state{state="Z"} = 0
2023-09-04T03:50:42.336229550Z node_processes_state{state="T"} = 0
2023-09-04T03:50:42.336229550Z node_processes_state{state="I"} = 0
2023-09-04T03:50:42.336229550Z node_processes_pids = 2
2023-09-04T03:50:42.336229550Z node_processes_max_processeses = 99999
[2023/09/04 03:50:43] [debug] [out flush] cb_destroy coro_id=0
[2023/09/04 03:50:43] [debug] [task] destroy task=0xffffad23c500 (task_id=0)
^C[2023/09/04 03:50:44] [engine] caught signal (SIGINT)
[2023/09/04 03:50:44] [ warn] [engine] service will shutdown in max 5 seconds
[2023/09/04 03:50:44] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] thread pause instance
[2023/09/04 03:50:44] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/04 03:50:44] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/09/04 03:50:44] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] thread pause instance
[2023/09/04 03:50:44] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2023/09/04 03:50:44] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] thread exit instance

edsiper · 2023-09-04T22:11:34Z

plugins/in_node_exporter_metrics/ne.c

@@ -701,6 +720,26 @@ static int in_ne_init(struct flb_input_instance *in,
                    }
                    ne_systemd_init(ctx);
                }
+                else if (strncmp(entry->str, "processes", 9) == 0) {
+                    if (ctx->processes_scrape_interval == 0) {
+                        flb_plg_debug(ctx->ins, "enabled metrics %s", entry->str);


it seems process collector is disabled but the message says enabled

This message is used when enabling processes metrics. Other messages for enabling metrics are also displayed in enabled circumstances. I mean, when "processes" string is included in metrics parameter, this message will be displayed.

cosmo0920 · 2023-09-06T10:54:19Z

This PR works well on CentOS 6 (x86_64 container):

# cat /etc/redhat-release 
CentOS release 6.10 (Final)
[root@d24f26fac50a build_centos6]# bin/fluent-bit -i node_exporter_metrics -pmetrics=processes -o stdout
Fluent Bit v2.1.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/06 10:52:36] [ info] [fluent bit] version=2.1.9, commit=, pid=1727
[2023/09/06 10:52:36] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/06 10:52:36] [ info] [cmetrics] version=0.6.3
[2023/09/06 10:52:36] [ info] [ctraces ] version=0.3.1
[2023/09/06 10:52:36] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] initializing
[2023/09/06 10:52:36] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] storage_strategy='memory' (memory only)
[2023/09/06 10:52:36] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.procfs = /proc
[2023/09/06 10:52:36] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.sysfs  = /sys
[2023/09/06 10:52:36] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] thread instance initialized
[2023/09/06 10:52:36] [ info] [sp] stream processor started
[2023/09/06 10:52:36] [ info] [output:stdout:stdout.0] worker #0 started
2023-09-06T10:52:40.332391348Z node_processes_threads = 6
2023-09-06T10:52:40.332391348Z node_processes_max_threads = 513122
2023-09-06T10:52:40.332391348Z node_processes_threads_state{thread_state="R"} = 1
2023-09-06T10:52:40.332391348Z node_processes_threads_state{thread_state="S"} = 5
2023-09-06T10:52:40.332391348Z node_processes_threads_state{thread_state="D"} = 0
2023-09-06T10:52:40.332391348Z node_processes_threads_state{thread_state="Z"} = 0
2023-09-06T10:52:40.332391348Z node_processes_threads_state{thread_state="T"} = 0
2023-09-06T10:52:40.332391348Z node_processes_threads_state{thread_state="I"} = 0
2023-09-06T10:52:40.332391348Z node_processes_state{state="R"} = 0
2023-09-06T10:52:40.332391348Z node_processes_state{state="S"} = 2
2023-09-06T10:52:40.332391348Z node_processes_state{state="D"} = 0
2023-09-06T10:52:40.332391348Z node_processes_state{state="Z"} = 0
2023-09-06T10:52:40.332391348Z node_processes_state{state="T"} = 0
2023-09-06T10:52:40.332391348Z node_processes_state{state="I"} = 0
2023-09-06T10:52:40.332391348Z node_processes_pids = 2
2023-09-06T10:52:40.332391348Z node_processes_max_processeses = 4194304
^C[2023/09/06 10:52:42] [engine] caught signal (SIGINT)
[2023/09/06 10:52:42] [ warn] [engine] service will shutdown in max 5 seconds
[2023/09/06 10:52:43] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/06 10:52:43] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/09/06 10:52:43] [ info] [output:stdout:stdout.0] thread worker #0 stopped

And CentOS 6 on vagrant box:

[vagrant@localhost centos6_rpms]$ cat /etc/redhat-release 
CentOS release 6.10 (Final)
[vagrant@localhost centos6_rpms]$ /path/to/bin/fluent-bit -i node_exporter_metrics -pmetrics=processes -o stdout
<snip>
[2023/09/07 04:23:53] [ info] [output:stdout:stdout.0] worker #0 started
2023-09-07T04:23:58.560399017Z node_processes_threads = 125
2023-09-07T04:23:58.560399017Z node_processes_max_threads = 14821
2023-09-07T04:23:58.560399017Z node_processes_threads_state{thread_state="R"} = 1
2023-09-07T04:23:58.560399017Z node_processes_threads_state{thread_state="S"} = 124
2023-09-07T04:23:58.560399017Z node_processes_threads_state{thread_state="D"} = 0
2023-09-07T04:23:58.560399017Z node_processes_threads_state{thread_state="Z"} = 0
2023-09-07T04:23:58.560399017Z node_processes_threads_state{thread_state="T"} = 0
2023-09-07T04:23:58.560399017Z node_processes_threads_state{thread_state="I"} = 0
2023-09-07T04:23:58.560399017Z node_processes_state{state="R"} = 0
2023-09-07T04:23:58.560399017Z node_processes_state{state="S"} = 115
2023-09-07T04:23:58.560399017Z node_processes_state{state="D"} = 0
2023-09-07T04:23:58.560399017Z node_processes_state{state="Z"} = 0
2023-09-07T04:23:58.560399017Z node_processes_state{state="T"} = 0
2023-09-07T04:23:58.560399017Z node_processes_state{state="I"} = 0
2023-09-07T04:23:58.560399017Z node_processes_pids = 115
2023-09-07T04:23:58.560399017Z node_processes_max_processeses = 32768
^C[2023/09/07 04:24:01] [engine] caught signal (SIGINT)
<snip>

github-actions bot added the docs-required label Aug 30, 2023

cosmo0920 temporarily deployed to pr August 30, 2023 09:16 — with GitHub Actions Inactive

cosmo0920 mentioned this pull request Aug 30, 2023

Process metrics for Linux #7870

Closed

cosmo0920 temporarily deployed to pr August 30, 2023 09:44 — with GitHub Actions Inactive

cosmo0920 force-pushed the cosmo0920-implement-process-metrics-on-node_exporter_metrics branch from eb36c69 to b289694 Compare August 31, 2023 02:01

cosmo0920 changed the title ~~in_node_exporter_metrics: implement process metrics~~ in_node_exporter_metrics: implement processes metrics Aug 31, 2023

cosmo0920 temporarily deployed to pr August 31, 2023 02:02 — with GitHub Actions Inactive

cosmo0920 temporarily deployed to pr August 31, 2023 02:27 — with GitHub Actions Inactive

cosmo0920 force-pushed the cosmo0920-implement-process-metrics-on-node_exporter_metrics branch from b289694 to 2f1f238 Compare August 31, 2023 10:12

cosmo0920 temporarily deployed to pr August 31, 2023 10:13 — with GitHub Actions Inactive

cosmo0920 temporarily deployed to pr August 31, 2023 10:44 — with GitHub Actions Inactive

cosmo0920 added 2 commits September 1, 2023 10:54

in_node_exporter_metrics: Implement processes metrics

9c77570

Signed-off-by: Hiroshi Hatake <[email protected]>

in_node_exporter_metrics: Check procfs exsitence before retrieving info

dd85693

Signed-off-by: Hiroshi Hatake <[email protected]>

cosmo0920 force-pushed the cosmo0920-implement-process-metrics-on-node_exporter_metrics branch from 2f1f238 to dd85693 Compare September 1, 2023 02:06

cosmo0920 marked this pull request as ready for review September 1, 2023 02:07

cosmo0920 requested review from edsiper, leonardo-albertovich, fujimotos and koleini as code owners September 1, 2023 02:07

cosmo0920 temporarily deployed to pr September 1, 2023 02:07 — with GitHub Actions Inactive

cosmo0920 temporarily deployed to pr September 1, 2023 02:34 — with GitHub Actions Inactive

edsiper requested changes Sep 4, 2023

View reviewed changes

edsiper merged commit cfa197c into master Oct 16, 2023

edsiper deleted the cosmo0920-implement-process-metrics-on-node_exporter_metrics branch October 16, 2023 19:10

cosmo0920 mentioned this pull request Oct 17, 2023

node_exporter_metrics: Fix macOS build #8048

Merged

7 tasks

leonardo-albertovich pushed a commit that referenced this pull request Nov 3, 2023

in_node_exporter_metrics: implement processes metrics (#7880)

ba2733d

BrewTestBot mentioned this pull request Nov 9, 2023

fluent-bit 2.2.0 Homebrew/homebrew-core#153758

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

in_node_exporter_metrics: implement processes metrics #7880

in_node_exporter_metrics: implement processes metrics #7880

cosmo0920 commented Aug 30, 2023 •

edited

Loading

cosmo0920 commented Aug 30, 2023

edsiper commented Sep 1, 2023

cosmo0920 commented Sep 2, 2023

cosmo0920 commented Sep 4, 2023 •

edited

Loading

edsiper Sep 4, 2023

cosmo0920 Sep 5, 2023 •

edited

Loading

cosmo0920 commented Sep 6, 2023 •

edited

Loading

in_node_exporter_metrics: implement processes metrics #7880

in_node_exporter_metrics: implement processes metrics #7880

Conversation

cosmo0920 commented Aug 30, 2023 • edited Loading

cosmo0920 commented Aug 30, 2023

edsiper commented Sep 1, 2023

cosmo0920 commented Sep 2, 2023

cosmo0920 commented Sep 4, 2023 • edited Loading

edsiper Sep 4, 2023

Choose a reason for hiding this comment

cosmo0920 Sep 5, 2023 • edited Loading

Choose a reason for hiding this comment

cosmo0920 commented Sep 6, 2023 • edited Loading

cosmo0920 commented Aug 30, 2023 •

edited

Loading

cosmo0920 commented Sep 4, 2023 •

edited

Loading

cosmo0920 Sep 5, 2023 •

edited

Loading

cosmo0920 commented Sep 6, 2023 •

edited

Loading