core: metrics: Provide capabilities for returning the count of hot reloaded in REST API and its metrics

[cosmo0920]

For HTTP REST API users, there is lack of capability to confirm that the status of hot-reloading.
It would be nice to have the endpoint which can handle the request for returning the count of the hot-reloading times.

Also, we need to provide the metrics for the status of hot-reloading counts.

In this PR, I added two capabilities.
One is providing an endpoint which can handle GET method to return the count of hot-reloading.
This can be called with:

$ curl -XGET <hostname>:2020/api/v2/reload 

The other is provide a v2 metrics which includes the count of hot-reloading.
This can be called by:

$ curl -XGET <hostname>:2020/api/v2/metrics

Then, the metrics can be displayed as follows:

<snip>
2023-09-06T07:25:45.332832481Z fluentbit_hot_reloaded_times{hostname="fluent-bit-dev"} = <the number of reloaded times>
<snip>

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
    Hot_Reload On

[INPUT]
    Name dummy
    Tag dummy.locals

[FILTER]
    Name record_modifier
    Match *
    Record hostname ${HOSTNAME}

[OUTPUT]
    Name  stdout
    Match *

• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

A documentation for newly added endpoint: fluent/fluent-bit-docs#1192
A documentation for v2 endpoint and v2 metrics: fluent/fluent-bit-docs#1193

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[patrick-stephens]

Can we add a docs PR @cosmo0920 to cover this on both the reload page and the metrics description page?

Do we get it via Fluent Bit metrics as well, i.e. not just the webserver?

[cosmo0920]

Can we add a docs PR @cosmo0920 to cover this on both the reload page and the metrics description page?

Definitely. I'll add them in fluent-bit doc.