filter_record_modifier: support metadata

[nokute78]

This patch is to support append key/value to metadata.

Key	Description
Metadata	Append fields to metadata. This parameter needs key and value pair.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [N/A] Documentation required for this feature

Backporting

• [N/A] Backport to latest stable release.

Configuration

[INPUT]
    Name dummy

[FILTER]
    Name record_modifier
    Match *
    Metadata meta_key meta_value
    Record rec_key rec_val

[OUTPUT]
    Name           stdout
    Match          *

Output will be:

[0] dummy.0: [[1710034120.954324214, {"meta_key"=>"meta_value"}], {"message"=>"dummy", "rec_key"=>"rec_val"}]

Valgrind output

d$ valgrind --leak-check=full bin/fluent-bit -c b.conf 
==45095== Memcheck, a memory error detector
==45095== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==45095== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==45095== Command: bin/fluent-bit -c b.conf
==45095== 
Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/03/10 10:28:36] [ info] [fluent bit] version=3.0.0, commit=9d9ac68a2b, pid=45095
[2024/03/10 10:28:37] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/10 10:28:37] [ info] [cmetrics] version=0.7.0
[2024/03/10 10:28:37] [ info] [ctraces ] version=0.4.0
[2024/03/10 10:28:37] [ info] [input:dummy:dummy.0] initializing
[2024/03/10 10:28:37] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/03/10 10:28:37] [ info] [output:stdout:stdout.0] worker #0 started
[2024/03/10 10:28:37] [ info] [sp] stream processor started
[0] dummy.0: [[1710034117.979425972, {"meta_key"=>"meta_value"}], {"message"=>"dummy", "rec_key"=>"rec_val"}]
[0] dummy.0: [[1710034118.980085361, {"meta_key"=>"meta_value"}], {"message"=>"dummy", "rec_key"=>"rec_val"}]
[0] dummy.0: [[1710034119.955219899, {"meta_key"=>"meta_value"}], {"message"=>"dummy", "rec_key"=>"rec_val"}]
^C[2024/03/10 10:28:41] [engine] caught signal (SIGINT)
[2024/03/10 10:28:41] [ warn] [engine] service will shutdown in max 5 seconds
[0] dummy.0: [[1710034120.954324214, {"meta_key"=>"meta_value"}], {"message"=>"dummy", "rec_key"=>"rec_val"}]
[2024/03/10 10:28:41] [ info] [input] pausing dummy.0
[2024/03/10 10:28:41] [ info] [engine] service has stopped (0 pending tasks)
[2024/03/10 10:28:41] [ info] [input] pausing dummy.0
[2024/03/10 10:28:41] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/03/10 10:28:42] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==45095== 
==45095== HEAP SUMMARY:
==45095==     in use at exit: 0 bytes in 0 blocks
==45095==   total heap usage: 1,906 allocs, 1,906 frees, 1,999,689 bytes allocated
==45095== 
==45095== All heap blocks were freed -- no leaks are possible
==45095== 
==45095== For lists of detected and suppressed errors, rerun with: -s
==45095== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[nokute78]

This patch is for metadata, but this plugin is filter_"record"_modifier.
Is it better to create a new plugin like filter_metadata_modifier ?

[edsiper]

@nokute78 we just merged the new content_modifier processor, I think they accomplish similar functionality, e.g:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "key2": "true", "key3": "delete me", "message": null, "http.url": "https://www.google.com/search?q=example"}'
      rate: 1

      processors:
        logs:
          - name: content_modifier
            action: insert
            context: metadata
            key: "new_key"
            value: "new_value"

[nokute78]

@edsiper Thank you for information. That's I want.