out_http: default the output format to JSON. #8493

pwhelan · 2024-02-16T22:15:15Z

Summary

Default out_http to JSON format so it works with in_http. Setting the output format to msgpack is still possible by setting it manually.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

Documentation required for this feature

Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Phillip Whelan <[email protected]>

pwhelan · 2024-02-16T22:15:42Z

Here is a valgrind log:

$ valgrind ./bin/fluent-bit -i dummy -o http -p host=localhost -p port=8889 -f 1
==997298== Memcheck, a memory error detector
==997298== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==997298== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==997298== Command: ./bin/fluent-bit -i dummy -o http -p host=localhost -p port=8889 -f 1
==997298==
Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

____________________
< Fluent Bit v2.2.2 >
 -------------------
          \
           \
            \          __---__
                    _-       /--______
               __--( /     \ )XXXXXXXXXXX\v.
             .-XXX(   O   O  )XXXXXXXXXXXXXXX-
            /XXX(       U     )        XXXXXXX\
          /XXXXX(              )--_  XXXXXXXXXXX\
         /XXXXX/ (      O     )   XXXXXX   \XXXXX\
         XXXXX/   /            XXXXXX   \__ \XXXXX
         XXXXXX__/          XXXXXX         \__---->
 ---___  XXX__/          XXXXXX      \__         /
   \-  --__/   ___/\  XXXXXX            /  ___--/=
    \-\    ___/    XXXXXX              '--- XXXXXX
       \-\/XXX\ XXXXXX                      /XXXXX
         \XXXXXXXXX   \                    /XXXXX/
          \XXXXXX      >                 _/XXXXX/
            \XXXXX--__/              __-- XXXX/
             -XXXXXXXX---------------  XXXXXX-
                \XXXXXXXXXXXXXXXXXXXXXXXXXX/
                  ""VXXXXXXXXXXXXXXXXXXV""

[2024/02/16 19:12:23] [ info] [fluent bit] version=3.0.0, commit=bf767458b2, pid=997298
[2024/02/16 19:12:23] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/02/16 19:12:23] [ info] [output:http:http.0] worker #1 started
[2024/02/16 19:12:23] [ info] [output:http:http.0] worker #0 started
[2024/02/16 19:12:23] [ info] [cmetrics] version=0.6.6
[2024/02/16 19:12:23] [ info] [ctraces ] version=0.4.0
[2024/02/16 19:12:23] [ info] [input:dummy:dummy.0] initializing
[2024/02/16 19:12:23] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/02/16 19:12:23] [ info] [sp] stream processor started
[2024/02/16 19:12:25] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:26] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:27] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:28] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:29] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:30] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:31] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:32] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:33] [ info] [output:http:http.0] localhost:8889, HTTP status=201
^C[2024/02/16 19:12:33] [engine] caught signal (SIGINT)
[2024/02/16 19:12:33] [ warn] [engine] service will shutdown in max 5 seconds
[2024/02/16 19:12:33] [ info] [input] pausing dummy.0
[2024/02/16 19:12:33] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:34] [ info] [engine] service has stopped (0 pending tasks)
[2024/02/16 19:12:34] [ info] [input] pausing dummy.0
[2024/02/16 19:12:34] [ info] [output:http:http.0] thread worker #0 stopping...
[2024/02/16 19:12:34] [ info] [output:http:http.0] thread worker #0 stopped
[2024/02/16 19:12:34] [ info] [output:http:http.0] thread worker #1 stopping...
[2024/02/16 19:12:34] [ info] [output:http:http.0] thread worker #1 stopped
==997298==
==997298== HEAP SUMMARY:
==997298==     in use at exit: 0 bytes in 0 blocks
==997298==   total heap usage: 3,068 allocs, 3,068 frees, 3,986,697 bytes allocated
==997298==
==997298== All heap blocks were freed -- no leaks are possible
==997298==
==997298== For lists of detected and suppressed errors, rerun with: -s
==997298== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

pwhelan · 2024-02-16T22:16:29Z

Here is a debug log:

$ ./bin/fluent-bit -v -i dummy -o http -p host=localhost -p port=8889 -f 1
Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

____________________
< Fluent Bit v2.2.2 >
 -------------------
          \
           \
            \          __---__
                    _-       /--______
               __--( /     \ )XXXXXXXXXXX\v.
             .-XXX(   O   O  )XXXXXXXXXXXXXXX-
            /XXX(       U     )        XXXXXXX\
          /XXXXX(              )--_  XXXXXXXXXXX\
         /XXXXX/ (      O     )   XXXXXX   \XXXXX\
         XXXXX/   /            XXXXXX   \__ \XXXXX
         XXXXXX__/          XXXXXX         \__---->
 ---___  XXX__/          XXXXXX      \__         /
   \-  --__/   ___/\  XXXXXX            /  ___--/=
    \-\    ___/    XXXXXX              '--- XXXXXX
       \-\/XXX\ XXXXXX                      /XXXXX
         \XXXXXXXXX   \                    /XXXXX/
          \XXXXXX      >                 _/XXXXX/
            \XXXXX--__/              __-- XXXX/
             -XXXXXXXX---------------  XXXXXX-
                \XXXXXXXXXXXXXXXXXXXXXXXXXX/
                  ""VXXXXXXXXXXXXXXXXXXV""

[2024/02/16 19:16:02] [ info] Configuration:
[2024/02/16 19:16:02] [ info]  flush time     | 1.000000 seconds
[2024/02/16 19:16:02] [ info]  grace          | 5 seconds
[2024/02/16 19:16:02] [ info]  daemon         | 0
[2024/02/16 19:16:02] [ info] ___________
[2024/02/16 19:16:02] [ info]  inputs:
[2024/02/16 19:16:02] [ info]      dummy
[2024/02/16 19:16:02] [ info] ___________
[2024/02/16 19:16:02] [ info]  filters:
[2024/02/16 19:16:02] [ info] ___________
[2024/02/16 19:16:02] [ info]  outputs:
[2024/02/16 19:16:02] [ info]      http.0
[2024/02/16 19:16:02] [ info] ___________
[2024/02/16 19:16:02] [ info]  collectors:
[2024/02/16 19:16:02] [ info] [fluent bit] version=3.0.0, commit=bf767458b2, pid=999168
[2024/02/16 19:16:02] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/02/16 19:16:02] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/02/16 19:16:02] [ info] [cmetrics] version=0.6.6
[2024/02/16 19:16:02] [ info] [ctraces ] version=0.4.0
[2024/02/16 19:16:02] [ info] [input:dummy:dummy.0] initializing
[2024/02/16 19:16:02] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/02/16 19:16:02] [debug] [dummy:dummy.0] created event channels: read=21 write=22
[2024/02/16 19:16:02] [debug] [http:http.0] created event channels: read=23 write=24
[2024/02/16 19:16:02] [ info] [output:http:http.0] worker #0 started
[2024/02/16 19:16:02] [ info] [sp] stream processor started
[2024/02/16 19:16:02] [ info] [output:http:http.0] worker #1 started
[2024/02/16 19:16:03] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.0
[2024/02/16 19:16:04] [debug] [task] created task=0x7ffff00222f0 id=0 OK
[2024/02/16 19:16:04] [debug] [output:http:http.0] task_id=0 assigned to thread #0
[2024/02/16 19:16:04] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.0
[2024/02/16 19:16:04] [debug] [upstream] KA connection #49 to localhost:8889 is connected
[2024/02/16 19:16:04] [debug] [http_client] not using http_proxy for header
[2024/02/16 19:16:04] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:16:04] [debug] [upstream] KA connection #49 to localhost:8889 is now available
[2024/02/16 19:16:04] [debug] [out flush] cb_destroy coro_id=0
[2024/02/16 19:16:04] [debug] [task] destroy task=0x7ffff00222f0 (task_id=0)
[2024/02/16 19:16:05] [debug] [task] created task=0x7ffff0022640 id=0 OK
[2024/02/16 19:16:05] [debug] [output:http:http.0] task_id=0 assigned to thread #1
[2024/02/16 19:16:05] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.0
[2024/02/16 19:16:05] [debug] [upstream] KA connection #50 to localhost:8889 is connected
[2024/02/16 19:16:05] [debug] [http_client] not using http_proxy for header
[2024/02/16 19:16:05] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:16:05] [debug] [upstream] KA connection #50 to localhost:8889 is now available
[2024/02/16 19:16:05] [debug] [out flush] cb_destroy coro_id=0
[2024/02/16 19:16:05] [debug] [task] destroy task=0x7ffff0022640 (task_id=0)
[2024/02/16 19:16:06] [debug] [task] created task=0x7ffff00228b0 id=0 OK
[2024/02/16 19:16:06] [debug] [output:http:http.0] task_id=0 assigned to thread #0
[2024/02/16 19:16:06] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.0
[2024/02/16 19:16:06] [debug] [upstream] KA connection #49 to localhost:8889 has been assigned (recycled)
[2024/02/16 19:16:06] [debug] [http_client] not using http_proxy for header
[2024/02/16 19:16:06] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:16:06] [debug] [upstream] KA connection #49 to localhost:8889 is now available
[2024/02/16 19:16:06] [debug] [out flush] cb_destroy coro_id=1
[2024/02/16 19:16:06] [debug] [task] destroy task=0x7ffff00228b0 (task_id=0)
[2024/02/16 19:16:07] [debug] [task] created task=0x7ffff0022aa0 id=0 OK
[2024/02/16 19:16:07] [debug] [output:http:http.0] task_id=0 assigned to thread #1
[2024/02/16 19:16:07] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.0
[2024/02/16 19:16:07] [debug] [upstream] KA connection #50 to localhost:8889 has been assigned (recycled)
[2024/02/16 19:16:07] [debug] [http_client] not using http_proxy for header
[2024/02/16 19:16:07] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:16:07] [debug] [upstream] KA connection #50 to localhost:8889 is now available
[2024/02/16 19:16:07] [debug] [out flush] cb_destroy coro_id=1
[2024/02/16 19:16:07] [debug] [task] destroy task=0x7ffff0022aa0 (task_id=0)
^C[2024/02/16 19:16:07] [engine] caught signal (SIGINT)
[2024/02/16 19:16:07] [debug] [task] created task=0x7ffff0022c50 id=0 OK
[2024/02/16 19:16:07] [debug] [output:http:http.0] task_id=0 assigned to thread #0
[2024/02/16 19:16:07] [ warn] [engine] service will shutdown in max 5 seconds
[2024/02/16 19:16:07] [ info] [input] pausing dummy.0
[2024/02/16 19:16:07] [debug] [upstream] KA connection #49 to localhost:8889 has been assigned (recycled)
[2024/02/16 19:16:07] [debug] [http_client] not using http_proxy for header
[2024/02/16 19:16:07] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:16:07] [debug] [upstream] KA connection #49 to localhost:8889 is now available
[2024/02/16 19:16:07] [debug] [out flush] cb_destroy coro_id=2
[2024/02/16 19:16:07] [debug] [task] destroy task=0x7ffff0022c50 (task_id=0)
[2024/02/16 19:16:08] [ info] [engine] service has stopped (0 pending tasks)
[2024/02/16 19:16:08] [ info] [input] pausing dummy.0
[2024/02/16 19:16:08] [ info] [output:http:http.0] thread worker #0 stopping...
[2024/02/16 19:16:08] [ info] [output:http:http.0] thread worker #0 stopped
[2024/02/16 19:16:08] [ info] [output:http:http.0] thread worker #1 stopping...
[2024/02/16 19:16:08] [ info] [output:http:http.0] thread worker #1 stopped

Signed-off-by: Phillip Whelan <[email protected]>

edsiper · 2024-03-12T15:24:37Z

@pwhelan thanks. would you mind to open a PR with docs mentioning this breaking change in the new config ?

out_http: default the output format to JSON.

81f63ce

Signed-off-by: Phillip Whelan <[email protected]>

pwhelan requested review from edsiper, leonardo-albertovich, fujimotos and koleini as code owners February 16, 2024 22:15

github-actions bot added the docs-required label Feb 16, 2024

pwhelan temporarily deployed to pr February 16, 2024 22:15 — with GitHub Actions Inactive

pwhelan temporarily deployed to pr February 16, 2024 22:36 — with GitHub Actions Inactive

pwhelan mentioned this pull request Feb 19, 2024

in_http: parse msgpack payloads emitted by out_http. #8499

Open

7 tasks

out_http: add test to make sure default settings work with in_http.

d03df8c

Signed-off-by: Phillip Whelan <[email protected]>

pwhelan temporarily deployed to pr February 22, 2024 20:08 — with GitHub Actions Inactive

pwhelan temporarily deployed to pr February 22, 2024 20:31 — with GitHub Actions Inactive

edsiper merged commit 1e2bb3e into master Mar 12, 2024
44 checks passed

edsiper deleted the pwhelan-out_http-json-default branch March 12, 2024 15:25

BrewTestBot mentioned this pull request Mar 21, 2024

fluent-bit 3.0.0 Homebrew/homebrew-core#166844

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

out_http: default the output format to JSON. #8493

out_http: default the output format to JSON. #8493

pwhelan commented Feb 16, 2024 •

edited

Loading

pwhelan commented Feb 16, 2024

pwhelan commented Feb 16, 2024

edsiper commented Mar 12, 2024

out_http: default the output format to JSON. #8493

out_http: default the output format to JSON. #8493

Conversation

pwhelan commented Feb 16, 2024 • edited Loading

Summary

pwhelan commented Feb 16, 2024

pwhelan commented Feb 16, 2024

edsiper commented Mar 12, 2024

pwhelan commented Feb 16, 2024 •

edited

Loading