Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_es: add missing condition to create static bulk header #7513

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

nokute78
Copy link
Collaborator

@nokute78 nokute78 commented Jun 4, 2023

In some cases out_es creates a bulk header j_index dynamically.
One of the cases is ctx->ra_id_key is not NULL.
https://github.com/fluent/fluent-bit/blob/v2.1.4/plugins/out_es/es.c#L537

Otherwise, it is created statically.
https://github.com/fluent/fluent-bit/blob/v2.1.4/plugins/out_es/es.c#L359
This patch is to add missing condition for static header.

Note: out_opensearch is already support similar condition.
https://github.com/fluent/fluent-bit/blob/v2.1.4/plugins/out_opensearch/opensearch.c#L346
https://github.com/fluent/fluent-bit/blob/v2.1.4/plugins/out_opensearch/opensearch.c#L483


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • [N/A] Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [N/A] Documentation required for this feature

Backporting

  • [N/A] Backport to latest stable release.

Debug/Valgrind output

$ valgrind --leak-check=full bin/flb-rt-out_elasticsearch 
==53972== Memcheck, a memory error detector
==53972== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==53972== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==53972== Command: bin/flb-rt-out_elasticsearch
==53972== 
Test long_index...                              [2023/06/04 18:50:06] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:06] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:06] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:06] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:06] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:06] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:06] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:06] [ info] [sp] stream processor started
[2023/06/04 18:50:06] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:07] [ info] [task] cleanup test task
[2023/06/04 18:50:08] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:09] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:09] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:09] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:10] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:10] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test div0_error...                              [2023/06/04 18:50:10] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:10] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:10] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:10] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:10] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:10] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:10] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:10] [ info] [sp] stream processor started
[2023/06/04 18:50:10] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:10] [ info] [task] cleanup test task
[2023/06/04 18:50:12] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:12] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:12] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:12] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:12] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:12] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test write_operation_index...                   [2023/06/04 18:50:12] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:12] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:12] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:12] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:12] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:12] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:12] [ info] [sp] stream processor started
[2023/06/04 18:50:12] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:12] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:13] [ info] [task] cleanup test task
[2023/06/04 18:50:14] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:15] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:15] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:15] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:15] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:15] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test write_operation_create...                  [2023/06/04 18:50:15] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:15] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:15] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:15] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:15] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:15] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:15] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:15] [ info] [sp] stream processor started
[2023/06/04 18:50:15] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:16] [ info] [task] cleanup test task
[2023/06/04 18:50:17] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:18] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:18] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:18] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:18] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:18] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test write_operation_update...                  [2023/06/04 18:50:18] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:18] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:18] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:18] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:18] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:18] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:18] [ info] [sp] stream processor started
[2023/06/04 18:50:18] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:18] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:19] [ info] [task] cleanup test task
[2023/06/04 18:50:20] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:21] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:21] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:21] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:21] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:21] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test write_operation_upsert...                  [2023/06/04 18:50:21] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:21] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:21] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:21] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:21] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:21] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:21] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:21] [ info] [sp] stream processor started
[2023/06/04 18:50:21] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:22] [ info] [task] cleanup test task
[2023/06/04 18:50:23] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:24] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:24] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:24] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:24] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:24] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test index_type...                              [2023/06/04 18:50:24] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:24] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:24] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:24] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:24] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:24] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:24] [ info] [sp] stream processor started
[2023/06/04 18:50:24] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:24] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:25] [ info] [task] cleanup test task
[2023/06/04 18:50:26] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:27] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:27] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:27] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:27] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:27] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test logstash_format...                         [2023/06/04 18:50:27] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:27] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:27] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:27] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:27] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:27] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:27] [ info] [sp] stream processor started
[2023/06/04 18:50:27] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:27] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:28] [ info] [task] cleanup test task
[2023/06/04 18:50:29] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:30] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:30] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:30] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:30] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:30] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test logstash_format_nanos...                   [2023/06/04 18:50:30] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:30] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:30] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:30] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:30] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:30] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:30] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:30] [ info] [sp] stream processor started
[2023/06/04 18:50:30] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:31] [ info] [task] cleanup test task
[2023/06/04 18:50:32] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:33] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:33] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:33] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:33] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:33] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test tag_key...                                 [2023/06/04 18:50:33] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:33] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:33] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:33] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:33] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:33] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:33] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:33] [ info] [sp] stream processor started
[2023/06/04 18:50:33] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:34] [ info] [task] cleanup test task
[2023/06/04 18:50:35] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:36] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:36] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:36] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:36] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:36] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test replace_dots...                            [2023/06/04 18:50:36] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:36] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:36] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:36] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:36] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:36] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:36] [ info] [sp] stream processor started
[2023/06/04 18:50:36] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:36] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:37] [ info] [task] cleanup test task
[2023/06/04 18:50:38] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:39] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:39] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:39] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:39] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:39] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test id_key...                                  [2023/06/04 18:50:39] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:39] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:39] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:39] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:39] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:39] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:39] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:39] [ info] [sp] stream processor started
[2023/06/04 18:50:39] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:40] [ info] [task] cleanup test task
[2023/06/04 18:50:41] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:42] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:42] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:42] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:42] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:42] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
Test logstash_prefix_separator...               [2023/06/04 18:50:42] [ info] [fluent bit] version=2.1.5, commit=3fdd42c6f2, pid=53972
[2023/06/04 18:50:42] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/04 18:50:42] [ info] [cmetrics] version=0.6.1
[2023/06/04 18:50:42] [ info] [ctraces ] version=0.3.1
[2023/06/04 18:50:42] [ info] [input:lib:lib.0] initializing
[2023/06/04 18:50:42] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/06/04 18:50:42] [ info] [output:es:es.0] worker #0 started
[2023/06/04 18:50:42] [ info] [sp] stream processor started
[2023/06/04 18:50:42] [ info] [output:es:es.0] worker #1 started
[2023/06/04 18:50:43] [ info] [task] cleanup test task
[2023/06/04 18:50:44] [ warn] [engine] service will shutdown in max 1 seconds
[2023/06/04 18:50:45] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/04 18:50:45] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/06/04 18:50:45] [ info] [output:es:es.0] thread worker #0 stopped
[2023/06/04 18:50:45] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/06/04 18:50:45] [ info] [output:es:es.0] thread worker #1 stopped
[ OK ]
SUCCESS: All unit tests have passed.
==53972== 
==53972== HEAP SUMMARY:
==53972==     in use at exit: 0 bytes in 0 blocks
==53972==   total heap usage: 20,663 allocs, 20,663 frees, 10,146,004 bytes allocated
==53972== 
==53972== All heap blocks were freed -- no leaks are possible
==53972== 
==53972== For lists of detected and suppressed errors, rerun with: -s
==53972== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)


Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@nokute78 nokute78 temporarily deployed to pr June 4, 2023 10:02 — with GitHub Actions Inactive
@nokute78 nokute78 temporarily deployed to pr June 4, 2023 10:02 — with GitHub Actions Inactive
@nokute78 nokute78 temporarily deployed to pr June 4, 2023 10:02 — with GitHub Actions Inactive
@nokute78 nokute78 temporarily deployed to pr June 4, 2023 10:22 — with GitHub Actions Inactive
Copy link
Contributor

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Dec 10, 2023
@nokute78
Copy link
Collaborator Author

Rebased using current master

@nokute78 nokute78 removed the Stale label Dec 15, 2023
Copy link
Contributor

github-actions bot commented Apr 7, 2024

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Apr 7, 2024
@nokute78 nokute78 removed the Stale label Apr 12, 2024
Copy link
Contributor

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Jul 14, 2024
@github-actions github-actions bot removed the Stale label Aug 16, 2024
Copy link
Contributor

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant