Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstd: introduce zstd compression support in Fluent Bit #9830

Closed
wants to merge 19 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
70a8c67
lib: zstd: add v1.5.6
robskillington Jan 13, 2025
c168031
zstd: add new compress/uncompress zstd interface
edsiper Jan 13, 2025
7348ccb
tests: internal: add unit test for zstd interface
edsiper Jan 13, 2025
a745400
build: link to zstd
edsiper Jan 13, 2025
fab6bdc
http_common: implement zstd wrapers
robskillington Jan 13, 2025
d5dca8f
out_opentelemetry: add support for zstd compression
edsiper Jan 14, 2025
753303c
http_client: add api to set zstd content-encoding
edsiper Jan 14, 2025
0360731
http_common: fix return value for zstd uncompress
edsiper Jan 14, 2025
58382e2
out_opentelemetry: set proper content encoding for HTTP/1.x
edsiper Jan 14, 2025
60d9103
workflows: macos: Build Intel Mac packages by macos-14-large runner
cosmo0920 Jan 15, 2025
aca2295
hash_table: added optional case insensitive mode
leonardo-albertovich Jan 17, 2025
a95f939
http_client: moved additional headers after the standard header set
leonardo-albertovich Jan 17, 2025
dd006bd
http_common: updated to use the hash table in caseless mode
leonardo-albertovich Jan 17, 2025
a691f5e
hash_table: add missing header to build on macOS
edsiper Jan 17, 2025
a565b0d
lib: monkey: upgrade to v1.8.2
edsiper Jan 17, 2025
56567e5
in_opentelemetry: adjust error log messages
edsiper Jan 17, 2025
f37d4cc
out_opentelemetry: set grpc flag when paylaod is compressed
edsiper Jan 17, 2025
391ce85
out_opentelemetry: do not set gRPC compression flag
edsiper Jan 21, 2025
6d9071f
in_opentelemetry: add missing group footer when processing protobuf
edsiper Jan 21, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
out_opentelemetry: add support for zstd compression
This patch adds a new compression mechanism called 'zstd', this is
available through the 'compress' configuration property, in the configuration file
it can be enabled with:

  pipeline:
     outputs:
      - name: opentelemetry
        match: '*'
        compress: zstd

Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
  • Loading branch information
edsiper committed Jan 14, 2025
commit d5dca8f70b07c5a49ed8dc5b2ef3e71ccb8e78d3
7 changes: 5 additions & 2 deletions plugins/out_opentelemetry/opentelemetry.c
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,9 @@ int opentelemetry_post(struct opentelemetry_context *ctx,
if (ctx->compress_gzip == FLB_TRUE) {
compression_algorithm = "gzip";
}
else if (ctx->compress_zstd == FLB_TRUE) {
compression_algorithm = "zstd";
}

result = flb_http_request_set_parameters(request,
FLB_HTTP_CLIENT_ARGUMENT_URI(http_uri),
Expand Down Expand Up @@ -407,7 +410,7 @@ int opentelemetry_post(struct opentelemetry_context *ctx,
if (ctx->log_response_payload &&
response->body != NULL &&
cfl_sds_len(response->body) > 0) {
flb_plg_info(ctx->ins, "%s:%i, HTTP status=%i\n%s",
flb_plg_info(ctx->ins, "%s:%i, HTTP status=%i%s",
ctx->host, ctx->port,
response->status, response->body);
}
Expand Down Expand Up @@ -840,7 +843,7 @@ static struct flb_config_map config_map[] = {
{
FLB_CONFIG_MAP_STR, "compress", NULL,
0, FLB_FALSE, 0,
"Set payload compression mechanism. Option available is 'gzip'"
"Set payload compression mechanism. Options available are 'gzip' and 'zstd'."
},
/*
* Logs Properties
Expand Down
5 changes: 4 additions & 1 deletion plugins/out_opentelemetry/opentelemetry.h
Original file line number Diff line number Diff line change
Expand Up @@ -156,9 +156,12 @@ struct opentelemetry_context {
/* instance context */
struct flb_output_instance *ins;

/* Compression mode (gzip) */
/* compression: gzip */
int compress_gzip;

/* compression: zstd */
int compress_zstd;

/* FLB/OTLP Record accessor patterns */
struct flb_record_accessor *ra_meta_schema;
struct flb_record_accessor *ra_meta_resource_id;
Expand Down
7 changes: 7 additions & 0 deletions plugins/out_opentelemetry/opentelemetry_conf.c
Original file line number Diff line number Diff line change
Expand Up @@ -404,6 +404,13 @@ struct opentelemetry_context *flb_opentelemetry_context_create(struct flb_output
if (strcasecmp(tmp, "gzip") == 0) {
ctx->compress_gzip = FLB_TRUE;
}
else if (strcasecmp(tmp, "zstd") == 0) {
ctx->compress_zstd = FLB_TRUE;
}
else {
flb_plg_error(ctx->ins, "Unknown compression method %s", tmp);
return NULL;
}
}

ctx->ra_observed_timestamp_metadata = flb_ra_create((char*)ctx->logs_observed_timestamp_metadata_key,
Expand Down
Loading