From e0707cffb22ec0a9f84af14d4e9d86f9df35d0b2 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Wed, 15 Nov 2023 19:20:10 +0900 Subject: [PATCH 1/2] in_kafka: Add buffer_chunk_size description Signed-off-by: Hiroshi Hatake --- pipeline/inputs/kafka.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pipeline/inputs/kafka.md b/pipeline/inputs/kafka.md index aac51620f..78610b11d 100644 --- a/pipeline/inputs/kafka.md +++ b/pipeline/inputs/kafka.md @@ -12,6 +12,8 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill | client\_id | Client id passed to librdkafka. | | | group\_id | Group id passed to librdkafka. | fluent-bit | | poll\_ms | Kafka brokers polling interval in milliseconds. | 500 | +| Chunk\_Size | Specify the size of buffer to store the incoming kafka messages. When this parameter is specified, do not poll them from brokers at once over the specified size and deferred to consume left messages from there at the next polling cycle. If not set, _Chunk\_Size_ is not effective. | | +| poll\_ms | Kafka brokers polling interval in milliseconds. | 500 | | rdkafka.{property} | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) | | ## Getting Started From e69894268b77877eda2b919094f6c0b713670e6f Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 17 Nov 2023 20:19:46 +0900 Subject: [PATCH 2/2] in_kafka: Add buffer_max_size parameter instead of chunk_size We need to halt the greedy consuming loop before reaching the exceeded limitation of chunk size. Without this parameter, in_kafka always consuming greedy from subscribing topics. Signed-off-by: Hiroshi Hatake --- pipeline/inputs/kafka.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/inputs/kafka.md b/pipeline/inputs/kafka.md index 78610b11d..358a46bc5 100644 --- a/pipeline/inputs/kafka.md +++ b/pipeline/inputs/kafka.md @@ -12,7 +12,7 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill | client\_id | Client id passed to librdkafka. | | | group\_id | Group id passed to librdkafka. | fluent-bit | | poll\_ms | Kafka brokers polling interval in milliseconds. | 500 | -| Chunk\_Size | Specify the size of buffer to store the incoming kafka messages. When this parameter is specified, do not poll them from brokers at once over the specified size and deferred to consume left messages from there at the next polling cycle. If not set, _Chunk\_Size_ is not effective. | | +| Buffer\_Max\_Size | Specify the maximum size of buffer per cycle to poll kafka messages from subscribed topics. To increase throughput, specify larger size. | 4M | | poll\_ms | Kafka brokers polling interval in milliseconds. | 500 | | rdkafka.{property} | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) | |