Could not connect to leader for partition/ Unknown Partition

[12ushan]

Describe the bug

Hi, as mentioned here in the fluentd discussion page.

To summarise: We have fluentd running in kubernetes sending logs to kafka (Amazon MSK). We're seeing errors intermittently mentioning "unknown partition"/Could not connect to leader for partition" where fluentd is trying to send to partition 14,15 or 16 for example when we only have 12 partitions on the topic. We've tried both kafka2 and rdkafka2 plugin as the type but get one of the error message mentioned above.

We have not been able to replicate this issue with a local setup so we're wondering if anyone has seen this issue at all?

To Reproduce

Use this plugin on fluentd running on kubernetes sending any logs to kafka on amazon MSK

Expected behavior

The plugin should fetch from AWS the partition numbers that exist on the topic

Your Environment

- Fluentd version: 1.15.3
- fluent-plugin-kafka version: 0.19.3
- ruby-kafka version: 1.5.0
- Operating system: Debian GNU/Linux 11
- Kernel version: Linux 5.10.217-205.860.amzn2.x86_64 x86_64
- Kubernetes version: v1.28.12-eks-a18cd3a 
- Amazon MSK running Apache Kafka Version 3.5.1

Your Configuration

<match **>
  <store>
    type kafka2
    brokers "kafka:9092"
    default_topic "logging.kubernetes.events"
    <buffer topic>
      flush_interval 3s
    </buffer>
    <format>
      @type json
    </format>
  </store>
</match>

Your Error Log

[error]: #0 Could not connect to leader for partition logging.kubernetes.events/19: no partition 19 in topic logging.kubernetes.events 

OR 

[warn]: #0 Send exception occurred: Local: Unknown partition (unknown_partition) at /fluentd/vendor/bundle/ruby/3.1.0/gems/rdkafka-0.18.0/lib/rdkafka/producer.rb:363:in `produce'

Additional context

We have seen the issue on older version of amazon msk too