fluent-bit:2.2.1 [ warn] [net] getaddrinfo(host='elasticsearch', err=3): DNS server returned general failure

[skyhos]

Bug Report

Describe the bug

1. Running fluent-bit:2.2.1 docker container with elasticsearch:8.11.3, kibana:8.11.3 containers via docker-compose, I get warning message and the log transferring does not happen.
2. The host server is Centos 7.
3. If I replace the fluent-bit:2.2.1 with the fluent-bit:2.2.0, then everything works fine.

The warning message:

[2024/01/02 16:48:39] [ info] [output:es:es.9] worker #1 started
[2024/01/02 16:48:39] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/01/02 16:48:39] [ info] [sp] stream processor started
[2024/01/02 16:52:44] [ warn] [net] getaddrinfo(host='elasticsearch', err=3): DNS server returned general failure
[2024/01/02 16:52:44] [ warn] [engine] failed to flush chunk '1-1704214360.901018925.flb', retry in 11 seconds: task_id=0, input=forward.0 > output=es.0 (out_id=0)
[2024/01/02 16:52:49] [ warn] [net] getaddrinfo(host='elasticsearch', err=3): DNS server returned general failure
[2024/01/02 16:52:49] [ warn] [engine] failed to flush chunk '1-1704214367.45588211.flb', retry in 9 seconds: task_id=1, input=forward.0 > output=es.0 (out_id=0)
[2024/01/02 16:52:55] [ warn] [net] getaddrinfo(host='elasticsearch', err=3): DNS server returned general failure
[2024/01/02 16:52:55] [ warn] [engine] failed to flush chunk '1-1704214360.901018925.flb', retry in 9 seconds: task_id=0, input=forward.0 > output=es.0 (out_id=0)
[2024/01/02 16:52:58] [ warn] [net] getaddrinfo(host='elasticsearch', err=3): DNS server returned general failure

fluent-bit.conf

[SERVICE]
    flush     5
    daemon    off
    http_server on
    log_level info
    parsers_file parsers.conf

[INPUT]
    Name   forward
    Listen 0.0.0.0
    Port   24224

[FILTER]
    name parser
    match *
    key_name log
    parser docker
    reserve_data true

@INCLUDE my-service.conf

my-service.conf

[OUTPUT]
    name                es
    match               my-service
    host                elasticsearch
    port                9200
    HTTP_User           fluentbit
    HTTP_Passwd         ${ELASTICSEARCH_PASSWORD}
    replace_dots        on
    retry_limit         false
    logstash_format     on
    logstash_prefix     my-service
    Suppress_Type_Name  on
    #Trace_Error       On
    #Trace_Output      On

docker-compose.yaml

version: '3.7'
networks:
  service_net_1:
    external: true
services:
  elasticsearch:
    container_name: elasticsearch
    image: elasticsearch:8.11.3
    networks:
      service_net_1:
        aliases:
          - elasticsearch
    environment:
      #node.name: elasticsearch
      ES_JAVA_OPTS: -Xms1024m -Xmx1024m
      discovery.type: single-node
    volumes:
      - ${CONF_DIR}/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ${DATA_DIR}/elasticsearch/data:/usr/share/elasticsearch/data
  kibana:
    container_name: kibana
    image: kibana:8.11.3
    networks:
      service_net_1:
        aliases:
          - kibana
    environment:
      - TZ=Asia/Seoul
    volumes:
      - ${CONF_DIR}/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml
  fluent-bit:
    container_name: fluent-bit
    image: fluent/fluent-bit:2.2.1
    networks:
      service_net_1:
        aliases:
          - fluent-bit
    command:
      - /fluent-bit/bin/fluent-bit
      - --config=/etc/fluent-bit/fluent-bit.conf
    ports:
      - '127.0.0.1:24224:24224'
      - '127.0.0.1:24224:24224/udp'
    volumes:
      - ${CONF_DIR}/fluent-bit/:/etc/fluent-bit/:ro
  my-service:
    container_name: my-service
    image: my-service-image:latest
    networks:
      service_net_1:
        aliases:
          - my-service
    logging:
      driver: 'fluentd'
      options:
        fluentd-address: localhost:24224
        fluentd-async-connect: "true"
        tag: my-service

[bluezheng]

meet same issue with 2.2.1, downgrade to 2.2.0 works

[nokute78]

c-ares was updated from v2.2.1 #8296
It may cause this issue.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.