Fluent bit fails to send logs collected by opentelemetry input source to opensearch output

[hoshiyosan]

Bug Report

Describe the bug
Fluent bit fails to send logs collected by opentelemetry input source to opensearch output.

To Reproduce

• Environment

Docker + current latest image of fluentbit (v3.0.6)

• Start an instance of opensearch + fluentbit using docker compose

version: "3.6"

services:  
  opensearch:
    image: opensearchproject/opensearch:latest
    container_name: opensearch
    environment:
      - cluster.name=opensearch
      - node.name=opensearch
      - discovery.seed_hosts=opensearch
      - cluster.initial_cluster_manager_nodes=opensearch
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD="*aK7Dz#Yx83W*nVVt"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data:/usr/share/opensearch/data
    ports:
      - 9200:9200 # REST API
    networks:
      - devstack

  fluent-bit:
    image: fluent/fluent-bit
    volumes:
      - ./confs/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    ports:
      - 4318:4318
    depends_on:
      - opensearch
    networks:
      - devstack

networks:
  devstack:
    external: false
    name: devstack

With the following ./confs/fluent-bit.conf :

[SERVICE]
    Log_Level       trace
    Daemon          off
    
[INPUT]
    name            opentelemetry
    listen          0.0.0.0
    port            4318
    Tag             opentelemetry

[OUTPUT]
    Name            opensearch
    Match           *
    Host            opensearch
    Port            9200
    Index           otel-v1-apm-span-000001
    tls             On
    tls.verify      Off
    Trace_Output    On
    # Don't worry these are dev credentials
    HTTP_User       admin
    HTTP_Passwd     *aK7Dz#Yx83W*nVVt

• Then send some logs to fluentbit opentelemetry input (e.g using python opentelemetry instrumentation)

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

resource = Resource(attributes={
    "service.name": "manila"
})

traceProvider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
traceProvider.add_span_processor(processor)

trace.set_tracer_provider(traceProvider)

tracer = trace.get_tracer("http.server")
with tracer.start_as_current_span("GET /v2/shares") as rollspan:
    rollspan.set_attribute("method", "POST")
    rollspan.set_attribute("path", "/v2/shares")

    rollspan.add_event("some event")

Expected behavior

Fluent Bit should send traces to opensearch POST /_bulk endpoint.

Actual behavior

• POST /_bulk endpoint is called with an empty payload.
• This is the same when using gRPC endpoint
• It looks like opentelemetry INPUT plugin works (using STDOUT as output shows that protobuf data from OTLP has been decoded properly)
• I suspect error might be related to how record conversion from msgpack to json is made in out_opensearch but I don't understand Fluent Bit code enough to be sure.

Maybe I misunderstood documentation as it lacks of a working example on how FluentBit can foward opentelemetry traces to opensearch

Thanks for any help

[leonardo-albertovich]

Hi @hoshiyosan, I'm currently looking into this issue and found that fluent-bit does properly send logs collected from an opentelemetry input to the opensearch output, however, it does not properly send traces regardless of the source to an opentelemetry output which is what your python snippet sends to fluent-bit.

The issue can be easily identified in line 906 of the opensearch output plugin where the variable pack which contains the results of the trace-to-json operation is overwritten with the uninitialized variable meant to hold the encoded log content (which in this case is never produced because we are processing a trace) chunk.

I am in the process of opening a PR to fix this and will update my response with the correct link once it's open.

Thanks for taking the time to report this issue.

Edit: The PR is #9158

[edsiper]

Fixed in #9158