Pod logs are not structured according to fluentbit.io/parser

[baby-gnu]

Bug Report

Describe the bug

I annotated the pod template of my ingress deployment with fluentbit.io/parser: k8s-nginx-ingress but the message seems not parsed.

To Reproduce

• On a kubernetes, inspired by kube-logging example
• Deploy Grafana and Loki
• Deploy kube-logging
• Configure K8S-Logging.Parser: "On"
• Example log message if applicable:

{
  "time": "2025-02-13T07:37:06.48966441Z",
  "stream": "stdout",
  "logtag": "F",
  "message": "10.42.1.0 - - [13/Feb/2025:07:37:06 +0000] \"POST /__meteor__/dynamic-import/fetch HTTP/2.0\" 200 1987 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:135.0) Gecko/20100101 Firefox/135.0\" 152 0.005 [eole3-laboite-3000] [] 10.42.1.60:3000 2004 0.005 200 c1926ab41c29c88aad473b2033faebb6",
  "kubernetes": {
    "pod_name": "ingress-nginx-controller-6986f544f4-dhw4s",
    "namespace_name": "eole3-nginx",
    "pod_id": "6e9fbf69-4c4e-4cd7-b1c4-7ca782b6dc79",
    "labels": {
      "app.kubernetes.io/component": "controller",
      "app.kubernetes.io/instance": "ingress-nginx",
      "app.kubernetes.io/managed-by": "Helm",
      "app.kubernetes.io/name": "ingress-nginx",
      "app.kubernetes.io/part-of": "ingress-nginx",
      "app.kubernetes.io/version": "1.11.2",
      "helm.sh/chart": "ingress-nginx-4.11.2",
      "pod-template-hash": "6986f544f4"
    },
    "annotations": {
      "fluentbit.io/parser_stdout-ingress-nginx": "k8s-nginx-ingress"
    },
    "host": "k3d-eole3-agent-0",
    "pod_ip": "10.42.4.63",
    "container_name": "controller",
    "docker_id": "0ca8a69797aaa4cbdaf1a87e1104c76c6024e0c81c6a933f02c92c2698b3aea2",
    "container_hash": "hub.eole.education/eole3/ingress-nginx-container-image/controller@sha256:0246021915691850ab79d6c848f77d911be17ebdd658b0436963d44c109e44c0",
    "container_image": "hub.eole.education/eole3/ingress-nginx-container-image/controller:v1.11.2-eole3.0"
  },
  "kubernetes_namespace": {
    "name": "eole3-nginx",
    "labels": {
      "kubernetes.io/metadata.name": "eole3-nginx"
    }
  }
}

• Steps to reproduce the problem:
  • On a kubernetes, inspired by kube-logging example
  • Deploy Grafana and Loki
  • Deploy kube-logging
  • Configure K8S-Logging.Parser: "On"
  • Annotate the pod template of ingress-nginx deployment

Expected behavior

The message should be parsed according to k8s-nginx-ingress parser and the matching parts should be labeled in loki.

Your Environment

• Version used: Fluentbit 3.2.5, Fluentd 5.1.1

• Configuration:

  my-logging.yaml

  ---
  apiVersion: logging.banzaicloud.io/v1beta1
  kind: Logging
  metadata:
    name: eole3
    namespace: logging
  spec:
    controlNamespace: logging
    errorOutputRef: loki
  ...
  ---
  apiVersion: logging.banzaicloud.io/v1beta1
  kind: ClusterOutput
  metadata:
    name: loki
    namespace: logging
  spec:
   loki:
     url: http://loki-gateway.eole3-loki
     extract_kubernetes_labels: true
     buffer:
       chunk_limit_size: 1MB
       timekey: 1m
       timekey_wait: 30s
       timekey_use_utc: true
  ...
  ---
  apiVersion: logging.banzaicloud.io/v1beta1
  kind: ClusterFlow
  metadata:
    name: eole3
    namespace: logging
  spec:
    globalOutputRefs:
      - loki
  ...
  ---
  apiVersion: logging.banzaicloud.io/v1beta1
  kind: FluentdConfig
  metadata:
    name: eole3
    namespace: logging
  spec:
    metrics:
      serviceMonitor: true
  ...
  ---
  apiVersion: logging.banzaicloud.io/v1beta1
  kind: FluentbitAgent
  metadata:
    name: eole3
    namespace: logging
  spec:
    metrics:
      serviceMonitor: true
    filterKubernetes:
      K8S-Logging.Parser: "On"
  ...

• Environment name and version (e.g. Kubernetes? What version?): Kubernetes k3d v5.6.0, k3s version v1.27.4-k3s1

• Server type and version:

• Operating System and version:

• Filters and plugins:

Additional context

I wonder how it's related to #9723 since Merge_Log is mandatory for pod annotation to work.