Vector Filter Not Working on EKS Cluster

[justinAtBeleb]

EDIT: Adding vital information.

We have an EKS Cluster that we have to monitor.
I have already installed Vector and the logs are being sent to Betterstack.
I'm trying to learn advanced filtering and have been testing some sample Transforms, but for some reason, they're not working.

The Problem

The integration works and logs are being recorded and sent out from Vector to Better Stack. However, the transforms are not working. The logs that should NOT have matched the filter condition are still being sent, and the remap is not reflecting in the logs, also.
Everything else works, except for the transforms.

Here is a sample Event Data:

{
    "dt": "2024-11-12T06:29:28.211962730Z",
    "file": "/var/log/pods/dev_analytic-service-79f65f9b58-w5xh5_723620b6-82aa-4828-8cc6-f9408249d7ba/analytic-service/0.log",
    "kubernetes": {
        "container_id": "removed_data_for_simplicity",
        "container_image": "removed_data_for_simplicity",
        "container_image_id": "removed_data_for_simplicity",
        "container_name": "analytic-service",
        "namespace_labels": {
            "kubernetes.io/metadata.name": "dev"
        },
        "node_labels": "removed_data_for_simplicity",
        "pod_annotations": {
            "fluentbit.io/exclude": "true",
            "kubectl.kubernetes.io/restartedAt": "2024-08-23T11:07:04+02:00"
        },
        "pod_ip": "10.0.16.52",
        "pod_ips": [
            "10.0.16.52"
        ],
        "pod_labels": {
            "app": "analytic-service",
            "pod-template-hash": "79f65f9b58"
        },
        "pod_name": "analytic-service-79f65f9b58-w5xh5",
        "pod_namespace": "dev",
        "pod_node_name": "ip-10-0-21-8.us-east-2.compute.internal",
        "pod_owner": "ReplicaSet/analytic-service-79f65f9b58",
        "pod_uid": "723620b6-82aa-4828-8cc6-f9408249d7ba",
        "stream": "stdout"
    },
    "message": {
        "ctx": "rpc",
        "data": {
            "action": "start"
        },
        "debug": {
            "event": "GET /healthz",
            "protocol": "http",
            "trace": "1c54a365-8f56-4524-acff-ee57904d3f91"
        },
        "level": 30,
        "message": "GET /healthz",
        "service": {
            "name": "analytic",
            "version": "1.3.15"
        },
        "time": 1731392968211
    },
    "platform": "Kubernetes"
}

And here is my Sources and Transforms Blocks in YAML.

 # Sources
    sources:
      better_stack_kubernetes_logs:
        type: kubernetes_logs
        extra_field_selector: "metadata.namespace!=monitoring,metadata.namespace!=kube-system,metadata.namespace!=prometheus"
        extra_label_selector: "app!=optimove-gateway"
    # Transforms
    transforms:
      extra_filters:
        type: filter
        inputs:
          - "better_stack_kubernetes_logs"
        condition: .message.message != "GET /healthz"
      example:
        type: remap
        inputs:
          - "*"
        source: |
          .extra = .message

The extra_field_selector and extra_label_selector are working properly, but transforms, are not.
The filter is not working despite it having the right source id and remap is also not working despite it having a simple reassignment line.

I even went to VRL Playground to check if that one line of code is incorrect, but it seems to be correct (as seen below).

I'm pretty sure it's just another simple mistake that I'm just overlooking, but I really can't seem to find what's wrong. I've checked the documentations again and again and again, but still can't see what's wrong here.

I hope someone can help me. It also does not help that the examples in the documentations are just very minimal. I wish there's a complete example somewhere for Kubernetes (or maybe I'm just blind)

[jszwedko]

Hi @justinAtBeleb ,

What do you mean by "not working"? It isn't clear above what isn't working. For example, does Vector fail to start with an error? Do you see errors during processing? Are the transforms being skipped? Something else?

It may be out-of-date (I see it hasn't been updated in a few years), but there is an example for EKS over here, https://github.com/vectordotdev/vector-demos/tree/main/eks-aggregator, that could be helpful for you.

[justinAtBeleb]

Oh dear. Sorry, I forgot to specify the most important part!

The Problem

The integration works and logs are being recorded and sent out from Vector to Better Stack. However, the transforms are not working. The logs that should NOT have matched the filter condition are still being sent, and the remap is not reflecting in the logs, also.
Everything else works, except for the transforms.

[justinAtBeleb]

I forgot, thank you for the link. I haven't fixed it, but looking at the codes in the examples is giving me an idea of what I might have done wrong. I'll test it out as soon as I get to open my work computer.

[jszwedko]

Gotcha, I see, thanks! Could you share your sink configuration as well? And any warnings or errors you see in Vector's output?