[Azure] [container_service] Duplicated documents for the kube_node_status_condition metric

[tetianakravchenko]

for the container_service data_stream: some documents will be dropped after enabling TSDB due to duplications, mainly for the kube_node_status_condition metric - since each condition is stored in separate document:

example: for the same node/status and condition there are multiple documents with the same metric value:

we checked together with @zmoog and could easily reproduce this behavior

cc @tommyers-elastic

[tommyers-elastic]

is node status condition a dimension? if so we will only drop identical docs, which seems ok to me?

[tetianakravchenko]

@tommyers-elastic yes, node, status and condition are dimensions. On the screenshot it is a "good" example, I will try to provide different example.

I think we shouldn't move de-duplication logic to the tsdb, instead it should be on the beats side

[tommyers-elastic]

we are going to look into the azure API response here to ensure we are not missing some dimensions. once that is confirmed we can come back to the discussion about where is the appropriate place to do the deduplication, if necessary.

[zmoog]

@tetianakravchenko, I think I now have a good understanding of why we're getting multiple documents with the same dimension values.

I created the PR elastic/beats#36491 and experimented with a solution. I hope to wrap this up soon.

[zmoog]

I'm expanding the commit message a little.

The integration/model sets up dimensions in the config:

    resources:
    - resource_group: ""
      resource_type: "Microsoft.ContainerService/managedClusters"
      metrics:
      - name: ["kube_node_status_condition"]
        namespace: "Microsoft.ContainerService/managedClusters"
        ignore_unsupported: true
        timegrain: "PT5M"
        dimensions:
        - name: "node"
          value: "*"
        - name: "status"
          value: "*"
        - name: "condition"
          value: "*"

When we have dimensions, the current implementation groups metrics response by timestamp + dimension (just one).

Grouping by timestamp + single dimension can sometimes lead to multiple documents with the same dimension values.

Here's an example where we have 10 metrics values right before entering the final groupings:

At this point, the current implementation starts grouping these 10 metric values by each single dimension. Here are screenshots of the groupings:

This grouping does not play well with TSDB because it expects all documents with the same timestamp to have
a unique combination of dimensions values.

Here's the result of the grouping: we end up having 1 + 2 + 10 = 13 documents in Elasticsearch:

[tetianakravchenko]

Hey @zmoog thank you for the detailed explanation!

I created the PR #7160 and experimented with a solution. I hope to wrap this up soon.

did you mean this PR - elastic/beats#36491 ?

Do you think it is the same reason for another issue as well - #7621 (note: there are 2 cases) ?

[zmoog]

I created the PR #7160 and experimented with a solution. I hope to wrap this up soon.

did you mean this PR - elastic/beats#36491 ?

Yep, I'm sorry for pasting the wrong link. I updated the comment with the correct one.

Do you think it is the same reason for another issue as well - #7621 (note: there are 2 cases)?

Uhm, I'm not sure, but probably not. I don't see dimensions in #7621; this problem happens only when we have them. Added a comment to the issue.