[pkg/ottl] Set method should allow dynamically setting map values to attributes

[rupeshnemade]

Is your feature request related to a problem? Please describe.

While using transform processor, we tried setting set kubernetes labels to a single attribute 'labels' in a body as below.

But Set method doesn't iterate through the map of values & instead needs all keys in array to be explicitly mentioned like below-
set(attributes["kubernetes.labels"], resource.attributes["pod_labels_app"])

Kubernetes labels is something dynamic things & we can't explicitly mention label name in OTEL processor rules.

Describe the solution you'd like

Set method in here should allow iterate through the map & fetch all values so that we can perform something like this-

set(attributes["kubernetes.labels"], resource.attributes["pod_labels_*"])

Describe alternatives you've considered

No response

Additional context

No response

[TylerHelmuth]

I can think of two ways to handle this situation:

1. Update all contexts used by the transform processor to understand that wildcard (*) in a key means "all fields in a map that match that key".
2. Create a new Factory Function that knows how to take a map and a string pattern and returns a map that contains all the fields from the input map that match the pattern. It would behave similarly to replace_all_patterns.

I would opt for the second because I like the flexibility that Factory Functions provide. I prefer putting business logic in the function vs the contexts.

@bogdandrutu @kentquirk @evan-bradley what do you think?

[evan-bradley]

I agree, I think a function would be better for this sort of operation. In addition to keeping business logic out of the contexts I think it also provides a good pattern for doing other types of matching (regex, filtering, etc.), and makes it more explicit what is going on.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley
• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley
• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[odubajDT]

Is this functionality still required? If yes, I would like to have a look

[TylerHelmuth]

@odubajDT some solution to make dynamic interactions in OTTL is still desired. I havent spent much time on this problem so I'm open to suggestions. I do know I don't want to change how set works, so the solution should be something else.

[odubajDT]

One of the solutions that comes into my mind is creating a separate Compile(input, pattern) function which will parse the input object and return the same object type but only with values/keys matching the pattern. Therefore setting up map should be possible like this

set(attributes["kubernetes.labels"], Compile(resource.attributes,"pod_labels_*"))

where resource.attributes will look like the following:

{
    "pod_labels_cpu": "cpu_label"
    "pod_labels_memory": "memory_label"
    "random_label":"random"
}

And the resulting attributes["kubernetes.labels"] will have the following content:

{
    "pod_labels_cpu": "cpu_label"
    "pod_labels_memory": "memory_label"
}

WDYT @evan-bradley @TylerHelmuth ?

Thanks for the opinions!

Just and additional though here, when implementing this function, would it makes sense to actually also implement a Flatten(input, delimiter) function which will be able to convert a map to a string with specified delimiter. Just thinking about the possible enhancements, since I recently ran into a use-case, where Flatten() function might be very useful. If you would agree, I would create a separate issue for this.

[odubajDT]

Created a draft implementation of the potential solution, will appreciate your opinions!

#35553

@evan-bradley @TylerHelmuth

[evan-bradley]

Sorry for the delay in taking a look at this.

We have keep_matching_keys now, would it be possible to do something like the following to solve this case?

- set(cache["attrs"], resource.attributes)
- keep_matching_keys(cache["attrs"], "pod_labels_.*")
- set(attributes["kubernetes.labels"], cache["attrs"])

I think it should be functionally equivalent to having a single Converter for both copying and modifying the map, and shouldn't be much less performant.

Just and additional though here, when implementing this function, would it makes sense to actually also implement a Flatten(input, delimiter) function which will be able to convert a map to a string with specified delimiter.

I think we should have this with the flatten Editor. Does that match what you have in mind?

[odubajDT]

Sorry for the delay in taking a look at this.

We have keep_matching_keys now, would it be possible to do something like the following to solve this case?

- set(cache["attrs"], resource.attributes)
- keep_matching_keys(cache["attrs"], "pod_labels_.*")
- set(attributes["kubernetes.labels"], cache["attrs"])

I think it should be functionally equivalent to having a single Converter for both copying and modifying the map, and shouldn't be much less performant.

That's probably true, do you think we should close the issue and the PR?

[TylerHelmuth]

@evan-bradley's solution is acceptable for now. Let's not close this issue tho until that example solution for this scenario is documented in the transformprocessor's readme. This is a common scenario but that solution isn't easy to think up unless you know OTTL well.

[odubajDT]

Opened an issue to document the example configuration #35893

@evan-bradley @TylerHelmuth