[processor/tailsampling] record sampling policy

[djluck]

Description

Adds support for optionally recording the policy (and any composite policy) associated with an inclusive tail processor sampling decision:

Link to tracking issue

Fixes
Resolves #35180.

Implementation notes

• This functionality lives behind a feature flag that is disabled by default
• The original issue described a solution where we might attach the attribute solely to the root span. I'm not sure I agree with the commenter that we can rely on this (e.g. we might decide to sample halfway through a long-running trace) so I have attached the attributes to all present scope spans. This feels like a decent trade off between complexity + network cost, as finding the highest non-root parent would require multiple passes of the spans and keeping all span ids in a set

Testing

• Added automated tests to verify enabling the flag both records the expected decision while not impacting existing logic
• Built a custom version and ran it in our preprod environment to ensure it was stable over a 1h period (still evaluating, will update PR with any further observations)

TODO

Does this require a CHANGELOG entry?

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: djluck / name: James Luck (de70892, 95b45b8, 849dbdf, 1eb3e55, b5a6787, 6aeabdc, b9a5e56, 0584b17)

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[djluck]

Hey @djaglowski, sorry to bother you but do you think you'd be able to take a look at this some time soon?

[jpkrohling]

I still prefer to set the policy at the root span, but I acknowledge that the root span might not be easy to find given the current structure we have. I think we can go with this first implementation and see what real-world effects it has on people.

hit "approve" too early :-)

[jpkrohling]

I like this change, it goes in the right direction, but it's missing unit tests for the newly introduced function, a changelog entry, and something in the README telling people how to use this.

[djluck]

Thanks for your feedback @jpkrohling, I'll work on resolving these comments over the next couple of days 👍

[djluck]

@jpkrohling thanks for the review, I've pushed fixes for all your comments. I think I missed Locking the tracedata mutex so I added this additionally.

Unsure why the pipeline is failing, seems unrelated to anything I touched (command missing error).

[jpkrohling]

I'm OK merging as is, the comments are nits.

[jpkrohling]

The code owners check seems to be failing for all jobs at the moment.

[djluck]

@jpkrohling thanks for the review, I've pushed fixes for your comments

[jpkrohling]

Linting seems to be failing:

tailsamplingprocessor/factory.go:12: File is not `gci`-ed with --skip-generated -s standard -s default -s prefix(github.com/open-telemetry/opentelemetry-collector-contrib) (gci)
	"github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor/internal/telemetry"

tailsamplingprocessor/factory.go:18: File is not `gci`-ed with --skip-generated -s standard -s default -s prefix(github.com/open-telemetry/opentelemetry-collector-contrib) (gci)
	"github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor/internal/metadata"
tailsamplingprocessor/processor_decisions_test.go:12: File is not `gci`-ed with --skip-generated -s standard -s default -s prefix(github.com/open-telemetry/opentelemetry-collector-contrib) (gci)

You can run make lint locally to test the changes before going to the CI.

[djluck]

@jpkrohling apologies, re-ran make lint, make fmt + make gci- I think we should be good now

[djluck]

Thanks again @jpkrohling for reviewing this change but I think I need to bother you one last time: I can see you've approved but I'm not authorized to merge. Is that something you could do?

[djluck]

Hey @jpkrohling! Just a friendly ping- if you've got time before Christmas and you're able to help merge this it would be appreciated. Otherwise I can follow up in the new year.