Add TBS examples to explain policy naunces #1239
Conversation
| - sample_rate: 1.0 # Fallback: always set a default | ||
| ``` | ||
|
|
||
| - Because Service A is the root of the trace, its policy (0.5) takes precedence over Service B's policy (0.3). |
There was a problem hiding this comment.
takes precedence
"takes precedence" implies some ordering. There is no ordering. It walks the policies from top to bottom, and match the first. service.name: B will never match. Service B is entirely out of the picture here.
| @@ -290,6 +290,50 @@ This example defines three tail-based sampling polices: | |||
| 2. Samples 1% of traces in `production` with the trace name `"GET /not_important_route"` | |||
| 3. Default policy to sample all remaining traces at 10%, e.g. traces in a different environment, like `dev`, or traces with any other name | |||
|
|
|||
| ### Example configuration B [_example_configuration_b] | |||
|
|
|||
| When a trace originates in Service A and then calls Service B (without errors), the sampling rate is determined by the service where the trace starts: | |||
There was a problem hiding this comment.
(without errors)
with and without error is irrelevant here. Removing it may be clearer
| @@ -272,7 +272,7 @@ Trace events are matched to policies in the order specified. Each policy list mu | |||
| Note that from version `9.0.0` APM Server has an unlimited storage limit, but will stop writing when the disk where the database resides reaches 80% usage. Due to how the limit is calculated and enforced, the actual disk space may still grow slightly over this disk usage based limit, or any configured storage limit. | |||
| :::: | |||
|
|
|||
| ### Example configuration [_example_configuration] | |||
| ### Example configuration A [_example_configuration_a] | |||
There was a problem hiding this comment.
I'm thinking about 1,2,3 instead of A,B,C since A,B,C are already used for service names.
| - Because Service A is the root of the trace, its policy (0.5) takes precedence over Service B's policy (0.3). | ||
| - If instead the trace began in Service B (and then passed to Service A), the policy for Service B would apply. | ||
|
|
||
| > **Key point**: Tail‑based sampling rules are evaluated at the *trace level* based on where the trace was initiated, not on downstream spans (*service level*). |
There was a problem hiding this comment.
| > **Key point**: Tail‑based sampling rules are evaluated at the *trace level* based on where the trace was initiated, not on downstream spans (*service level*). | |
| > **Key point**: Tail‑based sampling rules are evaluated at the *trace level* based on which service initiated the distributed trace, not the service of the transaction or span. |
|
|
||
| Policies targeting the trace (e.g. `trace.outcome: failure`) apply across all services and should appear before more specific, service‑level rules if you want them to take precedence. | ||
|
|
||
| > **Key point**: Define failure policy at the top to ensure capturing all failed traces, then define more specific policies for specific services to capture edge cases. |
There was a problem hiding this comment.
This is more like a opinion on how to use TBS, instead of explaining how Elastic APM picks the policy.
| - In Example A, traces from Service A are sampled at 20%, and all other failed traces (regardless of service) are sampled at 50%. | ||
| - In Example B, every failed trace is sampled at 20%, including those originating from Service A. | ||
|
|
||
| Policies targeting the trace (e.g. `trace.outcome: failure`) apply across all services and should appear before more specific, service‑level rules if you want them to take precedence. |
There was a problem hiding this comment.
I wonder if it is just easier to explain how it is done in our implementation, that is we walk the array and pick the first policy that applies. If none applies, the fallback will be used.
|
|
||
| ### Example configuration C [_example_configuration_c] | ||
|
|
||
| When you need to combine service‑specific policies with outcomes (e.g. failures), policy order defines specificity: |
There was a problem hiding this comment.
When you need to combine service‑specific policies with outcomes (e.g. failures), policy order defines specificity
The way I understand this sentence is there is something special to how we pick the policy when both service name and outcome are concerned. But it is not true. There is only 1 rule, that is we walk from top to bottom and pick the first applies.
I appreciate the example which makes it easier for users to understand how we pick the policy, but not in a way that implies special case handling in this sentence.
|
also looping in @colleenmcginnis for 👀 on language and style |
Closes elastic/apm-server#16253
Add two examples to TBS policy configurations:
servicedefinitions.traceandservicedefinitions.