Send sampling priority to lambda extension #6719

nhulston · 2025-02-26T17:50:23Z

Summary of changes

The issue here was that in Lambda, the _rootSpan in TraceContext.cs was never defined, so GetOrMakeSamplingDecision always returned 1 regardless of the tracer's sampling rules (e.g. the DD_TRACE_SAMPLING_RULES env var).

This change simply updates the function to take in span rather than hardcoding _rootSpan.

We send this sampling decision to the Lambda extension via the "x-datadog-sampling-priority" header in the /lambda/end-invocation request.

Reason for change

Fix sampling rules in .NET lambdas. Before this change, 100% of traces were sampled regardless of sampling rules.

Implementation details

Test coverage

Manual:

Other details

…verloaded method

datadog-datadog-prod-us1 · 2025-02-26T18:24:51Z

Datadog Report

All test runs 89a23d9 🔗

✅ 2 Total Test Services: 0 Failed, 2 Passed

Test Services

Service Name	Failed	Known Flaky	New Flaky	Passed	Skipped	Total Time	Test Service View
`dd-trace-dotnet`	0	0	0	552785	4421	34h 6m 53.56s	Link
`exploration_tests`	0	0	0	22085	3	2m 8.99s	Link

andrewlock · 2025-02-26T18:26:22Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6719) - mean (78ms)  : 72, 85
     .   : milestone, 78,
    master - mean (78ms)  : 72, 84
     .   : milestone, 78,

    section CallTarget+Inlining+NGEN
    This PR (6719) - mean (1,084ms)  : 1056, 1111
     .   : milestone, 1084,
    master - mean (1,093ms)  : 1058, 1128
     .   : milestone, 1093,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6719) - mean (116ms)  : 110, 122
     .   : milestone, 116,
    master - mean (120ms)  : 113, 127
     .   : milestone, 120,

    section CallTarget+Inlining+NGEN
    This PR (6719) - mean (738ms)  : 716, 759
     .   : milestone, 738,
    master - mean (751ms)  : 725, 776
     .   : milestone, 751,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6719) - mean (101ms)  : 96, 106
     .   : milestone, 101,
    master - mean (104ms)  : 96, 112
     .   : milestone, 104,

    section CallTarget+Inlining+NGEN
    This PR (6719) - mean (687ms)  : 663, 710
     .   : milestone, 687,
    master - mean (698ms)  : 675, 722
     .   : milestone, 698,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6719) - mean (191ms)  : 187, 196
     .   : milestone, 191,
    master - mean (191ms)  : 186, 195
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6719) - mean (1,106ms)  : 1080, 1132
     .   : milestone, 1106,
    master - mean (1,111ms)  : 1083, 1139
     .   : milestone, 1111,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6719) - mean (270ms)  : 266, 275
     .   : milestone, 270,
    master - mean (271ms)  : 266, 276
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (6719) - mean (868ms)  : 838, 897
     .   : milestone, 868,
    master - mean (868ms)  : 840, 896
     .   : milestone, 868,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6719) - mean (263ms)  : 258, 268
     .   : milestone, 263,
    master - mean (262ms)  : 257, 266
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (6719) - mean (850ms)  : 823, 878
     .   : milestone, 850,
    master - mean (845ms)  : 815, 876
     .   : milestone, 845,

andrewlock · 2025-02-26T18:47:47Z

Benchmarks Report for tracer 🐌

Benchmarks for #6719 compared to master:

1 benchmarks are faster, with geometric mean 1.130
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.14μs	46.6ns	373ns	0.0122	0.00407	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.5μs	60.3ns	490ns	0.0206	0.0103	0	5.8 KB
master	`StartStopWithChild`	net472	16.2μs	46ns	178ns	1.04	0.314	0.105	6.21 KB
#6719	`StartStopWithChild`	net6.0	8.33μs	45ns	305ns	0.0132	0.0044	0	5.61 KB
#6719	`StartStopWithChild`	netcoreapp3.1	10.5μs	60.1ns	485ns	0.0161	0.00536	0	5.8 KB
#6719	`StartStopWithChild`	net472	16.1μs	37.7ns	146ns	1.03	0.293	0.0895	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	510μs	438ns	1.64μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	665μs	487ns	1.82μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	845μs	472ns	1.77μs	0.422	3.3 KB
#6719	`WriteAndFlushEnrichedTraces`	net6.0	491μs	264ns	989ns	0	2.7 KB
#6719	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	654μs	221ns	855ns	0	2.7 KB
#6719	`WriteAndFlushEnrichedTraces`	net472	838μs	644ns	2.49μs	0.419	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	132μs	488ns	1.89μs	0.131	14.47 KB
master	`SendRequest`	netcoreapp3.1	150μs	189ns	732ns	0.224	17.27 KB
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#6719	`SendRequest`	net6.0	134μs	318ns	1.23μs	0.135	14.47 KB
#6719	`SendRequest`	netcoreapp3.1	154μs	200ns	776ns	0.229	17.27 KB
#6719	`SendRequest`	net472	7.91E‑05ns	7.91E‑05ns	0.000306ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	571μs	3μs	15μs	0.581	0	0	41.68 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	657μs	2.21μs	8.28μs	0.334	0	0	41.66 KB
master	`WriteAndFlushEnrichedTraces`	net472	843μs	4.01μs	16μs	8.62	2.59	0.431	53.26 KB
#6719	`WriteAndFlushEnrichedTraces`	net6.0	574μs	3μs	14.4μs	0.566	0	0	41.59 KB
#6719	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	678μs	3.07μs	19.2μs	0.329	0	0	41.75 KB
#6719	`WriteAndFlushEnrichedTraces`	net472	840μs	3.67μs	13.2μs	8.17	2.45	0.408	53.26 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.38μs	3.17ns	12.3ns	0.0146	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.85μs	1.33ns	5.17ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.06μs	1.12ns	4.33ns	0.157	0.00103	987 B
#6719	`ExecuteNonQuery`	net6.0	1.34μs	1.43ns	5.53ns	0.014	0	1.02 KB
#6719	`ExecuteNonQuery`	netcoreapp3.1	1.8μs	3.55ns	13.8ns	0.0135	0	1.02 KB
#6719	`ExecuteNonQuery`	net472	2.14μs	0.857ns	3.32ns	0.156	0.00108	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.25μs	0.566ns	2.19ns	0.0136	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.53μs	0.542ns	2.03ns	0.013	976 B
master	`CallElasticsearch`	net472	2.51μs	1.26ns	4.86ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.36μs	0.897ns	3.47ns	0.0129	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.65μs	0.506ns	1.75ns	0.014	1.02 KB
master	`CallElasticsearchAsync`	net472	2.71μs	3.53ns	12.7ns	0.167	1.05 KB
#6719	`CallElasticsearch`	net6.0	1.27μs	0.417ns	1.56ns	0.0139	976 B
#6719	`CallElasticsearch`	netcoreapp3.1	1.58μs	0.659ns	2.38ns	0.0129	976 B
#6719	`CallElasticsearch`	net472	2.51μs	2.02ns	7.84ns	0.157	995 B
#6719	`CallElasticsearchAsync`	net6.0	1.26μs	0.589ns	2.2ns	0.0132	952 B
#6719	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	2.21ns	8.54ns	0.0138	1.02 KB
#6719	`CallElasticsearchAsync`	net472	2.48μs	1.7ns	6.59ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.42μs	0.499ns	1.87ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.66μs	1.16ns	4.48ns	0.0125	952 B
master	`ExecuteAsync`	net472	1.84μs	0.382ns	1.43ns	0.145	915 B
#6719	`ExecuteAsync`	net6.0	1.41μs	0.534ns	1.93ns	0.0134	952 B
#6719	`ExecuteAsync`	netcoreapp3.1	1.64μs	0.57ns	2.13ns	0.0124	952 B
#6719	`ExecuteAsync`	net472	1.87μs	0.669ns	2.51ns	0.144	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.29μs	4.66ns	16.8ns	0.0323	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.44μs	4.27ns	16.6ns	0.0382	2.85 KB
master	`SendAsync`	net472	7.43μs	1.33ns	4.97ns	0.493	3.12 KB
#6719	`SendAsync`	net6.0	4.5μs	3.15ns	11.4ns	0.0314	2.31 KB
#6719	`SendAsync`	netcoreapp3.1	5.36μs	2.3ns	8.92ns	0.0376	2.85 KB
#6719	`SendAsync`	net472	7.52μs	2.4ns	8.98ns	0.496	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.59μs	0.583ns	2.18ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.12μs	0.64ns	2.31ns	0.0224	1.64 KB
master	`EnrichedLog`	net472	2.57μs	2.38ns	9.2ns	0.25	1.57 KB
#6719	`EnrichedLog`	net6.0	1.56μs	0.741ns	2.77ns	0.0226	1.64 KB
#6719	`EnrichedLog`	netcoreapp3.1	2.15μs	0.686ns	2.57ns	0.0226	1.64 KB
#6719	`EnrichedLog`	net472	2.6μs	1.06ns	3.97ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	114μs	218ns	844ns	0.0568	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	119μs	132ns	512ns	0	0	4.28 KB
master	`EnrichedLog`	net472	152μs	133ns	514ns	0.68	0.227	4.46 KB
#6719	`EnrichedLog`	net6.0	114μs	100ns	376ns	0.0572	0	4.28 KB
#6719	`EnrichedLog`	netcoreapp3.1	117μs	137ns	531ns	0	0	4.28 KB
#6719	`EnrichedLog`	net472	150μs	141ns	547ns	0.669	0.223	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.01μs	0.682ns	2.64ns	0.0302	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.23μs	2.49ns	9.63ns	0.0298	2.2 KB
master	`EnrichedLog`	net472	4.94μs	1.28ns	4.97ns	0.32	2.02 KB
#6719	`EnrichedLog`	net6.0	3.03μs	0.994ns	3.72ns	0.0304	2.2 KB
#6719	`EnrichedLog`	netcoreapp3.1	4.29μs	1.63ns	6.11ns	0.03	2.2 KB
#6719	`EnrichedLog`	net472	4.84μs	1.36ns	5.28ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.43μs	1.04ns	4.03ns	0.0158	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.75μs	0.812ns	3.14ns	0.0158	1.14 KB
master	`SendReceive`	net472	2.09μs	1.82ns	7.05ns	0.184	1.16 KB
#6719	`SendReceive`	net6.0	1.43μs	0.389ns	1.45ns	0.0165	1.14 KB
#6719	`SendReceive`	netcoreapp3.1	1.9μs	0.565ns	2.11ns	0.0152	1.14 KB
#6719	`SendReceive`	net472	2.1μs	1.42ns	5.52ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.91μs	0.744ns	2.79ns	0.0217	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.88μs	5.73ns	22.2ns	0.0216	1.65 KB
master	`EnrichedLog`	net472	4.25μs	4.3ns	16.6ns	0.323	2.04 KB
#6719	`EnrichedLog`	net6.0	2.87μs	1.62ns	6.06ns	0.0218	1.6 KB
#6719	`EnrichedLog`	netcoreapp3.1	3.95μs	0.782ns	2.71ns	0.0215	1.65 KB
#6719	`EnrichedLog`	net472	4.31μs	2.54ns	9.82ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6719

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.130	677.57	599.84

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	387ns	0.571ns	2.21ns	0.00809	576 B
master	`StartFinishSpan`	netcoreapp3.1	624ns	1.26ns	4.87ns	0.00773	576 B
master	`StartFinishSpan`	net472	677ns	1.06ns	4.09ns	0.0915	578 B
master	`StartFinishScope`	net6.0	477ns	0.556ns	2.15ns	0.00975	696 B
master	`StartFinishScope`	netcoreapp3.1	667ns	0.721ns	2.7ns	0.00923	696 B
master	`StartFinishScope`	net472	757ns	1.46ns	5.25ns	0.104	658 B
#6719	`StartFinishSpan`	net6.0	402ns	0.392ns	1.52ns	0.00817	576 B
#6719	`StartFinishSpan`	netcoreapp3.1	610ns	1.59ns	6.15ns	0.00798	576 B
#6719	`StartFinishSpan`	net472	598ns	1.35ns	5.22ns	0.0916	578 B
#6719	`StartFinishScope`	net6.0	497ns	0.843ns	3.26ns	0.00966	696 B
#6719	`StartFinishScope`	netcoreapp3.1	740ns	1.85ns	7.17ns	0.00934	696 B
#6719	`StartFinishScope`	net472	810ns	1.68ns	6.52ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	660ns	1.5ns	5.82ns	0.00986	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	899ns	1.47ns	5.7ns	0.00954	696 B
master	`RunOnMethodBegin`	net472	1.11μs	2.56ns	9.9ns	0.104	658 B
#6719	`RunOnMethodBegin`	net6.0	673ns	0.883ns	3.42ns	0.00974	696 B
#6719	`RunOnMethodBegin`	netcoreapp3.1	980ns	1.74ns	6.73ns	0.00928	696 B
#6719	`RunOnMethodBegin`	net472	1.08μs	2.22ns	8.59ns	0.105	658 B

nhulston added 3 commits February 26, 2025 12:48

pass span when making sampling decision in lambda

8b76e07

update TraceContext to have GetOrMakeSamplingDecision(Span? span) o…

2af11ec

…verloaded method

update SpanContext.cs comment

f091945

fix LambdaRequestBuilderTests

89a23d9

nhulston changed the title ~~[wip] Send sampling priority to lambda extension~~ Send sampling priority to lambda extension Feb 26, 2025

nhulston marked this pull request as ready for review February 26, 2025 21:48

nhulston requested review from a team as code owners February 26, 2025 21:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Send sampling priority to lambda extension #6719

Send sampling priority to lambda extension #6719

nhulston commented Feb 26, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Feb 26, 2025 •

edited

Loading

andrewlock commented Feb 26, 2025 •

edited

Loading

andrewlock commented Feb 26, 2025 •

edited

Loading

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6719

Raw results

Raw results

Send sampling priority to lambda extension #6719

Are you sure you want to change the base?

Send sampling priority to lambda extension #6719

Conversation

nhulston commented Feb 26, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-datadog-prod-us1 bot commented Feb 26, 2025 • edited Loading

Datadog Report

andrewlock commented Feb 26, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Feb 26, 2025 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6719

Raw results

Raw results

nhulston commented Feb 26, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Feb 26, 2025 •

edited

Loading

andrewlock commented Feb 26, 2025 •

edited

Loading

andrewlock commented Feb 26, 2025 •

edited

Loading