regenerate solutions as part of building the managed tracer #6608

vandonr · 2025-01-30T14:49:21Z

Summary of changes

the target RegenerateSolutions was not called by anything else. Added it to BuildManagedTracerHome so that it'd run on BuildTracerHome, which I assume(d) would regenerate everything that needs to

andrewlock · 2025-01-30T15:24:47Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6608) - mean (69ms)  : 67, 71
     .   : milestone, 69,
    master - mean (69ms)  : 65, 74
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6608) - mean (982ms)  : 952, 1012
     .   : milestone, 982,
    master - mean (981ms)  : 959, 1003
     .   : milestone, 981,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6608) - mean (108ms)  : 105, 110
     .   : milestone, 108,
    master - mean (108ms)  : 106, 111
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (6608) - mean (676ms)  : 660, 691
     .   : milestone, 676,
    master - mean (676ms)  : 660, 691
     .   : milestone, 676,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6608) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6608) - mean (630ms)  : 610, 649
     .   : milestone, 630,
    master - mean (637ms)  : 614, 660
     .   : milestone, 637,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6608) - mean (190ms)  : 187, 194
     .   : milestone, 190,
    master - mean (190ms)  : 185, 195
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6608) - mean (1,093ms)  : 1053, 1133
     .   : milestone, 1093,
    master - mean (1,087ms)  : 1064, 1110
     .   : milestone, 1087,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6608) - mean (277ms)  : 273, 281
     .   : milestone, 277,
    master - mean (276ms)  : 272, 281
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (6608) - mean (871ms)  : 837, 906
     .   : milestone, 871,
    master - mean (865ms)  : 832, 897
     .   : milestone, 865,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6608) - mean (265ms)  : 260, 270
     .   : milestone, 265,
    master - mean (264ms)  : 260, 268
     .   : milestone, 264,

    section CallTarget+Inlining+NGEN
    This PR (6608) - mean (849ms)  : 820, 877
     .   : milestone, 849,
    master - mean (846ms)  : 807, 885
     .   : milestone, 846,

datadog-ddstaging · 2025-01-30T15:44:47Z

Datadog Report

Branch report: vandonr/build
Commit report: 7996a40
Test service: dd-trace-dotnet

✅ 0 Failed, 246305 Passed, 2088 Skipped, 18h 59m 57.54s Total Time

andrewlock · 2025-01-30T16:14:46Z

Benchmarks Report for tracer 🐌

Benchmarks for #6608 compared to master:

1 benchmarks are faster, with geometric mean 1.211
2 benchmarks are slower, with geometric mean 1.144
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.98μs	45.1ns	296ns	0.0152	0.0076	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.1μs	55.5ns	314ns	0.0207	0.0103	0	5.8 KB
master	`StartStopWithChild`	net472	16.1μs	50.1ns	194ns	1.05	0.326	0.0953	6.21 KB
#6608	`StartStopWithChild`	net6.0	7.93μs	43.1ns	299ns	0.0158	0.00791	0	5.61 KB
#6608	`StartStopWithChild`	netcoreapp3.1	10.1μs	52.7ns	269ns	0.0148	0.00492	0	5.81 KB
#6608	`StartStopWithChild`	net472	16μs	34.8ns	130ns	1.03	0.297	0.0883	6.2 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	485μs	285ns	1.11μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	655μs	413ns	1.6μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	857μs	620ns	2.15μs	0.425	3.3 KB
#6608	`WriteAndFlushEnrichedTraces`	net6.0	489μs	343ns	1.33μs	0	2.7 KB
#6608	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	635μs	363ns	1.36μs	0	2.7 KB
#6608	`WriteAndFlushEnrichedTraces`	net472	854μs	803ns	3.11μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	127μs	595ns	2.38μs	0.194	14.47 KB
master	`SendRequest`	netcoreapp3.1	144μs	637ns	2.47μs	0.217	17.27 KB
master	`SendRequest`	net472	0.00201ns	0.000798ns	0.00299ns	0	0 b
#6608	`SendRequest`	net6.0	127μs	377ns	1.46μs	0.186	14.47 KB
#6608	`SendRequest`	netcoreapp3.1	147μs	296ns	1.15μs	0.217	17.27 KB
#6608	`SendRequest`	net472	0.00768ns	0.00261ns	0.00978ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	571μs	2.95μs	13.5μs	0.541	0	0	41.73 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	663μs	3.6μs	21μs	0.334	0	0	41.76 KB
master	`WriteAndFlushEnrichedTraces`	net472	837μs	2.99μs	11.2μs	8.08	2.55	0.425	53.29 KB
#6608	`WriteAndFlushEnrichedTraces`	net6.0	543μs	2.66μs	11.3μs	0.558	0	0	41.57 KB
#6608	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	664μs	3.78μs	27.8μs	0.321	0	0	41.64 KB
#6608	`WriteAndFlushEnrichedTraces`	net472	852μs	3.72μs	13.9μs	8.5	2.55	0.425	53.3 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.41μs	1.25ns	4.67ns	0.0147	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.82μs	2ns	7.76ns	0.0136	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.01μs	1.01ns	3.64ns	0.156	0.00101	987 B
#6608	`ExecuteNonQuery`	net6.0	1.42μs	1.37ns	5.31ns	0.0141	0	1.02 KB
#6608	`ExecuteNonQuery`	netcoreapp3.1	1.73μs	1.53ns	5.73ns	0.0139	0	1.02 KB
#6608	`ExecuteNonQuery`	net472	2.04μs	1.26ns	4.72ns	0.157	0.00102	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6608

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0	1.211	1,354.66	1,118.45

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.35μs	0.54ns	2.09ns	0.0136	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.64μs	0.932ns	3.36ns	0.0132	976 B
master	`CallElasticsearch`	net472	2.57μs	1.65ns	6.37ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.26μs	0.981ns	3.54ns	0.0132	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.59μs	0.944ns	3.4ns	0.0142	1.02 KB
master	`CallElasticsearchAsync`	net472	2.66μs	1.12ns	4.04ns	0.166	1.05 KB
#6608	`CallElasticsearch`	net6.0	1.12μs	0.966ns	3.61ns	0.0139	976 B
#6608	`CallElasticsearch`	netcoreapp3.1	1.58μs	0.869ns	3.25ns	0.0134	976 B
#6608	`CallElasticsearch`	net472	2.58μs	2.78ns	10.7ns	0.158	995 B
#6608	`CallElasticsearchAsync`	net6.0	1.24μs	0.855ns	3.31ns	0.013	952 B
#6608	`CallElasticsearchAsync`	netcoreapp3.1	1.7μs	1.36ns	5.28ns	0.0136	1.02 KB
#6608	`CallElasticsearchAsync`	net472	2.61μs	1.87ns	7.25ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.35μs	0.872ns	3.26ns	0.0135	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.72μs	2.08ns	7.8ns	0.012	952 B
master	`ExecuteAsync`	net472	1.85μs	0.467ns	1.81ns	0.145	915 B
#6608	`ExecuteAsync`	net6.0	1.22μs	0.975ns	3.65ns	0.013	952 B
#6608	`ExecuteAsync`	netcoreapp3.1	1.63μs	0.764ns	2.86ns	0.013	952 B
#6608	`ExecuteAsync`	net472	1.79μs	5.53ns	19.9ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.43μs	1.39ns	5.01ns	0.031	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.17μs	2.98ns	11.5ns	0.0386	2.85 KB
master	`SendAsync`	net472	7.36μs	1.09ns	4.07ns	0.494	3.12 KB
#6608	`SendAsync`	net6.0	4.38μs	1.38ns	5.36ns	0.0329	2.31 KB
#6608	`SendAsync`	netcoreapp3.1	5.3μs	2.66ns	10.3ns	0.0365	2.85 KB
#6608	`SendAsync`	net472	7.41μs	1.68ns	6.3ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.45μs	0.879ns	3.41ns	0.0231	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.06μs	0.847ns	3.05ns	0.0218	1.64 KB
master	`EnrichedLog`	net472	2.65μs	0.735ns	2.65ns	0.249	1.57 KB
#6608	`EnrichedLog`	net6.0	1.45μs	0.633ns	2.37ns	0.0233	1.64 KB
#6608	`EnrichedLog`	netcoreapp3.1	2.23μs	2.5ns	9.7ns	0.0223	1.64 KB
#6608	`EnrichedLog`	net472	2.62μs	0.336ns	1.21ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	116μs	76.3ns	285ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	119μs	109ns	379ns	0	0	4.28 KB
master	`EnrichedLog`	net472	150μs	94.2ns	365ns	0.679	0.226	4.46 KB
#6608	`EnrichedLog`	net6.0	115μs	107ns	413ns	0	0	4.28 KB
#6608	`EnrichedLog`	netcoreapp3.1	119μs	128ns	478ns	0	0	4.28 KB
#6608	`EnrichedLog`	net472	151μs	106ns	412ns	0.677	0.226	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.05μs	2.17ns	8.13ns	0.0306	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.26μs	0.929ns	3.35ns	0.0297	2.2 KB
master	`EnrichedLog`	net472	4.85μs	1.77ns	6.86ns	0.319	2.02 KB
#6608	`EnrichedLog`	net6.0	3.16μs	1.02ns	3.93ns	0.03	2.2 KB
#6608	`EnrichedLog`	netcoreapp3.1	4.18μs	0.973ns	3.51ns	0.0294	2.2 KB
#6608	`EnrichedLog`	net472	4.77μs	1.52ns	5.87ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.34μs	0.537ns	1.94ns	0.0159	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.74μs	1.94ns	7.49ns	0.0156	1.14 KB
master	`SendReceive`	net472	2.14μs	1.04ns	4.03ns	0.183	1.16 KB
#6608	`SendReceive`	net6.0	1.39μs	2.38ns	9.21ns	0.0158	1.14 KB
#6608	`SendReceive`	netcoreapp3.1	1.78μs	1.17ns	4.52ns	0.0151	1.14 KB
#6608	`SendReceive`	net472	2.1μs	1.35ns	5.04ns	0.184	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.81μs	1.08ns	4.19ns	0.0225	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.85μs	1.39ns	5.38ns	0.0231	1.65 KB
master	`EnrichedLog`	net472	4.47μs	2.69ns	10.4ns	0.323	2.04 KB
#6608	`EnrichedLog`	net6.0	2.74μs	0.657ns	2.37ns	0.0219	1.6 KB
#6608	`EnrichedLog`	netcoreapp3.1	3.82μs	1.13ns	4.38ns	0.0229	1.65 KB
#6608	`EnrichedLog`	net472	4.42μs	2.84ns	10.6ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6608

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	1.160	697.07	808.48
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.129	486.69	549.54

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	394ns	0.68ns	2.63ns	0.00797	576 B
master	`StartFinishSpan`	netcoreapp3.1	549ns	0.822ns	3.18ns	0.00775	576 B
master	`StartFinishSpan`	net472	633ns	1.31ns	5.09ns	0.0915	578 B
master	`StartFinishScope`	net6.0	487ns	1.12ns	4.35ns	0.00976	696 B
master	`StartFinishScope`	netcoreapp3.1	698ns	1.82ns	6.8ns	0.00952	696 B
master	`StartFinishScope`	net472	827ns	1.31ns	5.08ns	0.104	658 B
#6608	`StartFinishSpan`	net6.0	408ns	0.86ns	3.33ns	0.00818	576 B
#6608	`StartFinishSpan`	netcoreapp3.1	549ns	0.7ns	2.71ns	0.00785	576 B
#6608	`StartFinishSpan`	net472	602ns	1.26ns	4.88ns	0.0918	578 B
#6608	`StartFinishScope`	net6.0	550ns	0.772ns	2.99ns	0.0096	696 B
#6608	`StartFinishScope`	netcoreapp3.1	808ns	0.981ns	3.8ns	0.0094	696 B
#6608	`StartFinishScope`	net472	824ns	0.553ns	2.14ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	694ns	0.825ns	3.2ns	0.00981	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	903ns	1.37ns	5.29ns	0.00918	696 B
master	`RunOnMethodBegin`	net472	1.04μs	1.37ns	5.3ns	0.104	658 B
#6608	`RunOnMethodBegin`	net6.0	753ns	1.83ns	7.1ns	0.00986	696 B
#6608	`RunOnMethodBegin`	netcoreapp3.1	906ns	1.32ns	5.11ns	0.00948	696 B
#6608	`RunOnMethodBegin`	net472	1.03μs	1ns	3.87ns	0.104	658 B

## Summary of changes the target `RegenerateSolutions` was not called by anything else. Added it to `BuildManagedTracerHome` so that it'd run on `BuildTracerHome`, which I assume(d) would regenerate everything that needs to

regenerate solutions as part of building the managed tracer

7996a40

vandonr requested a review from a team as a code owner January 30, 2025 14:49

github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jan 30, 2025

bouwkast approved these changes Jan 30, 2025

View reviewed changes

andrewlock approved these changes Jan 30, 2025

View reviewed changes

vandonr merged commit 5bfb641 into master Jan 30, 2025
136 of 139 checks passed

vandonr deleted the vandonr/build branch January 30, 2025 16:20

github-actions bot added this to the vNext-v3 milestone Jan 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

regenerate solutions as part of building the managed tracer #6608

regenerate solutions as part of building the managed tracer #6608

vandonr commented Jan 30, 2025

andrewlock commented Jan 30, 2025

datadog-ddstaging bot commented Jan 30, 2025 •

edited

Loading

andrewlock commented Jan 30, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6608

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6608

Raw results

Raw results

regenerate solutions as part of building the managed tracer #6608

regenerate solutions as part of building the managed tracer #6608

Conversation

vandonr commented Jan 30, 2025

Summary of changes

andrewlock commented Jan 30, 2025

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Jan 30, 2025 • edited Loading

Datadog Report

andrewlock commented Jan 30, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6608

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6608

Raw results

Raw results

datadog-ddstaging bot commented Jan 30, 2025 •

edited

Loading