[Profiler] Properly initialize appDomainId #6630

kevingosse · 2025-02-05T15:12:44Z

Summary of changes

The contention profiler and the allocation profiler don't set the appDomainId if the managed thread can't be found (it can happen, for instance, if it has exited). Because the RawSample isn't zero'd at construction, this causes the field to contain garbage, which can cause crashes down the line.

Also, some error paths set the appDomainId to -1. I changed it to 0 because it's the value expected by the code that uses it and by the CLR.

Reason for change

This was found thanks to a crash dump in the CI, but there has been crash reports that we failed to understand until now.

gleocadie

LGTM

andrewlock · 2025-02-05T15:52:36Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6630) - mean (69ms)  : 66, 72
     .   : milestone, 69,
    master - mean (69ms)  : 65, 73
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6630) - mean (1,000ms)  : 977, 1023
     .   : milestone, 1000,
    master - mean (1,001ms)  : 978, 1024
     .   : milestone, 1001,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6630) - mean (102ms)  : 100, 104
     .   : milestone, 102,
    master - mean (102ms)  : 100, 105
     .   : milestone, 102,

    section CallTarget+Inlining+NGEN
    This PR (6630) - mean (681ms)  : 662, 700
     .   : milestone, 681,
    master - mean (676ms)  : 659, 693
     .   : milestone, 676,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6630) - mean (89ms)  : 87, 91
     .   : milestone, 89,
    master - mean (89ms)  : 87, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (6630) - mean (632ms)  : 615, 650
     .   : milestone, 632,
    master - mean (632ms)  : 616, 648
     .   : milestone, 632,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6630) - mean (191ms)  : 186, 195
     .   : milestone, 191,
    master - mean (191ms)  : 187, 195
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6630) - mean (1,114ms)  : 1087, 1142
     .   : milestone, 1114,
    master - mean (1,108ms)  : 1072, 1144
     .   : milestone, 1108,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6630) - mean (272ms)  : 267, 277
     .   : milestone, 272,
    master - mean (270ms)  : 264, 276
     .   : milestone, 270,

    section CallTarget+Inlining+NGEN
    This PR (6630) - mean (914ms)  : 714, 1114
     .   : milestone, 914,
    master - mean (863ms)  : 834, 892
     .   : milestone, 863,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6630) - mean (264ms)  : 258, 269
     .   : milestone, 264,
    master - mean (263ms)  : 259, 267
     .   : milestone, 263,

    section CallTarget+Inlining+NGEN
    This PR (6630) - mean (848ms)  : 817, 879
     .   : milestone, 848,
    master - mean (844ms)  : 806, 882
     .   : milestone, 844,

datadog-ddstaging · 2025-02-05T16:04:35Z

Datadog Report

Branch report: kevin/appdomainid
Commit report: ec12bb9
Test service: dd-trace-dotnet

❌ 39 Failed (0 Known Flaky), 245315 Passed, 2855 Skipped, 20h 49m 6.18s Total Time

❌ Failed Tests (39)

This report shows up to 5 failed tests.

IntegrationDisabled - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.MicrosoftDataSqlClientTests - Details
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.MicrosoftDataSqlClientTests - Details
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.MicrosoftDataSqlClientTests - Details
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.MicrosoftDataSqlClientTests - Details
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.MicrosoftDataSqlClientTests - Details
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```

andrewlock · 2025-02-05T20:28:06Z

Benchmarks Report for tracer 🐌

Benchmarks for #6630 compared to master:

All benchmarks have the same speed
1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.86μs	43ns	262ns	0.0203	0.00811	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.1μs	55.5ns	324ns	0.0199	0.00996	0	5.8 KB
master	`StartStopWithChild`	net472	15.9μs	38.1ns	142ns	1.04	0.307	0.0944	6.21 KB
#6630	`StartStopWithChild`	net6.0	7.81μs	44.2ns	334ns	0.0185	0.0074	0	5.61 KB
#6630	`StartStopWithChild`	netcoreapp3.1	10μs	55.9ns	340ns	0.0153	0.0051	0	5.8 KB
#6630	`StartStopWithChild`	net472	16.1μs	49.8ns	193ns	1.02	0.285	0.0813	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	492μs	486ns	1.82μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	700μs	282ns	1.05μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	848μs	626ns	2.34μs	0.422	3.3 KB
#6630	`WriteAndFlushEnrichedTraces`	net6.0	488μs	427ns	1.65μs	0	2.7 KB
#6630	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	664μs	369ns	1.43μs	0	2.7 KB
#6630	`WriteAndFlushEnrichedTraces`	net472	857μs	536ns	1.93μs	0.422	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	132μs	417ns	1.61μs	0.135	14.47 KB
master	`SendRequest`	netcoreapp3.1	147μs	396ns	1.53μs	0.22	17.27 KB
master	`SendRequest`	net472	0.00199ns	0.000288ns	0.00112ns	0	0 b
#6630	`SendRequest`	net6.0	132μs	232ns	867ns	0.198	14.47 KB
#6630	`SendRequest`	netcoreapp3.1	148μs	217ns	784ns	0.221	17.27 KB
#6630	`SendRequest`	net472	0.000681ns	0.000336ns	0.0013ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #6630

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.87 KB	41.53 KB	-340 B	-0.81%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	608μs	3.32μs	18.8μs	0.604	0	0	41.87 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	670μs	3.78μs	27.5μs	0.338	0	0	41.69 KB
master	`WriteAndFlushEnrichedTraces`	net472	840μs	2.83μs	10.6μs	8.22	2.47	0.411	53.34 KB
#6630	`WriteAndFlushEnrichedTraces`	net6.0	570μs	2.64μs	10.2μs	0.563	0	0	41.53 KB
#6630	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	665μs	3.6μs	19.7μs	0.324	0	0	41.74 KB
#6630	`WriteAndFlushEnrichedTraces`	net472	862μs	2.64μs	9.51μs	8.42	2.66	0.443	53.27 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.44μs	2.09ns	8.1ns	0.0138	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.75μs	2.38ns	9.22ns	0.0139	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.07μs	4.15ns	16.1ns	0.156	0.00103	987 B
#6630	`ExecuteNonQuery`	net6.0	1.31μs	1.69ns	6.56ns	0.0145	0	1.02 KB
#6630	`ExecuteNonQuery`	netcoreapp3.1	1.78μs	1.21ns	4.7ns	0.0135	0	1.02 KB
#6630	`ExecuteNonQuery`	net472	2.06μs	2.29ns	8.57ns	0.156	0.00103	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.21μs	0.404ns	1.51ns	0.0139	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.5μs	0.958ns	3.59ns	0.0128	976 B
master	`CallElasticsearch`	net472	2.46μs	2.36ns	9.14ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.37μs	1.01ns	3.76ns	0.013	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.71μs	0.965ns	3.61ns	0.0139	1.02 KB
master	`CallElasticsearchAsync`	net472	2.67μs	1.97ns	7.63ns	0.166	1.05 KB
#6630	`CallElasticsearch`	net6.0	1.29μs	0.562ns	2.1ns	0.0132	976 B
#6630	`CallElasticsearch`	netcoreapp3.1	1.51μs	1.06ns	3.81ns	0.0129	976 B
#6630	`CallElasticsearch`	net472	2.44μs	2.26ns	8.47ns	0.158	995 B
#6630	`CallElasticsearchAsync`	net6.0	1.26μs	0.328ns	1.14ns	0.0132	952 B
#6630	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	0.988ns	3.7ns	0.0139	1.02 KB
#6630	`CallElasticsearchAsync`	net472	2.74μs	1.48ns	5.72ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.44μs	1.02ns	3.96ns	0.013	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.71μs	0.954ns	3.7ns	0.012	952 B
master	`ExecuteAsync`	net472	1.86μs	0.559ns	2.09ns	0.145	915 B
#6630	`ExecuteAsync`	net6.0	1.38μs	0.699ns	2.71ns	0.0131	952 B
#6630	`ExecuteAsync`	netcoreapp3.1	1.69μs	1ns	3.89ns	0.0124	952 B
#6630	`ExecuteAsync`	net472	1.93μs	0.634ns	2.37ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.41μs	1.19ns	4.47ns	0.0331	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.35μs	3.47ns	13ns	0.0374	2.85 KB
master	`SendAsync`	net472	7.37μs	2.03ns	7.85ns	0.495	3.12 KB
#6630	`SendAsync`	net6.0	4.41μs	1.84ns	6.89ns	0.0334	2.31 KB
#6630	`SendAsync`	netcoreapp3.1	5.36μs	3.49ns	13.5ns	0.0374	2.85 KB
#6630	`SendAsync`	net472	7.5μs	1.84ns	7.12ns	0.494	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.58μs	0.493ns	1.78ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.19μs	1.07ns	3.99ns	0.0228	1.64 KB
master	`EnrichedLog`	net472	2.67μs	4.35ns	16.9ns	0.25	1.57 KB
#6630	`EnrichedLog`	net6.0	1.55μs	1.07ns	4ns	0.0234	1.64 KB
#6630	`EnrichedLog`	netcoreapp3.1	2.22μs	0.917ns	3.43ns	0.0223	1.64 KB
#6630	`EnrichedLog`	net472	2.66μs	1.73ns	6.7ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	112μs	212ns	822ns	0.0558	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	117μs	148ns	573ns	0.0582	0	4.28 KB
master	`EnrichedLog`	net472	151μs	82.1ns	307ns	0.681	0.227	4.46 KB
#6630	`EnrichedLog`	net6.0	111μs	138ns	515ns	0.056	0	4.28 KB
#6630	`EnrichedLog`	netcoreapp3.1	116μs	146ns	546ns	0.0582	0	4.28 KB
#6630	`EnrichedLog`	net472	149μs	85.1ns	295ns	0.669	0.223	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.11μs	0.496ns	1.72ns	0.0311	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.2μs	2.35ns	8.79ns	0.0294	2.2 KB
master	`EnrichedLog`	net472	5.03μs	1.01ns	3.79ns	0.32	2.02 KB
#6630	`EnrichedLog`	net6.0	2.91μs	1.93ns	7.22ns	0.0305	2.2 KB
#6630	`EnrichedLog`	netcoreapp3.1	4.22μs	2.14ns	8.29ns	0.0295	2.2 KB
#6630	`EnrichedLog`	net472	5.1μs	1.29ns	4.99ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.27μs	0.38ns	1.47ns	0.0159	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.77μs	0.486ns	1.68ns	0.015	1.14 KB
master	`SendReceive`	net472	2.12μs	2.85ns	11.1ns	0.183	1.16 KB
#6630	`SendReceive`	net6.0	1.38μs	0.67ns	2.59ns	0.0158	1.14 KB
#6630	`SendReceive`	netcoreapp3.1	1.74μs	3.8ns	14.7ns	0.0156	1.14 KB
#6630	`SendReceive`	net472	2.12μs	0.617ns	2.22ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.77μs	1.19ns	4.44ns	0.022	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.97μs	5.6ns	20.9ns	0.0217	1.65 KB
master	`EnrichedLog`	net472	4.3μs	3.26ns	12.6ns	0.323	2.04 KB
#6630	`EnrichedLog`	net6.0	2.75μs	0.97ns	3.76ns	0.0221	1.6 KB
#6630	`EnrichedLog`	netcoreapp3.1	3.97μs	1.89ns	7.31ns	0.0219	1.65 KB
#6630	`EnrichedLog`	net472	4.36μs	3.67ns	14.2ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	391ns	0.461ns	1.79ns	0.00817	576 B
master	`StartFinishSpan`	netcoreapp3.1	557ns	0.82ns	3.18ns	0.00784	576 B
master	`StartFinishSpan`	net472	584ns	1.27ns	4.91ns	0.0915	578 B
master	`StartFinishScope`	net6.0	515ns	0.487ns	1.89ns	0.00983	696 B
master	`StartFinishScope`	netcoreapp3.1	742ns	1.68ns	6.07ns	0.00932	696 B
master	`StartFinishScope`	net472	839ns	2.27ns	8.78ns	0.105	658 B
#6630	`StartFinishSpan`	net6.0	405ns	0.609ns	2.36ns	0.00816	576 B
#6630	`StartFinishSpan`	netcoreapp3.1	585ns	0.903ns	3.5ns	0.00788	576 B
#6630	`StartFinishSpan`	net472	564ns	0.906ns	3.51ns	0.0916	578 B
#6630	`StartFinishScope`	net6.0	507ns	0.453ns	1.75ns	0.00978	696 B
#6630	`StartFinishScope`	netcoreapp3.1	679ns	1.08ns	4.19ns	0.00937	696 B
#6630	`StartFinishScope`	net472	773ns	2.07ns	8.03ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	733ns	1.3ns	5.05ns	0.00951	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	907ns	1.48ns	5.73ns	0.00936	696 B
master	`RunOnMethodBegin`	net472	1.16μs	1.52ns	5.88ns	0.105	658 B
#6630	`RunOnMethodBegin`	net6.0	685ns	1.31ns	5.07ns	0.00964	696 B
#6630	`RunOnMethodBegin`	netcoreapp3.1	876ns	2.46ns	9.53ns	0.00919	696 B
#6630	`RunOnMethodBegin`	net472	1.08μs	1.72ns	6.67ns	0.104	658 B

kevingosse added type:bug area:profiler Issues related to the continous-profiler identified-by:crashtracking labels Feb 5, 2025

kevingosse requested a review from a team as a code owner February 5, 2025 15:12

andrewlock approved these changes Feb 5, 2025

View reviewed changes

gleocadie approved these changes Feb 5, 2025

View reviewed changes

Properly initialize appDomainId

c03aa18

gleocadie force-pushed the kevin/appdomainid branch from ec12bb9 to c03aa18 Compare February 19, 2025 09:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Profiler] Properly initialize appDomainId #6630

[Profiler] Properly initialize appDomainId #6630

kevingosse commented Feb 5, 2025

gleocadie left a comment

andrewlock commented Feb 5, 2025 •

edited

Loading

datadog-ddstaging bot commented Feb 5, 2025 •

edited

Loading

andrewlock commented Feb 5, 2025 •

edited

Loading

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6630

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

[Profiler] Properly initialize appDomainId #6630

Are you sure you want to change the base?

[Profiler] Properly initialize appDomainId #6630

Conversation

kevingosse commented Feb 5, 2025

Summary of changes

Reason for change

gleocadie left a comment

Choose a reason for hiding this comment

andrewlock commented Feb 5, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Feb 5, 2025 • edited Loading

Datadog Report

❌ Failed Tests (39)

andrewlock commented Feb 5, 2025 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6630

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

andrewlock commented Feb 5, 2025 •

edited

Loading

datadog-ddstaging bot commented Feb 5, 2025 •

edited

Loading

andrewlock commented Feb 5, 2025 •

edited

Loading