Skip to content

Commit 6a56abc

Browse files
thomhurstthomhurst
authored
chore: update benchmark results (#5175)
1 parent 95f26ee commit 6a56abc

18 files changed

Lines changed: 366 additions & 366 deletions

‎docs/docs/benchmarks/AsyncTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 2
77
# AsyncTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-03-16** from the latest CI run.
10+
This benchmark was automatically generated on **2026-03-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.201
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.19.57 | 556.2 ms | 556.4 ms | 10.41 ms |
20-
| NUnit | 4.5.1 | 715.8 ms | 717.0 ms | 6.10 ms |
21-
| MSTest | 4.1.0 | 646.7 ms | 645.3 ms | 4.57 ms |
22-
| xUnit3 | 3.2.2 | 780.8 ms | 782.3 ms | 5.29 ms |
23-
| **TUnit (AOT)** | 1.19.57 | 123.6 ms | 123.7 ms | 0.28 ms |
19+
| **TUnit** | 1.19.74 | 544.1 ms | 544.2 ms | 4.17 ms |
20+
| NUnit | 4.5.1 | 727.3 ms | 727.2 ms | 5.46 ms |
21+
| MSTest | 4.1.0 | 663.9 ms | 661.2 ms | 8.47 ms |
22+
| xUnit3 | 3.2.2 | 799.6 ms | 797.6 ms | 8.41 ms |
23+
| **TUnit (AOT)** | 1.19.74 | 123.4 ms | 123.4 ms | 0.14 ms |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
5858
xychart-beta
5959
title "AsyncTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 937
62-
bar [556.2, 715.8, 646.7, 780.8, 123.6]
61+
y-axis "Time (ms)" 0 --> 960
62+
bar [544.1, 727.3, 663.9, 799.6, 123.4]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-03-16T00:40:15.334Z*
75+
*Last generated: 2026-03-18T00:37:45.484Z*

‎docs/docs/benchmarks/BuildTime.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 8
77
# Build Performance Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-03-16** from the latest CI run.
10+
This benchmark was automatically generated on **2026-03-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.201
1313
:::
@@ -18,10 +18,10 @@ Compilation time comparison across frameworks:
1818

1919
| Framework | Version | Mean | Median | StdDev |
2020
|-----------|---------|------|--------|--------|
21-
| **TUnit** | 1.19.57 | 2.815 s | 2.808 s | 0.0275 s |
22-
| Build_NUnit | 4.5.1 | 2.536 s | 2.535 s | 0.0161 s |
23-
| Build_MSTest | 4.1.0 | 2.622 s | 2.618 s | 0.0252 s |
24-
| Build_xUnit3 | 3.2.2 | 2.528 s | 2.520 s | 0.0244 s |
21+
| **TUnit** | 1.19.74 | 2.814 s | 2.811 s | 0.0520 s |
22+
| Build_NUnit | 4.5.1 | 2.546 s | 2.547 s | 0.0297 s |
23+
| Build_MSTest | 4.1.0 | 2.588 s | 2.594 s | 0.0348 s |
24+
| Build_xUnit3 | 3.2.2 | 2.510 s | 2.511 s | 0.0334 s |
2525

2626
## 📈 Visual Comparison
2727

@@ -60,7 +60,7 @@ xychart-beta
6060
title "Build Time Comparison"
6161
x-axis ["Build_TUnit", "Build_NUnit", "Build_MSTest", "Build_xUnit3"]
6262
y-axis "Time (s)" 0 --> 4
63-
bar [2.815, 2.536, 2.622, 2.528]
63+
bar [2.814, 2.546, 2.588, 2.51]
6464
```
6565

6666
---
@@ -69,4 +69,4 @@ xychart-beta
6969
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7070
:::
7171

72-
*Last generated: 2026-03-16T00:40:15.337Z*
72+
*Last generated: 2026-03-18T00:37:45.486Z*

‎docs/docs/benchmarks/DataDrivenTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 3
77
# DataDrivenTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-03-16** from the latest CI run.
10+
This benchmark was automatically generated on **2026-03-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.201
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.19.57 | 471.01 ms | 470.80 ms | 5.509 ms |
20-
| NUnit | 4.5.1 | 601.13 ms | 602.09 ms | 10.413 ms |
21-
| MSTest | 4.1.0 | 604.81 ms | 602.20 ms | 12.744 ms |
22-
| xUnit3 | 3.2.2 | 649.09 ms | 646.62 ms | 8.654 ms |
23-
| **TUnit (AOT)** | 1.19.57 | 25.31 ms | 25.13 ms | 0.895 ms |
19+
| **TUnit** | 1.19.74 | 461.16 ms | 461.08 ms | 2.063 ms |
20+
| NUnit | 4.5.1 | 580.40 ms | 583.83 ms | 7.050 ms |
21+
| MSTest | 4.1.0 | 591.58 ms | 586.80 ms | 15.901 ms |
22+
| xUnit3 | 3.2.2 | 622.30 ms | 622.79 ms | 2.420 ms |
23+
| **TUnit (AOT)** | 1.19.74 | 25.03 ms | 24.91 ms | 1.283 ms |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
5858
xychart-beta
5959
title "DataDrivenTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 779
62-
bar [471.01, 601.13, 604.81, 649.09, 25.31]
61+
y-axis "Time (ms)" 0 --> 747
62+
bar [461.16, 580.4, 591.58, 622.3, 25.03]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-03-16T00:40:15.335Z*
75+
*Last generated: 2026-03-18T00:37:45.484Z*

‎docs/docs/benchmarks/MassiveParallelTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 4
77
# MassiveParallelTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-03-16** from the latest CI run.
10+
This benchmark was automatically generated on **2026-03-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.201
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.19.57 | 663.8 ms | 663.1 ms | 2.26 ms |
20-
| NUnit | 4.5.1 | 1,214.3 ms | 1,213.3 ms | 7.28 ms |
21-
| MSTest | 4.1.0 | 2,955.0 ms | 2,952.1 ms | 7.59 ms |
22-
| xUnit3 | 3.2.2 | 3,087.0 ms | 3,083.8 ms | 9.16 ms |
23-
| **TUnit (AOT)** | 1.19.57 | 229.2 ms | 229.4 ms | 0.58 ms |
19+
| **TUnit** | 1.19.74 | 666.0 ms | 661.0 ms | 10.76 ms |
20+
| NUnit | 4.5.1 | 1,217.1 ms | 1,215.6 ms | 8.81 ms |
21+
| MSTest | 4.1.0 | 2,949.8 ms | 2,950.1 ms | 10.58 ms |
22+
| xUnit3 | 3.2.2 | 3,083.7 ms | 3,084.7 ms | 5.54 ms |
23+
| **TUnit (AOT)** | 1.19.74 | 228.5 ms | 228.5 ms | 0.29 ms |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
5858
xychart-beta
5959
title "MassiveParallelTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 3705
62-
bar [663.8, 1214.3, 2955, 3087, 229.2]
61+
y-axis "Time (ms)" 0 --> 3701
62+
bar [666, 1217.1, 2949.8, 3083.7, 228.5]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-03-16T00:40:15.335Z*
75+
*Last generated: 2026-03-18T00:37:45.485Z*

‎docs/docs/benchmarks/MatrixTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 5
77
# MatrixTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-03-16** from the latest CI run.
10+
This benchmark was automatically generated on **2026-03-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.201
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.19.57 | 583.5 ms | 583.0 ms | 2.71 ms |
20-
| NUnit | 4.5.1 | 1,574.6 ms | 1,575.6 ms | 5.91 ms |
21-
| MSTest | 4.1.0 | 1,496.4 ms | 1,494.0 ms | 10.42 ms |
22-
| xUnit3 | 3.2.2 | 1,617.4 ms | 1,616.0 ms | 6.43 ms |
23-
| **TUnit (AOT)** | 1.19.57 | 129.7 ms | 129.7 ms | 0.35 ms |
19+
| **TUnit** | 1.19.74 | 584.6 ms | 582.5 ms | 4.68 ms |
20+
| NUnit | 4.5.1 | 1,625.7 ms | 1,626.0 ms | 7.43 ms |
21+
| MSTest | 4.1.0 | 1,537.9 ms | 1,539.1 ms | 9.49 ms |
22+
| xUnit3 | 3.2.2 | 1,674.6 ms | 1,671.7 ms | 9.32 ms |
23+
| **TUnit (AOT)** | 1.19.74 | 127.3 ms | 127.3 ms | 0.36 ms |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
5858
xychart-beta
5959
title "MatrixTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 1941
62-
bar [583.5, 1574.6, 1496.4, 1617.4, 129.7]
61+
y-axis "Time (ms)" 0 --> 2010
62+
bar [584.6, 1625.7, 1537.9, 1674.6, 127.3]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-03-16T00:40:15.336Z*
75+
*Last generated: 2026-03-18T00:37:45.485Z*

‎docs/docs/benchmarks/ScaleTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 6
77
# ScaleTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-03-16** from the latest CI run.
10+
This benchmark was automatically generated on **2026-03-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.201
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.19.57 | 500.75 ms | 500.68 ms | 5.874 ms |
20-
| NUnit | 4.5.1 | 628.64 ms | 632.06 ms | 8.338 ms |
21-
| MSTest | 4.1.0 | 592.78 ms | 591.68 ms | 13.569 ms |
22-
| xUnit3 | 3.2.2 | 656.79 ms | 658.93 ms | 11.776 ms |
23-
| **TUnit (AOT)** | 1.19.57 | 35.81 ms | 35.80 ms | 2.294 ms |
19+
| **TUnit** | 1.19.74 | 478.67 ms | 478.68 ms | 2.254 ms |
20+
| NUnit | 4.5.1 | 599.59 ms | 598.37 ms | 6.170 ms |
21+
| MSTest | 4.1.0 | 554.75 ms | 554.90 ms | 20.483 ms |
22+
| xUnit3 | 3.2.2 | 616.81 ms | 617.96 ms | 5.467 ms |
23+
| **TUnit (AOT)** | 1.19.74 | 35.96 ms | 36.08 ms | 2.485 ms |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
5858
xychart-beta
5959
title "ScaleTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 789
62-
bar [500.75, 628.64, 592.78, 656.79, 35.81]
61+
y-axis "Time (ms)" 0 --> 741
62+
bar [478.67, 599.59, 554.75, 616.81, 35.96]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-03-16T00:40:15.336Z*
75+
*Last generated: 2026-03-18T00:37:45.485Z*

‎docs/docs/benchmarks/SetupTeardownTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 7
77
# SetupTeardownTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-03-16** from the latest CI run.
10+
This benchmark was automatically generated on **2026-03-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.201
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.19.57 | 577.7 ms | 578.1 ms | 8.96 ms |
20-
| NUnit | 4.5.1 | 1,165.1 ms | 1,165.7 ms | 8.84 ms |
21-
| MSTest | 4.1.0 | 1,093.4 ms | 1,093.7 ms | 6.21 ms |
22-
| xUnit3 | 3.2.2 | 1,221.6 ms | 1,220.7 ms | 9.87 ms |
23-
| **TUnit (AOT)** | 1.19.57 | NA | NA | NA |
19+
| **TUnit** | 1.19.74 | 579.6 ms | 581.5 ms | 5.43 ms |
20+
| NUnit | 4.5.1 | 1,206.0 ms | 1,206.2 ms | 9.49 ms |
21+
| MSTest | 4.1.0 | 1,132.0 ms | 1,132.3 ms | 11.49 ms |
22+
| xUnit3 | 3.2.2 | 1,264.7 ms | 1,263.3 ms | 8.12 ms |
23+
| **TUnit (AOT)** | 1.19.74 | NA | NA | NA |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-16** from the latest CI
5858
xychart-beta
5959
title "SetupTeardownTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 1466
62-
bar [577.7, 1165.1, 1093.4, 1221.6, 0]
61+
y-axis "Time (ms)" 0 --> 1518
62+
bar [579.6, 1206, 1132, 1264.7, 0]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-03-16T00:40:15.336Z*
75+
*Last generated: 2026-03-18T00:37:45.486Z*

‎docs/docs/benchmarks/index.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 1
77
# Performance Benchmarks
88

99
:::info Last Updated
10-
These benchmarks were automatically generated on **2026-03-16** from the latest CI run.
10+
These benchmarks were automatically generated on **2026-03-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.201
1313
:::
@@ -37,7 +37,7 @@ These benchmarks compare TUnit against the most popular .NET testing frameworks:
3737

3838
| Framework | Version Tested |
3939
|-----------|----------------|
40-
| **TUnit** | 1.19.57 |
40+
| **TUnit** | 1.19.74 |
4141
| **xUnit v3** | 3.2.2 |
4242
| **NUnit** | 4.5.1 |
4343
| **MSTest** | 4.1.0 |
@@ -80,4 +80,4 @@ These benchmarks run automatically daily via [GitHub Actions](https://github.com
8080
Each benchmark runs multiple iterations with statistical analysis to ensure accuracy. Results may vary based on hardware and test characteristics.
8181
:::
8282

83-
*Last generated: 2026-03-16T00:40:15.337Z*
83+
*Last generated: 2026-03-18T00:37:45.486Z*

‎docs/static/benchmarks/AsyncTests.json‎

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"timestamp": "2026-03-16T00:40:15.335Z",
2+
"timestamp": "2026-03-18T00:37:45.484Z",
33
"category": "AsyncTests",
44
"environment": {
55
"benchmarkDotNetVersion": "BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.3 LTS (Noble Numbat)",
@@ -9,43 +9,43 @@
99
"results": [
1010
{
1111
"Method": "TUnit",
12-
"Version": "1.19.57",
13-
"Mean": "556.2 ms",
14-
"Error": "10.60 ms",
15-
"StdDev": "10.41 ms",
16-
"Median": "556.4 ms"
12+
"Version": "1.19.74",
13+
"Mean": "544.1 ms",
14+
"Error": "4.46 ms",
15+
"StdDev": "4.17 ms",
16+
"Median": "544.2 ms"
1717
},
1818
{
1919
"Method": "NUnit",
2020
"Version": "4.5.1",
21-
"Mean": "715.8 ms",
22-
"Error": "6.88 ms",
23-
"StdDev": "6.10 ms",
24-
"Median": "717.0 ms"
21+
"Mean": "727.3 ms",
22+
"Error": "5.84 ms",
23+
"StdDev": "5.46 ms",
24+
"Median": "727.2 ms"
2525
},
2626
{
2727
"Method": "MSTest",
2828
"Version": "4.1.0",
29-
"Mean": "646.7 ms",
30-
"Error": "5.15 ms",
31-
"StdDev": "4.57 ms",
32-
"Median": "645.3 ms"
29+
"Mean": "663.9 ms",
30+
"Error": "9.06 ms",
31+
"StdDev": "8.47 ms",
32+
"Median": "661.2 ms"
3333
},
3434
{
3535
"Method": "xUnit3",
3636
"Version": "3.2.2",
37-
"Mean": "780.8 ms",
38-
"Error": "6.77 ms",
39-
"StdDev": "5.29 ms",
40-
"Median": "782.3 ms"
37+
"Mean": "799.6 ms",
38+
"Error": "8.99 ms",
39+
"StdDev": "8.41 ms",
40+
"Median": "797.6 ms"
4141
},
4242
{
4343
"Method": "TUnit_AOT",
44-
"Version": "1.19.57",
45-
"Mean": "123.6 ms",
46-
"Error": "0.33 ms",
47-
"StdDev": "0.28 ms",
48-
"Median": "123.7 ms"
44+
"Version": "1.19.74",
45+
"Mean": "123.4 ms",
46+
"Error": "0.15 ms",
47+
"StdDev": "0.14 ms",
48+
"Median": "123.4 ms"
4949
}
5050
]
5151
}

0 commit comments

Comments
 (0)