Migrate test suite to xUnit v3 on Microsoft.Testing.Platform - #5949
Draft
hoyosjs wants to merge 7 commits into
Draft
Migrate test suite to xUnit v3 on Microsoft.Testing.Platform#5949hoyosjs wants to merge 7 commits into
hoyosjs wants to merge 7 commits into
Conversation
Migrates 11 test projects + TestHelpers/CommonTestRunner from xUnit v2 (Arcade classic xunit.console) to xUnit v3 self-running MTP executables. Deletes Skippable machinery; uses native Assert.Skip/SkipUnless. Updates eng/build.ps1 + eng/build.sh test filters to MTP syntax. SOS.UnitTests temporarily disabled in dirs.proj. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Migrate the last remaining test project (SOS.UnitTests) off the xUnit v2 Skippable* machinery to xUnit v3 running under Microsoft.Testing.Platform, and re-enable it in the test traversal. - SOS.cs / SOSRunner.cs: drop `using Xunit.Abstractions` (ITestOutputHelper is now in the Xunit namespace under v3) and rename [SkippableFact]/[SkippableTheory] to [Fact]/[Theory]. `Xunit.Extensions` stays because both files still throw SkipTestException, which emits the v3 dynamic-skip token. - xunit.runner.json / SOS.UnitTests.csproj: stop shipping xunit.runner.json as Content (it collided with the copy Arcade's XUnitV3 tooling injects, causing NETSDK1152 during publish). Instead point Arcade's XUnitCoreSettingsFile / XUnitDesktopSettingsFile at the project's own file, which both resolves the conflict and ensures the settings are honored under MTP. Merge Arcade's shadowCopy:false alongside the existing preEnumerateTheories:false. - global.json: add the .NET 10 test runner switch so standalone `dotnet test` drives Microsoft.Testing.Platform instead of VSTest. - dirs.proj: re-enable SOS.UnitTests in the traversal. Verified on Windows x64: full tree builds 0/0; MTP discovery finds 30 tests; SOSExceptionTests.DivZero passes across all runtime configs (full dump -> SOS-command -> assert pipeline); interpreter tests dynamically skip (not fail). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
When a -classfilter/-methodfilter is active, clear the results XML directory before the test run, then sum the assembly total counts across all result XMLs afterward and fail if zero tests ran. This replaces the timestamp/newer-file approach, which was not portable across the bash and pwsh scripts and could miscount stale per-TFM result files.
TestConfiguration.BaseDir defaulted to the current working directory
("."). Under the classic xunit.console runner the working directory was
the test output directory, so "." found Debugger.Tests.Config.txt and the
other assets. The Microsoft.Testing.Platform runner launches the test
executable with a different working directory, so default to
AppContext.BaseDirectory (the app base, where the assets are copied)
instead. Callers can still override it.
Remove the central NoWarn suppression of the xUnit v3 analyzers in src/tests/Directory.Build.props and address the flagged sites so TreatWarningsAsErrors passes: - xUnit1051 (29 sites): pass TestContext.Current.CancellationToken at call sites that already take a CancellationToken (Task.Delay/Run/Wait, StreamReader.ReadToEndAsync, Pipeline.StopAsync, and the CollectLinux TestArgs helper). No product signatures change - the analyzer only fires where a CancellationToken overload already exists. - xUnit1031 (1 site): the blocking streamTask.Wait is inside the skipped StartEventPipeSessionWithoutStackwalkTestAsync theory (the async fix is tied to the race in #4717); keep the blocking wait and suppress it with a scoped pragma, matching the existing pattern in src/tests/eventpipe/ThreadPoolEvents.cs.
Set TestDotNetRoot to $(ArtifactsDotnetTestDir) for MTP test projects so Arcade's XUnitV3 RunTests target launches each test executable with DOTNET_ROOT pointing at the test .NET install instead of the Arcade toolset SDK (.dotnet). This removes the dependency on the runtime that global.json's tools.runtimes staged into .dotnet.
Add an InstallHostRuntimeX86 target that stages the x86 shared framework into artifacts/dotnet-test/x86 on the Windows x86 test leg, matching the DOTNET_ROOT_X86 that Arcade's XUnitV3 RunTests target sets. A 32-bit apphost reads DOTNET_ROOT_X86 before DOTNET_ROOT and does not fall back when it is set, so the runtime must live in that subdirectory. With the test host now resolving its runtime from artifacts/dotnet-test, remove the tools.runtimes.dotnet/x86 entry from global.json that used to stage the same x86 framework into .dotnet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Move the repo's tests off the Arcade "classic"
xunit.consolerunner onto xUnit v3 built as self-contained Microsoft.Testing.Platform (MTP) executables. This is the direction Arcade and the broader .NET org are standardizing on, and it lets us drop custom test infrastructure that only existed to patch gaps in xUnit v2.What changed
Opt-in and runner
global.json: addedtest.runner: Microsoft.Testing.Platform.src/tests/Directory.Build.props: single opt-in for all test projects (TestRunnerName=XUnitV3,OutputType=Exe), guarded onIsTestProjectso it doesn't leak into product projects. Individual projects can disable viaEnableXUnitV3=false.Test projects and helpers
*.UnitTests/ test projects (NETCore.Client, Monitoring[.EventPipe], ExtensionCommands, DebugServices, SymbolStore, FileFormats, DbgShim, dotnet-trace, dotnet-counters, eventpipe, SOS.UnitTests) to the v3 APIs.Microsoft.Diagnostics.TestHelpers: updated output-helper plumbing (ITestOutputHelpershape changes, multiplex/file/indented loggers) for v3.SkippableFact/SkippableTheoryinfrastructure (discoverers, message buses, test cases) - xUnit v3 supports runtime skip natively viaAssert.Skip/SkipTestException, so the bespoke attributes are gone.Build/CI scripts
eng/build.ps1/eng/build.sh: test filters now use MTP options (--filter-method/--filter-class) instead of the old-method/-classconsole flags.--ignore-exit-code 8so projects in the traversal that legitimately match zero tests don't fail the run, plus a post-run guard that sums the result XMLs and fails if a filter matched nothing anywhere (catches typo'd filters that the per-project ignore would otherwise hide).x86 test host + global.json cleanup
artifacts/dotnet-test(viaTestDotNetRoot) instead of the Arcade toolset.dotnet.InstallHostRuntimeX86target ineng/InstallRuntimes.projstages the x86 shared framework intoartifacts/dotnet-test/x86, matching theDOTNET_ROOT_X86that Arcade sets for the MTP exe (a 32-bit apphost readsDOTNET_ROOT_X86first and won't fall back).tools.runtimes.dotnet/x86node fromglobal.json, which previously existed solely to stage that x86 runtime into.dotnet.Breaking changes
xunit.consoleinvocations. Anyone running tests directly, or tooling that shells the old console runner /-method/-classflags, must switch to the MTP equivalents.build.cmd -test/test.cmdand-methodfilter/-classfiltercontinue to work.SkippableFact/SkippableTheoryattributes are removed; skips use xUnit v3's native mechanism.Validation (beyond CI)
Output still lands in the trx/xml as before. The x86 host redirect was verified by repro on an x64 dev box (test projects are RID-agnostic, so a local
-architecture x86build emits x64 apphosts and can't exerciseDOTNET_ROOT_X86through the normal traversal): a genuinewin-x86apphost hosts correctly from the stagedartifacts/dotnet-test/x86subdir (52 tests pass), and an empty subdir hard-fails with no fallback - confirming the staging is load-bearing. The real CI x86 build/test legs are the final gate for that path.