Support Microsoft.Testing.Platform

[Youssef1313]

Currently, it seems that testing using Cake only supports VSTest. It doesn't support the newer testing platform, Microsoft.Testing.Platform (aka MTP)

[devlead]

It does using DotNetTest aliases
https://cakebuild.net/api/Cake.Common.Tools.DotNet/DotNetAliases/

[Youssef1313]

Thanks for the very fast reply @devlead.

Are you referring to https://cakebuild.net/api/Cake.Common.Tools.DotNet/DotNetAliases/47CF9794 ? The DotNetTestSettings seems to be VSTest-oriented. So currently MTP users will likely need to construct all the command-line arguments manually using ArgumentCustomization.

[devlead]

You can for now use this alias if you need any extra arguments not yet any settings for
https://cakebuild.net/api/Cake.Common.Tools.DotNet/DotNetAliases/E12827F8

[Youssef1313]

@devlead Yes it's possible to manually write all arguments, but I think it would be nice if there is a built-in support similar to the level of support for VSTest. Note that now all major frameworks support MTP (https://devblogs.microsoft.com/dotnet/mtp-adoption-frameworks).

[devlead]

What arguments are you missing?

Feel free to send a PR and we can get those out in next release.

[Youssef1313]

This may need some work on a design. For example, TestAdapterPath is not supported in MTP, as it's not relevant to the way MTP works. Loggers as well. For example, in MTP to enable TRX, you do --report-trx and optionally --report-trx-filename. Similar scenario for data collectors. So, I feel like MTP will need a new type and not be tied to DotNetTestSettings. Probably something like MicrosoftTestingPlatformSettings.

Also note that currently, when using dotnet test, arguments that are passed to MTP should come after --. For example:

dotnet test -c Debug --no-build -- --report-trx --report-trx-filename TestResults.trx

Note that build-specific arguments just follow dotnet test regularly (e.g, configuration, --no-build, basically anything that's not test-specific and is done for build time). However, test-specific arguments that should be passed to MTP should come after --. This is similar to dotnet run where arguments specific to dotnet run are added normally, while arguments that are passed down to the application being run by dotnet run are added after --.

Another note, the experience is hopefully going to change in .NET 10. See dotnet/sdk#45927

So, that's also an open question, how to design it such that it works with the current dotnet test experience and the future .NET 10 dotnet test experience.

[JustusGreiberORGADATA]

Also note that currently, when using dotnet test, arguments that are passed to MTP should come after --.

Also if you set the Verbosity on DotNetTestSettings it will be appended last, so if you set the arguments with ProcessArgumentBuilder it will result in the wrong order.

So something like this

context.DotNetTest(
    context.Solution.FullPath,
    // For Microsoft Testpaltform we need to append extra arguments:
    new ProcessArgumentBuilder()
        .Append("--report-trx"),
    new DotNetTestSettings
    {
        Configuration = "Debug",
        Verbosity = DotNetVerbosity.Diagnostic,
    }
);

Will result in something like this

dotnet test -c Debug -- --report-trx --report-trx-filename TestResults.trx --verbosity diagnostic

because common arguments are appended last:

cake/src/Cake.Common/Tools/DotNet/DotNetTool.cs

Lines 73 to 79 in d6c0cc3

	protected void RunCommand(TSettings settings, ProcessArgumentBuilder arguments, ProcessSettings processSettings)
	{
	// add arguments common to all commands last
	AppendCommonArguments(arguments, settings);
	Run(settings, arguments, processSettings, null);
	}

This will trigger the following error from the test platform exe:

MSTest v3.8.3 (UTC 17.03.2025) [win-x64 - .NET 8.0.15]
Unknown option '--verbosity'

[Cheesebaron]

Instead of using dotnet test you could just use dotnet run. I use it like:

var projectName = project.GetFilenameWithoutExtension();
var runSettings = new DotNetRunSettings
{
    NoBuild = true,
    Configuration = context.BuildConfiguration,
    Verbosity = context.VerbosityDotNet,
    ArgumentCustomization = args => args
        .Append("-- ")
        .Append($"--report-xunit-trx --report-xunit-trx-filename {projectName}.trx")
        .Append($"--report-ctrf --report-ctrf-filename {projectName}.ctrf.json")
        .Append($"--coverage --coverage-output {projectName}.coverage --coverage-output-format cobertura")
};

context.DotNetRun(project.FullPath, runSettings);

Would be super nice to have a Settings class that knows about all the Microsoft Testing Platform switches to avoid having to piece together your own cli args.

[Youssef1313]

Please note that in .NET 10 SDK, there is a new dotnet test implementation that's specifically made for MTP, which doesn't require the extra --. It will be opt-in via dotnet.config file:

[dotnet.test.runner]
name = "Microsoft.Testing.Platform"

As the old dotnet test will no longer be recommended, I think it's best if cake implementation focuses only on the new implementation in .NET 10.

[Frulfump]

Please note that in .NET 10 SDK, there is a new dotnet test implementation that's specifically made for MTP, which doesn't require the extra --. It will be opt-in via dotnet.config file:

[dotnet.test.runner]
name = "Microsoft.Testing.Platform"

As the old dotnet test will no longer be recommended, I think it's best if cake implementation focuses only on the new implementation in .NET 10.

Is there any docs for dotnet.config couldn't find one when searching for it.

[Youssef1313]

It's mentioned in `dotnet test command - .NET CLI and Testing with 'dotnet test'