diff --git a/docs/core/testing/microsoft-testing-platform-architecture-capabilities.md b/docs/core/testing/microsoft-testing-platform-architecture-capabilities.md index cae451d660702..6ddf21b7a5711 100644 --- a/docs/core/testing/microsoft-testing-platform-architecture-capabilities.md +++ b/docs/core/testing/microsoft-testing-platform-architecture-capabilities.md @@ -1,126 +1,389 @@ --- -title: Microsoft.Testing.Platform capabilities overview -description: Learn about Microsoft.Testing.Platform capabilities concept. +title: Microsoft.Testing.Platform capabilities system +description: Learn how the capabilities system enables feature negotiation and version-independent compatibility in Microsoft.Testing.Platform. author: MarcoRossignoli ms.author: mrossignoli -ms.date: 07/11/2024 +ms.date: 11/27/2025 +ai-usage: ai-assisted --- -# Microsoft.Testing.Platform capabilities +# Microsoft.Testing.Platform capabilities system -In the context of Microsoft.Testing.Platform, a *capability* refers to the *potential to perform a specific action or provide specific information*. It's a means for the testing framework and extensions to *declare* their *ability* to *operate* in a certain manner or provide specific information to the *requesters*. +The capabilities system is a core mechanism in Microsoft.Testing.Platform that enables components to discover and negotiate features at runtime. A *capability* represents the *ability to perform a specific action or provide specific information*. -The *requesters* can be any component involved in a test session, such as the platform, an extension, or the testing framework itself. +## Why capabilities matter -The primary objective of the capability system is to facilitate effective communication among the components involved in a test session, enabling them to exchange information and meet their respective needs accurately. +The capabilities system solves several key challenges in building extensible testing platforms: -## Guided example +- **Feature discovery**: Extensions can query what features a test framework supports before using them +- **Graceful degradation**: Extensions continue working even when optional features aren't available +- **Version compatibility**: Different versions of components work together based on mutual understanding +- **No breaking changes**: New features can be added without breaking existing code -Let's consider a hypothetical example to demonstrate the necessity of a capability system. +## How capabilities work -> [!NOTE] -> This example is purely for illustrative purposes and isn't currently implemented within Microsoft.Testing.Platform or any testing framework. +Components declare capabilities they support, and other components query for those capabilities before using features: -Imagine a situation where you have an extension that requires the testing framework to execute no more than one test at a time. Furthermore, after each test, the extension needs to know the CPU usage for that specific test. +```csharp +// Extension queries test framework capabilities +var capabilities = serviceProvider.GetRequiredService(); +var parallelism = capabilities.GetCapability(); -To accommodate the preceding scenario, you need to inquire from the testing framework if: +if (parallelism?.CanDisableParallelism == true) +{ + // Feature is supported, use it + parallelism.Enable(); +} +else +{ + // Feature not available, fallback or disable extension + return false; +} +``` -1. It has the capability to execute only one test at a time. -2. It can provide information regarding the amount of CPU consumed by each test. +## Capability interfaces -How can the extension determine if the testing framework has the ability to operate in this mode and provide CPU usage information for a test session? In Microsoft.Testing.Platform, this capability is represented by an implementation the `Microsoft.Testing.Platform.Capabilities.ICapability` interface: +The capability system is built on simple marker interfaces: ```csharp -// Base capabilities contracts - +// Base capability marker public interface ICapability { } +// Container for capabilities public interface ICapabilities where TCapability : ICapability { IReadOnlyCollection Capabilities { get; } } +``` -// Specific testing framework capabilities +### Test framework capabilities -public interface ITestFrameworkCapabilities : ICapabilities +Test frameworks declare their capabilities using specialized interfaces: + +```csharp +// Base for all test framework capabilities +public interface ITestFrameworkCapability : ICapability { } -public interface ITestFrameworkCapability : ICapability +// Container for test framework capabilities +public interface ITestFrameworkCapabilities : ICapabilities { } ``` -As you can see, the `ICapability` interface is a *marker* interface because it can represent *any capability*, and the actual implementation will be context dependent. You'll also observe the `ITestFrameworkCapability`, which inherits from `ICapability` to classify the capability. The capability system's generic nature allows for convenient grouping by context. The `ITestFrameworkCapability` groups all the capabilities implemented by the [testing framework](./microsoft-testing-platform-architecture-extensions.md#create-a-testing-framework). The `ICapabilities` interface reveals the *set* of all capabilities implemented by an extension. Similarly, for the base one, there's a context-specific testing framework called `ITestFrameworkCapabilities`. The `ITestFrameworkCapabilities` is provided to the platform during the [testing framework registration](./microsoft-testing-platform-architecture-extensions.md#register-a-testing-framework) process. +You provide these when [registering your test framework](microsoft-testing-platform-architecture-testframework.md#register-a-test-framework): + +```csharp +builder.RegisterTestFramework( + serviceProvider => new MyTestFrameworkCapabilities(), + (capabilities, serviceProvider) => new MyTestFramework(capabilities, serviceProvider)); +``` + +## Example: Designing a capability + +Let's design a capability for a hypothetical scenario where an extension needs a test framework to: + +1. Run tests one at a time (disable parallelism) +2. Provide CPU usage information for each test -To create a capability that addresses the aforementioned scenario, you define it as follows: +### Define the capability interface ```csharp public interface IDisableParallelismCapability : ITestFrameworkCapability { + // Query support bool CanDisableParallelism { get; } bool CanProvidePerTestCpuConsumption { get; } + + // Activate the feature void Enable(); } ``` -If the testing framework implements this interface, at runtime, the following can be queried: +### Implement in the test framework -* Verify if the testing framework has the ability to turn off parallelism `CanDisableParallelism = true`. -* Determine if the testing framework can supply CPU usage data `CanProvidePerTestCPUConsumption = true`. -* Request the testing adapter to activate this mode by invoking the `Enable()` method before the test session commences. +```csharp +internal class MyTestFrameworkCapabilities : ITestFrameworkCapabilities +{ + public IReadOnlyCollection Capabilities => + [ + new DisableParallelismCapability() + ]; +} -The hypothetical code fragment inside the extension could be something like: +internal class DisableParallelismCapability : IDisableParallelismCapability +{ + private bool _enabled; + + public bool CanDisableParallelism => true; + public bool CanProvidePerTestCpuConsumption => true; + + public void Enable() + { + _enabled = true; + // Configure test framework for sequential execution + } + + public bool IsEnabled => _enabled; +} +``` + +### Query and use in an extension ```csharp -IServiceProvider provider = null; // TODO: Get IServiceProvider. -var capabilities = serviceProvider.GetRequiredService(); +public class CpuMonitorExtension : IDataConsumer +{ + private readonly IDisableParallelismCapability? _parallelismCapability; + + public CpuMonitorExtension(IServiceProvider serviceProvider) + { + var capabilities = serviceProvider.GetRequiredService(); + + // Query for the capability + _parallelismCapability = capabilities.GetCapability(); + } + + public async Task IsEnabledAsync() + { + // Only enable if framework supports our requirements + if (_parallelismCapability is null) + { + Console.WriteLine("Test framework doesn't support CPU monitoring"); + return false; + } + + if (!_parallelismCapability.CanDisableParallelism) + { + Console.WriteLine("Cannot disable parallelism - CPU monitoring may be inaccurate"); + } + + if (_parallelismCapability.CanProvidePerTestCpuConsumption) + { + _parallelismCapability.Enable(); + return true; + } + + return false; + } + + // ... IDataConsumer implementation +} +``` + +## Built-in platform capabilities + +The platform provides several built-in capabilities that test frameworks can implement. + +### IBannerMessageOwnerCapability + +Allows test frameworks to provide custom banner messages instead of the platform default. -// Utilize the `GetCapability` API to search for the specific capability to query. -var capability = capabilities.GetCapability(); -if (capability is null) +```csharp +public interface IBannerMessageOwnerCapability : ITestFrameworkCapability { - // Capability not supported... + Task GetBannerMessageAsync(); } -else +``` + +**When to use**: When your test framework wants to display custom branding or version information at startup. + +**Example implementation**: + +```csharp +internal class MyBannerCapability : IBannerMessageOwnerCapability { - capability.Enable(); - if (capability.CanDisableParallelism) + private readonly IPlatformInformation _platformInfo; + + public MyBannerCapability(IPlatformInformation platformInfo) + { + _platformInfo = platformInfo; + } + + public Task GetBannerMessageAsync() { - // Do something... + return Task.FromResult( + $"MyTestFramework v2.0.0 on {_platformInfo.Name} {_platformInfo.Version}"); } +} +``` + +> [!TIP] +> Use the [`IPlatformInformation` service](microsoft-testing-platform-architecture-services.md#the-iplatforminformation-service) to access platform details when building your banner. + +## Capability ownership and distribution + +When you design a custom capability: + +### 1. Define the capability contract + +Create a separate "abstractions" package containing only the capability interfaces: + +``` +MyExtension.Abstractions +└── IMyCustomCapability.cs +``` + +This allows test frameworks to reference only the contract without pulling in your entire extension. + +### 2. Document the capability thoroughly + +Your capability documentation should include: + +- **Purpose**: What feature does this capability enable? +- **Requirements**: What must the test framework do to support it? +- **Data contracts**: What messages should be published to `IMessageBus`? +- **Activation**: How and when should `Enable()` be called? +- **Examples**: Working code samples + +### 3. Version the capability + +Follow semantic versioning for capability interfaces: + +- **Major version**: Breaking changes to the interface +- **Minor version**: New optional members (with default implementations if possible) +- **Patch version**: Documentation updates + +### 4. Example: TRX Report Capability + +The TRX report extension demonstrates this pattern: + +**Contract package**: `Microsoft.Testing.Extensions.TrxReport.Abstractions` + +- Contains capability interfaces +- Referenced by test frameworks + +**Implementation package**: `Microsoft.Testing.Extensions.TrxReport` - if (capability.CanProvidePerTestCpuConsumption) +- Contains the actual extension +- Referenced by test projects + +## Querying capabilities + +Extensions query capabilities from the service provider: + +```csharp +public class MyExtension : IDataConsumer +{ + private readonly ITestFrameworkCapabilities _frameworkCapabilities; + + public MyExtension(IServiceProvider serviceProvider) + { + // Get test framework capabilities + _frameworkCapabilities = serviceProvider + .GetRequiredService(); + } + + public async Task IsEnabledAsync() { - // Do something... + // Query for a specific capability + var capability = _frameworkCapabilities.GetCapability(); + + if (capability is null) + { + // Capability not supported + Console.WriteLine("MyCapability not supported by test framework"); + return false; + } + + // Check specific features + if (!capability.SupportsFeatureX) + { + Console.WriteLine("Feature X not supported - using fallback"); + } + + // Enable the capability + capability.Enable(); + return true; } } ``` -The preceding example illustrates how the capability infrastructure enables a powerful mechanism for communicating abilities between the components involved in a test session. While the sample demonstrates a capability specifically designed for the testing framework, any component can expose and implement extensions that inherit from `ICapability`. +## Best practices + +### For capability designers + +**DO**: + +- Design capabilities as simple, focused interfaces +- Use boolean properties to indicate feature support +- Provide an `Enable()` method for activating features +- Document expected `IMessageBus` message types +- Create separate abstractions packages +- Include working examples in documentation + +**DON'T**: + +- Include implementation details in capability interfaces +- Create overly complex capability hierarchies +- Make assumptions about how features will be used +- Break capability interfaces in minor versions -It's evident that not all details can be communicated through an interface. Considering the previous example, what should the extension expect if the `CanProvidePerTestCpuConsumption` is supported? What kind of custom information is expected to be transmitted via the [IMessageBus](./microsoft-testing-platform-architecture-services.md#the-imessagebus-service) by the testing framework? The solution is **documentation of the capability**. It's the responsibility of the capability *owner* to design, ship, and document it as clearly as possible to assist implementors who want to effectively *collaborate* with the extension that requires the specific capability. +### For test framework authors -For instance, the TRX report extension enables the testing framework to implement the necessary capability to accurately generate a TRX report. The extension to register is included in the NuGet package, but the capability to implement is found in the *contract only* NuGet package. +**DO**: + +- Implement capabilities your framework naturally supports +- Return empty collections if no capabilities are supported +- Document which capabilities you implement +- Test capability interactions thoroughly +- Respect capability contracts for message bus data + +**DON'T**: + +- Implement capabilities you can't fully support +- Break capability contracts +- Assume all extensions query capabilities + +### For extension authors + +**DO**: + +- Always check if capabilities exist before using them +- Provide fallback behavior when capabilities are missing +- Use `IsEnabledAsync()` to disable gracefully when requirements aren't met +- Document which capabilities your extension requires +- Test with test frameworks that don't support your capabilities + +**DON'T**: + +- Assume capabilities are always present +- Fail hard when capabilities are missing +- Query capabilities in performance-critical paths +- Modify capability objects after `Enable()` is called + +## Capability discovery + +Users can discover available capabilities using the `--info` command: + +```bash +./MyTests.exe --info +``` -In conclusion, let's summarize the primary aspects of the capability system: +This displays all registered extensions and their capabilities, helping users understand what features are available. -* It's essential for facilitating clear and stable communication between components. -* All capabilities should inherit from `ICapability` or an interface that inherits from it, and are exposed through a collection with the `ICapabilities` interface. -* It aids in the evolution of features without causing breaking changes. If a certain capability isn't supported, appropriate action can be taken. -* The responsibility of designing, shipping, and documenting the usage of a capability lies with the *capability owner*. Microsoft.Testing.Platform can also *own* a capability in the same way as any other extension. +## Summary -## Framework capabilities +The capabilities system enables: -The platform exposes a specialized interface named `ITestFrameworkCapability` that is the base of all capabilities exposed for test frameworks. These capabilities are provided when [registering the test framework to the platform](./microsoft-testing-platform-architecture-extensions.md#register-a-testing-framework). +- **Loose coupling**: Components communicate through well-defined contracts +- **Version independence**: Different versions work together +- **Graceful degradation**: Extensions adapt to available features +- **Extensibility**: New features without breaking changes -### `IBannerMessageOwnerCapability` +Key principles: -An optional [test framework capability](#framework-capabilities) that allows the test framework to provide the banner message to the platform. If the message is `null` or if the capability is n't present, the platform will use its default banner message. +1. Capabilities represent potential, not requirements +2. Extensions query before using features +3. Missing capabilities shouldn't break execution +4. Documentation is critical for capability adoption -This capability implementation allows you to abstract away the various conditions that the test framework may need to consider when deciding whether or not the banner message should be displayed. +## See also -The platform exposes the [`IPlatformInformation` service](./microsoft-testing-platform-architecture-services.md#the-iplatforminformation-service) to provide some information about the platform that could be useful when building your custom banner message. +- [Architecture overview](microsoft-testing-platform-architecture-overview.md) +- [Build a test framework](microsoft-testing-platform-architecture-testframework.md) +- [Build extensions](microsoft-testing-platform-extensions.md) +- [Platform services](microsoft-testing-platform-architecture-services.md) diff --git a/docs/core/testing/microsoft-testing-platform-architecture-extensions.md b/docs/core/testing/microsoft-testing-platform-architecture-extensions.md index 8b32fa972276d..d5fd012bdad94 100644 --- a/docs/core/testing/microsoft-testing-platform-architecture-extensions.md +++ b/docs/core/testing/microsoft-testing-platform-architecture-extensions.md @@ -1,57 +1,143 @@ --- -title: Microsoft.Testing.Platform extensions architecture overview -description: Learn about how to extend Microsoft.Testing.Platform. +title: Build extensions for Microsoft.Testing.Platform +description: Learn how to build custom extensions that enhance Microsoft.Testing.Platform with features like custom reports, code coverage, and monitoring. author: MarcoRossignoli ms.author: mrossignoli -ms.date: 07/11/2024 +ms.date: 11/27/2025 +ai-usage: ai-assisted --- -# Microsoft.Testing.Platform extensibility +# Build extensions for Microsoft.Testing.Platform -Microsoft.Testing.Platform consists of a [testing framework](#test-framework-extension) and any number of [extensions](#other-extensibility-points) that can operate *in-process* or *out-of-process*. +Microsoft.Testing.Platform is built on a flexible extension model that allows you to add custom functionality to the testing experience. Extensions can: -As outlined in the [architecture](./microsoft-testing-platform-architecture.md) section, Microsoft.Testing.Platform is designed to accommodate a variety of scenarios and extensibility points. The primary and essential extension is undoubtedly the [testing framework](#test-framework-extension) that your tests will utilize. Failing to register this results in startup error. **The [testing framework](#test-framework-extension) is the sole mandatory extension required to execute a testing session.** +- Process test results to generate reports +- Add command-line options for configuration +- Monitor test execution for hangs or crashes +- Collect code coverage or performance metrics +- Implement custom logging or diagnostic features -To support scenarios such as generating test reports, code coverage, retrying failed tests, and other potential features, you need to provide a mechanism that allows other extensions to work in conjunction with the [testing framework](#test-framework-extension) to deliver these features not inherently provided by the [testing framework](#test-framework-extension) itself. +This article explains how to build extensions that integrate with the platform. -In essence, the [testing framework](#test-framework-extension) is the primary extension that supplies information about each test that makes up the test suite. It reports whether a specific test has succeeded, failed, skipped, and can provide additional information about each test, such as a human-readable name (referred to as the display name), the source file, and the line where our test begins, among other things. +> [!TIP] +> For building a test framework (the core component that discovers and runs tests), see [Build a test framework](microsoft-testing-platform-architecture-testframework.md). -The extensibility point enables the utilization of information provided by the [testing framework](#test-framework-extension) to generate new artifacts or to enhance existing ones with additional features. A commonly used extension is the TRX report generator, which subscribes to the [TestNodeUpdateMessage](#the-testnodeupdatemessage-data) and generates an XML report file from it. +> [!NOTE] +> Complete working examples are available in the [Microsoft Test Framework repository](https://github.com/microsoft/testfx/tree/main/samples/public/TestingPlatformExamples). -As discussed in the [architecture](./microsoft-testing-platform-architecture.md), there are certain extension points that *cannot* operate within the same process as the [testing framework](#test-framework-extension). The reasons typically include: +## Extension types overview -* The need to modify the *environment variables* of the *test host*. Acting within the test host process itself is *too late*. -* The requirement to *monitor* the process from the outside because the *test host*, where tests and user code run, might have some *user code bugs* that render the process itself *unstable*, leading to potential *hangs* or *crashes*. In such cases, the extension would crash or hang along with the *test host* process. +Extensions fall into two categories based on where they run: -Due to these reasons, the extension points are categorized into two types: +### In-process extensions -1. *In-process extensions*: These extensions operate within the same process as the [testing framework](#test-framework-extension). +In-process extensions run inside the test host process alongside your tests. They have direct access to test execution but are affected if the test host crashes. - You can register *in-process extensions* via the `ITestApplicationBuilder.TestHost` property: +**Use cases:** - ```csharp - // ... - var builder = await TestApplication.CreateBuilderAsync(args); - builder.TestHost.AddXXX(...); - // ... - ``` +- Generate test reports (TRX, HTML, JSON) +- Consume and aggregate test results +- Add custom command-line options +- Implement test retry logic +- Collect telemetry -1. *Out-of-process extensions*: These extensions function in a separate process, allowing them to monitor the test host without being influenced by the test host itself. +**Register via:** - You can register *out-of-process extensions* via the `ITestApplicationBuilder.TestHostControllers`. +```csharp +builder.TestHost.AddXXX(...) +``` - ```csharp - var builder = await TestApplication.CreateBuilderAsync(args); - builder.TestHostControllers.AddXXX(...); - ``` +### Out-of-process extensions - Lastly, some extensions are designed to function in both scenarios. These common extensions behave identically in both *hosts*. You can register these extensions either through the *TestHost* and *TestHostController* interfaces or directly at the `ITestApplicationBuilder` level. An example of such an extension is the [ICommandLineOptionsProvider](#the-icommandlineoptionsprovider-extensions). +Out-of-process extensions run in a separate process (the test host controller) that monitors the test host. They remain stable even if the test host crashes. -## The `IExtension` interface +**Use cases:** -The `IExtension` interface serves as the foundational interface for all extensibility points within the testing platform. It's primarily used to obtain descriptive information about the extension and, most importantly, to enable or disable the extension itself. +- Collect code coverage (requires environment setup) +- Detect and report hangs +- Monitor resource usage +- Implement crash recovery +- Set environment variables before tests run -Consider the following `IExtension` interface: +**Register via:** + +```csharp +builder.TestHostControllers.AddXXX(...) +``` + +### Dual-context extensions + +Some extensions work in both contexts. They're registered once but can be used from either in-process or out-of-process contexts. + +**Examples:** + +- Command-line option providers +- Configuration providers + +**Register via:** + +```csharp +builder.CommandLine.AddProvider(...) +``` + +## Quick start: Build your first extension + +Here's a minimal example of a data consumer extension that logs test results: + +```csharp +using Microsoft.Testing.Platform.Extensions; +using Microsoft.Testing.Platform.Extensions.TestHost; +using Microsoft.Testing.Platform.Extensions.Messages; + +internal class SimpleTestLogger : IDataConsumer, ITestHostExtension +{ + // IExtension members - required for all extensions + public string Uid => "SimpleTestLogger"; + public string Version => "1.0.0"; + public string DisplayName => "Simple Test Logger"; + public string Description => "Logs test results to console"; + + public Task IsEnabledAsync() => Task.FromResult(true); + + // IDataConsumer members + public Type[] DataTypesConsumed => new[] { typeof(TestNodeUpdateMessage) }; + + public Task ConsumeAsync( + IDataProducer dataProducer, + IData value, + CancellationToken cancellationToken) + { + if (value is TestNodeUpdateMessage testUpdate) + { + var testNode = testUpdate.TestNode; + var state = testNode.Properties.Single(); + + Console.WriteLine($"Test: {testNode.DisplayName} - Status: {state.GetType().Name}"); + } + + return Task.CompletedTask; + } +} + +// Register the extension +public static async Task Main(string[] args) +{ + var builder = await TestApplication.CreateBuilderAsync(args); + + // Register test framework (required) + builder.RegisterTestFramework(/* ... */); + + // Register our custom extension + builder.TestHost.AddDataConsumer(sp => new SimpleTestLogger()); + + using var app = await builder.BuildAsync(); + return await app.RunAsync(); +} +``` + +## The IExtension base interface + +All extensions must implement the `IExtension` interface: ```csharp public interface IExtension @@ -64,41 +150,45 @@ public interface IExtension } ``` -* `Uid`: Represents the unique identifier for the extension. It's crucial to choose a unique value for this string to avoid conflicts with other extensions. - -* `Version`: Represents the version of the interface. Requires [**semantic versioning**](https://semver.org/). - -* `DisplayName`: A user-friendly name representation that will appear in logs and when you request information using the `--info` command line option. - -* `Description`: The description of the extension, that appears when you request information using the `--info` command line option. +### Implementation guidelines -* `IsEnabledAsync()`: This method is invoked by the testing platform when the extension is being instantiated. If the method returns `false`, the extension will be excluded. This method typically makes decisions based on the [configuration file](./microsoft-testing-platform-architecture-services.md#the-iconfiguration-service) or some [custom command line options](./microsoft-testing-platform-architecture-services.md#the-icommandlineoptions-service). Users often specify `--customExtensionOption` in the command line to opt into the extension itself. +**Uid** (Unique identifier): -## Test framework extension +- Must be unique across all extensions +- Use reverse domain notation: `"com.mycompany.myextension"` +- Avoid conflicts with built-in or third-party extensions -The test framework is the primary extension that provides the testing platform with the ability to discover and execute tests. The test framework is responsible for communicating the results of the tests back to the testing platform. The test framework is the only mandatory extension required to execute a testing session. +**Version**: -### Register a testing framework +- Must follow [semantic versioning](https://semver.org/) (major.minor.patch) +- Example: `"1.0.0"`, `"2.1.3"` -This section explains how to register the test framework with the testing platform. You register only one testing framework per test application builder using the `TestApplication.RegisterTestFramework` API as shown in [Microsoft.Testing.Platform architecture](./microsoft-testing-platform-architecture.md) documentation. +**DisplayName**: -The registration API is defined as follows: +- Human-readable name shown in logs and `--info` output +- Keep it concise: "TRX Report Generator" -```csharp -ITestApplicationBuilder RegisterTestFramework( - Func capabilitiesFactory, - Func adapterFactory); -``` +**Description**: -The `RegisterTestFramework` API expects two factories: +- Brief explanation shown with `--info` option +- Describe what the extension does: "Generates Visual Studio TRX format test reports" -1. `Func`: This is a delegate that accepts an object implementing the [`IServiceProvider`](./microsoft-testing-platform-architecture-services.md) interface and returns an object implementing the [`ITestFrameworkCapabilities`](./microsoft-testing-platform-architecture-capabilities.md) interface. The [`IServiceProvider`](./microsoft-testing-platform-architecture-services.md#the-imessagebus-service) provides access to platform services such as configurations, loggers, and command line arguments. +**IsEnabledAsync()**: - The [`ITestFrameworkCapabilities`](./microsoft-testing-platform-architecture-capabilities.md) interface is used to announce the capabilities supported by the testing framework to the platform and extensions. It allows the platform and extensions to interact correctly by implementing and supporting specific behaviors. For a better understanding of the [concept of capabilities](./microsoft-testing-platform-architecture-capabilities.md), refer to the respective section. +- Return `true` to enable the extension, `false` to disable +- Use for conditional activation based on configuration or command-line options +- Called when the extension is instantiated -1. `Func`: This is a delegate that takes in an [ITestFrameworkCapabilities](./microsoft-testing-platform-architecture-capabilities.md) object, which is the instance returned by the `Func`, and an [IServiceProvider](./microsoft-testing-platform-architecture-services.md#the-imessagebus-service) to provide access to platform services once more. The expected return object is one that implements the [ITestFramework](#test-framework-extension) interface. The `ITestFramework` serves as the execution engine that discovers and runs tests, and then communicates the results back to the testing platform. +Example: -The need for the platform to separate the creation of the [`ITestFrameworkCapabilities`](./microsoft-testing-platform-architecture-capabilities.md) and the creation of the [ITestFramework](#test-framework-extension) is an optimization to avoid creating the test framework if the supported capabilities are not sufficient to execute the current testing session. +```csharp +public async Task IsEnabledAsync() +{ + // Enable based on command-line option + var options = _serviceProvider.GetCommandLineOptions(); + return options.IsOptionSet("enable-custom-logger"); +} +``` Consider the following user code example, which demonstrates a test framework registration that returns an empty capability set: diff --git a/docs/core/testing/microsoft-testing-platform-architecture-overview.md b/docs/core/testing/microsoft-testing-platform-architecture-overview.md new file mode 100644 index 0000000000000..e96dfadc79920 --- /dev/null +++ b/docs/core/testing/microsoft-testing-platform-architecture-overview.md @@ -0,0 +1,283 @@ +--- +title: Microsoft.Testing.Platform architecture overview +description: Learn about the high-level architecture of Microsoft.Testing.Platform, including design principles, process models, and extension points. +author: MarcoRossignoli +ms.author: mrossignoli +ms.date: 11/27/2025 +ai-usage: ai-assisted +--- + +# Microsoft.Testing.Platform architecture overview + +Microsoft.Testing.Platform is a flexible, extensible testing platform built on key architectural principles that ensure reliability, performance, and compatibility across all .NET scenarios. This article provides a high-level overview of the platform's architecture to help you understand how components work together and decide which extension points to use. + +## Architectural principles + +The platform is built on several core principles: + +- **Determinism**: Running the same tests in different contexts (local, CI) produces the same results. The runtime doesn't rely on reflection or dynamic .NET runtime features to coordinate test runs. + +- **Runtime transparency**: The test runtime doesn't interfere with test framework code. It doesn't create isolated contexts like `AppDomain` or `AssemblyLoadContext`, and it doesn't use reflection or custom assembly resolvers. + +- **Compile-time registration**: Extensions, test frameworks, and both in-process and out-of-process extensions are registered at compile time to ensure determinism and detect inconsistencies early. + +- **Zero dependencies**: The platform core is a single .NET assembly, `Microsoft.Testing.Platform.dll`, with no dependencies other than the supported runtimes. + +- **Hostable**: The test runtime can be hosted in any .NET application—not just console apps. You can run tests in devices, browsers, or any context where .NET runs. + +- **Support all .NET form factors**: The platform supports current and future .NET form factors, including Native AOT. + +- **Performant**: The platform orchestrates test runs efficiently without bloating the runtime with non-essential code. + +- **Extensible**: Built on well-defined extension points that allow maximum customization of test execution. + +## Process architecture + +Microsoft.Testing.Platform uses a flexible process model to accommodate different testing scenarios. Understanding this model is key to choosing the right extension points. + +### Single-process execution + +In the simplest scenario, your test application runs as a single process: + +:::image type="content" source="./media/platform-testhost.png" lightbox="./media/platform-testhost.png" alt-text="Diagram showing a single test host process executing tests."::: + +This single process (the **test host**) contains: + +- Your test framework +- Your tests +- In-process extensions + +This model is sufficient when extensions don't need to: + +- Modify environment variables before the process starts +- Monitor the test process from outside (to detect hangs or crashes) + +### Multi-process execution + +When you register out-of-process extensions, the platform creates a multi-process architecture: + +:::image type="content" source="./media/platform-testhostcontroller-testhost.png" lightbox="./media/platform-testhostcontroller-testhost.png" alt-text="Diagram showing test host controller and test host processes."::: + +The **test host controller** (parent process): + +- Sets up the environment +- Spawns the **test host** (child process) with appropriate configuration +- Monitors test execution from outside +- Remains stable even if test code crashes the test host + +This model enables scenarios like: + +- Code coverage (requires environment variable setup) +- Hang detection (monitors process from outside) +- Crash recovery (parent survives child crashes) + +## Component types + +The platform consists of three main component types: + +### Test framework (mandatory) + +The test framework is the only mandatory component. It: + +- Discovers tests in your code +- Executes tests +- Reports results to the platform + +Examples: MSTest, NUnit, xUnit, TUnit + +For more information, see [Build a test framework](microsoft-testing-platform-architecture-testframework.md). + +### In-process extensions (optional) + +In-process extensions run inside the test host process alongside your tests. They can: + +- Consume test results as they're produced +- Add custom command-line options +- Participate in session lifecycle events +- Generate reports or artifacts + +Common uses: + +- TRX report generation +- Custom logging +- Test filtering +- Result aggregation + +For more information, see [In-process extensions](microsoft-testing-platform-extensions.md#in-process-extensions). + +### Out-of-process extensions (optional) + +Out-of-process extensions run in the test host controller, separate from your tests. They can: + +- Modify test host environment variables +- Monitor test execution externally +- Survive test host crashes +- Implement cross-cutting concerns safely + +Common uses: + +- Code coverage collection +- Hang detection +- Performance monitoring +- Resource cleanup + +For more information, see [Out-of-process extensions](microsoft-testing-platform-extensions.md#out-of-process-extensions). + +## Communication infrastructure + +The platform provides several mechanisms for components to communicate and share data. + +### Message bus + +The message bus is the central communication hub using a publish-subscribe pattern: + +:::image type="content" source="./media/message-bus.png" lightbox="./media/message-bus.png" alt-text="Diagram showing message bus with producers and consumers."::: + +- **Producers** publish data to the bus (test frameworks, extensions) +- **Consumers** subscribe to specific data types +- Enables loose coupling between components +- Supports version-independent evolution + +For more information, see [IMessageBus service](microsoft-testing-platform-architecture-services.md#the-imessagebus-service). + +### Capabilities + +The capability system allows components to declare and discover features: + +```csharp +// Extension queries test framework capabilities +var capabilities = serviceProvider.GetRequiredService(); +var parallelism = capabilities.GetCapability(); + +if (parallelism?.CanDisableParallelism == true) +{ + parallelism.Enable(); +} +``` + +This enables: + +- Feature negotiation between components +- Graceful degradation when features aren't available +- Version-independent compatibility + +For more information, see [Capabilities system](microsoft-testing-platform-architecture-capabilities.md). + +### Services + +The platform provides services through dependency injection: + +- **IConfiguration**: Access configuration from JSON files or environment variables +- **ICommandLineOptions**: Parse and access command-line arguments +- **IMessageBus**: Publish and subscribe to test data +- **ILoggerFactory**: Create loggers for diagnostic output +- **IOutputDevice**: Send information to the console or other output devices + +For more information, see [Platform services](microsoft-testing-platform-architecture-services.md). + +## Extension lifecycle + +Understanding when extensions are invoked helps you choose the right extension point. + +### High-level flow + +1. **Setup phase** (out-of-process): + - Environment variable providers configure test host environment + - Process lifetime handlers prepare for test host start + +2. **Test host startup** (in-process): + - Application lifecycle callbacks run first + - Session lifetime handlers initialize + - Test framework creates session + +3. **Test execution** (in-process): + - Test framework processes requests (discovery, execution) + - Data consumers receive and process test results + - Results flow through message bus + +4. **Teardown** (in-process): + - Test framework closes session + - Session lifetime handlers finalize + - Application lifecycle callbacks complete + - Extensions clean up + +5. **Monitoring** (out-of-process): + - Process lifetime handlers observe test host exit + +For a detailed execution order, see [Extension execution order](microsoft-testing-platform-extensions.md#extension-execution-order). + +## Choosing the right extension point + +Use this decision guide to select appropriate extension points: + +| If you need to... | Use this extension point | +|-------------------|--------------------------| +| Discover and run tests | [Test framework](microsoft-testing-platform-architecture-testframework.md) (mandatory) | +| Add command-line options | [ICommandLineOptionsProvider](microsoft-testing-platform-extensions.md#the-icommandlineoptionsprovider-extension) | +| Consume test results | [IDataConsumer](microsoft-testing-platform-extensions.md#the-idataconsumer-extension) | +| Run code before/after test session | [ITestSessionLifetimeHandler](microsoft-testing-platform-extensions.md#the-itestsessionlifetimehandler-extension) | +| Run code at process start/end | [ITestApplicationLifecycleCallbacks](microsoft-testing-platform-extensions.md#the-itestapplicationlifecyclecallbacks-extension) | +| Set environment variables | [ITestHostEnvironmentVariableProvider](microsoft-testing-platform-extensions.md#the-itesthostenvironmentvariableprovider-extension) | +| Monitor test process externally | [ITestHostProcessLifetimeHandler](microsoft-testing-platform-extensions.md#the-itesthostprocesslifetimehandler-extension) | + +## Registration patterns + +### Register a test framework + +```csharp +var builder = await TestApplication.CreateBuilderAsync(args); + +builder.RegisterTestFramework( + serviceProvider => new TestFrameworkCapabilities(), + (capabilities, serviceProvider) => new MyTestFramework(capabilities, serviceProvider)); + +using var app = await builder.BuildAsync(); +return await app.RunAsync(); +``` + +### Register in-process extensions + +```csharp +var builder = await TestApplication.CreateBuilderAsync(args); + +// Command-line options (available in both processes) +builder.CommandLine.AddProvider(() => new CustomCommandLineOptions()); + +// In-process extensions +builder.TestHost.AddDataConsumer(serviceProvider => new CustomDataConsumer()); +builder.TestHost.AddTestSessionLifetimeHandle(serviceProvider => new CustomLifetimeHandler()); + +using var app = await builder.BuildAsync(); +return await app.RunAsync(); +``` + +### Register out-of-process extensions + +```csharp +var builder = await TestApplication.CreateBuilderAsync(args); + +// Out-of-process extensions +builder.TestHostControllers.AddEnvironmentVariableProvider( + serviceProvider => new CustomEnvironmentVariableProvider()); +builder.TestHostControllers.AddProcessLifetimeHandler( + serviceProvider => new CustomProcessMonitor()); + +using var app = await builder.BuildAsync(); +return await app.RunAsync(); +``` + +## Next steps + +Now that you understand the high-level architecture, explore these areas based on your needs: + +- **Building a test framework**: [Test framework guide](microsoft-testing-platform-architecture-testframework.md) +- **Building extensions**: [Extensions guide](microsoft-testing-platform-extensions.md) +- **Understanding capabilities**: [Capabilities system](microsoft-testing-platform-architecture-capabilities.md) +- **Using platform services**: [Services reference](microsoft-testing-platform-architecture-services.md) +- **Working examples**: [Sample code](https://github.com/microsoft/testfx/tree/main/samples/public/TestingPlatformExamples) + +## See also + +- [Microsoft.Testing.Platform introduction](microsoft-testing-platform-intro.md) +- [Microsoft.Testing.Platform exit codes](microsoft-testing-platform-exit-codes.md) +- [Microsoft.Testing.Platform FAQ](microsoft-testing-platform-faq.md) diff --git a/docs/core/testing/microsoft-testing-platform-architecture-services.md b/docs/core/testing/microsoft-testing-platform-architecture-services.md index 9de66c0481eb5..fa8bd6f0ec8ca 100644 --- a/docs/core/testing/microsoft-testing-platform-architecture-services.md +++ b/docs/core/testing/microsoft-testing-platform-architecture-services.md @@ -1,77 +1,86 @@ --- -title: Microsoft.Testing.Platform services overview -description: Learn about the Microsoft.Testing.Platform available services. +title: Microsoft.Testing.Platform services reference +description: Comprehensive reference for services provided by Microsoft.Testing.Platform, including configuration, logging, message bus, and more. author: MarcoRossignoli ms.author: mrossignoli -ms.date: 07/11/2024 +ms.date: 11/27/2025 +ai-usage: ai-assisted --- -# Microsoft.Testing.Platform Services +# Microsoft.Testing.Platform services reference -The testing platform offers valuable services to both the testing framework and extension points. These services cater to common needs such as accessing the configuration, parsing and retrieving command-line arguments, obtaining the logging factory, and accessing the logging system, among others. `IServiceProvider` implements the _service locator pattern_ for the testing platform. +Microsoft.Testing.Platform provides services to test frameworks and extensions through dependency injection. Services handle common needs like configuration, logging, command-line parsing, and inter-component communication. -The `IServiceProvider` is derived directly from the base class library. +All services are accessed through `IServiceProvider`, which implements the service locator pattern. -```csharp -namespace System -{ - public interface IServiceProvider - { - object? GetService(Type serviceType); - } -} -``` +## Access services -The testing platform offers handy extension methods to access well-known service objects. All these methods are housed in a static class within the `Microsoft.Testing.Platform.Services` namespace. +Services are provided to your components through factory methods: ```csharp -public static class ServiceProviderExtensions -{ - public static TService GetRequiredService( - this IServiceProvider provider) - - public static TService? GetService( - this IServiceProvider provider) - - public static IMessageBus GetMessageBus( - this IServiceProvider serviceProvider) - - public static IConfiguration GetConfiguration( - this IServiceProvider serviceProvider) +// In test framework registration +builder.RegisterTestFramework( + serviceProvider => + { + // Access services here + var config = serviceProvider.GetConfiguration(); + var logger = serviceProvider.GetLoggerFactory().CreateLogger("MyFramework"); + + return new MyTestFrameworkCapabilities(); + }, + (capabilities, serviceProvider) => + { + // Or access services here + return new MyTestFramework(capabilities, serviceProvider); + }); - public static ICommandLineOptions GetCommandLineOptions( - this IServiceProvider serviceProvider) +// In extension registration +builder.TestHost.AddDataConsumer( + serviceProvider => + { + // Access services in extension factory + var messageBus = serviceProvider.GetMessageBus(); + var commandLine = serviceProvider.GetCommandLineOptions(); + + return new MyDataConsumer(messageBus, commandLine); + }); +``` - public static ILoggerFactory GetLoggerFactory( - this IServiceProvider serviceProvider) +## Service locator APIs - public static IOutputDevice GetOutputDevice( - this IServiceProvider serviceProvider) +The base `IServiceProvider` interface from .NET BCL: - // ... and more +```csharp +public interface IServiceProvider +{ + object? GetService(Type serviceType); } ``` -Most of the registration factories exposed by extension points provide access to the `IServiceProvider`: For example, when [registering the testing framework](./microsoft-testing-platform-architecture-extensions.md#register-a-testing-framework), the `IServiceProvider` is passed as a parameter to the factory method. +Platform extension methods for typed access: ```csharp -ITestApplicationBuilder RegisterTestFramework( - Func capabilitiesFactory, - Func adapterFactory); +public static class ServiceProviderExtensions +{ + // Generic service access + public static TService GetRequiredService(this IServiceProvider provider); + public static TService? GetService(this IServiceProvider provider); + + // Platform-specific services + public static IMessageBus GetMessageBus(this IServiceProvider provider); + public static IConfiguration GetConfiguration(this IServiceProvider provider); + public static ICommandLineOptions GetCommandLineOptions(this IServiceProvider provider); + public static ILoggerFactory GetLoggerFactory(this IServiceProvider provider); + public static IOutputDevice GetOutputDevice(this IServiceProvider provider); + public static IPlatformInformation GetPlatformInformation(this IServiceProvider provider); +} ``` -In the preceding code, both the `capabilitiesFactory` and the `adapterFactory` supply the `IServiceProvider` as a parameter. - -## The `IConfiguration` service - -The `IConfiguration` interface can be retrieved using the [`IServiceProvider`](#microsofttestingplatform-services) and provides access to the configuration settings for the testing framework and any extension points. By default, these configurations are loaded from: +## IConfiguration service -* Environment variables -* A JSON file named `[assemblyName].testingplatformconfig.json` located near the entry point assembly. +Provides access to configuration settings from JSON files and environment variables. -**The order of precedence is maintained, which means that if a configuration is found in the environment variables, the JSON file will not be processed.** - -The interface is a straightforward key-value pair of strings: +### Interface ```csharp public interface IConfiguration @@ -80,77 +89,149 @@ public interface IConfiguration } ``` -### JSON configuration file +### Configuration sources -The JSON file follows a hierarchical structure. To access child properties, you need to use the `:` separator. For example, consider a configuration for a potential testing framework like: +Configuration is loaded in order of precedence (first match wins): -```json +1. **Environment variables** (highest priority) +2. **JSON configuration file**: `[assemblyname].testingplatformconfig.json` + +### Access configuration + +```csharp +public class MyTestFramework : ITestFramework { - "CustomTestingFramework": { - "DisableParallelism": true - } + private readonly IConfiguration _config; + + public MyTestFramework(IServiceProvider serviceProvider) + { + _config = serviceProvider.GetConfiguration(); + + // Read settings + var timeout = _config["MyFramework:DefaultTimeout"]; + var parallelism = _config["MyFramework:MaxParallelism"]; + } } ``` -The code snippet would look something like this: +### JSON configuration format -```csharp -IServiceProvider serviceProvider = null; // Get the service provider... +Configuration files use hierarchical JSON. Access nested properties with `:` separator: -var configuration = serviceProvider.GetConfiguration(); +**File**: `MyTests.testingplatformconfig.json` -if (bool.TryParse(configuration["CustomTestingFramework:DisableParallelism"], out var value) && value is true) +```json { - // ... + "MyTestFramework": { + "DefaultTimeout": "30s", + "MaxParallelism": 4, + "Retry": { + "Enabled": true, + "MaxAttempts": 3 + } + } } ``` -In the case of an array, such as: +**Access in code**: + +```csharp +var timeout = _config["MyTestFramework:DefaultTimeout"]; // "30s" +var parallelism = _config["MyTestFramework:MaxParallelism"]; // "4" +var retryEnabled = _config["MyTestFramework:Retry:Enabled"]; // "true" +var maxAttempts = _config["MyTestFramework:Retry:MaxAttempts"]; // "3" +``` + +### Array configuration + +Arrays use zero-based indices: ```json { - "CustomTestingFramework": { - "Engine": [ - "ThreadPool", - "CustomThread" + "MyFramework": { + "IncludedCategories": [ + "Unit", + "Integration", + "E2E" ] } } ``` -The syntax to access to the fist element ("ThreadPool") is: - ```csharp -IServiceProvider serviceProvider = null; // Get the service provider... - -var configuration = serviceProvider.GetConfiguration(); - -var fistElement = configuration["CustomTestingFramework:Engine:0"]; +var first = _config["MyFramework:IncludedCategories:0"]; // "Unit" +var second = _config["MyFramework:IncludedCategories:1"]; // "Integration" ``` -### Environment variables +### Environment variable configuration -The `:` separator doesn't work with environment variable hierarchical keys on all platforms. `__`, the double underscore, is: +Use double underscore `__` instead of `:` for hierarchical keys: -* Supported by all platforms. For example, the `:` separator is not supported by [Bash](https://linuxhint.com/bash-environment-variables/), but `__` is. -* Automatically replaced by a `:` +**Windows**: -For instance, the environment variable can be set as follows (This example is applicable for Windows): +```cmd +setx MyFramework__DefaultTimeout=60s +setx MyFramework__MaxParallelism=8 +``` + +**Linux/macOS**: ```bash -setx CustomTestingFramework__DisableParallelism=True +export MyFramework__DefaultTimeout=60s +export MyFramework__MaxParallelism=8 ``` -You can choose not to use the environment variable configuration source when creating the `ITestApplicationBuilder`: +> [!NOTE] +> Double underscore (`__`) is supported on all platforms. Single colon (`:`) doesn't work in environment variable names on some systems. -```csharp -var options = new TestApplicationOptions(); +### Disable environment variable configuration -options.Configuration.ConfigurationSources.RegisterEnvironmentVariablesConfigurationSource = false; +```csharp +var options = new TestApplicationOptions +{ + Configuration = + { + ConfigurationSources = + { + RegisterEnvironmentVariablesConfigurationSource = false + } + } +}; var builder = await TestApplication.CreateBuilderAsync(args, options); ``` +### Best practices + +**DO**: + +- Use hierarchical naming for settings +- Prefix settings with your framework/extension name +- Document expected configuration keys +- Provide defaults for all settings +- Parse and validate configuration early + +**Example**: + +```csharp +private readonly TimeSpan _timeout; +private readonly int _maxParallelism; + +public MyTestFramework(IServiceProvider serviceProvider) +{ + var config = serviceProvider.GetConfiguration(); + + // Parse with defaults + _timeout = TimeSpan.TryParse( + config["MyFramework:DefaultTimeout"], + out var t) ? t : TimeSpan.FromSeconds(30); + + _maxParallelism = int.TryParse( + config["MyFramework:MaxParallelism"], + out var p) && p > 0 ? p : Environment.ProcessorCount; +} +``` + ## The `ICommandLineOptions` service The `ICommandLineOptions` service is utilized to fetch details regarding the command-line options that the platform has parsed. The APIs available include: diff --git a/docs/core/testing/microsoft-testing-platform-architecture-testframework.md b/docs/core/testing/microsoft-testing-platform-architecture-testframework.md new file mode 100644 index 0000000000000..1e450c1c03bb5 --- /dev/null +++ b/docs/core/testing/microsoft-testing-platform-architecture-testframework.md @@ -0,0 +1,709 @@ +--- +title: Build a test framework for Microsoft.Testing.Platform +description: Learn how to create a custom test framework that integrates with Microsoft.Testing.Platform to discover and execute tests. +author: MarcoRossignoli +ms.author: mrossignoli +ms.date: 11/27/2025 +ai-usage: ai-assisted +--- + +# Build a test framework for Microsoft.Testing.Platform + +A test framework is the core component that provides Microsoft.Testing.Platform with the ability to discover and execute tests. The test framework is responsible for: + +- Finding tests in your code +- Running those tests +- Reporting results back to the platform + +The test framework is the only **mandatory** component required to execute a testing session. This article explains how to build a custom test framework that integrates with the platform. + +> [!NOTE] +> Complete working examples are available in the [Microsoft Test Framework repository](https://github.com/microsoft/testfx/tree/main/samples/public/TestingPlatformExamples). + +## Test framework responsibilities + +Your test framework acts as an **in-process stateful server** with a well-defined lifecycle: + +:::image type="content" source="./media/test-framework-sequence-diagram.png" lightbox="./media/test-framework-sequence-diagram.png" alt-text="Sequence diagram showing test framework lifecycle from creation through disposal."::: + +The lifecycle consists of: + +1. **Creation**: Platform instantiates your test framework +2. **Session start**: Framework initializes for the test session +3. **Request processing**: Framework handles one or more requests (discovery, execution) +4. **Session close**: Framework finalizes the test session +5. **Disposal**: Framework cleans up resources + +## Register a test framework + +Register your test framework with the platform using the `RegisterTestFramework` API: + +```csharp +public static async Task Main(string[] args) +{ + ITestApplicationBuilder builder = await TestApplication.CreateBuilderAsync(args); + + builder.RegisterTestFramework( + serviceProvider => new TestingFrameworkCapabilities(), + (capabilities, serviceProvider) => new TestingFramework(capabilities, serviceProvider)); + + using ITestApplication app = await builder.BuildAsync(); + return await app.RunAsync(); +} +``` + +The `RegisterTestFramework` API requires two factory methods: + +### Capabilities factory + +```csharp +Func +``` + +This factory creates an object that describes your test framework's capabilities. The platform uses capabilities to: + +- Determine if your framework supports required features +- Enable extensions to adapt based on available features +- Avoid creating the framework if capabilities are insufficient + +Example: + +```csharp +internal class TestingFrameworkCapabilities : ITestFrameworkCapabilities +{ + public IReadOnlyCollection Capabilities => + [ + // Add capabilities your framework supports + ]; +} +``` + +For more information about capabilities, see [Capabilities system](microsoft-testing-platform-architecture-capabilities.md). + +### Test framework factory + +```csharp +Func +``` + +This factory creates your test framework implementation. It receives: + +- The `ITestFrameworkCapabilities` instance from the first factory +- An `IServiceProvider` for accessing platform services + +Example: + +```csharp +internal class TestingFramework : ITestFramework +{ + public TestingFramework( + ITestFrameworkCapabilities capabilities, + IServiceProvider serviceProvider) + { + // Store dependencies + // Access services like IConfiguration, IMessageBus, ILoggerFactory + } + + // Implement ITestFramework methods... +} +``` + +## Implement the ITestFramework interface + +The `ITestFramework` interface defines the contract for test frameworks: + +```csharp +public interface ITestFramework : IExtension +{ + Task CreateTestSessionAsync(CreateTestSessionContext context); + Task ExecuteRequestAsync(ExecuteRequestContext context); + Task CloseTestSessionAsync(CloseTestSessionContext context); +} +``` + +Your test framework must also implement the base `IExtension` interface: + +```csharp +public interface IExtension +{ + string Uid { get; } + string Version { get; } + string DisplayName { get; } + string Description { get; } + Task IsEnabledAsync(); +} +``` + +### Implement IExtension + +Provide descriptive information about your framework: + +```csharp +public string Uid => "MyTestFramework"; +public string Version => "1.0.0"; +public string DisplayName => "My Test Framework"; +public string Description => "A custom test framework for my tests"; + +public Task IsEnabledAsync() +{ + // Return true to enable the framework + // You can check configuration or command-line options here + return Task.FromResult(true); +} +``` + +### Implement CreateTestSessionAsync + +This method initializes your test framework at the start of the test session: + +```csharp +public Task CreateTestSessionAsync(CreateTestSessionContext context) +{ + // Session identifier for logging and correlation + SessionUid sessionId = context.SessionUid; + + // Information about who's running the tests (e.g., "testingplatform-console") + ClientInfo client = context.Client; + + // Cancellation token to honor + CancellationToken cancellationToken = context.CancellationToken; + + // Perform initialization (load configuration, set up state, etc.) + + return Task.FromResult(new CreateTestSessionResult + { + IsSuccess = true, + // Set WarningMessage or ErrorMessage if needed + }); +} +``` + +Return `IsSuccess = false` to halt test execution if initialization fails. + +### Implement ExecuteRequestAsync + +This method handles requests from the platform to discover or execute tests: + +```csharp +public async Task ExecuteRequestAsync(ExecuteRequestContext context) +{ + IRequest request = context.Request; + IMessageBus messageBus = context.MessageBus; + CancellationToken cancellationToken = context.CancellationToken; + + try + { + // Determine request type and handle accordingly + switch (request) + { + case DiscoverTestExecutionRequest discoverRequest: + await DiscoverTestsAsync(discoverRequest, messageBus, cancellationToken); + break; + + case RunTestExecutionRequest runRequest: + await RunTestsAsync(runRequest, messageBus, cancellationToken); + break; + + default: + throw new NotSupportedException($"Unsupported request type: {request.GetType()}"); + } + } + finally + { + // CRITICAL: Always call Complete() when done processing the request + context.Complete(); + } +} +``` + +> [!WARNING] +> You must call `context.Complete()` when you finish processing a request. Failing to do so causes the test application to hang indefinitely. + +### Implement CloseTestSessionAsync + +This method finalizes your test framework at the end of the test session: + +```csharp +public Task CloseTestSessionAsync(CloseTestSessionContext context) +{ + SessionUid sessionId = context.SessionUid; + CancellationToken cancellationToken = context.CancellationToken; + + // Perform cleanup (flush buffers, save state, etc.) + + return Task.FromResult(new CloseTestSessionResult + { + IsSuccess = true, + // Set WarningMessage or ErrorMessage if needed + }); +} +``` + +## Handle discovery requests + +The `DiscoverTestExecutionRequest` instructs your framework to discover tests and report them via the message bus: + +```csharp +private async Task DiscoverTestsAsync( + DiscoverTestExecutionRequest request, + IMessageBus messageBus, + CancellationToken cancellationToken) +{ + // Find all tests in your test assembly + foreach (var test in FindTests()) + { + var testNode = new TestNode + { + Uid = new TestNodeUid(test.UniqueId), + DisplayName = test.DisplayName, + Properties = new PropertyBag(DiscoveredTestNodeStateProperty.CachedInstance) + }; + + await messageBus.PublishAsync( + this, + new TestNodeUpdateMessage(request.Session.SessionUid, testNode)); + } +} +``` + +Key points: + +- **Unique ID**: Generate a stable, unique ID for each test that remains consistent across runs +- **Display name**: Provide a human-readable name +- **State property**: Use `DiscoveredTestNodeStateProperty` for discovered tests + +## Handle execution requests + +The `RunTestExecutionRequest` instructs your framework to execute tests and report results: + +```csharp +private async Task RunTestsAsync( + RunTestExecutionRequest request, + IMessageBus messageBus, + CancellationToken cancellationToken) +{ + // Execute all tests in your test assembly + foreach (var test in FindTests()) + { + var result = await test.RunAsync(cancellationToken); + + var testNode = new TestNode + { + Uid = new TestNodeUid(test.UniqueId), + DisplayName = test.DisplayName, + Properties = new PropertyBag(CreateStateProperty(result)) + }; + + await messageBus.PublishAsync( + this, + new TestNodeUpdateMessage(request.Session.SessionUid, testNode)); + } +} + +private IProperty CreateStateProperty(TestResult result) +{ + return result.Status switch + { + TestStatus.Passed => PassedTestNodeStateProperty.CachedInstance, + TestStatus.Failed => new FailedTestNodeStateProperty(result.Exception), + TestStatus.Skipped => SkippedTestNodeStateProperty.CachedInstance, + _ => throw new InvalidOperationException($"Unknown status: {result.Status}") + }; +} +``` + +## Report test results with TestNodeUpdateMessage + +The `TestNodeUpdateMessage` is the primary way to communicate test information to the platform: + +```csharp +public sealed class TestNodeUpdateMessage +{ + public TestNode TestNode { get; } + public TestNodeUid? ParentTestNodeUid { get; } +} + +public class TestNode +{ + public required TestNodeUid Uid { get; init; } + public required string DisplayName { get; init; } + public PropertyBag Properties { get; init; } = new(); +} +``` + +### The TestNode structure + +A `TestNode` represents a single test or test container with: + +- **Uid**: A unique, stable identifier (must be identical across runs and platforms) +- **DisplayName**: Human-readable name shown in test runners and reports +- **Properties**: A collection of properties describing the test's state and metadata +- **ParentTestNodeUid** (optional): Creates a tree structure for hierarchical tests + +### Test node properties + +The platform recognizes specific properties to determine test state and metadata. + +#### Required state properties (one per test) + +Each test node must have **exactly one** state property: + +**For discovery:** + +```csharp +DiscoveredTestNodeStateProperty.CachedInstance +``` + +**For execution (choose one):** + +```csharp +// Test passed +PassedTestNodeStateProperty.CachedInstance + +// Test failed (assertion failure) +new FailedTestNodeStateProperty(exception) + +// Test had an error (not an assertion failure) +new ErrorTestNodeStateProperty(exception) + +// Test was skipped +SkippedTestNodeStateProperty.CachedInstance + +// Test timed out +new TimeoutTestNodeStateProperty(exception) { Timeout = timeoutDuration } + +// Test was cancelled +new CancelledTestNodeStateProperty(exception) + +// Test is in progress (optional, for real-time updates) +InProgressTestNodeStateProperty.CachedInstance +``` + +#### Optional metadata properties + +Enhance test information with additional properties: + +**Source location:** + +```csharp +new TestFileLocationProperty( + filePath: "C:\\Tests\\MyTests.cs", + lineSpan: new LinePositionSpan( + new LinePosition(10, 0), + new LinePosition(20, 0))) +``` + +**Method identifier:** + +```csharp +new TestMethodIdentifierProperty( + assemblyFullName: "MyTests, Version=1.0.0.0", + @namespace: "MyNamespace", + typeName: "MyTestClass", + methodName: "MyTestMethod", + parameterTypeFullNames: new[] { "System.String", "System.Int32" }, + returnTypeFullName: "System.Void") +``` + +**Timing information:** + +```csharp +new TimingProperty( + globalTiming: new TimingInfo( + startTime: DateTimeOffset.Now, + endTime: DateTimeOffset.Now.AddSeconds(1), + duration: TimeSpan.FromSeconds(1))) +``` + +**Test traits/metadata:** + +```csharp +new TestMetadataProperty("Category", "Integration") +new TestMetadataProperty("Priority", "High") +``` + +**Custom key-value pairs:** + +```csharp +new KeyValuePairStringProperty("CustomData", "CustomValue") +``` + +### Example: Complete test reporting + +```csharp +private async Task ReportTestResultAsync( + TestInfo test, + TestResult result, + SessionUid sessionUid, + IMessageBus messageBus) +{ + var properties = new PropertyBag(); + + // Required: Test state + properties.Add(result.Status switch + { + TestStatus.Passed => PassedTestNodeStateProperty.CachedInstance, + TestStatus.Failed => new FailedTestNodeStateProperty(result.Exception, result.Message), + TestStatus.Skipped => SkippedTestNodeStateProperty.CachedInstance, + _ => throw new InvalidOperationException() + }); + + // Optional: Source location + if (test.SourceLocation != null) + { + properties.Add(new TestFileLocationProperty( + test.SourceLocation.FilePath, + test.SourceLocation.LineSpan)); + } + + // Optional: Timing + if (result.Timing != null) + { + properties.Add(new TimingProperty(result.Timing)); + } + + // Optional: Traits + foreach (var trait in test.Traits) + { + properties.Add(new TestMetadataProperty(trait.Key, trait.Value)); + } + + var testNode = new TestNode + { + Uid = new TestNodeUid(test.UniqueId), + DisplayName = test.DisplayName, + Properties = properties + }; + + await messageBus.PublishAsync( + this, + new TestNodeUpdateMessage(sessionUid, testNode)); +} +``` + +## Declare produced data types + +Your test framework must declare what types of data it publishes to the message bus: + +```csharp +public class TestingFramework : ITestFramework, IDataProducer +{ + public Type[] DataTypesProduced => new[] + { + typeof(TestNodeUpdateMessage), + // Add other types if you publish additional data + }; + + // ... ITestFramework implementation +} +``` + +## Use platform services + +Access platform services through the `IServiceProvider` passed to your factory: + +```csharp +public class TestingFramework : ITestFramework +{ + private readonly IConfiguration _configuration; + private readonly ILogger _logger; + + public TestingFramework( + ITestFrameworkCapabilities capabilities, + IServiceProvider serviceProvider) + { + // Access configuration + _configuration = serviceProvider.GetConfiguration(); + + // Create a logger + var loggerFactory = serviceProvider.GetLoggerFactory(); + _logger = loggerFactory.CreateLogger(); + + // Access command-line options + var commandLineOptions = serviceProvider.GetCommandLineOptions(); + } + + // ... implementation +} +``` + +For more information, see [Platform services](microsoft-testing-platform-architecture-services.md). + +## Implement async initialization + +If your framework requires async initialization, implement `IAsyncInitializableExtension`: + +```csharp +public class TestingFramework : ITestFramework, IAsyncInitializableExtension +{ + public async Task InitializeAsync() + { + // Perform async initialization + // This is called after the constructor + await LoadConfigurationAsync(); + await ConnectToDatabaseAsync(); + } + + // ... ITestFramework implementation +} +``` + +Similarly, implement `IAsyncCleanableExtension` for async cleanup: + +```csharp +public class TestingFramework : ITestFramework, IAsyncCleanableExtension +{ + public async Task CleanupAsync() + { + // Perform async cleanup + // This is called before disposal + await FlushBuffersAsync(); + await DisconnectFromDatabaseAsync(); + } + + // ... ITestFramework implementation +} +``` + +## Handle cancellation + +Always honor cancellation tokens to allow users to interrupt test execution: + +```csharp +private async Task RunTestAsync(TestInfo test, CancellationToken cancellationToken) +{ + // Check before starting + cancellationToken.ThrowIfCancellationRequested(); + + // Pass to async operations + await test.SetupAsync(cancellationToken); + + // Check periodically during long operations + if (cancellationToken.IsCancellationRequested) + { + return TestResult.Cancelled; + } + + await test.ExecuteAsync(cancellationToken); +} +``` + +## Error handling best practices + +### Distinguish between test failures and errors + +Use `FailedTestNodeStateProperty` for assertion failures: + +```csharp +try +{ + test.Execute(); +} +catch (AssertionException ex) +{ + // Assertion failure - expected test failure + return new FailedTestNodeStateProperty(ex); +} +``` + +Use `ErrorTestNodeStateProperty` for unexpected errors: + +```csharp +try +{ + test.Setup(); + test.Execute(); +} +catch (AssertionException ex) +{ + return new FailedTestNodeStateProperty(ex); +} +catch (Exception ex) +{ + // Unexpected error during setup or execution + return new ErrorTestNodeStateProperty(ex); +} +``` + +### Report framework errors + +If your framework encounters errors that prevent test execution, report them through session results: + +```csharp +public Task CreateTestSessionAsync(CreateTestSessionContext context) +{ + try + { + Initialize(); + return Task.FromResult(new CreateTestSessionResult { IsSuccess = true }); + } + catch (Exception ex) + { + return Task.FromResult(new CreateTestSessionResult + { + IsSuccess = false, + ErrorMessage = $"Failed to initialize: {ex.Message}" + }); + } +} +``` + +## Configuration and customization + +Allow users to configure your framework through: + +### Configuration files + +```csharp +var config = serviceProvider.GetConfiguration(); +var parallelism = config["MyTestFramework:Parallelism"]; +var timeout = config["MyTestFramework:DefaultTimeout"]; +``` + +Users can provide settings in `[assemblyname].testingplatformconfig.json`: + +```json +{ + "MyTestFramework": { + "Parallelism": 4, + "DefaultTimeout": "30s" + } +} +``` + +### Command-line options + +Implement `ICommandLineOptionsProvider` to add custom options. For more information, see [Command-line options provider](microsoft-testing-platform-extensions.md#the-icommandlineoptionsprovider-extension). + +## Test framework checklist + +Use this checklist to ensure your test framework is complete: + +- [ ] Implements `ITestFramework` with all required methods +- [ ] Implements `IExtension` with descriptive properties +- [ ] Implements `IDataProducer` declaring `TestNodeUpdateMessage` +- [ ] Implements `ITestFrameworkCapabilities` (can be empty initially) +- [ ] Generates stable, unique test IDs +- [ ] Calls `context.Complete()` after processing each request +- [ ] Publishes test results via `IMessageBus` +- [ ] Uses appropriate state properties (Passed, Failed, Skipped, etc.) +- [ ] Honors cancellation tokens +- [ ] Handles errors gracefully +- [ ] Returns meaningful session results +- [ ] Supports async initialization if needed +- [ ] Implements cleanup if needed +- [ ] Provides configuration options +- [ ] Includes diagnostic logging + +## Next steps + +- **Add command-line options**: [ICommandLineOptionsProvider](microsoft-testing-platform-extensions.md#the-icommandlineoptionsprovider-extension) +- **Understand capabilities**: [Capabilities system](microsoft-testing-platform-architecture-capabilities.md) +- **Use platform services**: [Platform services](microsoft-testing-platform-architecture-services.md) +- **View complete examples**: [Sample code](https://github.com/microsoft/testfx/tree/main/samples/public/TestingPlatformExamples) + +## See also + +- [Architecture overview](microsoft-testing-platform-architecture-overview.md) +- [Extensions guide](microsoft-testing-platform-extensions.md) +- [Platform services](microsoft-testing-platform-architecture-services.md) diff --git a/docs/core/testing/microsoft-testing-platform-architecture.md b/docs/core/testing/microsoft-testing-platform-architecture.md index 1385d0ccf05ff..67e37507651f8 100644 --- a/docs/core/testing/microsoft-testing-platform-architecture.md +++ b/docs/core/testing/microsoft-testing-platform-architecture.md @@ -1,167 +1,156 @@ --- -title: Microsoft.Testing.Platform architecture overview -description: Learn about Microsoft.Testing.Platform architecture. +title: Microsoft.Testing.Platform architecture +description: Navigation page for Microsoft.Testing.Platform architecture documentation, including guides for building test frameworks and extensions. author: MarcoRossignoli ms.author: mrossignoli -ms.date: 07/11/2024 +ms.date: 11/27/2025 +ai-usage: ai-assisted --- # Microsoft.Testing.Platform architecture -Welcome to our new test platform! To help you get acquainted with its capabilities, we'll start with a simple example that demonstrates how to register and run a test. This foundational example will give you a solid understanding of the core functionality and how to get started quickly. +This article provides links to comprehensive documentation about Microsoft.Testing.Platform's architecture, extensibility model, and services. -> [!NOTE] -> All of the concepts in this article are exemplified in the Microsoft Test Framework repository as a complete sample. For more information, see the [Sample code](https://github.com/microsoft/testfx/tree/main/samples/public/TestingPlatformExamples). +## Getting started -[Step 1: Register and Run a simple test application](#step-1-register-and-run-a-simple-test-application) +If you're new to Microsoft.Testing.Platform architecture, start here: -In this initial example, you walk through the basic steps to declare and run a test application. This straightforward approach ensures that you can immediately start using the platform with minimal setup. +- **[Architecture overview](microsoft-testing-platform-architecture-overview.md)**: High-level concepts, process models, and component types +- **[Microsoft.Testing.Platform introduction](microsoft-testing-platform-intro.md)**: Platform features and capabilities -[Step 2: Extending the Platform](#step-2-extending-the-platform) +## Building components -After you've discovered how to create your first test application, you explore an example of extension to cover partially the concepts surrounding the test application extensions. +### Test frameworks -[Step 3: Comprehensive Overview of Extension Points](#step-3-comprehensive-overview-of-extension-points) +Test frameworks are the core components that discover and execute tests: -Once you're comfortable with the basics, you delve into the various extension points. This will include: +- **[Build a test framework](microsoft-testing-platform-architecture-testframework.md)**: Complete guide to creating custom test frameworks + - Register test frameworks + - Handle discovery and execution requests + - Report test results + - Test node properties reference -1. **Platform and Test Framework Capabilities**: Understanding the full range of capabilities provided by the platform and the test framework, allowing you to leverage them effectively. +### Extensions -1. **Custom Test Framework**: How to write and register your custom test framework, enabling you to tailor the testing environment to your specific requirements. +Extensions add custom functionality to the platform: -1. **In-Process and Out-of-Process Extensions**: Detailed instructions on how to write and register both in-process and out-of-process extensions, offering flexibility in how you extend the platform. +- **[Build extensions](microsoft-testing-platform-extensions.md)**: Guide to building platform extensions + - In-process extensions (data consumers, lifetime handlers) + - Out-of-process extensions (environment providers, process monitors) + - Command-line option providers + - Extension execution order -1. **Order of Execution**: Clarifying the order of execution for the various extension points to ensure seamless integration and operation of your custom extensions. +## Core concepts -[Step 4: Available Services and Helpers](#step-4-available-services) +### Capabilities -Finally, you review an exhaustive list of the available services and helper functions within the platform. This section will serve as a reference to help you leverage all the tools at your disposal for creating robust and efficient test extensions. +The capabilities system enables feature negotiation between components: -By following this structured approach, you will gain a comprehensive understanding of our test platform, from basic usage to advanced customization. Let's get started and explore the full potential of what our platform can offer! +- **[Capabilities system](microsoft-testing-platform-architecture-capabilities.md)**: How components discover and declare features + - Query capabilities + - Design custom capabilities + - Built-in capabilities + - Best practices -## Step 1: Register and Run a simple test application +### Services -To introduce the architecture of the testing platform, this document will use the classic console application (for Windows) as the host. The samples in this document are written in C#, but you can use the testing platform with any language that supports the .NET Ecma specification, and run on any OS supported by .NET. To use the platform, simply reference the `Microsoft.Testing.Platform.dll` assembly, which can be consumed through the official NuGet package available at . +Platform services provide common functionality: -In a console project `Contoso.UnitTests.exe`, the following `Main` method defines the entry point: +- **[Services reference](microsoft-testing-platform-architecture-services.md)**: Comprehensive service documentation + - IConfiguration: Configuration management + - ICommandLineOptions: Command-line parsing + - IMessageBus: Inter-component communication + - ILoggerFactory: Diagnostic logging + - IOutputDevice: Console output + - IPlatformInformation: Platform details -```csharp -public static async Task Main(string[] args) -{ - ITestApplicationBuilder builder = await TestApplication.CreateBuilderAsync(args); - - builder.RegisterTestFramework(); - - using ITestApplication testApp = await builder.BuildAsync(); - - return await testApp.RunAsync(); -} -``` - -This code includes everything needed to execute a test session, except for registering a test framework such as MSTest through `RegisterTestFramework`. This code is shown and explained in later sections. - -> [!NOTE] -> The default behavior is to have the entry point automatically generated by the testing platform MSBuild task. This is done by adding the `Microsoft.Testing.Platform.MSBuild` package to your project. - -When `Contoso.UnitTests.exe` application is started a standard Windows process is created, and the testing platform interacts with the registered testing framework to execute the testing session. - -A single process is created to carry out this work: +## Quick reference - -:::image type="content" source="./media/platform-testhost.png" lightbox="./media/platform-testhost.png" alt-text="A diagram representing the test host process."::: +### Component registration patterns -The testing platform includes a built-in display device that writes the testing session information in the terminal, similar to: +**Test framework**: -```bash -Microsoft(R) Testing Platform Execution Command Line Tool -Version: 1.1.0+8c0a8fd8e (UTC 2024/04/03) -RuntimeInformation: win-x64 - .NET 9.0.0-preview.1.24080.9 -Copyright(c) Microsoft Corporation.  All rights reserved. -Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 5ms - Contoso.UnitTests.dll (win-x64 - .NET 9.0.0-preview.1.24080.9) +```csharp +builder.RegisterTestFramework( + sp => new MyFrameworkCapabilities(), + (cap, sp) => new MyFramework(cap, sp)); ``` -> [!NOTE] -> The known exit codes returned by the `ITestApplication.RunAsync()` call are detailed in [platform exit codes](./microsoft-testing-platform-exit-codes.md). - -## Step 2: Extending the platform - -Test runs commonly collect code coverage information, or similar information to evaluate code quality. Such workloads may require configuration before the test host process starts, for example setting environment variables. - -The testing platform accommodates this by having **out-of-process** extensions. When running with an out-of-process extensions, the testing platform will start multiple processes and it will manage them appropriately. - -The following example demonstrates how to register a code coverage feature using a **TestHostController** extension. +**In-process extension**: ```csharp -ITestApplicationBuilder builder = await TestApplication.CreateBuilderAsync(args); - -builder.RegisterTestFramework(); -builder.AddCodeCoverage(); - -using ITestApplication testApp = await builder.BuildAsync(); - -return await testApp.RunAsync(); +builder.TestHost.AddDataConsumer(sp => new MyConsumer(sp)); ``` -The `builder.AddCodeCoverage();` internally uses the `TestHostController` extensibility point, which is an out-of-process extensibility point. +**Out-of-process extension**: ```csharp -public static class TestApplicationBuilderExtensions -{ - public static ITestApplicationBuilder AddCodeCoverage( - this ITestApplicationBuilder builder) - { - builder.TestHostControllers - .AddEnvironmentVariableProvider(/* ... */); - - // ... - - return builder; - } -} +builder.TestHostControllers.AddProcessLifetimeHandler(sp => new MyMonitor(sp)); ``` -The parameters for the api `AddEnvironmentVariableProvider` will be explained in later sections. - -When running `Contoso.UnitTests.exe` this time, the testing platform detects that a `TestHostController` extension is registered. As a result, it starts another instance of the `Contoso.UnitTests.exe` process as a child process. This is done to properly set the environment variables as required by the extension registered with the `AddEnvironmentVariableProvider` API. +**Command-line options** (both contexts): -The process layout looks like this: - - -:::image type="content" source="./media/platform-testhostcontroller-testhost.png" lightbox="./media/platform-testhostcontroller-testhost.png" alt-text="A diagram representing the process layout of the test host and test host controller."::: - -> [!NOTE] -> The provided example assumes a console application layout, which handles the start process correctly and propagates all command line arguments to the child process. -> If you are using a different host, you need to ensure that the entry point code correctly forwards the process entry point (the "Main") to the appropriate code block. -> The runtime simply starts itself with the same command line arguments. +```csharp +builder.CommandLine.AddProvider(() => new MyOptions()); +``` -The above section provides a brief introduction to the architecture of the testing platform. The current extensibility points are divided into two categories: +### Extension types at a glance -1. **In process** extensions can be accessed through the `TestHost` property of the test application builder. In process means that they will run in the same process as the test framework. +| Extension | Context | Use case | +|-----------|---------|----------| +| IDataConsumer | In-process | Process test results, generate reports | +| ITestSessionLifetimeHandler | In-process | Run code before/after test session | +| ITestApplicationLifecycleCallbacks | In-process | Run code at process start/end | +| ICommandLineOptionsProvider | Both | Add custom command-line options | +| ITestHostEnvironmentVariableProvider | Out-of-process | Set environment variables for test host | +| ITestHostProcessLifetimeHandler | Out-of-process | Monitor test host externally | - ```csharp - ITestApplicationBuilder builder = await TestApplication.CreateBuilderAsync(args); - - builder.RegisterTestFramework(); - - builder.TestHost.AddXXX(/* ... */); - ``` +### Process architecture - As observed, the most crucial extension point is the in-process *testing framework* (`RegisterTestFramework`), which is the only **mandatory** one. +**Single process** (no out-of-process extensions): -1. **Out of process** extensions can be accessed through the `TestHostControllers` property of the test application builder. These extensions run in a separate process from the test framework to "observe" it. +``` +┌─────────────────┐ +│ Test Host │ +│ ┌───────────┐ │ +│ │ Tests │ │ +│ └───────────┘ │ +│ ┌───────────┐ │ +│ │Framework │ │ +│ └───────────┘ │ +│ ┌───────────┐ │ +│ │Extensions │ │ +│ └───────────┘ │ +└─────────────────┘ +``` - ```csharp - ITestApplicationBuilder builder = await TestApplication.CreateBuilderAsync(args); - - builder.TestHostControllers.AddXXX(/* ... */); - ``` +**Multi-process** (with out-of-process extensions): -## Step 3: Comprehensive Overview of Extension points +``` +┌───────────────────┐ +│ Test Host │ +│ Controller │ +│ ┌─────────────┐ │ +│ │ Out-of-proc │ │ +│ │ Extensions │ │ +│ └─────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────┐ │ +│ │ Test Host │ │ +│ │ (spawned) │ │ +│ └─────────────┘ │ +└───────────────────┘ +``` -Let's start by getting familiar with the concept of [capabilities](./microsoft-testing-platform-architecture-capabilities.md) before diving into the various [extensions points](./microsoft-testing-platform-architecture-extensions.md). +## Additional resources -## Step 4: Available services +- **[Sample code](https://github.com/microsoft/testfx/tree/main/samples/public/TestingPlatformExamples)**: Complete working examples +- **[Exit codes](microsoft-testing-platform-exit-codes.md)**: Platform exit code reference +- **[FAQ](microsoft-testing-platform-faq.md)**: Frequently asked questions -The testing platform offers valuable services to both the testing framework and extension points. These services cater to common needs such as accessing the configuration, parsing and retrieving command-line arguments, obtaining the logging factory, and accessing the logging system, among others. `IServiceProvider` implements the _service locator pattern_ for the testing platform. +## See also -All the services, helpers, and technical information about how to access and use these services is listed in [Microsoft.Testing.Platform Services documentation](./microsoft-testing-platform-architecture-services.md). +- [Microsoft.Testing.Platform vs VSTest](microsoft-testing-platform-vs-vstest.md) +- [MSTest runner](unit-testing-mstest-runner-intro.md) +- [NUnit runner](unit-testing-nunit-runner-intro.md) diff --git a/docs/core/testing/microsoft-testing-platform-config.md b/docs/core/testing/microsoft-testing-platform-config.md deleted file mode 100644 index 54c21fad0c2e5..0000000000000 --- a/docs/core/testing/microsoft-testing-platform-config.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -title: Microsoft.Testing.Platform config options -description: Learn how to configure Microsoft.Testing.Platform using configuration settings. -author: Evangelink -ms.author: amauryleve -ms.date: 08/15/2024 ---- - -# Microsoft.Testing.Platform configuration settings - -Microsoft.Testing.Platform supports the use of configuration files and environment variables to configure the behavior of the test platform. This article describes the configuration settings that you can use to configure the test platform. - -## testconfig.json - -The test platform uses a configuration file named *[appname].testconfig.json* to configure the behavior of the test platform. The *testconfig.json* file is a JSON file that contains configuration settings for the test platform. - -The *testconfig.json* file has the following structure: - -```json -{ - "platformOptions": { - "config-property-name1": "config-value1", - "config-property-name2": "config-value2" - } -} -``` - -The platform will automatically detect and load the *[appname].testconfig.json* file located in the output directory of the test project (close to the executable). - -When using [Microsoft.Testing.Platform.MSBuild](https://www.nuget.org/packages/Microsoft.Testing.Platform.MSBuild), you can simply create a *testconfig.json* file that will be automatically renamed to *[appname].testconfig.json* and moved to the output directory of the test project. - -Starting with Microsoft.Testing.Platform 1.5, you can use the command-line argument `--config-file` to specify the path to the *testconfig.json*. This file takes precedence over the *[appname].testconfig.json* file. - -> [!NOTE] -> The *[appname].testconfig.json* file will get overwritten on subsequent builds. - -## Environment variables - -Environment variables can be used to supply some runtime configuration information. - -> [!NOTE] -> Environment variables take precedence over configuration settings in the *testconfig.json* file. - -### `TESTINGPLATFORM_UI_LANGUAGE` environment variable - -Starting with Microsoft.Testing.Platform 1.5, this environment variable sets the language of the platform for displaying messages and logs using a locale value such as `en-us`. This language takes precedence over the Visual Studio and .NET SDK languages. The supported values are the same as for Visual Studio. For more information, see the section on changing the installer language in the [Visual Studio installation documentation](/visualstudio/install/install-visual-studio). diff --git a/docs/core/testing/microsoft-testing-platform-exit-codes.md b/docs/core/testing/microsoft-testing-platform-exit-codes.md index 9fc5046812a8a..c2197c97d9751 100644 --- a/docs/core/testing/microsoft-testing-platform-exit-codes.md +++ b/docs/core/testing/microsoft-testing-platform-exit-codes.md @@ -3,35 +3,339 @@ title: Microsoft.Testing.Platform exit codes description: Learn about the various Microsoft.Testing.Platform exit codes and their meaning. author: nohwnd ms.author: jajares -ms.date: 12/15/2023 +ms.date: 11/27/2025 ms.topic: reference --- # Microsoft.Testing.Platform exit codes -`Microsoft.Testing.Platform` uses known exit codes to communicate test failure or app errors. Exit codes start at `0` and are non-negative. Consider the following table that details the various exit codes and their corresponding reasons: +Microsoft.Testing.Platform uses exit codes to communicate the results of test execution. Understanding these exit codes helps you troubleshoot test failures, configure CI/CD pipelines, and automate testing workflows. + +## When to check exit codes + +Check exit codes in these scenarios: + +- **CI/CD pipelines fail** - Determine whether tests failed, crashed, or encountered configuration issues. +- **Automated scripts** - Make decisions based on specific test outcomes. +- **Debugging test runs** - Understand why a test session didn't complete as expected. +- **Configuring build systems** - Decide which exit codes should fail the build. + +## Exit code reference + +Microsoft.Testing.Platform uses exit codes starting at `0` and are non-negative. The following table details each exit code and its meaning: | Exit code | Details | |-----|----------| -| `0` | The `0` exit code indicates success. All tests that were chosen to run ran to completion and there were no errors. | -| `1` | The `1` exit code indicates unknown errors and acts as a _catch all_. To find additional error information and details, look in the output. | -| `2` | An exit code of `2` is used to indicate that there was at least one test failure. | -| `3` | The exit code `3` indicates that the test session was aborted. A session can be aborted using Ctrl+C, as an example. | -| `4` | The exit code `4` indicates that the setup of used extensions is invalid and the tests session cannot run. | -| `5` | The exit code `5` indicates that the command line arguments passed to the test app are invalid. | -| `6` | The exit code `6` indicates that the test session is using a non-implemented feature. | -| `7` | The exit code `7` indicates that a test session was unable to complete successfully, and likely crashed. It's possible that this was caused by a test session that was run via a test controller's extension point. | -| `8` | The exit code `8` indicates that the test session ran zero tests. | -| `9` | The exit code `9` indicates that the minimum execution policy for the executed tests was violated. | -| `10` | The exit code `10` indicates that the test adapter, Testing.Platform Test Framework, MSTest, NUnit, or xUnit, failed to run tests for an infrastructure reason unrelated to the test's self. An example is failing to create a fixture needed by tests. | -| `11` | The exit code `11` indicates that the test process will exit if dependent process exits. | -| `12` | The exit code `12` indicates that the test session was unable to run because the client does not support any of the supported protocol versions. | -| `13` | The exit code `13` indicates that the test session was stopped due to reaching the specified number of maximum failed tests using `--maximum-failed-tests` command-line option. For more information, see [the Options section in Microsoft.Testing.Platform overview](microsoft-testing-platform-intro.md#options) | - -To enable verbose logging and troubleshoot issues, see [Microsoft.Testing.Platform Diagnostics extensions](microsoft-testing-platform-extensions-diagnostics.md#built-in-options). - -## Ignore specific exit codes - -`Microsoft.Testing.Platform` is designed to be strict by default but allows for configurability. As such, it's possible for users to decide which exit codes should be ignored (an exit code of `0` will be returned instead of the original exit code). - -To ignore specific exit codes, use the `--ignore-exit-code` command line option or the `TESTINGPLATFORM_EXITCODE_IGNORE` environment variable. The valid format accepted is a semi-colon separated list of exit codes to ignore (for example, `--ignore-exit-code 2;3;8`). A common scenario is to consider that test failures shouldn't result in a nonzero exit code (which corresponds to ignoring exit-code `2`). +| `0` | Success. All tests that were chosen to run ran to completion, and there were no errors. | +| `1` | Unknown error. Acts as a _catch all_ for unexpected failures. Check the diagnostic output for details. For more information, see [Enable diagnostics](microsoft-testing-platform-options.md#diagnostic-options). | +| `2` | Test failure. At least one test failed during execution. Review the test output to identify which tests failed and why. | +| `3` | Test session aborted. The test session was canceled, typically by pressing Ctrl+C or through a termination signal. | +| `4` | Invalid extension setup. The configuration of test extensions is invalid, and the test session can't run. Check your project references and extension configuration. | +| `5` | Invalid command-line arguments. The arguments passed to the test app are invalid. Use `--help` to see available options. For more information, see [Configuration reference](microsoft-testing-platform-options.md). | +| `6` | Feature not implemented. The test session attempted to use a feature that isn't implemented. Check that all extensions support the features you're using. | +| `7` | Test host crash. The test session was unable to complete successfully and likely crashed. Check for unhandled exceptions or memory issues in your tests. Enable crash dumps for more information. For more information, see [Crash dump options](microsoft-testing-platform-options.md#crash-dump-options). | +| `8` | No tests ran. The test session completed but found zero tests to execute. This can happen when all tests are ignored, or filter expressions match no tests. | +| `9` | Minimum execution policy violated. The test session didn't meet the minimum expected number of tests configured with `--minimum-expected-tests`. | +| `10` | Test adapter infrastructure failure. The test framework (MSTest, NUnit, or xUnit) failed to run tests due to an infrastructure issue unrelated to test code. For example, failing to create a fixture needed by tests. | +| `11` | Dependent process exited. The test process exited because a dependent process (specified with `--exit-on-process-exit`) terminated. | +| `12` | Protocol version mismatch. The test session couldn't run because the client doesn't support any of the supported protocol versions. Update your test framework or tools. | +| `13` | Maximum failed tests reached. The test session was stopped after reaching the specified number of maximum failed tests using `--maximum-failed-tests`. For more information, see [Test execution options](microsoft-testing-platform-options.md#test-execution-options). | + +## Common scenarios and solutions + +This section provides practical guidance for the most frequently encountered exit codes. + +### Exit code 2: Test failures + +**What it means**: One or more tests failed during execution. + +**What to do**: + +1. **Review test output** to identify which tests failed: + + ```dotnetcli + dotnet run --project MyTests + ``` + +2. **Run only failed tests** using a filter: + + ```dotnetcli + dotnet run --project MyTests -- --filter "FullyQualifiedName~FailedTestName" + ``` + +3. **Enable detailed diagnostics** for more information: + + ```dotnetcli + dotnet run --project MyTests -- --diagnostic --diagnostic-verbosity Trace + ``` + +4. **Ignore in specific scenarios** where test failures shouldn't fail the build: + + ```dotnetcli + dotnet run --project MyTests -- --ignore-exit-code 2 + ``` + +For more information about filtering tests, see [Running selective unit tests](selective-unit-tests.md). + +### Exit code 8: No tests ran + +**What it means**: The test session completed but found no tests to execute. + +**Common causes**: + +- All tests are explicitly ignored or skipped +- Filter expression matched no tests +- Test discovery failed +- Tests aren't properly decorated with test attributes + +**Solutions**: + +1. **Check if tests are ignored**. Look for `[Ignore]` or similar attributes. + +2. **Verify filter expressions**: + + ```dotnetcli + # List all tests to see what's available + dotnet run --project MyTests -- --list-tests + ``` + +3. **Check test discovery**. Ensure tests have proper attributes like `[TestMethod]`, `[Test]`, or `[Fact]`. + +4. **Ignore this exit code for projects with skipped tests**: + + Add to your project file: + + ```xml + + $(TestingPlatformCommandLineArguments) --ignore-exit-code 8 + + ``` + +### Exit code 5: Invalid command-line arguments + +**What it means**: The arguments passed to the test executable are invalid. + +**Solutions**: + +1. **Check available options**: + + ```dotnetcli + dotnet run --project MyTests -- --help + ``` + +2. **Verify argument syntax**. Common mistakes include: + - Missing values for options that require them + - Using unsupported options + - Incorrect format for complex arguments like filters + +3. **Use response files** for complex command lines: + + Create a file named *test-args.rsp*: + + ```rsp + --filter "FullyQualifiedName~Calculator" + --timeout 5m + --diagnostic + ``` + + Then run: + + ```dotnetcli + dotnet run --project MyTests -- @test-args.rsp + ``` + +For more information, see [Configuration reference](microsoft-testing-platform-options.md). + +### Exit code 7: Test host crash + +**What it means**: The test process crashed during execution. + +**Solutions**: + +1. **Enable crash dumps** to capture the crash state: + + ```dotnetcli + dotnet add package Microsoft.Testing.Extensions.CrashDump + ``` + + Then run with crash dump enabled: + + ```dotnetcli + dotnet run --project MyTests -- --crashdump + ``` + +2. **Check for common crash causes**: + - Unhandled exceptions in test code + - Stack overflow + - Out of memory errors + - Native code crashes + +3. **Run tests in isolation** to identify the problematic test: + + ```dotnetcli + dotnet run --project MyTests -- --filter "FullyQualifiedName~SuspectedTest" + ``` + +For more information about crash dumps, see [Crash dump options](microsoft-testing-platform-options.md#crash-dump-options). + +### Exit code 13: Maximum failed tests reached + +**What it means**: The test session was stopped after reaching the specified maximum number of test failures. + +**What to do**: + +1. **Review the failed tests** to identify common issues. + +2. **Adjust the threshold** if needed: + + ```dotnetcli + dotnet run --project MyTests -- --maximum-failed-tests 10 + ``` + +3. **Remove the limit** to see all failures: + + Remove the `--maximum-failed-tests` argument from your command line. + +This feature is useful in CI/CD pipelines to fail fast when a certain number of tests fail, saving time and resources. + +## Configuring exit code behavior + +Microsoft.Testing.Platform is strict by default but allows you to configure which exit codes should be ignored. When an exit code is ignored, the process returns `0` instead. + +### Using command-line options + +Ignore specific exit codes using the `--ignore-exit-code` option: + +```dotnetcli +dotnet run --project MyTests -- --ignore-exit-code 2 +``` + +Ignore multiple exit codes with a semicolon-separated list: + +```dotnetcli +dotnet run --project MyTests -- --ignore-exit-code 2;8 +``` + +### Using environment variables + +Set the `TESTINGPLATFORM_EXITCODE_IGNORE` environment variable: + +**Windows (Command Prompt)**: + +```cmd +set TESTINGPLATFORM_EXITCODE_IGNORE=2;8 +dotnet run --project MyTests +``` + +**Windows (PowerShell)**: + +```powershell +$env:TESTINGPLATFORM_EXITCODE_IGNORE = "2;8" +dotnet run --project MyTests +``` + +**Linux/macOS**: + +```bash +export TESTINGPLATFORM_EXITCODE_IGNORE=2;8 +dotnet run --project MyTests +``` + +### Using MSBuild properties + +Set exit codes to ignore in your project file for consistent behavior across all test runs: + +```xml + + $(TestingPlatformCommandLineArguments) --ignore-exit-code 8 + +``` + +This approach is recommended for project-specific configuration, such as test projects where all tests are intentionally skipped. + +### Common use cases + +**Allow test failures without failing CI**: + +Useful for generating test reports even when tests fail: + +```dotnetcli +dotnet run --project MyTests -- --ignore-exit-code 2 --report-trx +``` + +**Ignore "no tests ran" for optional test projects**: + +Some projects might have tests conditionally compiled. Ignore exit code 8 to avoid failures when no tests are present: + +```xml + + $(TestingPlatformCommandLineArguments) --ignore-exit-code 8 + +``` + +**CI/CD pipeline examples**: + +**Azure Pipelines**: + +```yml +- task: CmdLine@2 + displayName: "Run Tests" + inputs: + script: 'dotnet run --project MyTests -- --ignore-exit-code 2' + continueOnError: true +``` + +**GitHub Actions**: + +```yml +- name: Run Tests + run: dotnet run --project MyTests -- --ignore-exit-code 2 + continue-on-error: true +``` + +For more information about configuration methods, see [Configuration reference](microsoft-testing-platform-options.md). + +## Troubleshooting tips + +When encountering unexpected exit codes: + +1. **Enable diagnostics** for detailed logging: + + ```dotnetcli + dotnet run --project MyTests -- --diagnostic --diagnostic-verbosity Trace + ``` + + The diagnostic log is saved to the *TestResults* directory by default. + +2. **Check the test output** for error messages and stack traces. + +3. **Run tests with `--list-tests`** to verify test discovery: + + ```dotnetcli + dotnet run --project MyTests -- --list-tests + ``` + +4. **Isolate the problem** by running a subset of tests: + + ```dotnetcli + dotnet run --project MyTests -- --filter "FullyQualifiedName~Namespace.Class" + ``` + +5. **Check for extension conflicts**. Review installed NuGet packages for testing extensions and ensure they're compatible versions. + +6. **Use `--info`** to see detailed platform information: + + ```dotnetcli + dotnet run --project MyTests -- --info + ``` + +For more information about diagnostics, see [Diagnostic options](microsoft-testing-platform-options.md#diagnostic-options). + +## See also + +- [Microsoft.Testing.Platform overview](microsoft-testing-platform-intro.md) +- [Run tests with Microsoft.Testing.Platform](microsoft-testing-platform-run-tests.md) +- [Configuration reference](microsoft-testing-platform-options.md) +- [Get started with Microsoft.Testing.Platform](microsoft-testing-platform-getting-started.md) +- [Running selective unit tests](selective-unit-tests.md) +- [Testing with dotnet test](unit-testing-with-dotnet-test.md) diff --git a/docs/core/testing/microsoft-testing-platform-extensions-code-coverage.md b/docs/core/testing/microsoft-testing-platform-extensions-code-coverage.md deleted file mode 100644 index 11c22b6828db5..0000000000000 --- a/docs/core/testing/microsoft-testing-platform-extensions-code-coverage.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -title: Microsoft.Testing.Platform Code Coverage extensions -description: Learn about the various Microsoft.Testing.Platform Code Coverage extensions and how to use them. -author: evangelink -ms.author: amauryleve -ms.date: 04/10/2024 -ai-usage: ai-assisted ---- - -# Code coverage extensions - -This article lists and explains all Microsoft.Testing.Platform extensions related to the code coverage capability. - -You can use the code coverage feature to determine what proportion of your project's code is being tested by coded tests such as unit tests. To effectively guard against bugs, your tests should exercise or *cover* a large proportion of your code. - -## Microsoft code coverage - -Microsoft Code Coverage analysis is possible for both managed (CLR) and unmanaged (native) code. Both static and dynamic instrumentation are supported. This extension is shipped as part of [Microsoft.Testing.Extensions.CodeCoverage](https://nuget.org/packages/Microsoft.Testing.Extensions.CodeCoverage) NuGet package. - -> [!NOTE] -> Unmanaged (native) code coverage is disabled in the extension by default. Use flags `EnableStaticNativeInstrumentation` and `EnableDynamicNativeInstrumentation` to enable it if needed. -> For more information about unmanaged code coverage, see [Static and dynamic native instrumentation](/visualstudio/test/customizing-code-coverage-analysis?#static-and-dynamic-native-instrumentation). - -> [!IMPORTANT] -> The package is shipped with Microsoft .NET library closed-source free to use licensing model. - -For more information about Microsoft code coverage, see its [GitHub page](https://github.com/microsoft/codecoverage). - -Microsoft Code Coverage provides the following options: - -| Option | Description | -|----------------------------|-------------------------------------------------------------------------------| -| `--coverage` | Collect the code coverage using dotnet-coverage tool. | -| `--coverage-output` | The name or path of the produced coverage file. By default, the file is `TestResults/.coverage`. | -| `--coverage-output-format` | Output file format. Supported values are: `coverage`, `xml`, and `cobertura`. Default is `coverage`. | -| `--coverage-settings` | [XML code coverage settings](../additional-tools/dotnet-coverage.md#settings). | - -For more information about the available options, see [settings](../additional-tools/dotnet-coverage.md#settings) and [samples](https://github.com/microsoft/codecoverage/tree/main/samples/Algorithms). - -> [!NOTE] -> The default value of `IncludeTestAssembly` in Microsoft.Testing.Extensions.CodeCoverage is `false`, while it used to be `true` in VSTest. This means that test projects are excluded by default. For more information, see [Code Coverage configuration](https://github.com/microsoft/codecoverage/blob/main/docs/configuration.md). - -## Version compatibility - -The following table shows the compatibility between different versions of Microsoft.Testing.Extensions.CodeCoverage and Microsoft.Testing.Platform: - -| Microsoft.Testing.Extensions.CodeCoverage | Microsoft.Testing.Platform | -|------------------------------------------|---------------------------| -| 18.1.x | 2.0.x | -| 18.0.x | 1.8.x | -| 17.14.x | 1.6.2 | - -> [!NOTE] -> For the best compatibility and latest features, it's recommended to use the latest versions of both packages together. - -## Coverlet - -> [!IMPORTANT] -> The `coverlet.collector` NuGet package is designed specifically for VSTest and cannot be used with `Microsoft.Testing.Platform`. - -There's currently no Coverlet extension, but you can use [Coverlet .NET global tool](https://github.com/coverlet-coverage/coverlet#net-global-tool-guide-suffers-from-possible-known-issue). - -Assuming you've already installed the Coverlet global tool, you can now run: - -```bash -coverlet .\bin\Debug\net8.0\TestProject2.dll --target "dotnet" --targetargs "test .\bin\Debug\net8.0\TestProject2.dll --no-build" -``` diff --git a/docs/core/testing/microsoft-testing-platform-extensions-diagnostics.md b/docs/core/testing/microsoft-testing-platform-extensions-diagnostics.md deleted file mode 100644 index 06b38ecbb09c3..0000000000000 --- a/docs/core/testing/microsoft-testing-platform-extensions-diagnostics.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -title: Microsoft.Testing.Platform diagnostics extensions -description: Learn about the various Microsoft.Testing.Platform diagnostics extensions and how to use them. -author: evangelink -ms.author: amauryleve -ms.date: 04/10/2024 ---- - -# Diagnostics extensions - -This article lists and explains all Microsoft.Testing.Platform extensions related to the diagnostics capability. - -## Built-in options - -The following [platform options](./microsoft-testing-platform-intro.md#options) provide useful information for troubleshooting your test apps: - -- `--info` -- `--diagnostic` -- `⁠-⁠-⁠diagnostic-⁠filelogger-⁠synchronouswrite` -- `--diagnostic-verbosity` -- `--diagnostic-output-fileprefix` -- `--diagnostic-output-directory` - -You can also enable the diagnostics logs using the environment variables: - -| Environment variable name | Description | -|--|--| -| `TESTINGPLATFORM_DIAGNOSTIC` | If set to `1`, enables the diagnostic logging. | -| `TESTINGPLATFORM_DIAGNOSTIC_VERBOSITY` | Defines the verbosity level. The available values are `Trace`, `Debug`, `Information`, `Warning`, `Error`, or `Critical`. | -| `TESTINGPLATFORM_DIAGNOSTIC_OUTPUT_DIRECTORY` | The output directory of the diagnostic logging, if not specified the file is generated in the default _TestResults_ directory. | -| `TESTINGPLATFORM_DIAGNOSTIC_OUTPUT_FILEPREFIX` | The prefix for the log file name. Defaults to `"log_"`. | -| `TESTINGPLATFORM_DIAGNOSTIC_FILELOGGER_SYNCHRONOUSWRITE` | Forces the built-in file logger to synchronously write logs. Useful for scenarios where you don't want to lose any log entries (if the process crashes). This does slow down the test execution. | - -> [!NOTE] -> Environment variables take precedence over the command line arguments. - -## Crash dump - -This extension allows you to create a crash dump file if the process crashes. This extension is shipped as part of [Microsoft.Testing.Extensions.CrashDump](https://nuget.org/packages/Microsoft.Testing.Extensions.CrashDump) NuGet package. - -To configure the crash dump file generation, use the following options: - -| Option | Description | -|--|--| -| `--crashdump` | Generates a dump file when the test host process crashes. Supported in .NET 6.0+. | -| `⁠-⁠-⁠crashdump-⁠filename` | Specifies the file name of the dump. | -| `--crashdump-type` | Specifies the type of the dump. Valid values are `Mini`, `Heap`, `Triage`, `Full`. Defaults as `Full`. For more information, see [Types of mini dumps](../diagnostics/collect-dumps-crash.md#types-of-mini-dumps). | - -> [!CAUTION] -> The extension isn't compatible with .NET Framework and will be silently ignored. For .NET Framework support, you enable the postmortem debugging with Sysinternals ProcDump. For more information, see [Enabling Postmortem Debugging: Window Sysinternals ProcDump](/windows-hardware/drivers/debugger/enabling-postmortem-debugging#window-sysinternals-procdump). The postmortem debugging solution will also collect process crash information for .NET so you can avoid the use of the extension if you're targeting both .NET and .NET Framework test applications. - -## Hang dump - -This extension allows you to create a dump file after a given timeout. This extension is shipped as part of [Microsoft.Testing.Extensions.HangDump](https://nuget.org/packages/Microsoft.Testing.Extensions.HangDump) package. - -To configure the hang dump file generation, use the following options: - -| Option | Description | -|--|--| -| `--hangdump` | Generates a dump file in case the test host process hangs. | -| `-⁠-hangdump-filename` | Specifies the file name of the dump. | -| `--hangdump-timeout` | Specifies the timeout after which the dump is generated. The timeout value is specified in one of the following formats:
`1.5h`, `1.5hour`, `1.5hours`
`90m`, `90min`, `90minute`, `90minutes`
`5400s`, `5400sec`, `5400second`, `5400seconds`. Defaults to `30m` (30 minutes). | -| `--hangdump-type` | Specifies the type of the dump. Valid values are `Mini`, `Heap`, `Triage`, `Full`. Defaults as `Full`. For more information, see [Types of mini dumps](../diagnostics/collect-dumps-crash.md#types-of-mini-dumps). | diff --git a/docs/core/testing/microsoft-testing-platform-extensions-fakes.md b/docs/core/testing/microsoft-testing-platform-extensions-fakes.md deleted file mode 100644 index 5951d1b734e11..0000000000000 --- a/docs/core/testing/microsoft-testing-platform-extensions-fakes.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: Microsoft.Testing.Platform Fakes extension -description: Learn about the various Microsoft.Testing.Platform Fakes extension capabilities and how to use it. -author: drognanar -ms.author: arturs -ms.date: 06/21/2024 ---- - -# Fakes extension - -The `Microsoft.Testing.Extensions.Fakes` extension provides support to execute a test project that makes use of `Microsoft Fakes`. - -[Microsoft Fakes](/visualstudio/test/isolating-code-under-test-with-microsoft-fakes) allows you to better test your code by either generating `Stub`s (for instance creating a testable implementation of `INotifyPropertyChanged`) or by `Shim`ing methods and static methods (replacing the implementation of `File.Open` with a one you can control in your tests). - -> [!NOTE] -> This extension requires a Visual Studio Enterprise installation with the minimum version of 17.11 preview 1 in order to work correctly. - -## Upgrade your project to the new extension - -To use the new extension with an existing project, update the existing `Microsoft.QualityTools.Testing.Fakes` reference with `Microsoft.Testing.Extensions.Fakes`. - -```diff -- -- False -- -+ -``` - -If you are using MSTest.Sdk 3.7 or later, use the `EnableMicrosoftTestingExtensionsFakes` property to enable the extension and don't add the package reference. diff --git a/docs/core/testing/microsoft-testing-platform-extensions-hosting.md b/docs/core/testing/microsoft-testing-platform-extensions-hosting.md deleted file mode 100644 index 66365572fb5e5..0000000000000 --- a/docs/core/testing/microsoft-testing-platform-extensions-hosting.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Microsoft.Testing.Platform hosting extensions -description: Learn about the various Microsoft.Testing.Platform hosting extensions and how to use them. -author: evangelink -ms.author: amauryleve -ms.date: 04/10/2024 ---- - -# Hosting extensions - -This article lists and explains all Microsoft.Testing.Platform extensions related to the hosting capability. - -## Hot reload - -Hot reload lets you modify your app's managed source code while the application is running, without the need to manually pause or hit a breakpoint. Simply make a supported change while the app is running and select the **Apply code changes** button in Visual Studio to apply your edits. - -> [!NOTE] -> The current version is limited to supporting hot reload in "console mode" only. There is currently no support for hot reload in Test Explorer for Visual Studio or Visual Studio Code. - -This extension is shipped as part of the [Microsoft.Testing.Extensions.HotReload](https://nuget.org/packages/Microsoft.Testing.Extensions.HotReload) package. - -> [!NOTE] -> The package is shipped with the restrictive Microsoft.Testing.Platform Tools license. -> The full license is available at . - -You can easily enable hot reload support by setting the `TESTINGPLATFORM_HOTRELOAD_ENABLED` environment variable to `"1"`. - -For SDK-style projects, you can add `"TESTINGPLATFORM_HOTRELOAD_ENABLED": "1"` in the `environmentVariables` section of the `launchSettings.json` file. The following snippet shows an example file: - -```json -{ - "profiles": { - "Contoso.MyTests": { - "commandName": "Project", - "environmentVariables": { - "TESTINGPLATFORM_HOTRELOAD_ENABLED": "1" - } - } - } -} -``` diff --git a/docs/core/testing/microsoft-testing-platform-extensions-output.md b/docs/core/testing/microsoft-testing-platform-extensions-output.md deleted file mode 100644 index a4ffd3554addf..0000000000000 --- a/docs/core/testing/microsoft-testing-platform-extensions-output.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: Microsoft.Testing.Platform output extensions -description: Learn about the various Microsoft.Testing.Platform output extensions and how to use them. -author: nohwnd -ms.author: jajares -ms.date: 08/26/2024 ---- - -# Output extensions - -This article lists and explains all Microsoft.Testing.Platform extensions related to the terminal output. - -## Terminal test reporter - -Terminal test reporter is the default implementation of status and progress reporting to the terminal (console). - -It comes built-in with **Microsoft.Testing.Platform**, and offers ANSI and non-ANSI mode, and progress indicator. - -### Output modes - -There are two output modes available: - -- `Normal`, the output contains the banner, reports full failures of tests, warning messages and writes summary of the run. - ![Output with 1 failed test and a summary](./media/test-output-and-summary.png) - -- `Detailed`, the same as `Normal` but it also reports `Passed` tests. - ![Output with 1 failed, and 1 passed test and a summary](./media/test-output-and-summary-with-passed.png) - -### ANSI - -Internally there are 2 different output formatters that are auto-detecting the terminal capability to handle [ANSI escape codes](/windows/console/console-virtual-terminal-sequences). - -- The ANSI formatter is used when the terminal is capable of rendering the escape codes. -- The non-ANSI formatter is used when the terminal cannot handle the escape codes, or when `--no-ansi` is used, or when output is redirected. - -The default is to auto-detect the capabilities. - -### Progress - -A progress indicator is written to terminal. The progress indicator, shows the number of passed tests, failed tests, and skipped tests, followed by the name of the tested assembly, its target framework and architecture. - -![A progress bar with 23 passed tests, 0 failed tests and 0 skipped tests](./media/test-progress-bar.png) - -The progress bar is written based on the selected mode: - -- ANSI, the progress bar is animated, sticking to the bottom of the screen and is refreshed every 500ms. The progress bar hides once test execution is done. -- non-ANSI, the progress bar is written to screen as is every 3 seconds. The progress remains in the output. - -### Options - -The available options are as follows: - -| Option | Description | -|-------------|----------------------------------------------------------------------------------------------------| -| no-progress | Disable reporting progress to screen. | -| no-ansi | Disable outputting ANSI escape characters to screen. | -| output | Output verbosity when reporting tests. Valid values are 'Normal', 'Detailed'. Default is 'Normal'. | diff --git a/docs/core/testing/microsoft-testing-platform-extensions-policy.md b/docs/core/testing/microsoft-testing-platform-extensions-policy.md deleted file mode 100644 index 8399ba024c622..0000000000000 --- a/docs/core/testing/microsoft-testing-platform-extensions-policy.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: Microsoft.Testing.Platform policy extensions -description: Learn about the various Microsoft.Testing.Platform policy extensions and how to use them. -author: evangelink -ms.author: amauryleve -ms.date: 04/10/2024 ---- - -# Policy extensions - -This article lists and explains all Microsoft.Testing.Platform extensions related to the policy capability. - -## Retry - -A .NET test resilience and transient-fault-handling extension. - -This extension is intended for integration tests where the test depends heavily on the state of the environment and could experience transient faults. - -This extension is shipped as part of [Microsoft.Testing.Extensions.Retry](https://nuget.org/packages/Microsoft.Testing.Extensions.Retry) package. - -> [!NOTE] -> The package is shipped with the restrictive Microsoft.Testing.Platform Tools license. -> The full license is available at . - -The available options are as follows: - -| Option | Description | -|---------------------------------------|--------------------------------------------------------------------------------------------------| -| `--retry-failed-tests` | Reruns any failed tests until they pass or until the maximum number of attempts is reached. | -| `--retry-failed-tests-max-percentage` | Avoids rerunning tests when the percentage of failed test cases crosses the specified threshold. | -| `--retry-failed-tests-max-tests` | Avoids rerunning tests when the number of failed test cases crosses the specified limit. | diff --git a/docs/core/testing/microsoft-testing-platform-extensions-test-reports.md b/docs/core/testing/microsoft-testing-platform-extensions-test-reports.md deleted file mode 100644 index c3de71910cd24..0000000000000 --- a/docs/core/testing/microsoft-testing-platform-extensions-test-reports.md +++ /dev/null @@ -1,56 +0,0 @@ ---- -title: Microsoft.Testing.Platform test reports extensions -description: Learn about the various Microsoft.Testing.Platform test reports extensions and how to use them. -author: evangelink -ms.author: amauryleve -ms.date: 04/10/2024 ---- - -# Test reports extensions - -This article lists and explains all Microsoft.Testing.Platform extensions related to the test report capability. - -A test report is a file that contains information about the execution and outcome of the tests. - -## Visual Studio test reports - -The Visual Studio test result file (or TRX) is the default format for publishing test results. This extension is shipped as part of [Microsoft.Testing.Extensions.TrxReport](https://nuget.org/packages/Microsoft.Testing.Extensions.TrxReport) package. - -The available options as follows: - -| Option | Description | -|--|--| -| `--report-trx` | Generates the TRX report. | -| `--report-trx-filename` | The name of the generated TRX report. The default name matches the following format `__.trx`. | - -The report is saved inside the default _TestResults_ folder that can be specified through the `--results-directory` command line argument. - -## Azure DevOps reports - -Azure DevOps report plugin enhances test running for developers that host their code on GitHub, but build on Azure DevOps build agents. It adds additional information to failures to show failure directly in GitHub PR. - -![Error annotation in GitHub PR files view](./media/test-azdoreport-failure.png) - -The extension is shipped in [Microsoft.Testing.Extensions.AzureDevOpsReport](https://nuget.org/packages/Microsoft.Testing.Extensions.AzureDevOpsReport) package. - -The available options as follows: - -| Option | Description | -|--|--| -| `--report-azdo` | Enable outputting errors / warnings in CI builds. | -| `--report-azdo-severity` | Severity to use for the reported event. Options are: `error` (default) and `warning`. | - -The extension automatically detects that it is running in continuous integration (CI) environment by checking the `TF_BUILD` environment variable. - -### Determining the line to report - -To highlight the correct line in code where failure occurred, AzureDevOps report plugin searches the error stacktrace for a file that exists in the current repository. - -To determine this it: - -- finds the repository root, this is done by searching the `.git` directory closest to the location from where the test application is started (as determined by `AppContext.BaseDirectory`). -- finds the first line in stack trace that has file location and line (the library needs to have debug symbols). -- excludes all files that end with `Assert.cs` to avoid showing details of your assertion implementations or wrappers. -- excludes all files that don't exist on disk (typically those are lines from external libraries that ship debug symbols e.g. MSTest). - -(These are internal details that serve to aid debugging the behavior and might change in the future.) diff --git a/docs/core/testing/microsoft-testing-platform-extensions-vstest-bridge.md b/docs/core/testing/microsoft-testing-platform-extensions-vstest-bridge.md deleted file mode 100644 index 233b7208f6e4e..0000000000000 --- a/docs/core/testing/microsoft-testing-platform-extensions-vstest-bridge.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -title: Microsoft.Testing.Platform VSTest bridge extension -description: Learn about the various Microsoft.Testing.Platform VSTest bridge extension and how to use it. -author: evangelink -ms.author: amauryleve -ms.date: 04/10/2024 ---- - -# VSTest Bridge extension - -This extension provides a compatibility layer with VSTest allowing the test frameworks depending on it to continue supporting running in VSTest mode (`vstest.console.exe`, usual `dotnet test`, `VSTest task` on AzDo, Test Explorers of Visual Studio and Visual Studio Code...). This extension is shipped as part of [Microsoft.Testing.Extensions.VSTestBridge](https://www.nuget.org/packages/Microsoft.Testing.Extensions.VSTestBridge) package. - -## Compatibility with VSTest - -The main purpose of this extension is to offer an easy and smooth upgrade experience for VSTest users by allowing a dual mode where the new platform is enabled and in parallel a compatibility mode is offered to allow the usual workflows to continue working. - -## Runsettings support - -This extension allows you to provide a [VSTest *.runsettings* file](/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file), but not all options in this file are picked up by the platform. We describe below the supported and unsupported settings, configuration options and alternatives for the most used VSTest configuration options. - -When enabled by the test framework, you can use `--settings ` to provide the `.runsettings` file. - -### RunConfiguration element - -The following **RunConfiguration** elements are not supported by `Microsoft.Testing.Platform`: - -| Node | Description | Reason / Workaround | -|------|-------------|---------------------| -| **MaxCpuCount** | This setting controls the level of parallelism on process-level. Use 0 to enable the maximum process-level parallelism.| When Microsoft.Testing.Platform is used with MSBuild, this option is [offloaded to MSBuild](/visualstudio/msbuild/building-multiple-projects-in-parallel-with-msbuild). When a single executable is run, this option has no meaning for Microsoft.Testing.Platform. | -| **ResultsDirectory** | The directory where test results are placed. The path is relative to the directory that contains the *.runsettings* file.| Use the command-line option `--results-directory` to determine the directory where the test results are going to be placed. If the specified directory doesn't exist, it's created. The default is `TestResults` in the directory that contains the test application. | -| **TargetFrameworkVersion** | This setting defines the framework version, or framework family to use to run tests.| This option is ignored. The `` or `` MSBuild properties determine the target framework of the application. The tests are hosted in the final application. | -| **TargetPlatform** | This setting defines the architecture to use to run tests. | `` determines the architecture of the final application that hosts the tests. | -| **TreatTestAdapterErrorsAsWarnings** | Suppresses test adapter errors to become warnings. | Microsoft.Testing.Platform allows only one type of tests to be run from a single assembly, and failure to load the test framework or other parts of infrastructure will become an un-skippable error, because it signifies that some tests could not be discovered or run. | -| **TestAdaptersPaths** | One or more paths to the directory where the TestAdapters are located| Microsoft.Testing.Platform does not use the concept of test adapters and does not allow dynamic loading of extensions unless they are part of the build, and are registered in `Program.cs`, either automatically via build targets or manually. | -| **TestCaseFilter** | A filter to limit tests which will run. | Starting with v1.6, this runsettings entry is now supported. Before this version, you should use `--filter` command line option instead. | -| **TestSessionTimeout** | Allows users to terminate a test session when it exceeds a given timeout.| There is no alternative option. | -| **DotnetHostPath** | Specify a custom path to dotnet host that is used to run the test host. | Microsoft.Testing.Platform is not doing any additional resolving of dotnet. It depends fully on how dotnet resolves itself, which can be controlled by environment variables such as [`DOTNET_HOST_PATH`](../tools/dotnet-environment-variables.md#dotnet_host_path). | -| **TreatNoTestsAsError** | Exit with non-zero exit code when no tests are discovered. | Microsoft.Testing.Platform will error by default when no tests are discovered or run in a test application. You can set how many tests you expect to find in the assembly by using `--minimum-expected-tests` command line parameter, which defaults to 1. | - -### DataCollectors element - -`Microsoft.Testing.Platform` is not using data collectors. Instead it has the concept of in-process and out-of-process extensions. Each extension is configured by its respective configuration file or through the command line. - -Most importantly [hang](microsoft-testing-platform-extensions-diagnostics.md#hang-dump) and [crash](microsoft-testing-platform-extensions-diagnostics.md#crash-dump) extension, and [code coverage](microsoft-testing-platform-extensions-code-coverage.md) extension. - -### LoggerRunSettings element - -Loggers in `Microsoft.Testing.Platform` are configured through command-line parameters or by settings in code. - -## VSTest filter support - -This extension also offer the ability to use VSTest filtering mechanism to discover or run only the tests that matches the filter expression. For more information, see the [Filter option details](../tools/dotnet-test.md#filter-option-details) section or for framework specific details see the [Running selective unit tests](./selective-unit-tests.md) page. - -When enabled by the test framework, you can use `--filter `. - -## TestRun parameters - -You can pass parameters to the test run by using the `--test-parameter` command line option in the format `key=value`. This option can be specified multiple times, one for each parameter to set. - -These parameters can then be accessed by the test framework in the test run: - -- for MSTest, use -- for NUnit, use [TestContext.TestParameters](https://docs.nunit.org/articles/nunit/writing-tests/TestContext.html#testparameters) diff --git a/docs/core/testing/microsoft-testing-platform-extensions.md b/docs/core/testing/microsoft-testing-platform-extensions.md index 62a9abec9b6d4..c6b881abdbff3 100644 --- a/docs/core/testing/microsoft-testing-platform-extensions.md +++ b/docs/core/testing/microsoft-testing-platform-extensions.md @@ -1,43 +1,354 @@ --- title: Microsoft.Testing.Platform extensions -description: Learn about the various Microsoft.Testing.Platform extensions and how to use them. -author: nohwnd -ms.author: jajares -ms.date: 12/15/2023 +description: Learn about the various Microsoft.Testing.Platform extensions that add capabilities like code coverage, test reports, crash dumps, and more. +author: Evangelink +ms.author: amauryleve +ms.date: 11/27/2025 +ai-usage: ai-assisted --- # Microsoft.Testing.Platform extensions -Microsoft.Testing.Platform can be customized through extensions. These extension are either built-in or can be installed as NuGet packages. Extensions installed through NuGet packages will auto-register the extensions they are holding to become available in test execution. +Microsoft.Testing.Platform provides a rich set of extensions that add capabilities to your test projects. This article describes all available extensions, their features, and how to use them. -Each and every extension is shipped with its own licensing model (some less permissive), be sure to refer to the license associated with the extensions you want to use. +> [!TIP] +> For a complete reference of all command-line options, environment variables, and configuration files, see [Configuration reference](microsoft-testing-platform-options.md). -## Extensions +## Code coverage -**[Code Coverage](./microsoft-testing-platform-extensions-code-coverage.md)** +Code coverage helps you determine what proportion of your project's code is being tested. To effectively guard against bugs, your tests should exercise or *cover* a large proportion of your code. -Extensions designed to provide code coverage support. +### Microsoft Code Coverage -**[Diagnostics](./microsoft-testing-platform-extensions-diagnostics.md)** +Microsoft Code Coverage analysis is possible for both managed (CLR) and unmanaged (native) code. Both static and dynamic instrumentation are supported. This extension is shipped as part of [Microsoft.Testing.Extensions.CodeCoverage](https://nuget.org/packages/Microsoft.Testing.Extensions.CodeCoverage) NuGet package. -Extensions offering diagnostics and troubleshooting functionalities. +> [!NOTE] +> Unmanaged (native) code coverage is disabled by default. Use flags `EnableStaticNativeInstrumentation` and `EnableDynamicNativeInstrumentation` to enable it if needed. +> For more information, see [Static and dynamic native instrumentation](/visualstudio/test/customizing-code-coverage-analysis#static-and-dynamic-native-instrumentation). -**[Hosting](./microsoft-testing-platform-extensions-hosting.md)** +> [!IMPORTANT] +> The package is shipped with Microsoft .NET library closed-source free to use licensing model. -Extensions affecting how the test execution is hosted. +**Available options:** -**[Policy](./microsoft-testing-platform-extensions-policy.md)** +| Option | Description | +|----------------------------|-------------------------------------------------------------------------------| +| `--coverage` | Collects the code coverage using dotnet-coverage tool. | +| `--coverage-output` | The name or path of the produced coverage file. By default, the file is `TestResults/.coverage`. | +| `--coverage-output-format` | Output file format. Supported values are: `coverage`, `xml`, and `cobertura`. Default is `coverage`. | +| `--coverage-settings` | [XML code coverage settings](../additional-tools/dotnet-coverage.md#settings). | -Extensions allowing to define policies around the test execution. +For more information about the available options, see [settings](../additional-tools/dotnet-coverage.md#settings) and [samples](https://github.com/microsoft/codecoverage/tree/main/samples/Algorithms). -**[Test Reports](./microsoft-testing-platform-extensions-test-reports.md)** +> [!NOTE] +> The default value of `IncludeTestAssembly` in Microsoft.Testing.Extensions.CodeCoverage is `false`, while it used to be `true` in VSTest. This means test projects are excluded by default. For more information, see [Code Coverage configuration](https://github.com/microsoft/codecoverage/blob/main/docs/configuration.md). -Extensions allowing to produce test report files that contains information about the execution and outcome of the tests. +**Version compatibility:** -**[VSTest Bridge](./microsoft-testing-platform-extensions-vstest-bridge.md)** +The following table shows the compatibility between different versions of Microsoft.Testing.Extensions.CodeCoverage and Microsoft.Testing.Platform: -This extension provides a compatibility layer with VSTest allowing the test frameworks depending on it to continue supporting running in VSTest mode (`vstest.console.exe`, usual `dotnet test`, `VSTest task` on AzDo, Test Explorers of Visual Studio and Visual Studio Code...). +| Microsoft.Testing.Extensions.CodeCoverage | Microsoft.Testing.Platform | +|------------------------------------------|---------------------------| +| 18.1.x | 2.0.x | +| 18.0.x | 1.8.x | +| 17.14.x | 1.6.2 | -**[Microsoft Fakes](./microsoft-testing-platform-extensions-fakes.md)** +> [!NOTE] +> For the best compatibility and latest features, use the latest versions of both packages together. -This extension provides support to execute a test project that makes use of `Microsoft Fakes`. +For more information about Microsoft code coverage, see its [GitHub page](https://github.com/microsoft/codecoverage). + +### Coverlet + +> [!IMPORTANT] +> The `coverlet.collector` NuGet package is designed specifically for VSTest and cannot be used with Microsoft.Testing.Platform. + +There's currently no Coverlet extension, but you can use [Coverlet .NET global tool](https://github.com/coverlet-coverage/coverlet#net-global-tool-guide-suffers-from-possible-known-issue). + +Assuming you've already installed the Coverlet global tool, you can run: + +```bash +coverlet .\bin\Debug\net8.0\TestProject2.dll --target "dotnet" --targetargs "test .\bin\Debug\net8.0\TestProject2.dll --no-build" +``` + +## Test reports + +Test reports are files that contain information about the execution and outcome of the tests. + +### Visual Studio test reports (TRX) + +The Visual Studio test result file (or TRX) is the default format for publishing test results. This extension is shipped as part of [Microsoft.Testing.Extensions.TrxReport](https://nuget.org/packages/Microsoft.Testing.Extensions.TrxReport) package. + +**Available options:** + +| Option | Description | +|--|--| +| `--report-trx` | Generates the TRX report. | +| `--report-trx-filename` | The name of the generated TRX report. The default name matches the following format `__.trx`. | + +The report is saved inside the default _TestResults_ folder that can be specified through the `--results-directory` command-line argument. + +### Azure DevOps reports + +Azure DevOps report plugin enhances test running for developers that host their code on GitHub, but build on Azure DevOps build agents. It adds additional information to failures to show failure directly in GitHub PR. + +![Error annotation in GitHub PR files view](./media/test-azdoreport-failure.png) + +The extension is shipped in [Microsoft.Testing.Extensions.AzureDevOpsReport](https://nuget.org/packages/Microsoft.Testing.Extensions.AzureDevOpsReport) package. + +**Available options:** + +| Option | Description | +|--|--| +| `--report-azdo` | Enables outputting errors and warnings in CI builds. | +| `--report-azdo-severity` | Severity to use for the reported event. Options are: `error` (default) and `warning`. | + +The extension automatically detects that it's running in continuous integration (CI) environment by checking the `TF_BUILD` environment variable. + +**Determining the line to report:** + +To highlight the correct line in code where failure occurred, AzureDevOps report plugin searches the error stacktrace for a file that exists in the current repository. + +To determine this it: + +- Finds the repository root, by searching the `.git` directory closest to the location from where the test application is started (as determined by `AppContext.BaseDirectory`). +- Finds the first line in stack trace that has file location and line (the library needs to have debug symbols). +- Excludes all files that end with `Assert.cs` to avoid showing details of your assertion implementations or wrappers. +- Excludes all files that don't exist on disk (typically those are lines from external libraries that ship debug symbols, for example, MSTest). + +## Diagnostics + +Diagnostics extensions help you troubleshoot test issues by collecting diagnostic logs and memory dumps. + +### Built-in diagnostic logging + +The following platform options provide useful information for troubleshooting your test apps: + +**Command-line options:** + +- `--info` - Displays advanced information about the .NET Test Application +- `--diagnostic` - Enables the diagnostic logging +- `--diagnostic-verbosity` - Defines the verbosity level (`Trace`, `Debug`, `Information`, `Warning`, `Error`, or `Critical`) +- `--diagnostic-output-directory` - The output directory of the diagnostic logging +- `--diagnostic-output-fileprefix` - The prefix for the log file name (defaults to `"log_"`) +- `--diagnostic-filelogger-synchronouswrite` - Forces synchronous write logs + +**Environment variables:** + +| Environment variable name | Description | +|--|--| +| `TESTINGPLATFORM_DIAGNOSTIC` | If set to `1`, enables the diagnostic logging. | +| `TESTINGPLATFORM_DIAGNOSTIC_VERBOSITY` | Defines the verbosity level. The available values are `Trace`, `Debug`, `Information`, `Warning`, `Error`, or `Critical`. | +| `TESTINGPLATFORM_DIAGNOSTIC_OUTPUT_DIRECTORY` | The output directory of the diagnostic logging, if not specified the file is generated in the default _TestResults_ directory. | +| `TESTINGPLATFORM_DIAGNOSTIC_OUTPUT_FILEPREFIX` | The prefix for the log file name. Defaults to `"log_"`. | +| `TESTINGPLATFORM_DIAGNOSTIC_FILELOGGER_SYNCHRONOUSWRITE` | Forces the built-in file logger to synchronously write logs. Useful for scenarios where you don't want to lose any log entries (if the process crashes). This does slow down test execution. | + +> [!NOTE] +> Environment variables take precedence over the command-line arguments. + +For more information, see [Configuration reference](microsoft-testing-platform-options.md). + +### Crash dump + +This extension allows you to create a crash dump file if the process crashes. This extension is shipped as part of [Microsoft.Testing.Extensions.CrashDump](https://nuget.org/packages/Microsoft.Testing.Extensions.CrashDump) NuGet package. + +**Available options:** + +| Option | Description | +|--|--| +| `--crashdump` | Generates a dump file when the test host process crashes. Supported in .NET 6.0 and later. | +| `--crashdump-filename` | Specifies the file name of the dump. | +| `--crashdump-type` | Specifies the type of the dump. Valid values are `Mini`, `Heap`, `Triage`, `Full`. Defaults as `Full`. For more information, see [Types of mini dumps](../diagnostics/collect-dumps-crash.md#types-of-mini-dumps). | + +> [!CAUTION] +> The extension isn't compatible with .NET Framework and is silently ignored. For .NET Framework support, enable the postmortem debugging with Sysinternals ProcDump. For more information, see [Enabling Postmortem Debugging: Window Sysinternals ProcDump](/windows-hardware/drivers/debugger/enabling-postmortem-debugging#window-sysinternals-procdump). The postmortem debugging solution also collects process crash information for .NET so you can avoid the use of the extension if you're targeting both .NET and .NET Framework test applications. + +### Hang dump + +This extension allows you to create a dump file after a given timeout. This extension is shipped as part of [Microsoft.Testing.Extensions.HangDump](https://nuget.org/packages/Microsoft.Testing.Extensions.HangDump) package. + +**Available options:** + +| Option | Description | +|--|--| +| `--hangdump` | Generates a dump file in case the test host process hangs. | +| `--hangdump-filename` | Specifies the file name of the dump. | +| `--hangdump-timeout` | Specifies the timeout after which the dump is generated. The timeout value is specified in one of the following formats:
`1.5h`, `1.5hour`, `1.5hours`
`90m`, `90min`, `90minute`, `90minutes`
`5400s`, `5400sec`, `5400second`, `5400seconds`. Defaults to `30m` (30 minutes). | +| `--hangdump-type` | Specifies the type of the dump. Valid values are `Mini`, `Heap`, `Triage`, `Full`. Defaults as `Full`. For more information, see [Types of mini dumps](../diagnostics/collect-dumps-crash.md#types-of-mini-dumps). | + +## Test retry policy + +A .NET test resilience and transient-fault-handling extension. + +This extension is intended for integration tests where the test depends heavily on the state of the environment and could experience transient faults. + +This extension is shipped as part of [Microsoft.Testing.Extensions.Retry](https://nuget.org/packages/Microsoft.Testing.Extensions.Retry) package. + +> [!NOTE] +> The package is shipped with the restrictive Microsoft.Testing.Platform Tools license. +> The full license is available at . + +**Available options:** + +| Option | Description | +|---------------------------------------|--------------------------------------------------------------------------------------------------| +| `--retry-failed-tests` | Reruns any failed tests until they pass or until the maximum number of attempts is reached. | +| `--retry-failed-tests-max-percentage` | Avoids rerunning tests when the percentage of failed test cases crosses the specified threshold. | +| `--retry-failed-tests-max-tests` | Avoids rerunning tests when the number of failed test cases crosses the specified limit. | + +## Terminal output + +Terminal test reporter is the default implementation of status and progress reporting to the terminal (console). It comes built-in with Microsoft.Testing.Platform, and offers ANSI and non-ANSI mode, and progress indicator. + +### Output modes + +There are two output modes available: + +- **`Normal`** - The output contains the banner, reports full failures of tests, warning messages and writes summary of the run. + ![Output with 1 failed test and a summary](./media/test-output-and-summary.png) + +- **`Detailed`** - The same as `Normal` but it also reports `Passed` tests. + ![Output with 1 failed, and 1 passed test and a summary](./media/test-output-and-summary-with-passed.png) + +### ANSI support + +Internally there are two different output formatters that auto-detect the terminal capability to handle [ANSI escape codes](/windows/console/console-virtual-terminal-sequences). + +- The ANSI formatter is used when the terminal is capable of rendering the escape codes. +- The non-ANSI formatter is used when the terminal cannot handle the escape codes, or when `--no-ansi` is used, or when output is redirected. + +The default is to auto-detect the capabilities. + +### Progress indicator + +A progress indicator is written to terminal. The progress indicator shows the number of passed tests, failed tests, and skipped tests, followed by the name of the tested assembly, its target framework and architecture. + +![A progress bar with 23 passed tests, 0 failed tests and 0 skipped tests](./media/test-progress-bar.png) + +The progress bar is written based on the selected mode: + +- **ANSI** - The progress bar is animated, sticking to the bottom of the screen and is refreshed every 500ms. The progress bar hides once test execution is done. +- **non-ANSI** - The progress bar is written to screen as is every 3 seconds. The progress remains in the output. + +**Available options:** + +| Option | Description | +|-------------|----------------------------------------------------------------------------------------------------| +| `--no-progress` | Disables reporting progress to screen. | +| `--no-ansi` | Disables outputting ANSI escape characters to screen. | +| `--output` | Output verbosity when reporting tests. Valid values are `Normal` and `Detailed`. Default is `Normal`. | + +## Hot reload + +Hot reload lets you modify your app's managed source code while the application is running, without the need to manually pause or hit a breakpoint. Simply make a supported change while the app is running and select the **Apply code changes** button in Visual Studio to apply your edits. + +> [!NOTE] +> The current version is limited to supporting hot reload in "console mode" only. There is currently no support for hot reload in Test Explorer for Visual Studio or Visual Studio Code. + +This extension is shipped as part of the [Microsoft.Testing.Extensions.HotReload](https://nuget.org/packages/Microsoft.Testing.Extensions.HotReload) package. + +> [!NOTE] +> The package is shipped with the restrictive Microsoft.Testing.Platform Tools license. +> The full license is available at . + +You can easily enable hot reload support by setting the `TESTINGPLATFORM_HOTRELOAD_ENABLED` environment variable to `"1"`. + +For SDK-style projects, you can add `"TESTINGPLATFORM_HOTRELOAD_ENABLED": "1"` in the `environmentVariables` section of the `launchSettings.json` file. The following snippet shows an example file: + +```json +{ + "profiles": { + "Contoso.MyTests": { + "commandName": "Project", + "environmentVariables": { + "TESTINGPLATFORM_HOTRELOAD_ENABLED": "1" + } + } + } +} +``` + +## Microsoft Fakes + +The `Microsoft.Testing.Extensions.Fakes` extension provides support to execute a test project that makes use of `Microsoft Fakes`. + +[Microsoft Fakes](/visualstudio/test/isolating-code-under-test-with-microsoft-fakes) allows you to better test your code by either generating `Stub`s (for instance creating a testable implementation of `INotifyPropertyChanged`) or by `Shim`ing methods and static methods (replacing the implementation of `File.Open` with one you can control in your tests). + +> [!NOTE] +> This extension requires a Visual Studio Enterprise installation with the minimum version of 17.11 preview 1 to work correctly. + +**Upgrade your project to the new extension:** + +To use the new extension with an existing project, update the existing `Microsoft.QualityTools.Testing.Fakes` reference with `Microsoft.Testing.Extensions.Fakes`. + +```diff +- +- False +- ++ +``` + +If you are using MSTest.Sdk 3.7 or later, use the `EnableMicrosoftTestingExtensionsFakes` property to enable the extension and don't add the package reference. + +## VSTest Bridge + +This extension provides a compatibility layer with VSTest allowing the test frameworks depending on it to continue supporting running in VSTest mode (`vstest.console.exe`, usual `dotnet test`, `VSTest task` on AzDo, Test Explorers of Visual Studio and Visual Studio Code). This extension is shipped as part of [Microsoft.Testing.Extensions.VSTestBridge](https://www.nuget.org/packages/Microsoft.Testing.Extensions.VSTestBridge) package. + +### Compatibility with VSTest + +The main purpose of this extension is to offer an easy and smooth upgrade experience for VSTest users by allowing a dual mode where the new platform is enabled and in parallel a compatibility mode is offered to allow the usual workflows to continue working. + +### Runsettings support + +This extension allows you to provide a [VSTest *.runsettings* file](/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file), but not all options in this file are picked up by the platform. We describe below the supported and unsupported settings, configuration options and alternatives for the most used VSTest configuration options. + +When enabled by the test framework, you can use `--settings ` to provide the `.runsettings` file. + +**RunConfiguration element:** + +The following **RunConfiguration** elements are not supported by Microsoft.Testing.Platform: + +| Node | Description | Reason / Workaround | +|------|-------------|---------------------| +| **MaxCpuCount** | This setting controls the level of parallelism on process-level. Use 0 to enable the maximum process-level parallelism.| When Microsoft.Testing.Platform is used with MSBuild, this option is [offloaded to MSBuild](/visualstudio/msbuild/building-multiple-projects-in-parallel-with-msbuild). When a single executable is run, this option has no meaning for Microsoft.Testing.Platform. | +| **ResultsDirectory** | The directory where test results are placed. The path is relative to the directory that contains the *.runsettings* file.| Use the command-line option `--results-directory` to determine the directory where the test results are placed. If the specified directory doesn't exist, it's created. The default is `TestResults` in the directory that contains the test application. | +| **TargetFrameworkVersion** | This setting defines the framework version, or framework family to use to run tests.| This option is ignored. The `` or `` MSBuild properties determine the target framework of the application. The tests are hosted in the final application. | +| **TargetPlatform** | This setting defines the architecture to use to run tests. | `` determines the architecture of the final application that hosts the tests. | +| **TreatTestAdapterErrorsAsWarnings** | Suppresses test adapter errors to become warnings. | Microsoft.Testing.Platform allows only one type of tests to be run from a single assembly, and failure to load the test framework or other parts of infrastructure becomes an un-skippable error, because it signifies that some tests could not be discovered or run. | +| **TestAdaptersPaths** | One or more paths to the directory where the TestAdapters are located| Microsoft.Testing.Platform doesn't use the concept of test adapters and doesn't allow dynamic loading of extensions unless they're part of the build, and are registered in `Program.cs`, either automatically via build targets or manually. | +| **TestCaseFilter** | A filter to limit tests which run. | Starting with v1.6, this runsettings entry is now supported. Before this version, you should use `--filter` command-line option instead. | +| **TestSessionTimeout** | Allows users to terminate a test session when it exceeds a given timeout.| There's no alternative option. | +| **DotnetHostPath** | Specifies a custom path to dotnet host that is used to run the test host. | Microsoft.Testing.Platform doesn't do any additional resolving of dotnet. It depends fully on how dotnet resolves itself, which can be controlled by environment variables such as [`DOTNET_HOST_PATH`](../tools/dotnet-environment-variables.md#dotnet_host_path). | +| **TreatNoTestsAsError** | Exit with non-zero exit code when no tests are discovered. | Microsoft.Testing.Platform errors by default when no tests are discovered or run in a test application. You can set how many tests you expect to find in the assembly by using `--minimum-expected-tests` command-line parameter, which defaults to 1. | + +**DataCollectors element:** + +Microsoft.Testing.Platform doesn't use data collectors. Instead it has the concept of in-process and out-of-process extensions. Each extension is configured by its respective configuration file or through the command line. + +Most importantly [hang](#hang-dump) and [crash](#crash-dump) extensions, and [code coverage](#code-coverage) extension. + +**LoggerRunSettings element:** + +Loggers in Microsoft.Testing.Platform are configured through command-line parameters or by settings in code. + +### VSTest filter support + +This extension also offers the ability to use VSTest filtering mechanism to discover or run only the tests that match the filter expression. For more information, see the [Filter option details](../tools/dotnet-test.md#filter-option-details) section or for framework specific details see the [Running selective unit tests](./selective-unit-tests.md) page. + +When enabled by the test framework, you can use `--filter `. + +### TestRun parameters + +You can pass parameters to the test run by using the `--test-parameter` command-line option in the format `key=value`. This option can be specified multiple times, one for each parameter to set. + +These parameters can then be accessed by the test framework in the test run: + +- For MSTest, use +- For NUnit, use [TestContext.TestParameters](https://docs.nunit.org/articles/nunit/writing-tests/TestContext.html#testparameters) + +## See also + +- [Microsoft.Testing.Platform overview](microsoft-testing-platform-intro.md) +- [Configuration reference](microsoft-testing-platform-options.md) +- [Get started with Microsoft.Testing.Platform](microsoft-testing-platform-getting-started.md) +- [Run tests with Microsoft.Testing.Platform](microsoft-testing-platform-run-tests.md) diff --git a/docs/core/testing/microsoft-testing-platform-getting-started.md b/docs/core/testing/microsoft-testing-platform-getting-started.md new file mode 100644 index 0000000000000..aeed61e604c77 --- /dev/null +++ b/docs/core/testing/microsoft-testing-platform-getting-started.md @@ -0,0 +1,308 @@ +--- +title: Get started with Microsoft.Testing.Platform +description: Learn how to create and run your first test project using Microsoft.Testing.Platform in under 5 minutes. +author: Evangelink +ms.author: amauryleve +ms.date: 11/27/2025 +--- + +# Get started with Microsoft.Testing.Platform + +This guide helps you create and run your first test project using Microsoft.Testing.Platform in under 5 minutes. + +## Prerequisites + +- [.NET SDK](https://dotnet.microsoft.com/download) or .NET Framework 4.6.2 or later +- A code editor (Visual Studio, Visual Studio Code, or any text editor) + +## Create your first test project + +The fastest way to get started is using the MSTest SDK, which comes with Microsoft.Testing.Platform built-in. + +### [.NET CLI](#tab/dotnetcli) + +Open a terminal and run: + +```dotnetcli +dotnet new mstest -n MyFirstTests +cd MyFirstTests +``` + +### [Visual Studio](#tab/visual-studio) + +1. Open Visual Studio +1. Select **Create a new project** +1. Search for "MSTest" +1. Select **MSTest Test Project** and select **Next** +1. Name your project "MyFirstTests" and select **Create** + +--- + +The project is created with Microsoft.Testing.Platform already configured and includes a sample test. + +## Run your tests + +### [.NET CLI](#tab/dotnetcli) + +Run the tests directly: + +```dotnetcli +dotnet run --project MyFirstTests +``` + +Or use `dotnet test`: + +```dotnetcli +dotnet test +``` + +You should see output similar to: + +```output +Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 20ms +``` + +### [Visual Studio](#tab/visual-studio) + +**Option 1: Using Test Explorer** + +1. Open **Test Explorer** (Test > Test Explorer) +1. Select **Run All Tests** (or press Ctrl+R, A) + +**Option 2: Run directly** + +1. Right-click the test project in Solution Explorer +1. Select **Set as Startup Project** +1. Press F5 to run + +### [Visual Studio Code](#tab/visual-studio-code) + +1. Install the [C# Dev Kit extension](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp) +1. Open the test project folder +1. Open the **Test Explorer** view +1. Select **Run All Tests** + +Or use the integrated terminal: + +```dotnetcli +dotnet run --project MyFirstTests +``` + +Or use `dotnet test`: + +```dotnetcli +dotnet test --project MyFirstTests +``` + +--- + +## Understanding the test project + +The generated test project includes: + +**Project file** (*MyFirstTests.csproj*): + +```xml + + + net8.0 + + +``` + +The `MSTest.Sdk` automatically configures Microsoft.Testing.Platform. + +**Sample test file** (*UnitTest1.cs*): + +```csharp +using Microsoft.VisualStudio.TestTools.UnitTesting; + +[TestClass] +public class UnitTest1 +{ + [TestMethod] + public void TestMethod1() + { + Assert.IsTrue(true); + } +} +``` + +## Write your first test + +Let's write a simple calculator test. Create a new file *CalculatorTests.cs*: + +```csharp +using Microsoft.VisualStudio.TestTools.UnitTesting; + +[TestClass] +public class CalculatorTests +{ + [TestMethod] + public void Add_TwoNumbers_ReturnsSum() + { + // Arrange + int a = 2; + int b = 3; + + // Act + int result = a + b; + + // Assert + Assert.AreEqual(5, result); + } +} +``` + +Run the tests again and you should see: + +```output +Passed! - Failed: 0, Passed: 2, Skipped: 0, Total: 2, Duration: 25ms +``` + +## Enable in an existing project + +If you have an existing test project using MSTest, NUnit, or xUnit.net, you can enable Microsoft.Testing.Platform: + +### [MSTest](#tab/mstest) + +Update your project file to use MSTest SDK: + +```xml + + + net8.0 + + +``` + +Or enable the MSTest runner: + +```xml + + true + Exe + +``` + +For more information, see [MSTest runner](unit-testing-mstest-runner-intro.md). + +### [NUnit](#tab/nunit) + +Add to your project file: + +```xml + + true + Exe + +``` + +For more information, see [NUnit runner](unit-testing-nunit-runner-intro.md). + +### [xUnit.net](#tab/xunit) + +xUnit.net v3 supports Microsoft.Testing.Platform. Add to your project file: + +```xml + + true + Exe + +``` + +For more information, see [xUnit.net documentation](https://xunit.net/docs/getting-started/v3/microsoft-testing-platform). + +--- + +## Next steps + +Now that you have tests running, explore these common scenarios: + +### Run specific tests + +Filter which tests to run: + +```dotnetcli +dotnet run --project MyFirstTests -- --filter "FullyQualifiedName~Calculator" +``` + +For more information, see [Running selective unit tests](selective-unit-tests.md). + +### Collect code coverage + +Add code coverage to see which code is tested: + +1. Add the package (if not using MSTest SDK): + + ```dotnetcli + dotnet add package Microsoft.Testing.Extensions.CodeCoverage + ``` + +1. Run with coverage: + + ```dotnetcli + dotnet run --project MyFirstTests -- --coverage + ``` + +For more information, see [Code coverage extensions](microsoft-testing-platform-extensions-code-coverage.md). + +### Generate test reports + +Create TRX reports for CI/CD: + +1. Add the package (if not using MSTest SDK): + + ```dotnetcli + dotnet add package Microsoft.Testing.Extensions.TrxReport + ``` + +1. Run with reports: + + ```dotnetcli + dotnet run --project MyFirstTests -- --report-trx + ``` + +For more information, see [Test reports extensions](microsoft-testing-platform-extensions-test-reports.md). + +### Debug failing tests + +Set breakpoints in your test code and debug: + +- **Visual Studio**: Right-click test in Test Explorer > **Debug** +- **Visual Studio Code**: Use F5 with the test project as startup +- **CLI**: Use `dotnet run` and attach a debugger + +### Run in CI/CD pipelines + +Tests run as regular executables in CI: + +```yml +- task: CmdLine@2 + displayName: "Run Tests" + inputs: + script: 'dotnet run --project MyFirstTests' +``` + +For more information, see [Microsoft.Testing.Platform overview](microsoft-testing-platform-intro.md#continuous-integration-ci). + +### Configure test execution + +Learn about all available options: + +- [Configuration reference](microsoft-testing-platform-options.md) - All command-line options, environment variables, and config files +- [Exit codes](microsoft-testing-platform-exit-codes.md) - Understanding test results + +### Learn your test framework + +Deep dive into writing tests with your chosen framework: + +- [MSTest](unit-testing-mstest-intro.md) - Microsoft's test framework +- [NUnit](unit-testing-csharp-with-nunit.md) - Popular cross-platform framework +- [xUnit.net](unit-testing-csharp-with-xunit.md) - Modern testing framework + +## See also + +- [Microsoft.Testing.Platform overview](microsoft-testing-platform-intro.md) +- [Configuration reference](microsoft-testing-platform-options.md) +- [Unit testing best practices](unit-testing-best-practices.md) diff --git a/docs/core/testing/microsoft-testing-platform-integration-dotnet-test.md b/docs/core/testing/microsoft-testing-platform-integration-dotnet-test.md deleted file mode 100644 index 75cb30d95fe02..0000000000000 --- a/docs/core/testing/microsoft-testing-platform-integration-dotnet-test.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: Use Microsoft.Testing.Platform in the VSTest mode of `dotnet test` -description: Learn how to run Microsoft.Testing.Platform tests through dotnet test. -author: nohwnd -ms.author: jajares -ms.date: 03/26/2025 ---- - -# Use Microsoft.Testing.Platform in the VSTest mode of `dotnet test` - -This article explains the integration of `dotnet test` for Microsoft.Testing.Platform, which is provided by Microsoft.Testing.Platform.MSBuild when running in the VSTest mode of `dotnet test`. - -Before diving into this article, it's recommended to first read [Testing with dotnet test](unit-testing-with-dotnet-test.md), which explains the two modes of `dotnet test` (VSTest and MTP modes). - -By default, `dotnet test` uses VSTest to run tests. To enable support for `Microsoft.Testing.Platform` in `dotnet test`, you have two options: - -1. Use `dotnet test` in VSTest mode and specify `true` MSBuild property in your project file. -2. Use `dotnet test` in MTP mode for more native support of MTP in `dotnet test`, which is only supported starting with the .NET 10 SDK. - -Both options are explained in detail in the [Testing with dotnet test](unit-testing-with-dotnet-test.md) article. - -> [!IMPORTANT] -> The rest of this article is specific to the VSTest mode of `dotnet test`. -> -> [!CAUTION] -> Starting with .NET 10 SDK, it's recommended to not use the VSTest mode of `dotnet test` when running with Microsoft.Testing.Platform. - -## Show failure per test - -By default, test failures are summarized into a _.log_ file, and a single failure per test project is reported to MSBuild. - -To show errors per failed test, specify `-p:TestingPlatformShowTestsFailure=true` on the command line, or add the `true` property to your project file. - -On command line: - -```dotnetcli -dotnet test -p:TestingPlatformShowTestsFailure=true -``` - -Or in project file: - -```xml - - - - net8.0 - enable - enable - - false - - Exe - true - - true - - - true - - - - - - -``` - -## Show complete platform output - -By default, all console output that the underlying test executable writes is captured and hidden from the user. This includes the banner, version information, and formatted test information. - -To show this information together with MSBuild output, use `false`. - -This option doesn't impact how the testing framework captures user output written by `Console.WriteLine` or other similar ways to write to the console. - -On command line: - -```dotnetcli -dotnet test -p:TestingPlatformCaptureOutput=false -``` - -Or in project file: - -```xml - - - - net8.0 - enable - enable - - false - - Exe - true - - true - - - false - - - - - - -``` - -> [!IMPORTANT] -> All examples above add properties like `EnableMSTestRunner`, `TestingPlatformDotnetTestSupport`, and `TestingPlatformCaptureOutput` in the csproj file. However, it's highly recommended that you set these properties in `Directory.Build.props`. That way, you don't have to add it to every test project file, and you don't risk introducing a new project that doesn't set these properties and end up with a solution where some projects are VSTest while others are Microsoft.Testing.Platform, which may not work correctly and is unsupported scenario. diff --git a/docs/core/testing/microsoft-testing-platform-options.md b/docs/core/testing/microsoft-testing-platform-options.md new file mode 100644 index 0000000000000..95db121173f38 --- /dev/null +++ b/docs/core/testing/microsoft-testing-platform-options.md @@ -0,0 +1,439 @@ +--- +title: Microsoft.Testing.Platform configuration reference +description: Complete reference of all configuration methods for Microsoft.Testing.Platform, including command-line options, environment variables, and configuration files. +author: Evangelink +ms.author: amauryleve +ms.date: 11/27/2025 +--- + +# Microsoft.Testing.Platform configuration reference + +This article provides a complete reference of all configuration methods available for Microsoft.Testing.Platform. You can configure tests using command-line options, environment variables, and configuration files. The configuration methods are organized by category, starting with core platform options followed by options provided by various features. + +> [!TIP] +> To see all available options for your specific test project, use the `--help` option: +> +> ```dotnetcli +> dotnet run --project YourTestProject -- --help +> ``` + +## Configuration methods + +Microsoft.Testing.Platform supports three configuration methods, with the following precedence (from highest to lowest): + +1. **Command-line arguments** - Options passed when running the test executable +2. **Environment variables** - System or user environment variables +3. **Configuration files** - Settings in *testconfig.json* file + +## Command-line options + +Command-line options are specified when running your test project. They provide the most flexibility and override all other configuration methods. + +### Platform options + +These core options are available in all test projects using Microsoft.Testing.Platform. + +#### General options + +- **`--help`** + + Prints out a description of how to use the command. + +- **`--info`** + + Displays advanced information about the .NET Test Application such as: + + - The platform. + - The environment. + - Each registered command line provider, such as its `name`, `version`, `description`, and `options`. + - Each registered tool, such as its `command`, `name`, `version`, `description`, and all command-line providers. + + This feature is useful to understand which features are registering the same command line option or the changes in available options between multiple versions of a feature or the platform. + +- **`@`** + + Specifies the name of the response file. The response file name must immediately follow the @ character with no white space between the @ character and the response file name. + + Options in a response file are interpreted as if they were present at that place in the command line. Each argument in a response file must begin and end on the same line. You can't use the backslash character (\\) to concatenate lines. Using a response file helps for very long commands that might exceed the terminal limits. You can combine a response file with inline command-line arguments. For example: + + ```console + ./TestExecutable.exe @"filter.rsp" --timeout 10s + ``` + + where *filter.rsp* can have the following contents: + + ```rsp + --filter "A very long filter" + ``` + + Or a single rsp file can be used to specify both timeout and filter as follows: + + ```console + ./TestExecutable.exe @"arguments.rsp" + ``` + + ```rsp + --filter "A very long filter" + --timeout 10s + ``` + +#### Test execution options + +- **`--list-tests`** + + Lists available tests. Tests aren't executed. + +- **`--timeout`** + + A global test execution timeout. Takes one argument as string in the format `[h|m|s]` where `` is float. + +- **`--minimum-expected-tests`** + + Specifies the minimum number of tests that are expected to run. By default, at least one test is expected to run. + +- **`--maximum-failed-tests`** + + Specifies the maximum number of tests failures that, when reached, stops the test run. Support for this switch requires framework authors to implement the `IGracefulStopTestExecutionCapability` capability. The exit code when reaching that amount of test failures is 13. For more information, see [Microsoft.Testing.Platform exit codes](microsoft-testing-platform-exit-codes.md). + + > [!NOTE] + > This feature is available in Microsoft.Testing.Platform starting with version 1.5. + +#### Configuration options + +- **`--config-file`** + + Specifies a [*testconfig.json*](microsoft-testing-platform-config.md) file. + +- **`--results-directory`** + + The directory where the test results are placed. If the specified directory doesn't exist, it's created. The default is `TestResults` in the directory that contains the test application. + +#### Diagnostic options + +- **`--diagnostic`** + + Enables the diagnostic logging. The default log level is `Trace`. The file is written in the output directory with the following name format, `log_[MMddHHssfff].diag`. + +- **`--diagnostic-verbosity`** + + Defines the verbosity level when the `--diagnostic` switch is used. The available values are `Trace`, `Debug`, `Information`, `Warning`, `Error`, or `Critical`. + +- **`--diagnostic-output-directory`** + + The output directory of the diagnostic logging, if not specified the file is generated in the default _TestResults_ directory. + +- **`--diagnostic-output-fileprefix`** + + The prefix for the log file name. Defaults to `"log_"`. + +- **`--diagnostic-filelogger-synchronouswrite`** + + Forces the built-in file logger to synchronously write logs. Useful for scenarios where you don't want to lose any log entries (if the process crashes). This does slow down the test execution. + +#### Process options + +- **`--exit-on-process-exit`** + + Exits the test process if dependent process exits. PID must be provided. + +- **`--ignore-exit-code`** + + Allows some non-zero exit codes to be ignored, and instead returned as `0`. For more information, see [Ignore specific exit codes](./microsoft-testing-platform-exit-codes.md#ignore-specific-exit-codes). + +### Code coverage options + +Options for collecting code coverage. These options require the [Microsoft.Testing.Extensions.CodeCoverage](https://nuget.org/packages/Microsoft.Testing.Extensions.CodeCoverage) NuGet package. + +- **`--coverage`** + + Collects the code coverage using dotnet-coverage tool. + +- **`--coverage-output`** + + The name or path of the produced coverage file. By default, the file is `TestResults/.coverage`. + +- **`--coverage-output-format`** + + Output file format. Supported values are: `coverage`, `xml`, and `cobertura`. Default is `coverage`. + +- **`--coverage-settings`** + + [XML code coverage settings](../additional-tools/dotnet-coverage.md#settings). + +For more information, see [Code coverage extensions](microsoft-testing-platform-extensions-code-coverage.md). + +### Test report options + +Options for generating test reports. + +#### TRX reports + +These options require the [Microsoft.Testing.Extensions.TrxReport](https://nuget.org/packages/Microsoft.Testing.Extensions.TrxReport) NuGet package. + +- **`--report-trx`** + + Generates the TRX report. + +- **`--report-trx-filename`** + + The name of the generated TRX report. The default name matches the following format `__.trx`. + +The report is saved inside the default _TestResults_ folder that can be specified through the `--results-directory` command line argument. + +For more information, see [Test reports extensions](microsoft-testing-platform-extensions-test-reports.md). + +#### Azure DevOps reports + +These options require the [Microsoft.Testing.Extensions.AzureDevOpsReport](https://nuget.org/packages/Microsoft.Testing.Extensions.AzureDevOpsReport) NuGet package. + +- **`--report-azdo`** + + Enables outputting errors and warnings in CI builds. + +- **`--report-azdo-severity`** + + Severity to use for the reported event. Options are: `error` (default) and `warning`. + +The extension automatically detects that it's running in continuous integration (CI) environment by checking the `TF_BUILD` environment variable. + +For more information, see [Test reports extensions](microsoft-testing-platform-extensions-test-reports.md). + +### Dump generation options + +Options for generating crash and hang dumps for troubleshooting. + +#### Crash dump options + +These options require the [Microsoft.Testing.Extensions.CrashDump](https://nuget.org/packages/Microsoft.Testing.Extensions.CrashDump) NuGet package. + +- **`--crashdump`** + + Generates a dump file when the test host process crashes. Supported in .NET 6.0 and later. + +- **`--crashdump-filename`** + + Specifies the file name of the dump. + +- **`--crashdump-type`** + + Specifies the type of the dump. Valid values are `Mini`, `Heap`, `Triage`, `Full`. Defaults as `Full`. For more information, see [Types of mini dumps](../diagnostics/collect-dumps-crash.md#types-of-mini-dumps). + +> [!CAUTION] +> The extension isn't compatible with .NET Framework and will be silently ignored. For .NET Framework support, enable the postmortem debugging with Sysinternals ProcDump. For more information, see [Enabling Postmortem Debugging: Window Sysinternals ProcDump](/windows-hardware/drivers/debugger/enabling-postmortem-debugging#window-sysinternals-procdump). + +#### Hang dump options + +These options require the [Microsoft.Testing.Extensions.HangDump](https://nuget.org/packages/Microsoft.Testing.Extensions.HangDump) NuGet package. + +- **`--hangdump`** + + Generates a dump file in case the test host process hangs. + +- **`--hangdump-filename`** + + Specifies the file name of the dump. + +- **`--hangdump-timeout`** + + Specifies the timeout after which the dump is generated. The timeout value is specified in one of the following formats: + - `1.5h`, `1.5hour`, `1.5hours` + - `90m`, `90min`, `90minute`, `90minutes` + - `5400s`, `5400sec`, `5400second`, `5400seconds` + + Defaults to `30m` (30 minutes). + +- **`--hangdump-type`** + + Specifies the type of the dump. Valid values are `Mini`, `Heap`, `Triage`, `Full`. Defaults as `Full`. For more information, see [Types of mini dumps](../diagnostics/collect-dumps-crash.md#types-of-mini-dumps). + +For more information, see [Diagnostics extensions](microsoft-testing-platform-extensions-diagnostics.md). + +### Test retry options + +Options for retrying failed tests. These options require the [Microsoft.Testing.Extensions.Retry](https://nuget.org/packages/Microsoft.Testing.Extensions.Retry) NuGet package. + +- **`--retry-failed-tests`** + + Reruns any failed tests until they pass or until the maximum number of attempts is reached. + +- **`--retry-failed-tests-max-percentage`** + + Avoids rerunning tests when the percentage of failed test cases crosses the specified threshold. + +- **`--retry-failed-tests-max-tests`** + + Avoids rerunning tests when the number of failed test cases crosses the specified limit. + +> [!NOTE] +> The package is shipped with the restrictive Microsoft.Testing.Platform Tools license. The full license is available at . + +For more information, see [Policy extensions](microsoft-testing-platform-extensions-policy.md). + +### Terminal output options + +Options for controlling how test results are displayed in the terminal. + +- **`--no-progress`** + + Disables reporting progress to screen. + +- **`--no-ansi`** + + Disables outputting ANSI escape characters to screen. + +- **`--output`** + + Output verbosity when reporting tests. Valid values are `Normal` and `Detailed`. Default is `Normal`. + +For more information, see [Output extensions](microsoft-testing-platform-extensions-output.md). + +### Framework-specific options + +Some options are specific to the test framework you're using. + +#### MSTest options + +- **`--filter `** + + Runs tests that match the given expression. For more information, see [Running selective unit tests](selective-unit-tests.md). + +- **`--settings `** + + Specifies a runsettings file for test configuration. For more information, see [Configure MSTest](unit-testing-mstest-configure.md). + +For more information, see [MSTest runner](unit-testing-mstest-runner-intro.md). + +#### NUnit options + +- **`--filter `** + + Runs tests that match the given expression. NUnit uses the same filter syntax as MSTest. For more information, see [Running selective unit tests](selective-unit-tests.md). + +For more information, see [NUnit runner](unit-testing-nunit-runner-intro.md). + +#### xUnit.net options + +xUnit.net provides its own set of filtering options: + +- **`--filter-class `** + + Runs tests in classes that match the pattern. + +- **`--filter-not-class `** + + Excludes tests in classes that match the pattern. + +- **`--filter-method `** + + Runs tests with methods that match the pattern. + +- **`--filter-not-method `** + + Excludes tests with methods that match the pattern. + +- **`--filter-namespace `** + + Runs tests in namespaces that match the pattern. + +- **`--filter-not-namespace `** + + Excludes tests in namespaces that match the pattern. + +- **`--filter-trait =`** + + Runs tests with traits that match the name and value. + +- **`--filter-not-trait =`** + + Excludes tests with traits that match the name and value. + +- **`--filter-query `** + + Runs tests matching the query filter language. For more information, see [Query Filter Language for xUnit.net](https://xunit.net/docs/query-filter-language). + +For more information, see [Microsoft.Testing.Platform documentation for xUnit.net](https://xunit.net/docs/getting-started/v3/microsoft-testing-platform). + +## Environment variables + +Environment variables provide a way to configure tests globally without modifying command-line arguments or configuration files. Environment variables take precedence over configuration files but are overridden by command-line arguments. + +### Platform environment variables + +- **`TESTINGPLATFORM_TELEMETRY_OPTOUT`** or **`DOTNET_CLI_TELEMETRY_OPTOUT`** + + Set to `1` to disable telemetry. For more information, see [Microsoft.Testing.Platform telemetry](microsoft-testing-platform-telemetry.md). + +- **`TESTINGPLATFORM_UI_LANGUAGE`** + + Sets the language of the platform for displaying messages and logs using a locale value such as `en-us`. This language takes precedence over the Visual Studio and .NET SDK languages. The supported values are the same as for Visual Studio. For more information, see [Visual Studio installation documentation](/visualstudio/install/install-visual-studio). + +- **`TESTINGPLATFORM_EXITCODE_IGNORE`** + + Allows some non-zero exit codes to be ignored. The valid format is a semi-colon separated list of exit codes (for example, `2;3;8`). + +### Diagnostic environment variables + +- **`TESTINGPLATFORM_DIAGNOSTIC`** + + If set to `1`, enables the diagnostic logging. + +- **`TESTINGPLATFORM_DIAGNOSTIC_VERBOSITY`** + + Defines the verbosity level. The available values are `Trace`, `Debug`, `Information`, `Warning`, `Error`, or `Critical`. + +- **`TESTINGPLATFORM_DIAGNOSTIC_OUTPUT_DIRECTORY`** + + The output directory of the diagnostic logging, if not specified the file is generated in the default _TestResults_ directory. + +- **`TESTINGPLATFORM_DIAGNOSTIC_OUTPUT_FILEPREFIX`** + + The prefix for the log file name. Defaults to `"log_"`. + +- **`TESTINGPLATFORM_DIAGNOSTIC_FILELOGGER_SYNCHRONOUSWRITE`** + + Forces the built-in file logger to synchronously write logs. Useful for scenarios where you don't want to lose any log entries (if the process crashes). This does slow down the test execution. + +### Hot reload environment variable + +- **`TESTINGPLATFORM_HOTRELOAD_ENABLED`** + + Set to `1` to enable hot reload support. Requires the [Microsoft.Testing.Extensions.HotReload](https://nuget.org/packages/Microsoft.Testing.Extensions.HotReload) NuGet package. For more information, see [Hosting extensions](microsoft-testing-platform-extensions-hosting.md). + +## Configuration file + +The *testconfig.json* file provides a way to configure the test platform using a JSON configuration file. Configuration files have the lowest precedence and are overridden by both environment variables and command-line arguments. + +### File location and naming + +The test platform uses a configuration file named *[appname].testconfig.json* to configure the behavior of the test platform. The platform automatically detects and loads the *[appname].testconfig.json* file located in the output directory of the test project (close to the executable). + +When using [Microsoft.Testing.Platform.MSBuild](https://www.nuget.org/packages/Microsoft.Testing.Platform.MSBuild), you can simply create a *testconfig.json* file that's automatically renamed to *[appname].testconfig.json* and moved to the output directory of the test project. + +> [!NOTE] +> The *[appname].testconfig.json* file gets overwritten on subsequent builds. + +Starting with Microsoft.Testing.Platform 1.5, you can use the command-line argument `--config-file` to specify the path to a *testconfig.json* file. This file takes precedence over the *[appname].testconfig.json* file. + +### File structure + +The *testconfig.json* file is a JSON file that contains configuration settings for the test platform. The file has the following structure: + +```json +{ + "platformOptions": { + "config-property-name1": "config-value1", + "config-property-name2": "config-value2" + } +} +``` + +For more information about available configuration options, see the individual feature documentation. + +## See also + +- [Microsoft.Testing.Platform overview](microsoft-testing-platform-intro.md) +- [Microsoft.Testing.Platform exit codes](microsoft-testing-platform-exit-codes.md) +- [Running selective unit tests](selective-unit-tests.md) +- [Code coverage extensions](microsoft-testing-platform-extensions-code-coverage.md) +- [Test reports extensions](microsoft-testing-platform-extensions-test-reports.md) +- [Diagnostics extensions](microsoft-testing-platform-extensions-diagnostics.md) diff --git a/docs/core/testing/microsoft-testing-platform-run-tests.md b/docs/core/testing/microsoft-testing-platform-run-tests.md new file mode 100644 index 0000000000000..6f5553281aa59 --- /dev/null +++ b/docs/core/testing/microsoft-testing-platform-run-tests.md @@ -0,0 +1,396 @@ +--- +title: Run tests with Microsoft.Testing.Platform +description: Learn about the different methods to run tests with Microsoft.Testing.Platform and when to use each approach. +author: Evangelink +ms.author: amauryleve +ms.date: 11/27/2025 +--- + +# Run tests with Microsoft.Testing.Platform + +Microsoft.Testing.Platform test projects are built as executables that can be run directly or through various tools. This article explains the different execution methods and when to use each. + +> [!TIP] +> If you're new to Microsoft.Testing.Platform, start with the [Getting Started guide](microsoft-testing-platform-getting-started.md) to create and run your first test project. + +## Execution methods + +Microsoft.Testing.Platform supports multiple ways to run tests, each suited for different scenarios: + +| Method | Best for | Pros | Cons | +|--------|----------|------|------| +| [Direct execution](#direct-execution) | Production deployments, Native AOT | No dependencies, fastest startup | Requires separate build | +| [`dotnet run`](#dotnet-run) | Local development | Auto-rebuilds, finds project | Slower due to build check | +| [`dotnet exec`](#dotnet-exec) | Scripting, automation | Fast, explicit control | Requires exact path | +| [`dotnet test`](#dotnet-test) | CI/CD, existing workflows | Familiar, works with both platforms | Additional layer | +| [Visual Studio](#visual-studio) | Interactive testing, debugging | Rich UI, test management | IDE required | +| [Visual Studio Code](#visual-studio-code) | Cross-platform development | Lightweight, integrated | Extension required | +| [CI/CD pipelines](#continuous-integration) | Automated testing | Reliable, reproducible | Environment-specific setup | + +### Direct execution + +Run the test executable directly. This is the most straightforward method and works exactly like any other .NET application. + +```bash +# Windows +.\MyTests.exe + +# Linux/macOS +./MyTests +``` + +**When to use:** + +- Production-like scenarios +- Native AOT deployments +- When you need complete control over the process +- Debugging specific runtime issues + +**Requirements:** + +- Project must be built first (`dotnet build` or `dotnet publish`) +- Test project must have `Exe` + +For more information about publishing, see [.NET application publishing](../deploying/index.md). + +### dotnet run + +The `dotnet run` command builds and runs your test project in one step. + +```dotnetcli +dotnet run --project MyTests +``` + +Pass arguments to the test executable using `--`: + +```dotnetcli +dotnet run --project MyTests -- --filter "FullyQualifiedName~Calculator" +``` + +**When to use:** + +- Local development and iteration +- When you want automatic rebuilds +- Testing during active code changes + +**Benefits:** + +- Automatically finds the project in the current directory +- Rebuilds only when needed +- Combines build and run in one command + +For more information, see [dotnet run](../tools/dotnet-run.md). + +### dotnet exec + +The `dotnet exec` command (or just `dotnet`) runs an already-built test assembly. + +```dotnetcli +dotnet exec MyTests.dll +``` + +Or the shorter form: + +```dotnetcli +dotnet MyTests.dll +``` + +Pass arguments directly: + +```dotnetcli +dotnet MyTests.dll --filter "FullyQualifiedName~Calculator" +``` + +**When to use:** + +- Tests are already built +- Scripting and automation scenarios +- When you need fast execution without build checks + +> [!NOTE] +> Use the `.dll` file, not the `.exe` file. Using the executable results in an error about duplicate assemblies. + +For more information, see [dotnet exec](../tools/dotnet.md#options-for-running-an-application-with-the-exec-command). + +### dotnet test + +The `dotnet test` command provides compatibility with existing testing workflows. It works with both Microsoft.Testing.Platform and VSTest. + +```dotnetcli +dotnet test +``` + +Microsoft.Testing.Platform supports two modes with `dotnet test`: + +**For .NET 10 SDK and later (recommended):** + +Add to *global.json*: + +```json +{ + "test": { + "runner": "Microsoft.Testing.Platform" + } +} +``` + +Then use arguments directly: + +```dotnetcli +dotnet test --filter "FullyQualifiedName~Calculator" --report-trx +``` + +**For .NET 9 SDK and earlier (VSTest mode):** + +> [!CAUTION] +> Starting with .NET 10 SDK, it's recommended to not use the VSTest mode of `dotnet test` when running with Microsoft.Testing.Platform. + +Set in your project file or *Directory.Build.props*: + +```xml + + true + +``` + +Use extra `--` to separate platform arguments: + +```dotnetcli +dotnet test -- --filter "FullyQualifiedName~Calculator" --report-trx +``` + +**When to use:** + +- Existing CI/CD pipelines using `dotnet test` +- When you need MSBuild integration +- Running multiple test projects in a solution + +**VSTest mode configuration options:** + +When using VSTest mode (.NET 9 SDK and earlier), you can configure these additional options: + +**Show failure per test:** + +By default, test failures are summarized into a *.log* file, and a single failure per test project is reported to MSBuild. To show errors per failed test: + +```dotnetcli +dotnet test -p:TestingPlatformShowTestsFailure=true +``` + +Or in *Directory.Build.props*: + +```xml + + true + +``` + +**Show complete platform output:** + +By default, all console output from the test executable is captured and hidden. To show the banner, version information, and formatted test information: + +```dotnetcli +dotnet test -p:TestingPlatformCaptureOutput=false +``` + +Or in *Directory.Build.props*: + +```xml + + false + +``` + +> [!TIP] +> Set these properties in *Directory.Build.props* instead of individual project files. This ensures consistency across all test projects and avoids mixing VSTest and Microsoft.Testing.Platform projects in the same solution. + +For complete information about using `dotnet test` with Microsoft.Testing.Platform, see [Testing with dotnet test](unit-testing-with-dotnet-test.md). + +### Visual Studio + +Visual Studio provides rich support for running and debugging tests through Test Explorer. + +**Using Test Explorer:** + +1. Open **Test Explorer** (Test > Test Explorer or Ctrl+E, T) +1. Tests are discovered automatically after building +1. Select tests and click **Run** or **Debug** +1. Use keyboard shortcuts: Ctrl+R, A to run all tests + +**Running as startup project:** + +1. Right-click the test project in Solution Explorer +1. Select **Set as Startup Project** +1. Press F5 to debug or Ctrl+F5 to run + +**Benefits:** + +- Visual test results and history +- Run/debug individual tests or groups +- Live test results while editing code +- Integration with code coverage tools + +**Requirements:** + +- Visual Studio 2022 version 17.14 or later for full Microsoft.Testing.Platform support + +For more information, see [Run unit tests with Test Explorer](/visualstudio/test/run-unit-tests-with-test-explorer). + +### Visual Studio Code + +Visual Studio Code supports running and debugging tests through the C# Dev Kit extension. + +**Setup:** + +1. Install the [C# Dev Kit extension](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp) +1. Open your test project folder +1. Build the project (Ctrl+Shift+B) + +**Using Test Explorer:** + +1. Open the **Test Explorer** view (Testing icon in the Activity Bar) +1. Tests are discovered after building +1. Select tests and click **Run Test** or **Debug Test** + +**Using the integrated terminal:** + +```dotnetcli +dotnet run --project MyTests +``` + +**Benefits:** + +- Cross-platform support +- Lightweight compared to Visual Studio +- Integrated terminal for advanced scenarios +- Support for debugging with breakpoints + +For more information, see [Testing in Visual Studio Code](https://code.visualstudio.com/docs/csharp/testing). + +### Continuous integration + +Microsoft.Testing.Platform tests run as regular executables in CI/CD environments. No special test runner or task is required. + +**Azure Pipelines:** + +```yml +- task: CmdLine@2 + displayName: "Run Tests" + inputs: + script: 'dotnet run --project tests/MyTests/MyTests.csproj' +``` + +**GitHub Actions:** + +```yml +- name: Run Tests + run: dotnet run --project tests/MyTests/MyTests.csproj +``` + +**GitLab CI:** + +```yml +test: + script: + - dotnet run --project tests/MyTests/MyTests.csproj +``` + +**When to use:** + +- Automated testing in build pipelines +- Continuous integration workflows +- Deployment validation + +**Best practices:** + +- Use `dotnet run` or `dotnet exec` for reliability +- Check exit codes to determine test success +- Generate reports for pipeline integration (use `--report-trx`) +- Store test results as artifacts + +For more information about exit codes, see [Microsoft.Testing.Platform exit codes](microsoft-testing-platform-exit-codes.md). + +## Passing arguments + +All execution methods support passing command-line arguments to configure test execution: + +**Direct execution:** + +```bash +./MyTests.exe --filter "FullyQualifiedName~Calculator" --report-trx +``` + +**dotnet run:** + +```dotnetcli +dotnet run --project MyTests -- --filter "FullyQualifiedName~Calculator" +``` + +**dotnet exec:** + +```dotnetcli +dotnet MyTests.dll --filter "FullyQualifiedName~Calculator" +``` + +**dotnet test (.NET 10+):** + +```dotnetcli +dotnet test --filter "FullyQualifiedName~Calculator" --report-trx +``` + +For a complete list of available options, see [Configuration reference](microsoft-testing-platform-options.md). + +## Debugging tests + +All execution methods support debugging: + +**Visual Studio:** + +- Use Test Explorer's **Debug** option for specific tests +- Set test project as startup project and press F5 +- Attach to running test process + +**Visual Studio Code:** + +- Use Test Explorer's **Debug Test** option +- Set up launch configuration for test project +- Use F5 with test project + +**Command line:** + +- Use `dotnet run` and attach debugger to process +- Set `DOTNET_WAIT_FOR_DEBUGGER=1` environment variable to wait for debugger on startup + +For more information about debugging, see [Debug unit tests](/visualstudio/test/debug-unit-tests-with-test-explorer). + +## Choosing an execution method + +Use this decision tree to choose the best execution method: + +```mermaid +graph TD + A[How do you want to run tests?] --> B{Local development?} + B -->|Yes| C{Using IDE?} + B -->|No| D{CI/CD pipeline?} + + C -->|Visual Studio| E[Use Test Explorer] + C -->|VS Code| F[Use Test Explorer + C# Dev Kit] + C -->|No IDE| G{Iterating on code?} + + G -->|Yes| H[Use dotnet run] + G -->|No| I[Use direct execution or dotnet exec] + + D -->|Yes| J{Existing dotnet test?} + D -->|No| K[Use dotnet run or direct execution] + + J -->|Yes| L[Use dotnet test with platform support] + J -->|No| M[Use dotnet run] +``` + +## See also + +- [Get started with Microsoft.Testing.Platform](microsoft-testing-platform-getting-started.md) +- [Configuration reference](microsoft-testing-platform-options.md) +- [Testing with dotnet test](unit-testing-with-dotnet-test.md) +- [Microsoft.Testing.Platform exit codes](microsoft-testing-platform-exit-codes.md) +- [Running selective unit tests](selective-unit-tests.md) diff --git a/docs/core/testing/unit-testing-mstest-runner-intro.md b/docs/core/testing/unit-testing-mstest-runner-intro.md index c799e9c54c2ca..df9e444831dc8 100644 --- a/docs/core/testing/unit-testing-mstest-runner-intro.md +++ b/docs/core/testing/unit-testing-mstest-runner-intro.md @@ -44,7 +44,7 @@ Consider the following example project file: true diff --git a/docs/core/testing/unit-testing-mstest-sdk.md b/docs/core/testing/unit-testing-mstest-sdk.md index df51a1d4f10a1..55dadd72c14f6 100644 --- a/docs/core/testing/unit-testing-mstest-sdk.md +++ b/docs/core/testing/unit-testing-mstest-sdk.md @@ -60,7 +60,7 @@ When you `build` the project, all the needed components are restored and install You don't need anything else to build and run your tests and you can use the same tooling (for example, `dotnet test` or Visual Studio) used by a ["classic" MSTest project](./unit-testing-csharp-with-mstest.md). > [!IMPORTANT] -> By switching to the `MSTest.Sdk`, you opt in to using the [MSTest runner (enables Microsoft.Testing.Platform for MSTest)](./unit-testing-mstest-runner-intro.md), including with [dotnet test](./microsoft-testing-platform-integration-dotnet-test.md). That requires modifying your CI and local CLI calls, and also impacts the available entries of the _.runsettings_. You can use `MSTest.Sdk` and still keep the old integrations and tools by instead switching the [runner](#select-the-runner). +> By switching to the `MSTest.Sdk`, you opt in to using the [MSTest runner (enables Microsoft.Testing.Platform for MSTest)](./unit-testing-mstest-runner-intro.md), including with [dotnet test](./microsoft-testing-platform-run-tests.md#dotnet-test). That requires modifying your CI and local CLI calls, and also impacts the available entries of the _.runsettings_. You can use `MSTest.Sdk` and still keep the old integrations and tools by instead switching the [runner](#select-the-runner). > By default, MSTest.Sdk sets `EnableMSTestRunner` and `TestingPlatformDotnetTestSupport` to true. For more information about dotnet test and its different modes for running Microsoft.Testing.Platform, see [Testing with dotnet test](./unit-testing-with-dotnet-test.md). ## Test utility helper libraries diff --git a/docs/core/testing/unit-testing-nunit-runner-intro.md b/docs/core/testing/unit-testing-nunit-runner-intro.md index 9babc27877976..2efd3d0bde896 100644 --- a/docs/core/testing/unit-testing-nunit-runner-intro.md +++ b/docs/core/testing/unit-testing-nunit-runner-intro.md @@ -31,7 +31,7 @@ Consider the following example project file: true diff --git a/docs/core/testing/unit-testing-with-dotnet-test.md b/docs/core/testing/unit-testing-with-dotnet-test.md index c57dc5e2871da..5ba521c782dcf 100644 --- a/docs/core/testing/unit-testing-with-dotnet-test.md +++ b/docs/core/testing/unit-testing-with-dotnet-test.md @@ -65,7 +65,7 @@ The following list outlines the command-line options of `dotnet test` command in > > ``` -For more information specific to running MTP projects in VSTest mode of `dotnet test`, see [Use Microsoft.Testing.Platform with VSTest mode of `dotnet test`](./microsoft-testing-platform-integration-dotnet-test.md). +For more information about running tests with Microsoft.Testing.Platform, including VSTest mode configuration, see [Run tests with Microsoft.Testing.Platform](./microsoft-testing-platform-run-tests.md#dotnet-test). #### Advanced technical details