fix(defer-enumeration): report placeholder as a container with an aggregate result

The IDE run of a deferred test's placeholder node showed "Inconclusive: Test has
not run" because the placeholder produced no result when its children were the
real results. Reporting it always-Passed (the original behaviour) is wrong too —
it would show green even when a child fails.

Now the placeholder is reported as a container: InProgress when expansion starts,
then a final result after the children run that aggregates them — passed only if
every case passes, failed if any case fails, skipped if all were skipped. This
resolves the IDE node without masking child failures. The placeholder therefore
adds one entry to flat result counts (TRX/console) per deferred test; engine
tests and docs updated to reflect this.

Author: thomhurst

TUnit.Engine.Tests/DeferEnumerationTests.cs

@@ -5,9 +5,10 @@ namespace TUnit.Engine.Tests;
 
 /// <summary>
 /// Validates DeferEnumeration: a data source marked with it is NOT expanded during discovery (one
-/// placeholder node is shown) and is expanded into the real cases at runtime. The placeholder is a
-/// container and is not reported as its own result, so the run-time counts below are exactly the number
-/// of data cases. Discovery-time collapse is covered by manual verification / list-tests.
+/// placeholder node is shown) and is expanded into the real cases at runtime. The placeholder is reported
+/// as a container whose result aggregates its cases, so the run-time counts below are the number of data
+/// cases plus one for the placeholder (e.g. a 10-row source => 10 cases + 1 placeholder = 11). Discovery-time
+/// collapse is covered by manual verification / list-tests.
 /// </summary>
 public class DeferEnumerationTests(TestMode testMode) : InvokableTestBase(testMode)
 {
@@ -18,34 +19,35 @@ await RunTestsWithFilter(
             "/*/*/DeferEnumerationTests/Deferred",
             [
                 result => result.ResultSummary.Outcome.ShouldBe("Completed"),
-                result => result.ResultSummary.Counters.Total.ShouldBe(10),
-                result => result.ResultSummary.Counters.Passed.ShouldBe(10),
+                result => result.ResultSummary.Counters.Total.ShouldBe(11),
+                result => result.ResultSummary.Counters.Passed.ShouldBe(11),
                 result => result.ResultSummary.Counters.Failed.ShouldBe(0)
             ]);
     }
 
     [Test]
     public async Task Deferred_Test_Honours_Repeat()
     {
-        // [Repeat(2)] => 3 runs per case; 10 cases => 30 results (the placeholder is not counted).
+        // [Repeat(2)] => 3 runs per case; 10 cases => 30 cases + 1 placeholder container = 31.
         await RunTestsWithFilter(
             "/*/*/DeferEnumerationTests/DeferredWithRepeat",
             [
                 result => result.ResultSummary.Outcome.ShouldBe("Completed"),
-                result => result.ResultSummary.Counters.Total.ShouldBe(30),
-                result => result.ResultSummary.Counters.Passed.ShouldBe(30),
+                result => result.ResultSummary.Counters.Total.ShouldBe(31),
+                result => result.ResultSummary.Counters.Passed.ShouldBe(31),
                 result => result.ResultSummary.Counters.Failed.ShouldBe(0)
             ]);
     }
 
     [Test]
     public async Task Deferred_Data_Source_Error_Surfaces_At_Runtime_Without_Crashing_Discovery()
     {
-        // The throwing data source must not crash discovery; the error surfaces as a failed result.
+        // The throwing data source must not crash discovery; the error surfaces as a failed case, and the
+        // placeholder container aggregates to failed too (failed case + failed container = 2).
         await RunTestsWithFilter(
             "/*/*/DeferEnumerationErrorTests/*",
             [
-                result => result.ResultSummary.Counters.Failed.ShouldBe(1)
+                result => result.ResultSummary.Counters.Failed.ShouldBe(2)
             ]);
     }
 }

TUnit.Engine/TestSessionCoordinator.cs

@@ -59,14 +59,18 @@ public async Task ExecuteTests(
 
         // Expand any deferred-enumeration placeholders into their real cases before counting/scheduling,
         // so the children flow through the normal pipeline (correct hooks + lifecycle counting).
-        await ExpandDeferredPlaceholdersAsync(testList, cancellationToken);
+        var expandedPlaceholders = await ExpandDeferredPlaceholdersAsync(testList, cancellationToken);
 
         InitializeEventReceivers(testList, cancellationToken);
 
         try
         {
             await PrepareTestOrchestrator(testList, cancellationToken);
             await ExecuteTestsCore(testList, cancellationToken);
+
+            // Children have now run: resolve each placeholder container to the aggregate of its cases so the
+            // IDE node the user ran gets a result (rather than "not run") that reflects its children.
+            await ReportDeferredPlaceholderResultsAsync(expandedPlaceholders);
         }
         finally
         {
@@ -101,18 +105,20 @@ private void InitializeEventReceivers(List<AbstractExecutableTest> testList, Can
 
     /// <summary>
     /// Replaces every <see cref="DeferredEnumerationExecutableTest"/> in the list with the real test cases
-    /// produced by enumerating its data source. The placeholder is a container, not a test: its real results
-    /// are the children (added to the list, scheduled like any other test, and nested under the placeholder
-    /// via their ParentTestId), so it is NOT reported as a passed result on success — that would inflate test
-    /// counts and show the node green even when its children fail. A result is reported only if the expansion
-    /// itself throws, so that failure stays visible on the placeholder node. (Per-row data errors surface as
-    /// their own failed child via the standard data-generation-error path.)
+    /// produced by enumerating its data source. The placeholder is reported as a running container now (so the
+    /// IDE node the user ran shows as in-progress rather than "not run") and resolved to the aggregate of its
+    /// children later by <see cref="ReportDeferredPlaceholderResultsAsync"/>; the children are added to the
+    /// list, scheduled like any other test, and nested under the placeholder via their ParentTestId. If the
+    /// expansion itself throws, the placeholder is reported failed immediately. (Per-row data errors surface
+    /// as their own failed child via the standard data-generation-error path.) The returned list pairs each
+    /// successfully-expanded placeholder with its children so its final result can be reported post-run.
     /// </summary>
 #if NET8_0_OR_GREATER
     [UnconditionalSuppressMessage("Trimming", "IL2026", Justification = "Reflection mode is not used in AOT/trimmed scenarios")]
     [UnconditionalSuppressMessage("AOT", "IL3050", Justification = "Reflection mode is not used in AOT scenarios")]
 #endif
-    private async Task ExpandDeferredPlaceholdersAsync(List<AbstractExecutableTest> testList, CancellationToken cancellationToken)
+    private async Task<List<(AbstractExecutableTest Placeholder, IReadOnlyList<AbstractExecutableTest> Children)>> ExpandDeferredPlaceholdersAsync(
+        List<AbstractExecutableTest> testList, CancellationToken cancellationToken)
     {
         List<DeferredEnumerationExecutableTest>? placeholders = null;
         foreach (var test in testList)
@@ -125,32 +131,86 @@ private async Task ExpandDeferredPlaceholdersAsync(List<AbstractExecutableTest>
 
         if (placeholders is null)
         {
-            return;
+            return [];
         }
 
         // Drop all placeholders in a single pass so none are scheduled as real tests (their Create/Invoke
         // throw); their children are added back below.
         testList.RemoveAll(static t => t is DeferredEnumerationExecutableTest);
 
+        var expanded = new List<(AbstractExecutableTest, IReadOnlyList<AbstractExecutableTest>)>(placeholders.Count);
+
         foreach (var placeholder in placeholders)
         {
+            placeholder.StartTime = DateTimeOffset.UtcNow;
+            placeholder.State = TestState.Running;
+            await _messageBus.InProgress(placeholder.Context);
+
             try
             {
                 var children = await _deferredTestExpander.ExpandAsync(placeholder, cancellationToken);
                 testList.AddRange(children);
+                expanded.Add((placeholder, children));
             }
             catch (Exception ex)
             {
                 // Expansion itself failed (as opposed to a per-row data error, which becomes a failed
                 // child). Surface it on the placeholder node so the failure is visible.
                 await _logger.LogErrorAsync($"Failed to expand deferred test '{placeholder.TestId}': {ex}");
-                placeholder.StartTime = DateTimeOffset.UtcNow;
-                await _messageBus.InProgress(placeholder.Context);
                 placeholder.EndTime = DateTimeOffset.UtcNow;
                 placeholder.SetResult(TestState.Failed, ex);
                 await _messageBus.Failed(placeholder.Context, ex, placeholder.StartTime.GetValueOrDefault());
             }
         }
+
+        return expanded;
+    }
+
+    /// <summary>
+    /// Reports the final result for each deferred placeholder once its children have executed: the placeholder
+    /// is a container whose outcome is the aggregate of its cases — failed if any case failed, skipped if every
+    /// case was skipped, otherwise passed. This resolves the IDE node the user ran without masking child
+    /// failures (which a fixed "passed" would).
+    /// </summary>
+    private async Task ReportDeferredPlaceholderResultsAsync(
+        List<(AbstractExecutableTest Placeholder, IReadOnlyList<AbstractExecutableTest> Children)> expandedPlaceholders)
+    {
+        foreach (var (placeholder, children) in expandedPlaceholders)
+        {
+            placeholder.EndTime = DateTimeOffset.UtcNow;
+
+            var failedCount = 0;
+            var skippedCount = 0;
+            foreach (var child in children)
+            {
+                switch (child.State)
+                {
+                    case TestState.Failed or TestState.Timeout or TestState.Cancelled:
+                        failedCount++;
+                        break;
+                    case TestState.Skipped:
+                        skippedCount++;
+                        break;
+                }
+            }
+
+            if (failedCount > 0)
+            {
+                var exception = new Exception($"{failedCount} of {children.Count} deferred test case(s) failed.");
+                placeholder.SetResult(TestState.Failed, exception);
+                await _messageBus.Failed(placeholder.Context, exception, placeholder.StartTime.GetValueOrDefault());
+            }
+            else if (children.Count > 0 && skippedCount == children.Count)
+            {
+                placeholder.State = TestState.Skipped;
+                await _messageBus.Skipped(placeholder.Context, "All deferred test cases were skipped");
+            }
+            else
+            {
+                placeholder.SetResult(TestState.Passed);
+                await _messageBus.Passed(placeholder.Context, placeholder.StartTime.GetValueOrDefault());
+            }
+        }
     }
 
     private async Task PrepareTestOrchestrator(List<AbstractExecutableTest> testList, CancellationToken cancellationToken)

docs/docs/writing-tests/defer-enumeration.md

@@ -26,7 +26,7 @@ With the flag set:
 `DeferEnumeration` is available on any data source attribute (`[MethodDataSource]`, `[ClassDataSource]`, custom `DataSourceGenerator` attributes, etc.). If **any** data source on a test sets it, the entire test's case expansion is deferred. It has no effect on `[Arguments]` (a single inline row, so there is nothing to defer).
 
 :::info
-The placeholder is a **container**, not a test: it is not reported as its own result, so it does not count toward your test totals, and the individual cases (nested under it) carry the real pass/fail results. If the data source itself throws while enumerating, the error surfaces as a failed result at run time (just as a non-deferred data source error would) instead of failing discovery for the whole assembly.
+The placeholder is reported as a **container**: the individual cases (nested under it) carry the real pass/fail results, and the placeholder's own result aggregates them — it passes only if every case passes, and fails if any case fails. Because it is reported, it adds one extra entry to flat result counts (TRX/console) per deferred test. If the data source itself throws while enumerating, the error surfaces as a failed result at run time (just as a non-deferred data source error would) instead of failing discovery for the whole assembly.
 :::
 
 :::warning Trade-offs