Port test.fail's session outcome with NUnit's E2E.Fail property (#1553)

docs-site/src/content/docs/coding-agents.mdx

@@ -60,7 +60,7 @@ claude mcp add e2e -- e2e mcp
 
 Each node in `observe` starts its line with a node id (`#n42 button "Save"`), and an action takes that id as its `target`. `observe` shows the whole screen under a revision (`Current screen (revision b3, path /, 12 nodes):`). An action reports only what changed since the screen the agent last received: the lines that were `added`, `changed` (with what each read before), or `removed`. An id stays the same for as long as its element exists, so a node that is not listed as removed is still on screen under the id the agent has. When the screen changed mostly, the action sends it whole again.
 
-`open_session {test: "Shop.CheckoutTests.MemberChecksOut"}` runs one NUnit test of the config's `testAssembly` in the session and stops on the screen where it passed or failed. The opening text names the failing line with the code around it, the screen's nodes nearest a failed locator, and the last steps. A `[OneTimeSetUp]` that failed is the outcome, `HOOK_FAILED in beforeAll`, with no steps and no screen (`No screen to show`), and a test that ran out of time is `TEST_TIMEOUT`. A task the test left faulted and never awaited fails it with its error (`Also failed:` for any further ones); one that faults after the body ended is reported at `close_session` as `UNHANDLED_REJECTION`, and one that faults after the session closed is a warning on the server's stderr. `test_lines` returns the lines the agent adds after the test stopped. Name the test by its full name or as `file:line`, the line its attributes or signature is on (not a line inside the body). The name must select one test: several fail with `TEST_AMBIGUOUS`, and a test that starts from a saved session (`[E2ESession]`) fails with `SESSION_TEST_UNSUPPORTED`. If the test's timeout cut its body short, the session still opens on the screen the body left, and the steps the body would have run after the cut do not run. The agent reads the failure, checks the screen, and fixes the test, instead of clicking through the flow again. Build the test project first. The assembly loads again on every `open_session`, so a rebuilt test runs as it is now, with no server restart. Tests take turns: a second `open_session {test}` waits until the test before it has ended, and cancelling it while it waits runs none of its test.
+`open_session {test: "Shop.CheckoutTests.MemberChecksOut"}` runs one NUnit test of the config's `testAssembly` in the session and stops on the screen where it passed or failed. The opening text names the failing line with the code around it, the screen's nodes nearest a failed locator, and the last steps. A `[OneTimeSetUp]` that failed is the outcome, `HOOK_FAILED in beforeAll`, with no steps and no screen (`No screen to show`), and a test that ran out of time is `TEST_TIMEOUT`. A test marked as expected to fail with NUnit's property `E2E.Fail` (`[Property("E2E.Fail", "reason")]`, or set in its body) reads `failed as expected` with `Expected to fail: <reason>` when its body fails, and `UNEXPECTED_PASS` when it passes; the mark does not change the verdict of `dotnet test`. A task the test left faulted and never awaited fails it with its error (`Also failed:` for any further ones); one that faults after the body ended is reported at `close_session` as `UNHANDLED_REJECTION`, and one that faults after the session closed is a warning on the server's stderr. `test_lines` returns the lines the agent adds after the test stopped. Name the test by its full name or as `file:line`, the line its attributes or signature is on (not a line inside the body). The name must select one test: several fail with `TEST_AMBIGUOUS`, and a test that starts from a saved session (`[E2ESession]`) fails with `SESSION_TEST_UNSUPPORTED`. If the test's timeout cut its body short, the session still opens on the screen the body left, and the steps the body would have run after the cut do not run. The agent reads the failure, checks the screen, and fixes the test, instead of clicking through the flow again. Build the test project first. The assembly loads again on every `open_session`, so a rebuilt test runs as it is now, with no server restart. Tests take turns: a second `open_session {test}` waits until the test before it has ended, and cancelling it while it waits runs none of its test.
 
 When the engine records video, the catalog also has `start_recording {name?}` and `stop_recording`. The session records only between them. `stop_recording` saves the video to `.e2e/videos/<session>/<n>[-<name>].<ext>` in the project and returns each file's absolute path, or the URL where a hosted provider keeps it. `close_session` saves a recording still running. Recordings are not masked, so a secret typed into a plain input shows in the video.
 

skills/e2e-dotnet/SKILL.md

@@ -115,7 +115,7 @@ working directory it resolved against.
 `open_session` names the app URL, the port a free-port `url` was assigned, and the
 app command (its executable, readiness URL, and `env` and `inheritEnv` names; no
 `args`, no `env` values). The config reads the MCP server's environment, not
-your shell's. `open_session {test}` opens with how the test ended: `HOOK_FAILED in beforeAll` (no steps, no screen) when a `[OneTimeSetUp]` failed, `TEST_TIMEOUT` when it ran out of time, the error of a task it left faulted and unobserved (`UNHANDLED_REJECTION` at `close_session` when that task faults after the body), and `Also failed:` lines for later failures (the `mcp` topic has the detail).
+your shell's. `open_session {test}` opens with how the test ended: `HOOK_FAILED in beforeAll` (no steps, no screen) when a `[OneTimeSetUp]` failed, `TEST_TIMEOUT` when it ran out of time, the error of a task it left faulted and unobserved (`UNHANDLED_REJECTION` at `close_session` when that task faults after the body), and `Also failed:` lines for later failures (the `mcp` topic has the detail). A test marked as expected to fail with NUnit's property `E2E.Fail` (`[Property("E2E.Fail", "reason")]`, or set in the body) reads `failed as expected` with `Expected to fail: <reason>` when its body fails, and `UNEXPECTED_PASS` when it passes; the mark does not change the verdict of `dotnet test`.
 
 ```csharp
 using E2E;

skills/e2e-dotnet/references/mcp.md

@@ -94,7 +94,7 @@ tool with the same name, use the `e2e` server's.
 
 | Tool | Does |
 | --- | --- |
-| `open_session` | Loads the config (`config` names another file; default the nearest `e2e.config.json`), boots the browser or device, opens the app (its URL, or launches a device's pinned app; it installs no build, so a local device needs `app.bundleId`), and returns the session id, the catalog, and the first observation. Its summary names the app URL, a port assigned for `:0`, and the app command's readiness URL and `env` names; e2e starts nothing else, so a database the app uses runs outside it. The config reads the MCP server's environment, not your shell's. `target` must name the config's target when given. `headed: true` shows the browser when the user wants to watch. `test` (`Namespace.Class.Method`) runs that one NUnit test of the config's `testAssembly` first, as `dotnet test` would (set-up, body, tear-down of the test's own hooks, replay cache as configured), and hands the session over on the screen where it passed or failed: the opening text leads with the outcome and how long it took, then for a failure the error code and message (`HOOK_FAILED in beforeAll: beforeAll failed: ...` when a `[OneTimeSetUp]` failed, so no test ran and the session opens on no screen, `No screen to show: ...`; `TEST_TIMEOUT: test timed out after N ms in phase body` when the test ran out of time; the task's own error when a task the test left faulted unobserved fails during the test, `UNHANDLED_REJECTION` at `close_session` when it faults after the body, and as a stderr warning when it faults after the session closed; `Also failed:` lines for the later ones), the test line it came from with the code around it, the URL, the nodes on screen nearest what a failed locator asked for, and the last steps. `test` must name exactly one test: a name that selects several fails `TEST_AMBIGUOUS` (the message lists them), and a test that starts from a saved session (`[E2ESession]`) fails `SESSION_TEST_UNSUPPORTED`; run it with `dotnet test`. Tests take turns in one server: a second `open_session {test}` waits until the test before it has ended, and a cancel while it waits runs none of its test. `test_lines` then returns only the lines after the test stopped. To fix a failing test, open it this way, look at the screen, and put the lines you ran in the test; close the session, rebuild the test project (restart the server for the new build), and open it again. Build the test project first. The test's own steps print no test line. |
+| `open_session` | Loads the config (`config` names another file; default the nearest `e2e.config.json`), boots the browser or device, opens the app (its URL, or launches a device's pinned app; it installs no build, so a local device needs `app.bundleId`), and returns the session id, the catalog, and the first observation. Its summary names the app URL, a port assigned for `:0`, and the app command's readiness URL and `env` names; e2e starts nothing else, so a database the app uses runs outside it. The config reads the MCP server's environment, not your shell's. `target` must name the config's target when given. `headed: true` shows the browser when the user wants to watch. `test` (`Namespace.Class.Method`) runs that one NUnit test of the config's `testAssembly` first, as `dotnet test` would (set-up, body, tear-down of the test's own hooks, replay cache as configured), and hands the session over on the screen where it passed or failed. A test marked as expected to fail, with NUnit's property `E2E.Fail` (`[Property("E2E.Fail", "reason")]`, or set in the body), reads `failed as expected` when its body fails, and `UNEXPECTED_PASS` when it passes; the mark does not change `dotnet test`'s verdict: the opening text leads with the outcome and how long it took, then for a failure the error code and message (`HOOK_FAILED in beforeAll: beforeAll failed: ...` when a `[OneTimeSetUp]` failed, so no test ran and the session opens on no screen, `No screen to show: ...`; `TEST_TIMEOUT: test timed out after N ms in phase body` when the test ran out of time; the task's own error when a task the test left faulted unobserved fails during the test, `UNHANDLED_REJECTION` at `close_session` when it faults after the body, and as a stderr warning when it faults after the session closed; `Also failed:` lines for the later ones), the test line it came from with the code around it, the URL, the nodes on screen nearest what a failed locator asked for, and the last steps. `test` must name exactly one test: a name that selects several fails `TEST_AMBIGUOUS` (the message lists them), and a test that starts from a saved session (`[E2ESession]`) fails `SESSION_TEST_UNSUPPORTED`; run it with `dotnet test`. Tests take turns in one server: a second `open_session {test}` waits until the test before it has ended, and a cancel while it waits runs none of its test. `test_lines` then returns only the lines after the test stopped. To fix a failing test, open it this way, look at the screen, and put the lines you ran in the test; close the session, rebuild the test project (restart the server for the new build), and open it again. Build the test project first. The test's own steps print no test line. |
 | `tools` | The catalog: one line per tool with its argument names (`?` marks optional), the first sentence of its description, and `[read-only]` where it changes nothing. `tools {tool}` shows the full description and the JSON Schema of its arguments. |
 | `call` | Runs one catalog tool: `call {tool: "tap", args: {target: "n42"}}`, where `n42` is the node id that starts a line of the newest observation (`#n42 button "Save"`). Arguments are checked against the tool's schema first; a wrong one fails with `INVALID_ARGUMENT` naming the field. |
 | `close_session` | Ends the session, disposes the browser or device engine, and releases a hosted device the session leased, each within the config's `cleanupTimeout`. A cleanup that fails or times out is a `Cleanup:` line in the result, with the error's code when it has one (`CLEANUP_TIMEOUT` for a release that outlived the budget). With no session open and no `session` argument, it says `No session is open.` |

src/E2E.Cli/Mcp/Session.cs

@@ -438,7 +438,7 @@ [.. secrets.Select(secret => secret.Name)],
             var text = OpeningText(live, target, app, config, await FirstScreenAsync(live, target, ran is not null, abort.Token).ConfigureAwait(false));
             if (ran is not null)
             {
-                text = TestOutcome.Describe(ran.TestName, ran.Failure, elapsed, session.StepRecords, session.Evidence, session.Redactor.Redact, config.ProjectRoot, ran.Phase, ran.Secondary) + "\n" + text;
+                text = TestOutcome.Describe(ran.TestName, ran.Failure, elapsed, session.StepRecords, session.Evidence, session.Redactor.Redact, config.ProjectRoot, ran.Phase, ran.Secondary, ran.ExpectedFailure) + "\n" + text;
             }
 
             // A first screen that came back despite the cancel must not become a live session nobody asked for.

src/E2E.Cli/Mcp/TestOutcome.cs

@@ -26,16 +26,25 @@ internal static class TestOutcome
 
     /// <summary>
     /// The opening lines of a session that ran the test named <paramref name="name"/>, before its summary and catalog. <paramref name="steps"/> and <paramref name="evidence"/> are the session's; <paramref name="elapsed"/> is
-    /// how long the run took, set-up and tear-down included. Upstream's <c>describeTestOutcome</c>.
+    /// how long the run took, set-up and tear-down included. <paramref name="expectedFailure"/> is the reason the test is marked as expected to fail, or
+    /// null. Upstream's <c>describeTestOutcome</c>.
     /// </summary>
-    public static string Describe(string name, Exception? failure, TimeSpan elapsed, IReadOnlyList<StepRecord> steps, FailureEvidence? evidence, Func<string, string> redact, string projectRoot, RunPhase phase = RunPhase.Body, IReadOnlyList<Exception>? secondary = null)
+    public static string Describe(string name, Exception? failure, TimeSpan elapsed, IReadOnlyList<StepRecord> steps, FailureEvidence? evidence, Func<string, string> redact, string projectRoot, RunPhase phase = RunPhase.Body, IReadOnlyList<Exception>? secondary = null, string? expectedFailure = null)
     {
         ArgumentNullException.ThrowIfNull(steps);
         ArgumentNullException.ThrowIfNull(redact);
         ArgumentNullException.ThrowIfNull(projectRoot);
         var error = failure is null ? null : ErrorSerializer.Serialize(failure, projectRoot, redact);
-        var status = failure is null ? "ended without failing" : failure is E2EException { Code: "TEST_TIMEOUT" } ? "timed out" : "failed";
+
+        // A body that fails as its test is marked expected fails "as expected"; a hook's failure is not the marked failure, so it reads "failed".
+        var expected = expectedFailure is not null && phase == RunPhase.Body && error is { Code: not "UNEXPECTED_PASS" };
+        var status = failure is null ? "ended without failing" : failure is E2EException { Code: "TEST_TIMEOUT" } ? "timed out" : expected ? "failed as expected" : "failed";
         List<string> lines = ["Test \"" + name + "\" " + status + " after " + Format.FormatTime(elapsed) + "; " + MarkdownText.Plural(steps.Count, "step") + " ran."];
+        if (expectedFailure is not null && error?.Code != "TEST_TIMEOUT")
+        {
+            lines.Add("Expected to fail: " + expectedFailure);
+        }
+
         if (error is not null)
         {
             lines.Add(error.Code + PhaseText(phase) + ": " + error.Message);

src/E2E.NUnit/NUnitSessionTestRunner.cs

@@ -122,8 +122,25 @@ void Walk(ITest test)
     /// </summary>
     private static SessionTestOutcome? Outcome(SessionHandover.Request request, string name, ITestResult result, List<Exception> stray)
     {
+        var expected = ExpectedFailure(result, name);
         if (request.Outcome is { } handed)
         {
+            // A test marked as expected to fail that passes is a failure of its own (upstream's UNEXPECTED_PASS).
+            // A teardown that failed after a passing body is a failure of its own: the mark covers the body only, so it is not expected.
+            if (expected is not null && handed.Failure is null)
+            {
+                if (result.ResultState.Status == TestStatus.Passed)
+                {
+                    handed = handed with { Failure = new TestException("UNEXPECTED_PASS", "test is marked test.fail (\"" + expected + "\") but passed") };
+                }
+                else
+                {
+                    handed = handed with { Failure = new TestException("TEST_FAILED", string.IsNullOrWhiteSpace(result.Message) ? "The test failed." : result.Message) };
+                    expected = null;
+                }
+            }
+
+            handed = handed with { ExpectedFailure = expected };
             if (stray.Count == 0)
             {
                 return handed;
@@ -135,18 +152,29 @@ void Walk(ITest test)
         }
 
         return HookFailure(result, name) is { } message
-            ? new SessionTestOutcome(name, null, null, new TestException("HOOK_FAILED", "beforeAll failed: " + message), RunPhase.BeforeAll, [.. stray])
+            ? new SessionTestOutcome(name, null, null, new TestException("HOOK_FAILED", "beforeAll failed: " + message), RunPhase.BeforeAll, [.. stray], ExpectedFailure: expected)
             : null;
     }
 
+    /// <summary>
+    /// The reason the test <paramref name="name"/> is marked as expected to fail, or null. The mark is the NUnit property <see cref="ExpectedFailureProperty"/>:
+    /// <c>[Property("E2E.Fail", "reason")]</c> on the test, or <c>TestContext.CurrentContext.Test.Properties.Set</c> in its body (upstream's <c>fail</c> option
+    /// and <c>test.fail</c>). NUnit has no expected-failure outcome, so this reads the mark only; the run's own verdict is not inverted.
+    /// </summary>
+    private static string? ExpectedFailure(ITestResult root, string name) =>
+        FindResult(root, name)?.Test.Properties.Get(ExpectedFailureProperty)?.ToString();
+
+    /// <summary>The NUnit property a test sets to be marked as expected to fail (see <see cref="ExpectedFailure"/>).</summary>
+    public const string ExpectedFailureProperty = "E2E.Fail";
+
+    private static ITestResult? FindResult(ITestResult each, string name) => !each.Test.IsSuite && string.Equals(each.Test.FullName, name, StringComparison.Ordinal)
+        ? each
+        : each.Children.Select(child => FindResult(child, name)).FirstOrDefault(found => found is not null);
+
     /// <summary>The message of the <c>OneTimeSetUp</c> that failed the test <paramref name="name"/>, or null when its result says no such thing.</summary>
     private static string? HookFailure(ITestResult root, string name)
     {
-        ITestResult? Find(ITestResult each) => !each.Test.IsSuite && string.Equals(each.Test.FullName, name, StringComparison.Ordinal)
-            ? each
-            : each.Children.Select(Find).FirstOrDefault(found => found is not null);
-
-        var message = Find(root)?.Message;
+        var message = FindResult(root, name)?.Message;
         const string Prefix = "OneTimeSetUp: ";
         if (message is null || !message.StartsWith(Prefix, StringComparison.Ordinal))
         {