Add AI issue replication mode to MAUI Copilot - #37445
Conversation
Extend the MAUI Copilot pipeline with a guarded replicate mode that reconstructs public issues, records platform evidence, verifies a targeted failing test, and publishes a draft reproduction PR through isolated trusted jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
|
Azure Pipelines: 1 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Skill Validation Results
❌ Skill Validation Results —
|
Compile-time exclude the Azure evidence task for review runs so missing replication infrastructure cannot invalidate the existing review workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Add a bounded portrait/landscape Appium plan action so device rotation scenarios can be reproduced without granting shell access to the agent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Exclude the Azure publisher task when its service connection is not configured, then fail explicitly after candidate validation so reproduction artifacts remain testable without exposing credentials. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
|
Warning 🔍 Automated review could not completeA stage of the reviewer pipeline could not finish, so no review summary was produced for this run — most often the PR's target branch failing to build. This is usually a transient infrastructure issue or a pre-existing break on the base branch, not a problem with your change. Please re-comment 🔍 Automated message from the .NET MAUI Copilot reviewer pipeline · build log |
Accept GitHub's exact production user-asset S3 host while retaining strict initial URLs, bounded raster validation, and safe re-encoding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Platform provisioning can modify tracked generated assets, so restore tracked files to the pinned main commit immediately before the trusted replication orchestrator validates the worktree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Install Appium under the agent temp directory so trusted provisioning artifacts cannot contaminate replication git-state validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
|
Validated both automated failures against
No PR code change is indicated by either failure. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Require self-consistent XAML names and trimmed Appium values so bounded retries can repair empirical failures without introducing compile or schema errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Validate every executable test source, resolve exact verifier targets, bind signatures to targeted assertion messages, preserve host device identity, and make recorder/Appium retries reliable across Android, iOS, and Catalyst. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Publish replication evidence through the existing public asset branch and create same-repository draft PRs with the trusted checkout credential. Also harden Android text proof and Catalyst capture recovery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Pass the source root to the trusted device runner, use native bounded Catalyst screen recording, and require Android native text locators for visible controls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Keep Git command diagnostics out of the publisher function return stream so the manifest receives exactly one commit SHA. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Run 14994604 (dotnet#33037 ios) authored a HostApp page declaring Title, which hides Page.Title. MAUI builds warnings as errors, so CS0108 failed the build, but the verifier records a build break only as an infrastructure failure with an empty actualFailureMessage. The agent therefore retried without ever learning which member was wrong. Recover distinct compiler diagnostics from the verification console and quote them, and tell the agent up front that warning-level diagnostics break this build and that hiding an inherited MAUI member is not allowed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
#177 was opened against the target repository's default branch, which is not the commit the reproduction was validated on. The resulting diff listed nine unrelated infrastructure files beside the one added test, which undermines the evidence the PR exists to present. Publish the validated baseline as a per-PR base branch and open the pull request against it, so the diff is exactly the add-only patch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The previous commit pushed the validated baseline into the target repository as a per-PR base branch, but MauiBot publishes from its own fork and has no write access to kubaflo/maui, so that push would have failed and blocked publication entirely. Fetch the pull request base and build the reproduction branch on it instead. The commit still contains only the validated add-only patch, so the diff is exactly the added test, and no branch is created in the target repository. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The locator fallback added in de71ccf used string.Contains(char). The trusted runner is compiled from inside the repository tree, so the repository analyzer rules apply and CA1307 is an error, which cost runs 14995313, 14995314, and 14995316 all five sandbox attempts each. Use the StringComparison overload and assert the char overload never returns. Verified by compiling the template in-tree, where the previous form reports CA1307 and the new form builds clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The review of #173 identified three defects that generalize beyond that reproduction: - The page signalled progress by reassigning Border.AutomationId. MAUI allows AutomationId to be set only once, so the test could fail with InvalidOperationException for a reason unrelated to the reported crash. Forbid it in both agent contracts and reject it in the source guard. - Assert.DoesNotThrow wrapped both the tap and the wait, collapsing managed exceptions, driver errors, timeouts, and process death into one indistinguishable failure, so it could not establish the reported crash. Forbid broad exception wrappers as the oracle. - A Windows-only claim was not reliably scoped to Windows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Reviews of #164 and #173 both judged the recording against a standard it cannot meet: for a defect with no visible symptom the video can only show the app-reported verdict. Say so in the body, and name the trusted targeted test failure as the authoritative proof. Also require the test to disclose when the reported defect is a static invariant that already holds before the recorded action, so the recording and the oracle describe the same causality (#170). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Reproduction runs for issue 36694 burned every Sandbox attempt because the guard rejected the reported System.Runtime.InteropServices.COMException type as native code, leaving process termination as the only available oracle even though the Windows app stays open. Allow the fully qualified exception type, keep every real interop form blocked, and tell the agent to prove a named managed exception instead of assertAppClosed. Test planning for issue 33037 also exhausted its attempts because the repo already contains Issue33037.cs and the naming rule forces the issue number into the file name. List the colliding paths in the plan prompt so the agent chooses a distinct new file up front. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
A force-killed simctl capture produced an MP4 whose stream map matched no streams, so issue 33037 on iOS lost a whole Sandbox regeneration attempt to a recorder fault even though the generated app was correct. Send a second interrupt before force-killing an iOS recorder, and classify decode and recorder-exit faults as transient infrastructure so they retry without consuming a semantic attempt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Issue 35410 repeated x:FactoryMethod on consecutive Sandbox attempts because the rejection listed only the allowed directives. Keep the directive blocked and name the supported code-behind assignment in both the rejection and the Sandbox prompt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Issue 37427 lost three Sandbox attempts to compiler errors the agent never saw, because the failure summary truncated to the build command banner before reaching the diagnostics. Extract diagnostics from the prepare log too. Issue 36851 requires an unpackaged unit-test host, so the packaged Sandbox can never reproduce it. The agent reasoned this correctly but had no channel to report it, so the run burned every attempt and failed hard. Accept a written block declaration from attempt 3 onward and record it as an unsupported scenario rather than a failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Reviews of #179 and #180 validated both reproductions but raised a provenance defect: the body named the validated baseline as the commit parent, which stopped being true once reproduction commits moved onto the base branch tip. Name both commits and their distinct roles. The same reviews rejected the recording as exact-head evidence because its on-screen text comes from the Sandbox app rather than the committed test, so state that provenance directly. Also require per-control failure messages and bounded eventual assertions for native state that settles after the trigger. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The publication stage already knows how to report a not-reproduced outcome on the issue, but that path was unreachable: the replicate task rethrew after writing the blocked candidate, so the stage was skipped and a correct empirical answer surfaced as a red build. Issue 36694 on Windows exercised this after the COMException guard fix let the agent use the faithful oracle and the defect genuinely did not occur. Finish successfully for sandbox_not_reproduced and unsupported_scenario, which are conclusive answers rather than pipeline defects, and keep every inconclusive or infrastructure code failing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The third independent validation verdict (PR 170, issue 36800) confirmed the reproduction was sound but raised three quality gaps: the failure message reported only a boolean verdict without the measured range, exact comparisons over device metrics invite rounding flakiness, and the recorded drag was not causally required by a scroll-offset-invariant assertion. Require the numbers behind each verdict, a tolerance on device-derived floats, and either a before/after transition assertion or no decorative interaction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
A locator timeout reported only the identifiers that were searched for, so every retry re-guessed a name the app never had: issue 37429 on Android spent all five attempts cycling through 'Group 1', '- Group', and androidText variants of the same absent element, and 37335 failed the same way. Capture the addressable attributes from the live page source when a wait times out and surface that bounded inventory in the failure the agent reads, so the next attempt selects a real element or assigns an explicit AutomationId. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Issue 37532 reproduced on device and recorded cleanly, then spent all five authoring attempts on a device test that kept passing: the defect only appears on the second Shell flyout visit, which a handler-level device test can never perform. Repair is forbidden from changing testType, so every attempt repeated the same passing result and the run ended red despite valid evidence. Detect a test that ran correctly but did not fail, and allow one re-plan at a tier that exercises the recorded path, clearing the abandoned files first. Also teach the planner to pick the tier by what the recording required and to make intermittent reports deterministic or declare them blocked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Six independent validation verdicts confirmed the reproductions but flagged the same three defects the prompts already forbade in general terms: PR 151 seeded its sentinel with the passing value 4, PR 152 asserted the literal '07:30' without arranging fr-FR or 24-hour time, and PR 148 introduced an explicit purple Style for an issue about the platform-default appearance. Reviewers also noted that several recordings changed nothing on screen except an app-authored status label. Replace the abstract guidance with the concrete failure each reviewer observed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Reviewers of PRs 150 and 162 recorded that the claim these tests ran on Android hardware was inaccurate: both are host unit tests, but the body listed only the platform, so the platform line and the device recording together implied the committed test executed on the device. Name the host explicitly and, for host-executed tests, say that the recording is on-device evidence of the reported issue rather than of the test running. Also cover the existing .MacCatalyst.cs rejection from the authoring path, which PR 156 predates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The manifest filter is an issue-keyed class token by design, and reviewers of PRs 150, 158, and 163 all found it does not select the test: the runners needed the fully qualified name or an XHarness display-name selector, so the documented filter could not be used to reproduce the result. Carry the resolved class and method from the verifier metadata into the manifest and name the exact test in the body, falling back to the filter when a manifest predates the fields. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Build 14997672 posted a MAUI_COPILOT_NOT_REPRODUCED comment on the public dotnet#36694 and applied s/try-latest-version, which made the policy service ping the reporter twice, while the replicate flow was still being hardened. Gate the outcome step behind a new PublishIssueOutcome parameter that defaults to false, and drop the dotnet/maui default from -Repository so an omission fails instead of silently writing to the public tracker. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Builds 14997683 and 14997708 spent all five Windows attempts on NoSuchWindowException. The app under test had closed mid-scenario, so every later lookup failed against a dead window and the agent kept re-guessing locators. For 36298 that termination was the reported defect. Report a lost window or exited process as REPLICATION_APP_TERMINATED, naming it as the possible reproduction, and surface it past the failure summary truncation. Build 14997687 alternated between a CS1503 build break and a not-reproduced run because the summary was overwritten each attempt and repeats were compared only against the immediately previous attempt. Keep a signature-keyed history of every distinct failure and show it, so a revision cannot fix the newest failure by reintroducing an earlier one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
PR 155 reported a preview that stopped before the defect appeared. The preview trimmed the recording to its first seconds, which usually shows only the app launching, while the reported defect is demonstrated by the final step of the reproduction. Compress the whole recording into the preview budget instead of truncating it, so the preview always ends where the defect is shown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Build 14997689 declared verified regression 37418 non-reproducible and publicly told the reporter to try the latest version. Its attempts had alternated between a CS0246 build break and a run that observed no defect, so the pipeline never actually answered the question; the classification looked only at the final attempt. Record why each attempt failed and treat a non-reproduction as conclusive only when every attempt reached the device and observed no defect. Anything else is inconclusive, which reports a pipeline defect instead of telling a reporter their verified bug does not exist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Build 14999054 crashed on its first failed attempt because an ordered dictionary exposes Contains rather than ContainsKey. The regression test asserted only that the source contained the call, so it never executed it. Move the lookup behind a function and cover it by running it against a real ordered dictionary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The review of #180 rejected the submitted media partly because it described simulator-shaped evidence as on-device. Every replication pool runs an emulator, a simulator, or the build host, so that claim was never accurate. Report the actual execution surface and the recorded device identifier throughout the pull request body. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The review of #180 rejected the submitted clip partly because it starts after the focus and tap, so the recording never shows the state the interaction changed. A quarter-second lead-in is not enough to read the starting state. Keep 1.5 seconds before the first Appium action. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Every reviewer verdict so far has flagged that a reproduction proved by a single execution is not evidence of a deterministic defect. #181 put it plainly: 'one attempt, one simulator geometry ... the result can depend on caret landing position/device width'. Run the verified test twice and require both executions to fail at the same assertion. The trusted validator now rejects a candidate that completed fewer runs than it requested, that reports inconsistent runs, or that is missing a console log for any completed run, and the pull request body states how many independent executions failed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Build 14999466 spent five attempts on 37263, twice watched the Sandbox report no defect, and still failed the run red because one attempt never rendered its result element. Demanding that every attempt observe no defect means an unreproducible issue can never be concluded. Accept a non-reproduction when at least two attempts observed no defect and nothing was lost to a build break or to the app dying, and classify a bare locator timeout as an attempt that observed nothing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Build 14999468 abandoned iOS issue 37089 as an unsupported scenario: the reported SwipeView trigger holds one finger down while moving right, out of the CollectionView row, and back before release, and the plan could express only single cardinal swipes. Add a dragPath action that presses a located element and moves one pointer through two to four bounded screen-fraction segments before releasing. The trusted plan reader rejects a malformed or oversized path and confines the action to Android and iOS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Commit 37885e8 taught the pull request body to name the class and method, but never carried them into the validated document. The real document is a PSCustomObject deserialised from JSON, so reading a property it does not have throws under StrictMode: build 14999470 reproduced iOS issue 37013, verified the failing test, and then lost the whole publication to it. Every hashtable fixture hid the fault because a missing key is silently null. Emit both names from the validator, read them defensively in the publisher, and cover the deserialised shape under the publisher's own strict mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Issue 36298 reports an unhandled ArgumentException. Every recorded attempt did terminate the app with exactly that exception, yet every generated plan still finished by asserting text against a window that no longer existed, so the reproduction read as automation flakiness and the run went red without a candidate. When the app terminates on a platform that can prove closure and the issue text reports a crash, the reproduction is the termination. Say so plainly to the agent and reject any later plan that ends with anything other than the trusted app-closure assertion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Important
This PR changes trusted CI automation only. Product behavior is validated through the guarded reproduction tests generated by successful pilot runs.
Summary
Modeselector tomaui-copilot, preservingreviewas the default and adding manualreplicaterunsMAUI_REPRODUCTION_ISSUEmatches the exact issuereview-tests-assets-v2branch and create a same-repository draft PR using the trusted checkout credentialNo new credentials or services
Replication reuses infrastructure already present in
maui-copilot:review-tests-assets-v2branch with commit-pinnedraw.githubusercontent.comURLsNo replication-specific token, Azure Storage account/container/service connection, public base URL, or bot fork is required. Candidate validation completes before the checkout credential is extracted, generated code is never executed in the credentialed job, and
GH_TOKENis cleared infinally.Security boundaries
Validation
Pester (.github/scripts)is greeneng/pipelines/ci-copilot.ymlparses andgit diff --checkpassesscreencaptureproduced a valid H.264 MOV in a local smoke testpull_request_targetworkflow refuses to check out fork codePilot hardening
androidTextlocators and require them for controls with stable visible textscreencapture, followed by trusted ffmpeg normalization and validationreview-tests-assets-v2and the trusted checkout credential instead of adding secrets or Azure StorageCompleted live rollout: 20 validated reproduction PRs
Every entry below is a draft PR with an add-only issue guard, an intended targeted assertion failure, and publicly reachable commit-pinned MP4/GIF evidence.
The autonomous rollout target is complete: 20/20 validated draft reproduction PRs across Android, iOS, and Mac Catalyst. Windows execution remains independently blocked by the hosted image's unpackaged Windows App SDK bootstrap COM failure
0x80040154; it was not needed to meet the validated rollout target.