Skip to content

Add AI issue replication mode to MAUI Copilot - #37445

Draft
kubaflo wants to merge 151 commits into
dotnet:improved-reviewerfrom
kubaflo:copilot/replicate-issues-pipeline
Draft

Add AI issue replication mode to MAUI Copilot#37445
kubaflo wants to merge 151 commits into
dotnet:improved-reviewerfrom
kubaflo:copilot/replicate-issues-pipeline

Conversation

@kubaflo

@kubaflo kubaflo commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Important

This PR changes trusted CI automation only. Product behavior is validated through the guarded reproduction tests generated by successful pilot runs.

Summary

  • add a first-position Mode selector to maui-copilot, preserving review as the default and adding manual replicate runs
  • reconstruct public MAUI issues from sanitized issue text, inline snippets, and validated raster screenshots without downloading reporter repositories, archives, binaries, scripts, packages, or videos
  • build and deploy a bounded Sandbox scenario, execute a declarative Appium plan through a trusted interpreter, and record validated MP4/GIF/PNG evidence on Android, iOS, Mac Catalyst, or Windows
  • author the lightest guarded failing test in unit/XAML, device, then UI order; normal CI no-ops unless MAUI_REPRODUCTION_ISSUE matches the exact issue
  • validate the add-only patch, exact targeted failure, media, manifests, and trusted execution before publication
  • publish evidence to the existing public asset-only review-tests-assets-v2 branch and create a same-repository draft PR using the trusted checkout credential

No new credentials or services

Replication reuses infrastructure already present in maui-copilot:

  • the GitHub service-connection credential persisted only in the clean trusted publisher job
  • the existing public review-tests-assets-v2 branch with commit-pinned raw.githubusercontent.com URLs
  • existing Android/iOS/macOS/Windows pools, emulator/simulator setup, Copilot token, and pipeline artifacts

No replication-specific token, Azure Storage account/container/service connection, public base URL, or bot fork is required. Candidate validation completes before the checkout credential is extracted, generated code is never executed in the credentialed job, and GH_TOKEN is cleared in finally.

Security boundaries

  • issue content, generated code, Appium plans, logs, patches, and media are untrusted until independently validated
  • Copilot receives repository reads and exact-file writes only; it has no shell, URL, GitHub MCP, Azure, Git, build, test, device, recording, or publication capability
  • generated C#/XAML is capability-scanned before credentialless execution
  • Appium input is bounded JSON interpreted by checked-in trusted C#
  • generated tests are add-only, issue-numbered, and guarded before assertions or lifecycle work
  • the publisher starts from a clean checkout pinned to the validated baseline and independently rechecks staged files and evidence before any push

Validation

  • focused replication, publication, recorder, verifier, pipeline, telemetry, and review Pester suites pass; GitHub Pester (.github/scripts) is green
  • changed PowerShell passes error-level PSScriptAnalyzer
  • eng/pipelines/ci-copilot.yml parses and git diff --check passes
  • the trusted Appium interpreter compiles and reaches its expected missing-runtime-environment boundary
  • native Catalyst screencapture produced a valid H.264 MOV in a local smoke test
  • review-mode regression 14984949 succeeded
  • Skill Validation remains unrelatedly blocked because its trusted pull_request_target workflow refuses to check out fork code

Pilot hardening

  • iOS: pass the source checkout root explicitly to the trusted staged device-test runner
  • Android: support constrained native androidText locators and require them for controls with stable visible text
  • Mac Catalyst: replace the hanging AVFoundation ffmpeg capture with bounded native screencapture, followed by trusted ffmpeg normalization and validation
  • publication: reuse review-tests-assets-v2 and the trusted checkout credential instead of adding secrets or Azure Storage

Completed live rollout: 20 validated reproduction PRs

Every entry below is a draft PR with an add-only issue guard, an intended targeted assertion failure, and publicly reachable commit-pinned MP4/GIF evidence.

Platform Issue Run Draft PR
Mac Catalyst #35516 14985805 #37453
Android #10792 14985778 #37454
Android #37440 14985825 #37456
iOS #31059 14985980 #37457
Mac Catalyst #17210 14986056 #37458
Android #11740 14986058 #37459
Mac Catalyst #37229 14986173 #37460
Mac Catalyst #37187 14986257 #37461
Mac Catalyst #35511 14986306 #37462
iOS #37407 14986307 #37463
iOS #19064 14986304 #37464
iOS #37013 14986331 #37465
Mac Catalyst #35512 14986351 #37466
Android #37418 14986305 #37467
iOS #37264 14986391 #37468
Mac Catalyst #35667 14986423 #37469
Mac Catalyst #35624 14986444 #37470
Mac Catalyst #36230 14986524 #37471
Android #36834 14986530 #37472
iOS #36826 14986617 #37473

The autonomous rollout target is complete: 20/20 validated draft reproduction PRs across Android, iOS, and Mac Catalyst. Windows execution remains independently blocked by the hosted image's unpackaged Windows App SDK bootstrap COM failure 0x80040154; it was not needed to meet the validated rollout target.

Extend the MAUI Copilot pipeline with a guarded replicate mode that reconstructs public issues, records platform evidence, verifies a targeted failing test, and publishes a draft reproduction PR through isolated trusted jobs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
@kubaflo
kubaflo deployed to copilot-pat-pool August 15, 2026 16:18 — with GitHub Actions Active
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@kubaflo
kubaflo deployed to copilot-pat-pool August 15, 2026 16:18 — with GitHub Actions Active
@github-actions

github-actions Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Skill Validation Results

@kubaflo — new skill validation results are available based on this last commit: 924a44a.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

Overall Failed Static Failed LLM Needs attention Skills 0 Agents 0

Skill Validation Results924a44a · Add AI issue replication mode to MAUI Copilot · 2026-08-18T01:04:29Z

❌ Static Checks Failed

Skills: 0 | Eval specs linted:

⏭️ LLM Evaluation: Skipped

No changed skills with eval specs found.

🔍 Full results and investigation steps

@kubaflo
kubaflo deployed to copilot-pat-pool August 15, 2026 16:19 — with GitHub Actions Active
@kubaflo
kubaflo deployed to copilot-pat-pool August 15, 2026 16:20 — with GitHub Actions Active
Copilot CI added 3 commits August 15, 2026 18:20
Compile-time exclude the Azure evidence task for review runs so missing replication infrastructure cannot invalidate the existing review workflow.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Add a bounded portrait/landscape Appium plan action so device rotation scenarios can be reproduced without granting shell access to the agent.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Exclude the Azure publisher task when its service connection is not configured, then fail explicitly after candidate validation so reproduction artifacts remain testable without exposing credentials.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
@MauiBot

MauiBot commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Warning

🔍 Automated review could not complete

A stage of the reviewer pipeline could not finish, so no review summary was produced for this run — most often the PR's target branch failing to build. This is usually a transient infrastructure issue or a pre-existing break on the base branch, not a problem with your change.

Please re-comment /review to retry on a fresh agent.

🔍 Automated message from the .NET MAUI Copilot reviewer pipeline · build log

Copilot CI added 3 commits August 15, 2026 18:38
Accept GitHub's exact production user-asset S3 host while retaining strict initial URLs, bounded raster validation, and safe re-encoding.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Platform provisioning can modify tracked generated assets, so restore tracked files to the pinned main commit immediately before the trusted replication orchestrator validates the worktree.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Install Appium under the agent temp directory so trusted provisioning artifacts cannot contaminate replication git-state validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
@kubaflo

kubaflo commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Validated both automated failures against 5e6970d:

  • Skill Validation: the static checks never ran. actions/checkout@v4 refused to check out fork content from the trusted pull_request_target workflow (allow-unsafe-pr-checkout: false). This is a workflow-wide fork-handling regression; the same refusal occurs on other fork PRs. I ran the workflow’s pinned Vally lint command locally at this exact head and all 14 eval specs pass. Opting into unsafe checkout here would not fix this run anyway—the trusted workflow comes from the default branch and needs a separate security-reviewed workflow change.
  • MauiBot review: build 14984529 failed fast in Validate PR base branch because this stacked PR targets improved-reviewer, which the reviewer supports neither for workload selection nor merge-base derivation. Re-commenting /review would repeat the same failure; it should be retried after the PR is retargeted to a supported branch.

No PR code change is indicated by either failure.

Copilot CI and others added 10 commits August 15, 2026 19:19
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Require self-consistent XAML names and trimmed Appium values so bounded retries can repair empirical failures without introducing compile or schema errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Validate every executable test source, resolve exact verifier targets, bind signatures to targeted assertion messages, preserve host device identity, and make recorder/Appium retries reliable across Android, iOS, and Catalyst.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Copilot CI added 3 commits August 15, 2026 23:28
Publish replication evidence through the existing public asset branch and create same-repository draft PRs with the trusted checkout credential. Also harden Android text proof and Catalyst capture recovery.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Pass the source root to the trusted device runner, use native bounded Catalyst screen recording, and require Android native text locators for visible controls.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Keep Git command diagnostics out of the publisher function return stream so the manifest receives exactly one commit SHA.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Copilot CI and others added 30 commits August 17, 2026 21:38
Run 14994604 (dotnet#33037 ios) authored a HostApp page declaring Title, which
hides Page.Title. MAUI builds warnings as errors, so CS0108 failed the
build, but the verifier records a build break only as an infrastructure
failure with an empty actualFailureMessage. The agent therefore retried
without ever learning which member was wrong.

Recover distinct compiler diagnostics from the verification console and
quote them, and tell the agent up front that warning-level diagnostics
break this build and that hiding an inherited MAUI member is not allowed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
#177 was opened against the target repository's default
branch, which is not the commit the reproduction was validated on. The
resulting diff listed nine unrelated infrastructure files beside the one
added test, which undermines the evidence the PR exists to present.

Publish the validated baseline as a per-PR base branch and open the pull
request against it, so the diff is exactly the add-only patch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The previous commit pushed the validated baseline into the target
repository as a per-PR base branch, but MauiBot publishes from its own
fork and has no write access to kubaflo/maui, so that push would have
failed and blocked publication entirely.

Fetch the pull request base and build the reproduction branch on it
instead. The commit still contains only the validated add-only patch, so
the diff is exactly the added test, and no branch is created in the
target repository.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The locator fallback added in de71ccf used string.Contains(char).
The trusted runner is compiled from inside the repository tree, so the
repository analyzer rules apply and CA1307 is an error, which cost runs
14995313, 14995314, and 14995316 all five sandbox attempts each.

Use the StringComparison overload and assert the char overload never
returns. Verified by compiling the template in-tree, where the previous
form reports CA1307 and the new form builds clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The review of #173 identified three defects that generalize
beyond that reproduction:

- The page signalled progress by reassigning Border.AutomationId. MAUI
  allows AutomationId to be set only once, so the test could fail with
  InvalidOperationException for a reason unrelated to the reported crash.
  Forbid it in both agent contracts and reject it in the source guard.
- Assert.DoesNotThrow wrapped both the tap and the wait, collapsing
  managed exceptions, driver errors, timeouts, and process death into one
  indistinguishable failure, so it could not establish the reported
  crash. Forbid broad exception wrappers as the oracle.
- A Windows-only claim was not reliably scoped to Windows.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Reviews of #164 and #173 both judged the recording against a
standard it cannot meet: for a defect with no visible symptom the video
can only show the app-reported verdict. Say so in the body, and name the
trusted targeted test failure as the authoritative proof.

Also require the test to disclose when the reported defect is a static
invariant that already holds before the recorded action, so the
recording and the oracle describe the same causality (#170).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Reproduction runs for issue 36694 burned every Sandbox attempt because the
guard rejected the reported System.Runtime.InteropServices.COMException type
as native code, leaving process termination as the only available oracle even
though the Windows app stays open. Allow the fully qualified exception type,
keep every real interop form blocked, and tell the agent to prove a named
managed exception instead of assertAppClosed.

Test planning for issue 33037 also exhausted its attempts because the repo
already contains Issue33037.cs and the naming rule forces the issue number
into the file name. List the colliding paths in the plan prompt so the agent
chooses a distinct new file up front.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
A force-killed simctl capture produced an MP4 whose stream map matched no
streams, so issue 33037 on iOS lost a whole Sandbox regeneration attempt to a
recorder fault even though the generated app was correct. Send a second
interrupt before force-killing an iOS recorder, and classify decode and
recorder-exit faults as transient infrastructure so they retry without
consuming a semantic attempt.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Issue 35410 repeated x:FactoryMethod on consecutive Sandbox attempts because
the rejection listed only the allowed directives. Keep the directive blocked
and name the supported code-behind assignment in both the rejection and the
Sandbox prompt.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Issue 37427 lost three Sandbox attempts to compiler errors the agent never
saw, because the failure summary truncated to the build command banner before
reaching the diagnostics. Extract diagnostics from the prepare log too.

Issue 36851 requires an unpackaged unit-test host, so the packaged Sandbox can
never reproduce it. The agent reasoned this correctly but had no channel to
report it, so the run burned every attempt and failed hard. Accept a written
block declaration from attempt 3 onward and record it as an unsupported
scenario rather than a failure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Reviews of #179 and #180 validated both reproductions but raised a
provenance defect: the body named the validated baseline as the commit parent,
which stopped being true once reproduction commits moved onto the base branch
tip. Name both commits and their distinct roles.

The same reviews rejected the recording as exact-head evidence because its
on-screen text comes from the Sandbox app rather than the committed test, so
state that provenance directly. Also require per-control failure messages and
bounded eventual assertions for native state that settles after the trigger.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The publication stage already knows how to report a not-reproduced outcome on
the issue, but that path was unreachable: the replicate task rethrew after
writing the blocked candidate, so the stage was skipped and a correct
empirical answer surfaced as a red build. Issue 36694 on Windows exercised
this after the COMException guard fix let the agent use the faithful oracle
and the defect genuinely did not occur.

Finish successfully for sandbox_not_reproduced and unsupported_scenario, which
are conclusive answers rather than pipeline defects, and keep every
inconclusive or infrastructure code failing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The third independent validation verdict (PR 170, issue 36800) confirmed the
reproduction was sound but raised three quality gaps: the failure message
reported only a boolean verdict without the measured range, exact comparisons
over device metrics invite rounding flakiness, and the recorded drag was not
causally required by a scroll-offset-invariant assertion.

Require the numbers behind each verdict, a tolerance on device-derived floats,
and either a before/after transition assertion or no decorative interaction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
A locator timeout reported only the identifiers that were searched for, so
every retry re-guessed a name the app never had: issue 37429 on Android spent
all five attempts cycling through 'Group 1', '- Group', and androidText
variants of the same absent element, and 37335 failed the same way.

Capture the addressable attributes from the live page source when a wait times
out and surface that bounded inventory in the failure the agent reads, so the
next attempt selects a real element or assigns an explicit AutomationId.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Issue 37532 reproduced on device and recorded cleanly, then spent all five
authoring attempts on a device test that kept passing: the defect only appears
on the second Shell flyout visit, which a handler-level device test can never
perform. Repair is forbidden from changing testType, so every attempt repeated
the same passing result and the run ended red despite valid evidence.

Detect a test that ran correctly but did not fail, and allow one re-plan at a
tier that exercises the recorded path, clearing the abandoned files first. Also
teach the planner to pick the tier by what the recording required and to make
intermittent reports deterministic or declare them blocked.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Six independent validation verdicts confirmed the reproductions but flagged
the same three defects the prompts already forbade in general terms: PR 151
seeded its sentinel with the passing value 4, PR 152 asserted the literal
'07:30' without arranging fr-FR or 24-hour time, and PR 148 introduced an
explicit purple Style for an issue about the platform-default appearance.
Reviewers also noted that several recordings changed nothing on screen except
an app-authored status label.

Replace the abstract guidance with the concrete failure each reviewer observed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Reviewers of PRs 150 and 162 recorded that the claim these tests ran on Android
hardware was inaccurate: both are host unit tests, but the body listed only the
platform, so the platform line and the device recording together implied the
committed test executed on the device.

Name the host explicitly and, for host-executed tests, say that the recording
is on-device evidence of the reported issue rather than of the test running.
Also cover the existing .MacCatalyst.cs rejection from the authoring path,
which PR 156 predates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The manifest filter is an issue-keyed class token by design, and reviewers of
PRs 150, 158, and 163 all found it does not select the test: the runners
needed the fully qualified name or an XHarness display-name selector, so the
documented filter could not be used to reproduce the result.

Carry the resolved class and method from the verifier metadata into the
manifest and name the exact test in the body, falling back to the filter when
a manifest predates the fields.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Build 14997672 posted a MAUI_COPILOT_NOT_REPRODUCED comment on the public
dotnet#36694 and applied s/try-latest-version, which made the policy
service ping the reporter twice, while the replicate flow was still being
hardened.

Gate the outcome step behind a new PublishIssueOutcome parameter that
defaults to false, and drop the dotnet/maui default from -Repository so an
omission fails instead of silently writing to the public tracker.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Builds 14997683 and 14997708 spent all five Windows attempts on
NoSuchWindowException. The app under test had closed mid-scenario, so
every later lookup failed against a dead window and the agent kept
re-guessing locators. For 36298 that termination was the reported defect.

Report a lost window or exited process as REPLICATION_APP_TERMINATED,
naming it as the possible reproduction, and surface it past the failure
summary truncation.

Build 14997687 alternated between a CS1503 build break and a
not-reproduced run because the summary was overwritten each attempt and
repeats were compared only against the immediately previous attempt.
Keep a signature-keyed history of every distinct failure and show it, so
a revision cannot fix the newest failure by reintroducing an earlier one.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
PR 155 reported a preview that stopped before the defect appeared. The
preview trimmed the recording to its first seconds, which usually shows
only the app launching, while the reported defect is demonstrated by the
final step of the reproduction.

Compress the whole recording into the preview budget instead of
truncating it, so the preview always ends where the defect is shown.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Build 14997689 declared verified regression 37418 non-reproducible and
publicly told the reporter to try the latest version. Its attempts had
alternated between a CS0246 build break and a run that observed no
defect, so the pipeline never actually answered the question; the
classification looked only at the final attempt.

Record why each attempt failed and treat a non-reproduction as conclusive
only when every attempt reached the device and observed no defect.
Anything else is inconclusive, which reports a pipeline defect instead of
telling a reporter their verified bug does not exist.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Build 14999054 crashed on its first failed attempt because an ordered
dictionary exposes Contains rather than ContainsKey. The regression test
asserted only that the source contained the call, so it never executed it.

Move the lookup behind a function and cover it by running it against a
real ordered dictionary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The review of #180 rejected the submitted media partly
because it described simulator-shaped evidence as on-device. Every
replication pool runs an emulator, a simulator, or the build host, so
that claim was never accurate.

Report the actual execution surface and the recorded device identifier
throughout the pull request body.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
The review of #180 rejected the submitted clip partly because
it starts after the focus and tap, so the recording never shows the state
the interaction changed. A quarter-second lead-in is not enough to read
the starting state.

Keep 1.5 seconds before the first Appium action.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Every reviewer verdict so far has flagged that a reproduction proved by a
single execution is not evidence of a deterministic defect. #181
put it plainly: 'one attempt, one simulator geometry ... the result can
depend on caret landing position/device width'.

Run the verified test twice and require both executions to fail at the same
assertion. The trusted validator now rejects a candidate that completed
fewer runs than it requested, that reports inconsistent runs, or that is
missing a console log for any completed run, and the pull request body
states how many independent executions failed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Build 14999466 spent five attempts on 37263, twice watched the Sandbox
report no defect, and still failed the run red because one attempt never
rendered its result element. Demanding that every attempt observe no defect
means an unreproducible issue can never be concluded.

Accept a non-reproduction when at least two attempts observed no defect and
nothing was lost to a build break or to the app dying, and classify a bare
locator timeout as an attempt that observed nothing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Build 14999468 abandoned iOS issue 37089 as an unsupported scenario: the
reported SwipeView trigger holds one finger down while moving right, out of
the CollectionView row, and back before release, and the plan could express
only single cardinal swipes.

Add a dragPath action that presses a located element and moves one pointer
through two to four bounded screen-fraction segments before releasing. The
trusted plan reader rejects a malformed or oversized path and confines the
action to Android and iOS.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Commit 37885e8 taught the pull request body to name the class and method,
but never carried them into the validated document. The real document is a
PSCustomObject deserialised from JSON, so reading a property it does not have
throws under StrictMode: build 14999470 reproduced iOS issue 37013, verified
the failing test, and then lost the whole publication to it. Every hashtable
fixture hid the fault because a missing key is silently null.

Emit both names from the validator, read them defensively in the publisher,
and cover the deserialised shape under the publisher's own strict mode.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Issue 36298 reports an unhandled ArgumentException. Every recorded
attempt did terminate the app with exactly that exception, yet every
generated plan still finished by asserting text against a window that
no longer existed, so the reproduction read as automation flakiness
and the run went red without a candidate.

When the app terminates on a platform that can prove closure and the
issue text reports a crash, the reproduction is the termination. Say so
plainly to the agent and reject any later plan that ends with anything
other than the trusted app-closure assertion.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-ai-agents Copilot CLI agents, agent skills, AI-assisted development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants