ci: the stuck-job watchdog reports success while it cannot read runner inventory

[wallstop]

Problem

The stuck-job watchdog has been reporting success while unable to do the one thing it exists for. It cannot list runners at all, logs an error, takes no action, and exits 0.

Evidence

Run 30644904581, job Audit queued runs and recover dispatcher-stuck ones, conclusion success:

Queued runs older than 300s: 2
gh: Resource not accessible by integration (HTTP 403)
WARN: org-level runner list unavailable (likely 403). Falling back to repo runners.
gh: Resource not accessible by integration (HTTP 403)
ERROR: repo-level runner list also failed; cannot evaluate idle runners. No action issued.

Two runs were queued past the threshold and neither was evaluated.

What it failed to catch

Performance Numbers run 30613031270 (b09a854f) has been queued since 2026-07-31T07:30Z, with both benchmark legs waiting on [self-hosted, Windows, RAM-64GB, fast]. Behind it, the run for 6a70d1f8 has sat pending since 09:27Z, blocked by concurrency: Performance Numbers-refs/heads/master with cancel-in-progress: false.

So the published dispatch-throughput numbers have not refreshed since 2e58122c at 01:38Z, and the automation meant to notice was inert the whole time. Self-hosted runner registration preflight passed for that run at 07:30, which only proves a runner with those labels was registered, not online and idle.

Root cause

Two independent causes, both real.

1. The watchdog cannot read runner inventory. It runs with GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} and permissions: actions: write, contents: write. Both runner endpoints it tries need more than that:

• GET /orgs/{org}/actions/runners is org-scoped. GITHUB_TOKEN is repository-scoped and can never reach it.
• GET /repos/{owner}/{repo}/actions/runners needs administration: read, which is not in the permissions: block.

Adding administration: read alone does not fix it: this repository has zero repository-level runners (GET /repos/.../actions/runners returns total_count: 0); every runner is registered at the organization in group "Default". The repo-level fallback would succeed and report no idle runners, which reads identically to "no runner matches" and still issues no action.

The credential that can answer is the one check-unity-runner-availability already uses: secrets.BUILD_LOCK_READER_APP_ID / secrets.BUILD_LOCK_READER_APP_PRIVATE_KEY. Whether that App carries organization_self_hosted_runners: read needs checking before wiring it in.

2. The failure is silent. ERROR: repo-level runner list also failed; cannot evaluate idle runners. No action issued. is written to the step summary and the job then exits 0. Every other fail-closed gate in this repository treats "the check could not run" as a failure, not as a pass -- see the changes-detector gates in ci.yml and the ENOENT/missing-git rule in .llm/context.md. A recovery automation that cannot function is exactly the case that must be loud.

Proposed fix

1. Mint a token for the runner listing from the reader App (actions/create-github-app-token@v3 with app-id + private-key, the pattern already used elsewhere in this repository), and confirm it can read organization runners. Keep GITHUB_TOKEN for the cancel/redispatch calls so the write scope stays where it is.
2. Make the inventory failure fail the job. If the noise of a five-minute schedule failing is a concern, the schedule can back off, but a green run must mean the audit happened.
3. Add a contract test asserting the watchdog cannot conclude success on a path that issued no action because inventory was unavailable.

Also worth deciding

Neither cause explains why ELI-MACHINE (the only runner carrying fast) has been unavailable to accept the benchmark legs for eight hours. That is an operations question, but the watchdog is the thing that was supposed to surface it.

Acceptance criteria

• A watchdog run that cannot enumerate runners does not report success.
• A run queued past the threshold whose label set no online runner satisfies is reported distinctly from a dispatcher-stuck run, and both appear in the step summary.
• The Performance Numbers queue on master drains, or the reason it cannot is visible without reading a job log.

Related

• ci: a superseded push keeps the Unity concurrency group, so the current SHA cannot start #311 -- superseded Unity runs holding a concurrency group; the same pending-behind-queued shape, different cause.

[wallstop]

Fixed in 66feaae (#331).

The watchdog now fails closed on every input its verdict depends on — the queued-run listing, the organization runner inventory, per-run job listings, and the cancel-cap state branch. Previously an unreadable inventory produced a green run, which is how a Performance Numbers run sat queued for ten hours while the automation meant to notice it reported success every five minutes.

Runner inventory is organization-scoped and is now read with the build-lock reader App token (organization_self_hosted_runners: read); the job's own GITHUB_TOKEN is repository-scoped and could never reach /orgs/{org}/actions/runner-groups. The repository-level fallback the workflow used to attempt was worse than none: this repository registers zero repository-level runners, so the call succeeded with an empty inventory that read identically to "no runner matches".

Starvation is also now reported from every branch that can observe it — including the busy and dispatcher-stuck ones — since a busy or cancellable sibling does not make a starved one less starved.

Verified by executing the workflow's own run: block against stubbed gh/git across ~41 truth-table rows, with each behaviour pinned by mutation: ignoring an inventory-read failure, tolerating an empty group set, and swapping the reader token for GITHUB_TOKEN are all caught.