Fix CI fix agent's blind context pre-fetch

[ytdb-ci-bot]

Motivation:

The autonomous CI fix agent has been starting every run with no failure data at all, and nothing in its logs said so. Its job runs on a self-hosted Hetzner image that does not ship the GitHub CLI, and the workflow installs only jq, so all six gh calls in the "Gather CI failure context" step exited 127. Each call was written as gh ... 2>/dev/null || echo '<fallback>' > file, so the agent received run.json as {}, jobs.json with an empty job list, logs.txt saying no logs were available, and four empty arrays — indistinguishable, from inside the agent, from a run that genuinely had nothing to report.

This was the state on run 30921482302, where the pre-fetch produced exactly those seven placeholder files. The agent had a 6-hour budget and no visible failure to investigate.

The same commit that introduced the pre-fetch (#1247, retiring the CI personal access tokens) also moved PR creation into a gh pr create in the "Publish fix" step. Before that, all gh usage sat inside the agent step, where the agent held a token and could install whatever it needed on the fly — which is why the missing CLI went unnoticed for so long. Once the calls moved into fixed shell steps, nobody was left to self-heal them, and the publish path would push a fix branch and then fail to open a pull request, leaving an orphan branch behind.

Planned changes:

Current state. gh is absent on the runner and every fetch failure is silently swallowed. The agent cannot tell a missing datum from an empty one, because [] is both the fallback and the correct value for a develop-push failure, a job with no annotations, and a repo with no open fix PR.

What changes. gh is installed and verified alongside jq, so both consumers work. Fetch failures stay non-fatal but are no longer silent: each is surfaced as a warning annotation carrying the command's real exit code and stderr, and each datum's outcome is recorded in a new fetch-status.json in the context directory. Missing run metadata — the one datum the agent cannot work without — aborts the job before the agent starts rather than handing it an empty object. The publish path checks gh and the App token before the push instead of after.

How. The status file is the design centre. Placeholders are ambiguous by construction, so the outcome is recorded as a fact rather than left to be inferred from file contents: ok means the value is real even when empty, partial: means the entries present are real but incomplete, and anything else means treat the file as absent. Derived datums seed their status from their parent, so a datum whose loop never ran is reported as not attempted rather than as ok. The agent prompt keys its degraded path off that file, bounds the fallback to one local reproduce run, and terminates in a new "Insufficient failure context" classification.

Key decisions. Installing the distro gh package was chosen over rewriting the six calls as curl against the REST API: the calls are the correct ones and well commented, and rewriting them would mean re-deriving pagination, redirect handling for logs, and the camelCase field names the prompt documents — new bugs for no gain, when the actual defect is a missing prerequisite. Ubuntu 24.04 ships gh 2.45.0, which supports every flag used here, and the file already installs jq the same way. The stale-summary problem was resolved by moving the agent summary into the per-job temp directory rather than by clearing a fixed /tmp path, so the runner's own per-job wipe provides the guarantee.

Out of scope. Two pre-existing exposures surfaced during review and are recorded rather than fixed: the remaining PID-named /tmp files in this workflow are pre-plantable as symlinks, and the agent can write $GITHUB_PATH to influence binaries that later steps invoke. Closing the second properly means moving "Publish fix" into its own job so it does not share a process environment with the agent — a larger change than this fix should carry. The CI_FIX_RESULT_DIR half of that exposure is closed here by pinning the path in the consuming steps' env:.

Risks & accepted trade-offs. Making missing run metadata fatal converts a silent, expensive failure into a loud, immediate one; three retries with backoff keep a transient 5xx from tripping it. gh is installed unpinned from the distro archive, which fails safe: a version too old for run view --json makes every fetch fail, and the new gate then aborts before the agent starts.

Verification approach. The premise was confirmed on the runner image itself, and the new shell was exercised against a stubbed gh across eight scenarios rather than reasoned about.

Tracks:

N/A (single-track)

Root Cause

.github/workflows/ci-failure-fix-agent.yml runs entirely on [self-hosted, type-cpx42, image-x86-system-ubuntu-24.04]. That image has jq only because the workflow's "Install jq" step put it there; gh was never installed and is not in the image. Verified directly on a runner of that type: command -v gh fails, while apt-cache policy gh shows 2.45.0-1ubuntu0.3 available but Installed: (none).

Every gh invocation therefore exited 127 with command not found. In the gather step each call redirected stderr to /dev/null and fell back to a placeholder on failure, so the exit code never reached the log and the agent silently received a complete set of empty files. In the publish step gh pr create would fail the same way, but only after git push had already succeeded.

Changes

All in .github/workflows/ci-failure-fix-agent.yml.

• Install and verify the prerequisites. "Install jq" becomes "Install CLI prerequisites (jq, gh)", installing whichever is missing and failing with a named error if either is still absent afterwards. Both apt-get calls are || true so an apt failure reaches that error instead of aborting with only apt's output.
• Surface fetch failures. A fetch helper captures stderr, emits a ::warning:: with the real exit code, writes the fallback, and records the outcome. It always returns success, because the step runs under bash -e and a bare non-zero return would abort the pre-fetch and discard everything already gathered.
• Record outcomes in fetch-status.json. Per-datum ok / partial: / failed: / not attempted:, built from a TSV accumulator so a successful retry overrides its own earlier failure. Derived datums (annotations, pr-comments) seed from their parent, and both are recorded after their merge so a merge failure cannot leave ok over an empty file.
• Fail fast on missing run metadata, after three retries, with a context digest and a warning when a failed run reports zero failed jobs.
• Harden the inputs. The run id is extracted as the segment after the last /runs/ with query and fragment stripped first, then validated as all-digits; a pasted job URL previously yielded the numeric job id and fetched the wrong run. Captured stderr is stripped of CR/TAB, folded to one line, and %-escaped on the annotation path only.
• Fix annotation data loss. gh api writes its error body to stdout as well, so appending unconditionally mixed an object into the array stream; jq -s add then exited 5 and the fallback replaced the file with [], discarding every successful job's annotations. Both loops now append only on success and pass --paginate, since the annotation naming the failing test can sit past the 30-entry first page.
• Harden the publish path. gh and the App token are checked before the push; branch and existing_pr come from the already-validated step outputs rather than a second raw read of the sentinel; CI_FIX_RESULT_DIR / CI_FIX_RESULT_FILE are pinned in the consuming steps' env:, and both sentinel readers now resolve the same file.
• Close the tokenless bracket structurally. Installing gh removed an accidental second layer of it — an absent GH_TOKEN alone does not stop gh, which falls through to hosts.yml and the keyring — so the agent step points GH_CONFIG_DIR at an empty directory. GH_DEBUG is pinned empty at job level, covering the token-bearing publish step too.
• Update the agent prompt. The three-state semantics, the status key for every context file, a degraded path scoped to the failure-detail datums and bounded to one reproduce run, and a new "Insufficient failure context" classification terminating in a no-fix sentinel. The completeness claims that promised the context is always whole now point at the status file.

Test plan

• Root cause reproduced on the runner image: command -v gh fails there, and apt-get install -y -qq gh installs 2.45.0 cleanly.
• Flag support verified for every command used — run view --json/--log-failed, api, pr list --json, pr create --body-file — from the shipped man pages, and the requested JSON field names confirmed present in the binary.
• YAML parses; bash -n clean on all 13 run: blocks.
• Gather step exercised against a stubbed gh across eight scenarios: total failure (aborts with exit 1 and names 127), partial degradation, parent-failure propagation, partial per-job failure preserving good data, all-per-job-failure downgrading to failed:, malformed-JSON merge failure, a pasted job URL, and hostile stderr attempting to forge ::error:: / ::stop-commands::.
• Run-id extraction checked across six URL shapes including ?redirect=/runs/999.
• No temp files left behind on any path, including the new abort.
• Dimensional review completed: three iterations across code quality, security, hook/script safety, and prompt design. Two defects in the fix itself were found by the harness and one by review — a wrong exit code from reading $? after a failed if, a numeric job id passing run-id validation, and derived statuses reporting ok for loops that never ran.
• Unit suite run on develop with this change applied — see the note below.

No Java changed. The diff is one GitHub Actions workflow file, so the coverage gate has no changed lines to measure and Spotless (Java-only) does not apply.

[github-actions]

Test Count Gate Results

✅ No baseline available yet — gate skipped (first run).

[github-actions]

Coverage Gate Results

Thresholds: 85% line, 70% branch

✅ No changed Java files — coverage gate skipped.