Fix/geak revalidation dispatch - #1221
Merged
Merged
Conversation
Spare geak_fallback rebench tasks from phase and explore-family cancellation, scope revalidation idempotency keys by macro_cycle, and add regression tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Do not spare GEAK rebench tasks on CLOSE transitions, cancel them at close entry, block duplicate enqueues while one is in flight, and gate 2b promote on geak_pending task tracking. Parameterize phase-boundary spares via spare_queued, move policy helpers to geak_rebench, revert cancel_family exemption, and extend regression tests plus CHANGELOG. Co-authored-by: Cursor <cursoragent@cursor.com>
Treat empty geak_pending as untracked-by-design for promote and failure paths, apply the orphan gate to no_material and no_promote, and finalize geak_pending when CLOSE cancels queued rebench tasks. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
CI E2E report — ✅ Succeeded
|
…-dispatch Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # CHANGELOG.md
The phase boundary into CLOSE already cancels the queued rebench, so keying the settle on what the close sequencer just cancelled never fired and left geak_pending stuck at awaiting_rebench. Drive it off state instead, leaving a queued or running rebench alone. Also apply the orphan gate to the 2b inconclusive branch, which otherwise let an untracked rebench replay through the GEAK harness and clear the tracked slot. Co-authored-by: Cursor <cursoragent@cursor.com>
…g gate. Reusing one key per macro-cycle handed the cancelled row back from create_or_return_existing, which KERNEL read as rebench_unavailable and left the win audit-only for the rest of the cycle; keys now step past settled attempts, bounded per cycle. Pruning the explore family settles the slot instead of leaving it awaiting a task that will never run. Narrow the apply gate so an empty slot is only a resume signal when the resume flag is set, a settled slot rejects late results, and a legacy placeholder no longer absorbs a cycle-scoped rebench. CLOSE now stops a running rebench too, since the phase only writes reports and the task would otherwise hold the GPU lane against the post-opt roofline. Ignored results are recorded as an observation rather than dropped with only a log line. Co-authored-by: Cursor <cursoragent@cursor.com>
The final report and the mission view only recognised awaiting_rebench, so a measured candidate left at rebench_unavailable or rebench_cancelled produced neither a fact nor a warning and the session read as if nothing was found — exactly the event this branch exists to surface. Both statuses now warn and name the drop reason. Also document why the phase-boundary spare is a deny-list and how the orphan gate covers its surplus, and why the pending slot can hold a key. Co-authored-by: Cursor <cursoragent@cursor.com>
…-dispatch Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/final.py
_drain_geak_rebench_for_close only ran from the CLOSE sequencer, so a run that timed out cancelled the queued rebench but left a running one writing back through the grace window and the slot at awaiting_rebench — the report then promised a rebench whose task had already been cancelled. Both wind-down paths now share the drain. Settling also rebuilt the slot from scratch, dropping the self-reported numbers the report needs to say what was abandoned; it now edits the verdict fields in place and only discards the dead task id. Co-authored-by: Cursor <cursoragent@cursor.com>
The file is maintained at release cut, not per fix: none of the last 13 merged fix commits touch it, and carrying an Unreleased entry here only made CHANGELOG.md the one file that conflicted on every rebase onto main.
xiaofei-zheng
approved these changes
Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A GEAK e2e win only reaches the headline after a same-harness rebench revalidates it. Across 68 audited sessions, 19 of 26 eligible candidates (73%) never completed that rebench, and the quality gate never once fired — the candidates were not rejected on merit, the rebench never ran to completion.
Two defects caused it:
exploretask and no phase after KERNEL allows that kind, soKERNEL_AGENT -> SWEEPcancelled the task KERNEL had just dispatched — 475ms after creation in the sampled session."geak-revalidate"is unique per session, so from macro-cycle 2 onwardcreate_or_return_existingreturned the settled first-cycle row, which KERNEL read asrebench_unavailable. All 20 affected sessions sat atmacro_cycle >= 2.Making the rebench survive and retry then exposes four more:
CLOSE, where it runs a full serve-and-bench that starves the post-opt roofline of the GPU lane and can rewrite the headline after the report is written. The wall-clock timeout path is worse: it never reaches the CLOSE sequencer, so it cancels queued work but leaves a running rebench writing back through the grace window and the slot claiming a rebench is still coming.rebench_unavailablewas already invisible.Fix
cancel_queued_not_allowedtakes aspare_queuedpredicate; the GEAK policy spares the rebench at every boundary exceptCLOSE. Deny-list, not allow-list: the phase set changes and a missing allow-list entry silently restores the bug, while a surplus costs one wasted bench that the gate in (4) refuses anyway._enqueue_geak_revalidationreturns early while a rebench is in flight, and a result is applied only whengeak_pendingstill tracks the finishing task. Applies to all four 2b decisions, including the inconclusive branch, which otherwise reached the 2a harness replay and cleared the tracked slot. Ignored results emit ageak_rebench_result_ignoredobservation.-r1,-r2, bounded per cycle). Pruning the explore family settles the slot instead of leaving KERNEL held open.rebench_unavailableand the newrebench_cancelledboth emit a warning naming the drop reason, in the final report and the mission view.GEAK policy lives in
orchestrator/phases/geak_rebench.py.TaskRegistrystays generic, receiving only thespare_queuedhook and afind_by_idempotency_keylookup.RCA:
Desgin-DOCS/hyperloom-geak-revalidation-dispatch-rca.mdTests
26 regression tests in
test_geak_revalidation_dispatch.pycovering per-cycle keys, re-dispatch after cancel, theSWEEPspare and the cancel on both wind-down paths, duplicate-dispatch suppression, prune settling, audit fields surviving a settle, orphan and late rejection across all four decisions, and report rendering for dropped candidates.pytest src/hyperloom -n 12 # 14339 passed, 21 failedThe 21 failures are pre-existing and environment-induced (credentials / gateway auth), each reproduced on a clean
origin/mainworktree.test_objective.py::test_run_closing_phase_writes_report_on_time_exhaustedis a pre-existing flake driven by its 6msmax_minutes; measured 8 runs at pass=6 fail=2 on both this branch andorigin/main.Breaking changes
None.
rebench_cancelledis a new additivegeak_pending.status; readers treat unknown statuses as non-pending. Sessions on the old key resume correctly — the slot holds a task id, which matches directly.