Skip to content

Fix/geak revalidation dispatch - #1221

Merged
xiaofei-zheng merged 11 commits into
mainfrom
fix/geak-revalidation-dispatch
Aug 19, 2026
Merged

Fix/geak revalidation dispatch#1221
xiaofei-zheng merged 11 commits into
mainfrom
fix/geak-revalidation-dispatch

Conversation

@lishuoshuo-amd

@lishuoshuo-amd lishuoshuo-amd commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Problem

A GEAK e2e win only reaches the headline after a same-harness rebench revalidates it. Across 68 audited sessions, 19 of 26 eligible candidates (73%) never completed that rebench, and the quality gate never once fired — the candidates were not rejected on merit, the rebench never ran to completion.

Two defects caused it:

  1. The phase boundary cancelled the rebench. It runs as an explore task and no phase after KERNEL allows that kind, so KERNEL_AGENT -> SWEEP cancelled the task KERNEL had just dispatched — 475ms after creation in the sampled session.
  2. The idempotency key was a fixed string. "geak-revalidate" is unique per session, so from macro-cycle 2 onward create_or_return_existing returned the settled first-cycle row, which KERNEL read as rebench_unavailable. All 20 affected sessions sat at macro_cycle >= 2.

Making the rebench survive and retry then exposes four more:

  1. A rebench spared at every boundary also survives into CLOSE, where it runs a full serve-and-bench that starves the post-opt roofline of the GPU lane and can rewrite the headline after the report is written. The wall-clock timeout path is worse: it never reaches the CLOSE sequencer, so it cancels queued work but leaves a running rebench writing back through the grace window and the slot claiming a rebench is still coming.
  2. Per-cycle keys plus no in-flight check allow two live rebenches; the orphan's result is taken at face value.
  3. A cancelled attempt cannot be replaced within its cycle, because its key returns the cancelled row.
  4. A candidate abandoned without revalidation produces no fact and no warning in the report — the session reads as if nothing was found. This predates the branch: rebench_unavailable was already invisible.

Fix

  1. cancel_queued_not_allowed takes a spare_queued predicate; the GEAK policy spares the rebench at every boundary except CLOSE. Deny-list, not allow-list: the phase set changes and a missing allow-list entry silently restores the bug, while a surplus costs one wasted bench that the gate in (4) refuses anyway.
  2. Keys are scoped per macro-cycle.
  3. Both wind-down paths — the CLOSE sequencer and the wall-clock closing phase — share one drain that cancels queued and running rebench, then settles the slot. Settling is driven by state, not by what the caller just cancelled, since the phase boundary has already cancelled the queued case by then; it edits the verdict fields in place so the candidate's self-reported numbers survive for the report.
  4. _enqueue_geak_revalidation returns early while a rebench is in flight, and a result is applied only when geak_pending still tracks the finishing task. Applies to all four 2b decisions, including the inconclusive branch, which otherwise reached the 2a harness replay and cleared the tracked slot. Ignored results emit a geak_rebench_result_ignored observation.
  5. Keys step past settled attempts (-r1, -r2, bounded per cycle). Pruning the explore family settles the slot instead of leaving KERNEL held open.
  6. rebench_unavailable and the new rebench_cancelled both emit a warning naming the drop reason, in the final report and the mission view.

GEAK policy lives in orchestrator/phases/geak_rebench.py. TaskRegistry stays generic, receiving only the spare_queued hook and a find_by_idempotency_key lookup.

RCA: Desgin-DOCS/hyperloom-geak-revalidation-dispatch-rca.md

Tests

26 regression tests in test_geak_revalidation_dispatch.py covering per-cycle keys, re-dispatch after cancel, the SWEEP spare and the cancel on both wind-down paths, duplicate-dispatch suppression, prune settling, audit fields surviving a settle, orphan and late rejection across all four decisions, and report rendering for dropped candidates.

pytest src/hyperloom -n 12
# 14339 passed, 21 failed

The 21 failures are pre-existing and environment-induced (credentials / gateway auth), each reproduced on a clean origin/main worktree. test_objective.py::test_run_closing_phase_writes_report_on_time_exhausted is a pre-existing flake driven by its 6ms max_minutes; measured 8 runs at pass=6 fail=2 on both this branch and origin/main.

Breaking changes

None. rebench_cancelled is a new additive geak_pending.status; readers treat unknown statuses as non-pending. Sessions on the old key resume correctly — the slot holds a task id, which matches directly.

Spare geak_fallback rebench tasks from phase and explore-family cancellation,
scope revalidation idempotency keys by macro_cycle, and add regression tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lishuoshuo-amd
lishuoshuo-amd requested a review from a team as a code owner August 18, 2026 10:50
lishuoshuo-amd and others added 3 commits August 18, 2026 20:02
Do not spare GEAK rebench tasks on CLOSE transitions, cancel them at close
entry, block duplicate enqueues while one is in flight, and gate 2b promote
on geak_pending task tracking. Parameterize phase-boundary spares via
spare_queued, move policy helpers to geak_rebench, revert cancel_family
exemption, and extend regression tests plus CHANGELOG.

Co-authored-by: Cursor <cursoragent@cursor.com>
Treat empty geak_pending as untracked-by-design for promote and failure
paths, apply the orphan gate to no_material and no_promote, and finalize
geak_pending when CLOSE cancels queued rebench tasks.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

Copy link
Copy Markdown

CI E2E report — ✅ Succeeded

item value
result ✅ Succeeded
model Qwen/Qwen3-0.6B (dense)
resources 1× GPU, TP=1
PR branch fix/geak-revalidation-dispatch
commit a9f93fa6e1e8bf051551ee0476f8426a43764f37
session_id 68c45c93-83ea-455b-9578-1053e2847d77
queue → dispatch 0s
run time 167m 13s
total 167m 13s

details

lishuoshuo-amd and others added 7 commits August 19, 2026 14:15
…-dispatch

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	CHANGELOG.md
The phase boundary into CLOSE already cancels the queued rebench, so keying
the settle on what the close sequencer just cancelled never fired and left
geak_pending stuck at awaiting_rebench. Drive it off state instead, leaving a
queued or running rebench alone. Also apply the orphan gate to the 2b
inconclusive branch, which otherwise let an untracked rebench replay through
the GEAK harness and clear the tracked slot.

Co-authored-by: Cursor <cursoragent@cursor.com>
…g gate.

Reusing one key per macro-cycle handed the cancelled row back from
create_or_return_existing, which KERNEL read as rebench_unavailable and left
the win audit-only for the rest of the cycle; keys now step past settled
attempts, bounded per cycle. Pruning the explore family settles the slot
instead of leaving it awaiting a task that will never run.

Narrow the apply gate so an empty slot is only a resume signal when the resume
flag is set, a settled slot rejects late results, and a legacy placeholder no
longer absorbs a cycle-scoped rebench. CLOSE now stops a running rebench too,
since the phase only writes reports and the task would otherwise hold the GPU
lane against the post-opt roofline. Ignored results are recorded as an
observation rather than dropped with only a log line.

Co-authored-by: Cursor <cursoragent@cursor.com>
The final report and the mission view only recognised awaiting_rebench, so a
measured candidate left at rebench_unavailable or rebench_cancelled produced
neither a fact nor a warning and the session read as if nothing was found —
exactly the event this branch exists to surface. Both statuses now warn and
name the drop reason.

Also document why the phase-boundary spare is a deny-list and how the orphan
gate covers its surplus, and why the pending slot can hold a key.

Co-authored-by: Cursor <cursoragent@cursor.com>
…-dispatch

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/final.py
_drain_geak_rebench_for_close only ran from the CLOSE sequencer, so a run that
timed out cancelled the queued rebench but left a running one writing back
through the grace window and the slot at awaiting_rebench — the report then
promised a rebench whose task had already been cancelled. Both wind-down paths
now share the drain.

Settling also rebuilt the slot from scratch, dropping the self-reported numbers
the report needs to say what was abandoned; it now edits the verdict fields in
place and only discards the dead task id.

Co-authored-by: Cursor <cursoragent@cursor.com>
The file is maintained at release cut, not per fix: none of the last 13
merged fix commits touch it, and carrying an Unreleased entry here only made
CHANGELOG.md the one file that conflicted on every rebase onto main.

@ZhengGong-amd ZhengGong-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

@xiaofei-zheng
xiaofei-zheng merged commit df82cb2 into main Aug 19, 2026
26 of 28 checks passed
@xiaofei-zheng
xiaofei-zheng deleted the fix/geak-revalidation-dispatch branch August 19, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants