fix(operator): settle applies left behind their own settled operations

[aparajon]

Why this matters

An operation drive finishes in two writes: it records the terminal state on its operation row, then projects that outcome onto the parent apply. A drive that stops between those two writes — crash, eviction, deploy-time termination — leaves every operation terminal and the parent non-terminal, permanently.

Nothing picks that up. The operation-level claim arms all look for an operation that still has work left, and this apply has none. Meanwhile the one-active-apply guard sees a parent still running and refuses every new apply for that target. So the cost is not a stale row: it is a database no one can change until someone repairs it by hand.

BEFORE — the drive crashed after settling its operations, before telling the parent

  apply_operations                      applies
  ┌─────────────────────┐               ┌──────────────────────────────┐
  │ op 1   completed    │               │ apply-42   running  (stale)  │
  │ op 2   completed    │    ──✗──▶     └──────────────────────────────┘
  └─────────────────────┘                no claim arm matches settled
    all work is done                     operations, so nothing finds it;
                                         the one-active-apply guard blocks
                                         every new apply for this target

                    │
                    │  heartbeat stale for a full lease window
                    │  → the new claim arm leases the parent and
                    │    re-runs the projection the drive never finished
                    ▼

AFTER — the parent carries the state its operations already prove

  apply_operations                      applies
  ┌─────────────────────┐               ┌──────────────────────────────┐
  │ op 1   completed    │    ──✔──▶     │ apply-42   completed         │
  │ op 2   completed    │               └──────────────────────────────┘
  └─────────────────────┘                terminal summary posted, pending
                                         control requests completed, and
                                         the target accepts applies again

Work discovery runs exclusively through the operation ladder, and no operation-level claim arm matches a fully settled operation set — so nothing ever finds this parent, and the one-active-apply guard keeps the target blocked until an operator intervenes by hand. The parent needs its own repair arm.

What it does

Adds a claim arm for exactly that shape. FindNextApplyForOperationProjection claims one non-terminal apply whose operation rows have all settled and whose heartbeat has gone stale, rotating the lease onto it like the other claim primitives. The operator then runs the existing projection under that lease, publishes the terminal summary, and completes any pending control requests — so the parent reaches the state its operations already derive, and the target is released.

Notes on scope:

• The staleness gate is the safety property. A driver mid-projection still holds a fresh lease, so a live rollout can never match. The arm only sees parents whose driver has stopped reporting.
• It runs before the new-work arms, because an orphaned parent blocks its whole target — repairing it unblocks a database rather than tidying a row. On a healthy plane it matches nothing and costs one indexed query per tick.
• It derives, it does not rewrite. The operation rows are the input; only the parent changes. If the projection somehow declines to swap, that is logged as a warning rather than retried in a loop — the claim refreshed the heartbeat, so the apply is not reconsidered until the window elapses again.
• Applies with no operation rows are excluded (there is nothing to project from), as are parents that already have their own resume paths.
• Each repair increments operation_projection_repairs_total, labelled with the derived state. A sustained rate means drives are dying mid-projection — the investigation is driver lifecycle, not this path.

Design notes

Relation to the stranded-operation reaper. The reaper repairs the opposite direction of the same crash window: a parent that settled while a child row stayed pending, mirroring the parent's recorded verdict down onto pending, unleased operation rows. This arm derives the parent up from its already-settled operations. The two can never match the same apply — the reaper requires a terminal parent while this claim requires a non-terminal one, and a pending child simultaneously qualifies rows for the reaper and disqualifies the parent here. Their cadences differ for the same reason: reaped rows are dead history under a settled verdict, so the reaper is a slow maintenance pass; a stranded parent blocks its target's whole apply queue, so this arm runs on every driver tick, gated only by lease staleness.

Why a repair arm rather than one transaction. The completion write could terminalize the operation row and project the parent atomically, but this arm is needed regardless: a drive can crash before writing anything, so the engine's terminal truth always outruns storage. A naive same-transaction version also reintroduces the stranded shape without any crash — two siblings that each commit terminal and then read the other as still running would both skip the projection — so the atomic form requires every finisher to serialize on the parent row. That design belongs to the native multi-operation completion write; when it lands, this arm demotes to a safety net that should never fire.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This pull request adds an operator recovery path for “operation-projection orphans”: applies whose child apply_operations are already terminal, but whose parent applies row is still non-terminal due to a crash between (1) terminalizing the operation row(s) and (2) projecting that terminal outcome onto the parent. The new claim arm detects this stranded shape (gated by stale heartbeat), claims the parent lease, re-derives the apply state from operation rows, publishes the terminal summary, and completes any pending control requests—unblocking the one-active-apply guard for that target.

Changes:

• Added a new storage claim primitive FindNextApplyForOperationProjection to find/lease stranded applies whose operation rows are all terminal and whose heartbeat is stale.
• Added an operator recovery tick arm that runs this repair before claiming new operation work, and records a dedicated metric for repairs.
• Added unit/integration coverage for the claim predicate and the end-to-end operator behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/storage/storage.go	Extends ApplyStore with a new claim method for operation→apply projection repair.
pkg/storage/mysqlstore/applies.go	Implements the MySQL transactional claim query (stale heartbeat + all ops terminal + has ops) and lease rotation.
pkg/storage/mysqlstore/applies_test.go	Adds focused unit tests for claim eligibility gates (staleness, terminal ops, terminal parent, owner required).
pkg/metrics/metrics.go	Adds claim-failure reason keys and a new counter for operation-projection repairs (labeled by derived terminal state).
pkg/api/operator.go	Adds a recovery arm to claim stranded parents and re-derive/apply terminal state + summary + control request completion.
integration/operator_test.go	Adds an integration test validating the operator settles a stranded parent once its lease becomes stale.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.