fix(operator): gate stop-reconciliation claims on lease staleness

[aparajon]

Why this matters

When an operator tells SchemaBot to stop an apply, the stop is recorded durably and a background driver later picks it up to actually halt the work. Drivers check for pending stops first on every poll, before any other work, so a stop never waits behind new schema changes.

The problem: that check matched any apply with a pending stop — including one a driver had just picked up. If a stop kept failing to reconcile (say, halting the work errors on every attempt), every driver re-grabbed the same apply on every poll, and each grab used up the whole poll. One stuck stop could monopolize every driver in the fleet, and the actual schema-change work behind it never ran.

What it does

A stop is now handled by one driver at a time instead of every driver on every poll. Picking it up takes a lease on the apply; while that lease is fresh, every other poll skips the apply and moves on to real work.

one apply's stop keeps failing to reconcile

BEFORE                                    AFTER
──────                                    ─────
poll 1: grab the stop, attempt fails      poll 1: a driver takes the stop and its
poll 2: grab it again, fails                      lease; the attempt fails
poll 3: grab it again, fails              poll 2+: everyone else sees the fresh
...                                                lease, skips it, and does normal
no other work ever runs                            schema-change work

                                          the stop is retried when the owner's
                                          lease goes stale (crashed driver), or
                                          immediately if the operator re-issues it

Two details make this safe:

• A re-issued stop jumps the queue. Asking again re-opens the request and refreshes its timestamp, which makes the apply claimable immediately — the operator shouldn't wait out a staleness window they know nothing about. A webhook redelivery of a still-pending request leaves the timestamp untouched, so only a genuine new ask has this effect.
• A failed claim doesn't burn the poll. If the claim query itself errors (a storage failure), the driver did no work and owns nothing, so it falls through to normal work instead of a persistent storage error starving everything behind it. Failures after a successful claim still pace their retry by the staleness window — that driver holds the lease.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adjusts the operator stop-reconciliation claim logic to be once-per-stop-request (instead of once-per-poll) by gating reclaims on lease freshness vs. control-request recency and lease heartbeat staleness, preventing a wedged stop reconciliation from monopolizing driver ticks and starving other claim rungs.

Changes:

• Gate FindNextApplyForStopReconciliation to only match a pending stop request when the apply is unclaimed, the stop request is newer than the lease, or the lease heartbeat is stale.
• Ensure recoverApplyPendingStop does not consume the driver tick when the claim itself errors, allowing the claim ladder to proceed.
• Add focused storage + operator tests covering lease-gated reclaim, stop re-issue behavior, and non-consuming claim errors.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
pkg/storage/mysqlstore/applies.go	Adds lease/updated_at gating to stop-reconciliation claiming to prevent once-per-poll reclaims.
pkg/storage/mysqlstore/applies_test.go	Adds coverage for lease-staleness reclaim, stop re-issue reclaim, and queue fairness under fresh leases.
pkg/api/operator.go	Changes stop-reconciliation claim errors to fall through (do not consume the tick) to avoid starving later claim rungs.
pkg/api/operator_test.go	Adds a unit test verifying claim errors don’t consume the driver tick.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.