fix(storage): terminal apply states are immutable

[aparajon]

Why this matters

Applies().Update was last-writer-wins: any caller holding a stale in-memory snapshot could write an active state over a terminal row, silently resurrecting a completed, failed, or stopped apply. A resurrected apply re-enters driver claiming with no operator action behind it — the storage layer must refuse the transition regardless of which caller races.

What it does

• Update refuses terminal-to-active transitions in the WHERE clause itself: when the new state is active, the write matches only rows that are still active, so a concurrent terminal verdict can never be overwritten from a snapshot.
• The ambiguous zero-rows result (idempotent no-op vs guarded refusal vs missing row) is resolved by a read-committed re-read: a terminal row surfaces the distinct ErrApplyTerminalStateImmutable, a missing row surfaces ErrApplyNotFound, and a still-active row is a benign no-op.
• Terminal-to-terminal writes, including same-state refreshes, stay allowed. A settled apply re-enters the active lifecycle only through the dedicated guarded transition of claiming a stopped apply.
• The stop-before-start normalization in the API layer reloads the row when its write is refused and proceeds from the newer verdict instead of overwriting it; guard errors carry the apply identifier, not the internal row ID.

caller snapshot (Running)          concurrent driver
        |                                 |
        |                          write Failed (terminal)
   Update(state=Running) ---------------> WHERE state IN (active)
        |                                 -> 0 rows
   re-read row: terminal
        -> ErrApplyTerminalStateImmutable (never resurrected)

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR enforces “terminal apply state immutability” as a storage-layer safety invariant, preventing stale writers from moving an apply from a terminal verdict back into the active lifecycle, and hardens the stop-before-start normalization path with a CAS write to avoid clobbering concurrent state advances.

Changes:

• Add a storage guard in Applies().Update that refuses terminal → active transitions with a distinct storage.ErrApplyTerminalStateImmutable.
• Disambiguate “0 rows affected” outcomes for guarded active-state updates by re-reading the row to distinguish missing vs terminal vs idempotent no-op.
• Update the stop-before-start handler to use UpdateDerivedState (CAS) and add coverage for the concurrent-advance scenario.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pkg/storage/storage.go	Documents new Update() behavior around terminal immutability.
pkg/storage/mysqlstore/applies.go	Implements terminal→active guard + zero-rows disambiguation helper.
pkg/storage/mysqlstore/applies_test.go	Adds/updates tests for terminal immutability and Update() missing-row behavior.
pkg/storage/errors.go	Introduces ErrApplyTerminalStateImmutable.
pkg/api/handlers_test.go	Extends staticApplyStore test double with UpdateDerivedState CAS semantics.
pkg/api/control_handlers.go	Uses UpdateDerivedState to CAS the stop-before-start normalization write.
pkg/api/control_handlers_test.go	Adds tests covering the CAS stop-before-start behavior and concurrent advance handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.