fix(engine): keep operator stops clean and never re-execute direct DDL on resume

[aparajon]

Why this matters

Direct execution runs non-revertible native DDL, so the operator-control seams around it have to be exact. Two were not.

First, stops didn't stay stopped. An operator stop that landed during ALTER routing or executor setup overwrote the clean stopped state with a spurious failure — the schema change became unresumable. Worse, a stop that landed during the size gate (the row-count check that decides whether a table is small enough to run natively) was misreported as "row count unavailable" and miscounted under blocked_size_unknown — the exact signal the runbook maps to connectivity problems.

Second, nothing consulted the record of completed direct statements. A resume — or a volume change, which is internally a stop followed by a start — re-ran the whole plan from the start and re-executed direct DDL that had already landed. In the worst case this was silent: an unnamed FOREIGN KEY re-runs successfully as a second identical constraint, diverging the target from the declared schema under a passing apply.

What it does

• Stops stay stopped. Every direct-execution path — ALTER routing, target connection, dedicated-connection acquisition, session lock-wait setup — now distinguishes an operator stop from a genuine failure, matching the convention the Spirit-runner path already follows. A cancelled size gate is reported as an interruption error, kept distinct from the fail-closed blocked verdict, so it is never misdiagnosed or miscounted.
• A direct statement executes exactly once. The executor consults each statement's recorded lifecycle before running it: completed statements are skipped (skipped_completed outcome); a failed statement re-runs (MySQL DDL is atomic — it left no effect); a statement an earlier run left mid-flight fails the resume closed (blocked_outcome_unknown) with guidance to inspect the table and re-plan, because MySQL may have finished the DDL server-side after the connection dropped. Progress entries are reused across resumes, so a statement never renders twice.
• Plan/apply connection symmetry. The apply-time size gate and executor now connect with the verbatim credentials DSN the plan-time gate uses, so DSN parameters such as TLS survive to apply time instead of being dropped by a rebuild from parsed parts; the GTID probe reuses the shared DSN builder.
• Docs match the code. The routing doc now states the direct-before-engine execution order and the fail-closed consequence for cross-partition dependencies, qualifies exactly when a stale verdict fails (at ALTER routing, after the CREATE phase), adds a "Stop, resume, and retry" section, and corrects the metric-coverage claim.

                     BEFORE                                 AFTER
stop lands during    reported as a failure — change         stays cleanly stopped and resumable;
routing or setup     unresumable; a stop in the size        a stopped size gate is an interruption,
                     gate counted as blocked_size_unknown   never a blocked verdict

resume reaches a     re-executes it — non-revertible        completed  ─▶ skip (never runs twice)
direct statement     DDL runs twice (an unnamed             failed     ─▶ re-run (atomic: no effect)
with a lifecycle     FOREIGN KEY silently duplicates)       mid-flight ─▶ fail closed; re-plan

Northstar: every operator control operation behaves identically whether it lands during an online copy or a direct statement — stops are always clean, and non-revertible DDL never runs twice.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR hardens Spirit’s direct-execution path so operator stop/resume behavior is consistent with engine-driven execution, and ensures non-revertible native DDL is never re-executed on resume by consulting per-statement lifecycle state.

Changes:

• Preserve clean Stopped state across all direct-execution seams (routing, target connect, dedicated-conn acquisition, session lock-wait setup, and size gate cancellation).
• Add per-direct-statement “claim” logic so resumes skip already-completed statements and fail closed on unknown outcomes (preventing unsafe re-execution).
• Extend observability/docs and add integration tests covering stop/resume edge cases and per-statement lifecycle tracking.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/metrics/metrics.go	Adds new direct-execution metric outcomes for resume skip and fail-closed unknown-outcome cases.
pkg/engine/spirit/spirit.go	Threads verbatim credentials DSN into apply execution; updates plan-time refused-statement verdict handling to propagate cancellation distinctly.
pkg/engine/spirit/execution.go	Propagates DSN into execution/resume paths and ensures stop handling during ALTER routing is treated as an operator interruption, not a failure.
pkg/engine/spirit/direct.go	Implements cancellation-aware size gate, direct-execution stop checks at multiple seams, and per-statement lifecycle “claim” logic to prevent double execution on resume.
pkg/engine/spirit/control.go	Ensures resume/start forwards stored DSN into the resume path.
pkg/engine/spirit/spirit_integration_test.go	Updates test call sites for the new execute signature including DSN.
pkg/engine/spirit/pending_drops_integration_test.go	Updates test call sites for the new execute signature including DSN.
pkg/engine/spirit/direct_integration_test.go	Adds/updates integration tests for stop/resume behavior and per-statement direct lifecycle handling.
docs/direct-execution.md	Documents routing/execution order, stop/resume semantics, and expanded metric outcomes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.