Publish immutable config snapshots; atomic settings persistence, timing validation, payment/build-retry fixes

[pk910]

Immutable config snapshots

The settings service no longer mutates one shared config.Config in place. It now publishes immutable snapshots: every applied change builds a fresh Config generation (shallow copy — all fields are values) and swaps it in atomically via atomic.Pointer. Modules hold the *config.Service instead of a raw *config.Config and load exactly one snapshot per operation (HTTP request, scheduler tick, build, reconcile pass), threading it down the call stack.

This fixes the data race reported in #161 — a UI/API settings write racing an unsynchronized hot-path read could produce a torn string read (a remote crash primitive, since the write path is unauthenticated by default on --api-port) — but fixes it for all field types at once, including the JSON-marshal read paths (GET /api/config, the SSE config event) that a field-level fix could not cover. As a bonus, multi-field reads are now coherent within one settings generation (e.g. a plan freeze can no longer observe half of a SetMany batch).

Design notes:

• Greppable rule: *config.Service may live in struct fields; *config.Config (and section pointers) appear only as function parameters and locals — storing a snapshot freezes that consumer on a stale generation.
• Config stays a plain marshal-able struct: viper/YAML loading, json.Marshal, and test construction are unchanged; no mutexes inside config structs (no copylocks hazards).
• Field registry unchanged — recompute points the existing closures at the staging copy before publishing.
• One-shot commands and tests use config.NewStaticService(cfg).
• The builder key registry moved after settings-service construction in cmd/run.go (its fleet targets are mutable settings); payload_builder.GetConfig() now returns the latest snapshot; the no-op UpdateConfig was removed.
• New race regression test drives real SetMany batches against concurrent readers and asserts snapshot coherence (not just detector silence).

Included work from other PRs

• Supersedes Make string-typed mutable settings race-safe against concurrent reads #161 — the underlying race report and reachability analysis were correct (thanks @damilolaedwards); this PR takes the approach one level up instead of per-field mutexes, which left the marshal paths racy and tripped copylocks.
• Supersedes Make settings persistence atomic and validate timing invariants #169 (ported, co-authored): db.PutSettings batch-transaction upsert; SetMany now validates the batch's resulting timing invariants (ValidateTimingBounds: inverted bid window, negative / past-deadline reveal time), persists durably, and only then applies and publishes — a failure leaves memory and state-db untouched and is reported to the caller. Startup validates operator timing config hard; a persisted override violating the bounds (saved by an older release) is kept but warned about. Per-slot action-plan overrides deliberately stay free of these bounds — chaos scenarios belong in plans, not the global baseline.
• Supersedes Make MarkRevealed and RecordWonBid order-independent #170 (ported, co-authored): MarkRevealed / RecordWonBid order independence, reimplemented for the per-key PaymentTracker (the PR predated the key-fleet work and no longer applied). An early reveal defers its deduction per key; the late won-bid report applies it immediately instead of orphaning a pending payment for two epochs.
• Supersedes Clear the candidate build marker on a failed attempt #171 (ported, co-authored): executeCandidateBuild clears the per-candidate started marker on both failure paths, so a transient engine/transform error no longer blocks every retry of that (slot, parent-tuple) for the rest of the slot.

Testing

go build ./..., go vet ./... (clean), and the full go test -race ./... suite pass after every commit. Regression tests are included for each ported fix, each verified to fail against the pre-fix code.

[redpandabot]

Summary

The PR converts the settings service to immutable atomic config snapshots (fixing the #161 torn-read race across all field types and the JSON/marshal paths), makes SetMany batches durably atomic via a single DB transaction with pre-commit timing validation, and adds an early-reveal payment-accounting fix plus a build-retry fix. The migration is thorough and correct: no post-publication mutation of snapshots, no leaked locks, and the retry/payment fixes are concurrency-safe. One low-severity concern remains around the interplay of keep-but-warn persisted overrides and whole-config timing validation.

Issues

• 🟡 pkg/config/settings.go:291 — A single stale persisted timing value wedges every unrelated settings write — ValidateTimingBounds runs against the whole scratch copy, not just the keys the batch touches. Combined with NewService's keep-but-warn policy for a persisted override that violates the bounds, one bad stored value (e.g. a reveal.time_ms saved by an older release, or a state-db reused across networks with a shorter slot duration than the one that wrote it) makes EVERY subsequent SetMany fail with 400 — including unrelated writes and the service enable toggles — until the offending key is explicitly fixed in the same batch. The error does name the key, so it is recoverable, and the whole-config semantics are clearly intentional; the specific sharp edge is that the violating value can linger from a valid-until-now config (per-slot duration is startup-only, not re-validated on reload), silently blocking the settings API for products that reuse a state-db across devnets. Consider re-validating persisted layers against the current slot duration at startup and surfacing the offending key in the batch-rejection error, or validating only the touched keys plus their cross-field partners.

---

_{Reviewed @ 74e11ef9
"When in doubt, leave it out." — Joshua Porter}