feat(updates): add host updater engine

[mcharles-square]

Reviewable diff: +3633/-0 across 5 files (excludes generated, test, and story files).

Summary

Adds the dormant host updater engine that will power one-click upgrades: a single-flight daemon, durable operation state, and a narrow Unix-socket HTTP interface. Its trigger contract accepts optional caller-generated operation IDs, making same-ID retries idempotent while retaining target-only compatibility for older callers. It stages and preflights a release while Fleet remains online, then crosses a marker-protected activation boundary with bounded commands, crash recovery, and coordinated updater self-restart. The binary is not packaged, installed, or connected to Fleet API in this PR, so merging it exposes no operator-facing behavior.

Stack: The update-notification foundation #841 → #842 → #843 → #844 → #845 is merged. The one-click phase is #835 (merged) → #836 → #837 → #838 → #839 → #840. This is 2/6 and its diff is relative to main; #835 supplies release metadata plus the preflight/activation contract. Fleet API bridging, packaging, installation, and client exposure remain in #837–#840.

How it works

At startup, the daemon validates and canonicalizes every component of its state, install, executable, and socket trust boundaries before loading or binding anything. The install chain may contain root plus one consistent deployment-admin UID—the owner #839 has already verified controls the same rootful Docker daemon—while the updater executable remains root/daemon-controlled. It then takes a lifetime advisory lock, reconciles only activation swaps proven by a durable marker, and removes crash-abandoned artifacts whose exact names are derived from the operation ID. Operation logs are confined and retained within fixed count/byte bounds; the local API refuses live or ambiguous socket owners and explicitly sets the bound socket identity and mode. A trigger accepts only a stable or RC tag, binds an optional canonical operation ID to that target, deduplicates concurrent same-ID admission, and constructs artifact URLs from the fixed Proto Fleet GitHub Releases origin.

The manager downloads the archive and SHA-256 sidecar, verifies transfer integrity, safely extracts bounded regular-file content, retains the updater payload through a private file descriptor, and preserves deployment configuration including the operator-owned HA node identity. Commands have phase deadlines, whole-process-group cancellation, and bounded output. After preflight, the staged tree and its parent are synchronized under a cancellable activation-length preparation deadline while the operation is still in Preflight; only then can the marker-backed, non-cancelable swap begin. Shutdown cancels pre-activation work and waits for terminal state before releasing the daemon lock.

After #835's preflight succeeds with Fleet still online and the staged tree is durable, the manager atomically persists an activation marker and rotates deployment to deployment.previous before installing the new tree. Pre-command failures are reconciled before terminal state is written; a restart restores the previous tree only when the marker proves the two-rename gap, never merely because an old backup exists. Success and failure become terminal only after that exact transition is durable; if the state write fails, memory retains the last nonterminal recovery context and blocks retriggers until startup reconciliation. Once Fleet is healthy, the protected updater candidate must execute --version successfully on the host and report the exact target tag. Before replacement, the manager hard-links the working executable into a retained rollback slot and durably writes a protected sibling handoff marker. The matching argv identifies the first candidate attempt; returned startup errors restore .previous, while a signal, kill, or power loss leaves the marker for the next supervisor start to reconcile before manager initialization. Secured-socket readiness durably removes the marker. Failed exec, missing destination entries, and restore/marker-clear crash windows are idempotently recoverable without consuming the backup; ordinary starts and post-ready failures cannot downgrade the updater.

flowchart LR
  C["Local Unix-socket caller"] --> A["Bounded updater API"]
  A --> M["Single-flight manager"]
  M --> G["Pinned GitHub Releases origin"]
  M --> S["Durable state, marker, lock, and retained logs"]
  M --> D["Staged, active, and previous deployments"]
  M --> R["run-fleet.sh preflight and activation"]
  M --> X["Verified replacement and durable startup handoff"]

Loading

sequenceDiagram
  participant C as Local caller
  participant U as Host updater
  participant G as GitHub Releases
  participant R as run-fleet.sh

  U->>U: Acquire daemon lock
  U->>U: Reconcile marker-proven swap, then clean stale artifacts
  C->>U: POST /v1/upgrade with target tag
  U->>G: Download archive and SHA-256 sidecar
  U->>U: Verify, safely extract, and preserve configuration
  U->>R: --preflight-only with deadline

  alt Preflight fails or shutdown begins
    U->>U: Bounded cleanup and persisted failure
  else Preflight passes
    U->>U: Sync staged tree under cancellable deadline
    U->>U: Persist swap marker and activating state
    U->>U: Fsync and rotate deployment directories
    alt Pre-command swap fails or process stops
      U->>U: Reconcile only the marker-proven layout
    else Swap is durable
      U->>U: Persist recovery command and consume marker
      U->>R: --skip-build with activation deadline
      alt Fleet fails to become healthy
        U->>U: Keep forward deployment and recovery details
      else Fleet is healthy
        U->>U: Smoke-test updater and retain rollback binary
        U->>U: Persist handoff marker, replace updater, and persist success
        U->>U: Drain API, release lock, and exec with one-shot handoff
        alt Exec fails, startup returns, or restart finds pending marker
          U->>U: Restore previous updater and exit nonzero
        else Manager initializes and secured socket binds
          U->>U: Consume rollback eligibility
        end
      end
    end
  end

  C->>U: GET /v1/status
  U-->>C: Persisted operation status

Loading

Areas of the code involved

Area / package / file	What changed	Why it matters for review
server/internal/updaterapi	Transport-neutral operation and error types	Defines the complete local privilege-boundary contract
server/internal/updater/manager.go	Validation, download, staging, marker-gated activation recovery, locking, deadlines, confined log retention, exact stale-artifact cleanup, and self-refresh	Primary security, durability, and lifecycle review surface
server/internal/updater/self_update_handoff.go	Protected sibling marker plus idempotent executable restoration/commit state machine	Review crash ordering across replacement, supervisor restart, and secured-socket readiness
server/internal/updater/server.go	Bounded HTTP API plus validated socket-directory ownership and explicit socket identity/mode	Review request limits, local filesystem authentication, and duplicate-daemon behavior
server/cmd/fleet-updater	Signal handling, graceful draining, production configuration, and canonical replacement-process exec	Pins production downloads and coordinates listener/operation/lock handoff without re-resolving configured symlinks
server/internal/updater/*_test.go	Adversarial archive, activation, locking, socket, deadline, cleanup, process-group, and recovery coverage	Tests — review alongside the manager invariants

Key technical decisions & trade-offs

• Publisher trust: Production is pinned to the official GitHub Releases origin. The sidecar detects corruption, not an independently compromised publisher; independent signing needs a separate release-key custody, rotation, revocation, and bootstrap design rather than a same-origin key added only here.
• Activation provenance: A file-fsynced, atomically installed marker exists only around the two-rename/pre-command window. Automatic restoration requires that marker; terminal state or the mere presence of deployment.previous is insufficient because forward migrations may already exist.
• Crash cleanup: Operation artifacts use exact SHA-256-derived direct-child names. Startup and pre-trigger sweeps run under the daemon lock, after layout reconciliation, unlink non-directories without following them, and recursively remove only exact reserved staging directories.
• Forward recovery: Once the activation command begins, shutdown waits instead of canceling because run-fleet.sh may have applied forward-only migrations. A 45-minute deadline is the liveness bound, and failed starts keep the new deployment plus an actionable recovery command.
• Daemon serialization: A host-state advisory lock is acquired before state load and held through operation drain; socket cleanup occurs only after proving ECONNREFUSED on an existing socket.
• Filesystem trust: State, install, executable, and socket chains reject unrelated owners and unsafe write modes, validate configured plus resolved paths, and continue through canonical paths. Install content may be owned by root plus one consistent rootful-Docker admin; state/socket/executable paths remain root/daemon-controlled. Protected final nodes cannot be symlinks, and the bound socket receives the daemon's explicit UID/GID and mode 0660.
• Durable completion: Success or failure is published as terminal—and can make the manager reusable—only after that exact state is durable. A write failure restores the last nonterminal state and recovery command, blocking another upgrade or self-restart until startup reconciliation.
• Release asset contract: build(updates): package host updater artifacts #838 builds the versioned updater into deployment/updater/proto-fleet-updater and publishes a .sha256 sidecar for every architecture archive; feat(updates): bootstrap host updater installs #839 installs that payload and service. The production-bundle contract test belongs with that packaging workflow rather than this dormant engine.
• Resource bounds: Preflight, staged-tree durability, activation, and cleanup have bounded phases; cancellation kills the command process group and is checked between file/directory syncs before the non-cancelable swap. An individual kernel fsync remains inherently non-preemptible. Command output is capped at 64 MiB, while confined retention keeps the current plus newest prior logs within eight files and 256 MiB.
• Self-refresh ordering: The protected updater payload replaces the installed binary only after Fleet is healthy and the host kernel executes its exact-version smoke test. A hard-linked previous binary and fsynced sibling marker are installed before replacement; a canonical argv value distinguishes the first candidate attempt, and the marker survives signals, kills, power loss, missing destination entries, and supervisor restarts. Restoration retains the backup and is idempotent across crashes. Only real manager initialization plus secured production-socket readiness durably removes rollback eligibility—without adding a second supervisor or a duplicate startup probe.
• Execution privilege: Supported one-click installs already grant the deployment owner rootful Docker administration, which is host-administrator-equivalent. Any future relaxation of feat(updates): bootstrap host updater installs #839's eligibility gate must revisit this boundary.
• Downstream service lifecycle: feat(updates): bootstrap host updater installs #839 must set systemd TimeoutStopSec above the 45-minute activation bound (with margin or infinity) so service shutdown preserves this PR's graceful-wait contract.

Testing & validation

• DB_PASSWORD=fleet GOWORK=off go test ./...
• GOWORK=off go test ./cmd/fleet-updater ./internal/updater
• GOWORK=off go test -race ./cmd/fleet-updater ./internal/updater
• GOWORK=off go vet ./cmd/fleet-updater ./internal/updater
• golangci-lint run -c .golangci.yaml
• Linux amd64 static build and git diff --check
• Coverage includes traversal/size rejection, checksum failure, preserved secrets/TLS/HA node configuration and restrictive mode, failed-preflight cleanup deadlines, cancellable/deadline-bounded staged-tree durability before marker creation, marker write atomicity, pre-command rollback provenance, terminal-state reconciliation, crash-abandoned artifact cleanup/confinement, state/install/deployment/executable/socket ownership and mode policies, symlink rejection/canonical retarget resistance, canonical self-exec signaling, explicit socket identity, symlink-safe log confinement and count/byte retention, queued/terminal-state persistence failures, daemon-lock retention, live/stale socket handling, shutdown races, activation deadlines, real descendant-process termination, candidate version/format rejection, protected handoff/argv mismatch rejection, pre-rename and post-restore crash reconciliation, missing executable restoration, readiness commit, degraded self-refresh, and supervisor restart handoff.
• Fleet API integration, systemd packaging, installer eligibility, and browser behavior are intentionally deferred to feat(updates): bridge Fleet API to host updater #837–feat(updates): add one-click upgrade experience #840.

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (7cef8985e99c202c5d82125a49071a6f8266ab9e...60a77091cbf978335af514c18f66f2f31028b58f, exact PR three-dot diff)
• Model: gpt-5.6-sol

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: MEDIUM

Findings

[MEDIUM] Required checksum assets are not produced by the release pipeline

• Category: Reliability
• Location: server/internal/updater/manager.go:1013
• Description: Every upgrade requires a per-bundle .sha256 release asset. The checked-in artifact and release workflows only package and upload the .tar.gz; they do not create or publish this sidecar. Tests use a synthetic server that supplies one, so they do not exercise the actual release contract.
• Impact: Canonical GitHub release upgrades fail with a 404 while downloading the checksum and never reach verification or staging.
• Recommendation: Generate and upload the expected sidecar alongside every architecture bundle before enabling the updater. Add an integration test that validates the updater’s requested asset names against the release manifest.

[MEDIUM] Command deadlines can hang on inherited output pipes

• Category: Reliability
• Location: server/internal/updater/manager.go:110
• Description: The runner kills the original process group on cancellation but leaves exec.Cmd.WaitDelay unset. A child that detaches into another process group while retaining stdout or stderr can survive cancellation, keeping the pipes open and causing cmd.Run() to wait indefinitely.
• Impact: Preflight, activation, or candidate validation can remain stuck after its deadline. The operation never becomes terminal, the process lock remains held, and shutdown can hang.
• Recommendation: Set a bounded WaitDelay so inherited pipes are forcibly closed after cancellation, and use a cgroup or equivalent containment if all descendants must be terminated. Test with a detached setsid child retaining stdout.

Notes

The diff contains no mining-pool, wallet, worker-credential, protobuf, frontend, or database changes. Test execution was unavailable because the workspace is read-only.

---

_{Generated by Codex Security Review |
Triggered by: @mcharles-square |
Review workflow run}

[Copilot]

Pull request overview

Adds a new (currently dormant) host updater daemon to the server codebase, including a Unix-socket HTTP API contract, socket security/ownership enforcement, and a crash-safe self-update handoff mechanism. This forms the foundation for later one-click upgrade wiring and packaging work in the stacked PRs.

Changes:

• Introduces updaterapi transport-neutral operation/status types for the local updater protocol.
• Adds a Unix-socket HTTP server with strict socket-directory trust validation and explicit socket mode/ownership.
• Adds self-update handoff marker logic and a new fleet-updater command entrypoint with restart/rollback handling.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
server/internal/updaterapi/types.go	Defines the JSON operation model (phases, status, trigger, error response) shared across the local updater boundary.
server/internal/updater/server.go	Implements Unix-socket HTTP endpoints and socket-path trust/permission enforcement.
server/internal/updater/server_test.go	Tests socket binding/ownership readiness and stale/live socket handling behavior.
server/internal/updater/self_update_handoff.go	Implements durable self-update handoff markers and idempotent rollback/commit semantics.
server/internal/updater/self_update_handoff_test.go	Exercises crash windows and mismatch/symlink hardening for the self-update handoff.
server/internal/updater/manager.go	Core updater manager logic (state, locking, activation boundaries, validation, etc.).
server/internal/updater/manager_test.go	Extensive coverage for staging/activation, recovery, cleanup, and durability invariants.
server/cmd/fleet-updater/main.go	Adds the updater daemon binary entrypoint, signal handling, socket readiness commit, and exec-based self-restart.
server/cmd/fleet-updater/main_test.go	Tests argv rewriting for self-update handoff and startup-failure rollback behavior.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5c8287b47

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d52cfb6426

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4b144a8260

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a7736eec63

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a8b988c4e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".