Skip to content

JobRunner orphan-marking has no instance ownership — a 2nd server kills the 1st's live jobs #110

Description

@Sootopolis

Problem

On startup JobRunner.live calls JobRun.markOrphansAsFailed, which runs an unconditional update with no instance-ownership filter:

UPDATE job_run SET status = 'Failed', completed_at = NOW(), error = 'Service restarted'
WHERE status = 'Running'

There is no instance_id / owner column on job_run, no lease, and no heartbeat. So if a second CcasServer starts against the same DB while a first server has jobs in flight, the second server's startup marks the first server's live jobs as Failed — while their fibers keep running. The DB then disagrees with reality (a "Failed" job is still executing), and the first server's later state transitions / the idx_job_run_running_unique partial index produce knock-on conflicts.

markOrphansAsFailed is correct for the single-server case it was written for (a real restart genuinely orphans the previous process's Running rows). It is unsafe only under concurrent servers.

Evidence

  • JobRun.markOrphansAsFailed — unconditional WHERE status = 'Running', connectZIO (single statement), no ownership filter.
  • JobRunner layer init calls it during construction, before the live runner exists, with no distributed lock / leader election.
  • Schema (job_run) has no instance_id / owner / started_by column; no heartbeat or lease table anywhere.
  • Surfaced by the multi-instance concurrency audit (2026-06-23).

Scope (when multi-server is actually pursued — gated under #60)

  • Add an instance identity (e.g. a per-process UUID column on job_run, or a lease/heartbeat table).
  • Scope orphan-marking to the current instance, or to rows whose owning instance's lease has expired.
  • Add a concurrent-instance test.

Status

Single server per DB is the documented supported model, so this is latent, not a live bug. Blocks safe multi-server hosting (#60). Not to be fixed until multi-server is pursued.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:serverBackend HTTP server, jobs, schedulerbugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions