Problem
On startup JobRunner.live calls JobRun.markOrphansAsFailed, which runs an unconditional update with no instance-ownership filter:
UPDATE job_run SET status = 'Failed', completed_at = NOW(), error = 'Service restarted'
WHERE status = 'Running'
There is no instance_id / owner column on job_run, no lease, and no heartbeat. So if a second CcasServer starts against the same DB while a first server has jobs in flight, the second server's startup marks the first server's live jobs as Failed — while their fibers keep running. The DB then disagrees with reality (a "Failed" job is still executing), and the first server's later state transitions / the idx_job_run_running_unique partial index produce knock-on conflicts.
markOrphansAsFailed is correct for the single-server case it was written for (a real restart genuinely orphans the previous process's Running rows). It is unsafe only under concurrent servers.
Evidence
JobRun.markOrphansAsFailed — unconditional WHERE status = 'Running', connectZIO (single statement), no ownership filter.
JobRunner layer init calls it during construction, before the live runner exists, with no distributed lock / leader election.
- Schema (
job_run) has no instance_id / owner / started_by column; no heartbeat or lease table anywhere.
- Surfaced by the multi-instance concurrency audit (2026-06-23).
Scope (when multi-server is actually pursued — gated under #60)
- Add an instance identity (e.g. a per-process UUID column on
job_run, or a lease/heartbeat table).
- Scope orphan-marking to the current instance, or to rows whose owning instance's lease has expired.
- Add a concurrent-instance test.
Status
Single server per DB is the documented supported model, so this is latent, not a live bug. Blocks safe multi-server hosting (#60). Not to be fixed until multi-server is pursued.
Problem
On startup
JobRunner.livecallsJobRun.markOrphansAsFailed, which runs an unconditional update with no instance-ownership filter:There is no
instance_id/ownercolumn onjob_run, no lease, and no heartbeat. So if a secondCcasServerstarts against the same DB while a first server has jobs in flight, the second server's startup marks the first server's live jobs as Failed — while their fibers keep running. The DB then disagrees with reality (a "Failed" job is still executing), and the first server's later state transitions / theidx_job_run_running_uniquepartial index produce knock-on conflicts.markOrphansAsFailedis correct for the single-server case it was written for (a real restart genuinely orphans the previous process'sRunningrows). It is unsafe only under concurrent servers.Evidence
JobRun.markOrphansAsFailed— unconditionalWHERE status = 'Running',connectZIO(single statement), no ownership filter.JobRunnerlayer init calls it during construction, before the live runner exists, with no distributed lock / leader election.job_run) has noinstance_id/owner/started_bycolumn; no heartbeat or lease table anywhere.Scope (when multi-server is actually pursued — gated under #60)
job_run, or a lease/heartbeat table).Status
Single server per DB is the documented supported model, so this is latent, not a live bug. Blocks safe multi-server hosting (#60). Not to be fixed until multi-server is pursued.