ci: a build-lock holder's reservation disappears mid-run, so the cleanup gate fails a clean leg

[wallstop]

Problem

A licensed Unity leg acquired the organization build lock, did its work, and then failed Require confirmed Unity cleanup because the release found the lock already free. The holder's own reservation had disappeared while it was working.

Evidence

Run 30645211053, job Unity 2021.3.45f1 playmode, on PR #326. The leg ran 16:01:03Z -> 16:02:54Z. Tests passed; the only failing step is the cleanup gate.

Acquire, at 16:00:57Z:

Lock repository: Ambiguous-Interactive/ambiguous-organization-build-lock
Holder: Ambiguous-Interactive/DxMessaging:30645211053:unity-tests:2021.3.45f1-playmode
Max concurrent holders: 2
Expired cooldown reservation fd74b0fb-ee46-45c6-9dd8-33c10cf21d94 for runner DAD-MACHINE.
Attempt 1: lock is free queue-position=1 max-holders=2
Acquired wallstop-organization-builds.

Release, roughly 68 seconds later at 16:02:05Z:

Release build lock wallstop-organization-builds
Lock is already free.
No release needed for wallstop-organization-builds.

Gate:

##[error]the central release did not report a safe holder-release result; the central
release did not confirm holder ownership removal acquired=true classification=true
cleanup=confirmed/healthy cleanup-reason=cleanup-confirmed release=success/noop
released=false release-health=healthy release-reason=cleanup-confirmed
reservation=missing/none incident=none

acquired=true and reservation=missing: the acquire succeeded and by release time the reservation was gone.

Why it is not the change under test

• The pull request touches no lock code. Its acquire, release, classify and gate steps are byte-identical to master.
• The immediately preceding commit on the same branch, abaabb68, passed all nine legs of the same matrix twenty minutes earlier (30643646679).
• Scanning the last eight completed unity-tests runs, the only other legs to fail this step are the four in 30642224168, and those failed for a different reason: they aborted at Require current PR head before setup before acquisition ever ran, which is ci: a Unity leg that aborts before lock acquisition still fails Require confirmed Unity cleanup #327.

So this shape -- acquire succeeds, reservation vanishes, release no-ops -- is new and has one occurrence.

Leads

• The acquire step expired a stale cooldown reservation belonging to a different runner (DAD-MACHINE) in the same call that took the lock, so the state was already carrying entries the sweep considered dead.
• max-holders=2 and the lock is organization-wide. A concurrent acquirer from another repository running its own expiry sweep is the obvious candidate for removing this holder's reservation.
• No other DxMessaging workflow held the lock: max-parallel: 1 serializes the matrix, and no Unity Benchmarks, Performance Numbers, or Release run was active in that window.
• The preceding leg (2021.3.45f1 editmode) completed at 16:01:02Z, one second before this leg started. If the lock-state branch write for that release landed after this leg's acquire write, it would clobber the new reservation. That ordering is worth checking directly against the lock-state branch history.

Impact

A leg that did its work correctly and cleaned up correctly is reported red, and the red is on the cleanup gate -- the signal that must stay trustworthy, because the four-layer license-return guarantee depends on a reader believing it. A gate that can cry wolf trains people to re-run it.

Where the fix belongs

Ambiguous-Interactive/ambiguous-organization-build-lock, in the acquire/release state handling, not in this repository. Filed here so the DxMessaging-side evidence is not lost.

Acceptance criteria

• A holder whose reservation is removed by another actor's sweep is distinguishable from a holder that never had one, and the gate reports the two differently.
• Concurrent acquire and release against the lock-state branch cannot lose a reservation written by the other.
• A leg whose tests passed and whose license was returned does not fail the cleanup gate because of lock bookkeeping alone.

Related

• ci: a Unity leg that aborts before lock acquisition still fails Require confirmed Unity cleanup #327 -- the other way this gate produces a failure that does not correspond to uncleaned resources.

[wallstop]

Root cause: the reaper matched the wrong matrix leg, and quarantined a live holder

Not a concurrent acquirer from another repository, and not a lost lock-state write. The lock-state branch history names the culprit directly.

The lock-state timeline

16:00:10Z  e59fed98  Acquire  wallstop-organization-builds
16:00:54Z  5565abcc  Release  wallstop-organization-builds
16:01:41Z  5ed7d440  Acquire  wallstop-organization-builds     <- this leg
16:01:44Z  2ab1cc43  Reap capacity-critical stale state        <- 3 seconds later
16:03:46Z  1b08bf8d  Acquire  wallstop-organization-builds

2ab1cc43 is what removed the reservation. Its diff nulls holder, empties holders, and files the holder as a quarantine reservation:

-  "holder": {
-    "holderId": "...DxMessaging:30645211053:unity-tests:2021.3.45f1-playmode",
-    "acquiredAt": "2026-07-31T16:00:58.333Z",
-    "runnerId": "DAD-MACHINE"
-  },
+  "holder": null,
+  "holders": [],
+  "reservations": [
     {
+      "state": "quarantine",
+      "reason": "scheduled stale reaping: holder job 91204876077 is completed",

Job 91204876077 is the wrong leg

Both legs of run 30645211053 attempt 1 ran on DAD-MACHINE, serialized by max-parallel: 1:

job	name	started	completed
91204876077	Unity 2021.3.45f1 editmode	15:59:30Z	16:01:02Z
91204876143	Unity 2021.3.45f1 playmode	16:01:03Z	16:02:54Z

The lock holder was 91204876143, the playmode leg, which was 43 seconds into its work when the reaper ran. The reaper evaluated 91204876077, the editmode leg that had finished one second before playmode started, found it completed, and declared the live holder stale.

Why the lookup picked it

getIdentityJobStatus in build-lock.js resolves the holder's job by a time window around holder.acquiredAt:

const matchingJobs = jobs.filter((job) => {
  if (!job || job.runner_name !== identity.runnerId) return false;
  const startedAt = parseTime(job.started_at);
  const completedAt = parseTime(job.completed_at);
  return startedAt > 0 && startedAt <= observedTime && (completedAt <= 0 || observedTime <= completedAt);
});
if (matchingJobs.length !== 1) { /* fail closed, keep run status */ }

With observedTime = acquiredAt = 16:00:58.333Z:

• editmode: 15:59:30 <= 16:00:58.333 and 16:00:58.333 <= 16:01:02 -> matches
• playmode: 16:01:03 <= 16:00:58.333 is false -> does not match

Exactly one job matched, so matchingJobs.length !== 1 did not fire and the reaper acted with full confidence on the wrong job.

The premise the window rests on -- that the holding job's started_at precedes its own acquiredAt -- does not hold here. The holder recorded queuedAt: 16:00:55.276Z and acquiredAt: 16:00:58.333Z, both before GitHub's started_at: 16:01:03Z for the same job. The acquire action stamps its own clock from inside the job; the jobs API started_at is a separate record at whole-second granularity. A few seconds of skew is all it takes, and a max-parallel: 1 matrix guarantees a same-runner predecessor sitting exactly in that window.

This is a lock-safety bug, not only a false red

The gate going red is the visible symptom. The real problem is that a live holder was removed from the lock state. From 16:01:44Z the lock under-counted its active holders while the playmode leg kept working and kept its Unity seat, so the lock could admit one more holder than max-holders allows. That is the shape of #83 (two concurrent holders sharing one Unity entitlement seat).

Reproduction conditions

Any max-parallel: 1 matrix on a self-hosted runner where leg N+1 acquires the lock within a few seconds of leg N completing on the same machine. That is the normal steady state of unity-tests, which is why this will recur.

Suggested fix

Stop inferring the job from a time window; the identity is already known.

1. Best: have acquire-build-lock record the numeric job id at acquire time and store it on the holder, so the reaper looks it up directly and never guesses.
2. Cheap and sufficient: match job.name against the holder's holder-id-suffix. holderId already carries 2021.3.45f1-playmode, which uniquely identifies the matrix leg; the editmode job could never have matched.
3. Minimum hardening if the window survives at all: treat a candidate whose completed_at is within a few seconds of observedTime as ambiguous and take the existing fail-closed path, rather than accepting it as the sole match.

Filed upstream as Ambiguous-Interactive/ambiguous-organization-build-lock#169.