Stuck Job Watchdog #1969

Workflow file for this run

.github/workflows/stuck-job-watchdog.yml at 2fc8bcd

	name: Stuck Job Watchdog

	# Recovery automation for the known GitHub Actions self-hosted dispatcher bug
	# documented at https://github.com/orgs/community/discussions/186811:
	# self-hosted runners report Online/Idle but `runner_id` stays at 0 for
	# 7+ minutes, so a queued job whose label set is satisfied by an idle runner
	# nonetheless waits indefinitely until manually re-run.
	#
	# FAIL CLOSED (#328). A green run means the audit ran to completion. Every
	# input the verdict depends on -- the queued-run listing, the organization
	# runner inventory, a per-run job listing, and the cancel-cap state branch --
	# fails the job when it cannot be read, instead of logging and exiting 0.
	# Before this rule the inventory read was unauthorized on both endpoints the
	# watchdog tried and the job still reported success, so a `Performance Numbers`
	# run sat queued for ten hours while the automation meant to notice it reported
	# green every five minutes.
	#
	# Runner inventory is ORGANIZATION-scoped and is read with the build-lock
	# reader App (`organization_self_hosted_runners: read`), the same credential
	# `check-unity-runner-availability` already uses. The job's own GITHUB_TOKEN is
	# repository-scoped and can never reach `/orgs/{org}/actions/runner-groups`.
	# The repository-level fallback this workflow used to attempt was worse than
	# no fallback: this repository registers zero repository-level runners, so the
	# call would succeed and report an empty inventory, which reads identically to
	# "no runner matches" and still issues no action.
	#
	# The inventory walk enumerates ALL organization runner groups rather than
	# filtering with `visible_to_repository`. That parameter is documented only as a
	# "string" with no stated format, and the two plausible readings -- repository
	# name and numeric repository id -- are not distinguishable from the docs. Guess
	# wrong and the call either 4xxs or returns an empty list, and BOTH land in
	# `fail_closed` below, so this job would fail every one of its 288 daily runs
	# while reporting a credential or outage problem it does not have. The
	# organization registers a single group ("Default") today, so the filter would
	# select exactly what the unfiltered call already returns: it is a live failure
	# risk in exchange for nothing.
	#
	# The group names are logged for the day that stops being true. Enumerating all
	# groups over-counts capacity if a second, restricted group is ever added, and
	# over-counting capacity is what could turn a legitimately queued run into a
	# wrongful cancel -- so if a second name ever appears in the summary, narrow this
	# walk before trusting it, using an id confirmed against a live response rather
	# than against the documentation.
	#
	# Detection requires ALL of the following to be true:
	# * The workflow run is `status: queued` AND older than MIN_QUEUE_AGE_SECONDS
	# (default 300s / 5 min; round-2 false-positive guards below -- zero
	# in-progress jobs, self-run exclusion, excluded-workflow list -- make
	# the previous conservative 600s buffer unnecessary).
	# * No job in the run is `status: in_progress` - a run with even one
	# in-progress job is by definition holding/using a runner, not
	# dispatcher-stuck. This is what prevents false positives for matrix cells
	# or jobs waiting while another job from the same run is actively running.
	# * At least one job in the run is `status: queued`.
	# * At least one ONLINE, NOT-BUSY runner's labels satisfy a queued job's
	# label requirements (superset match, case-insensitive on both sides).
	# * The run's workflow file is NOT in the exclusion list (`release.yml`
	# is hard-excluded by default; additional entries may be added via the
	# `WATCHDOG_EXCLUDED_WORKFLOWS` repo variable, whitespace-separated).
	# * The run id is NOT the watchdog's own run id.
	#
	# A queued run that no ONLINE runner can satisfy is NOT dispatcher-stuck --
	# cancelling it would destroy work the fleet will pick up as soon as the
	# machine reconnects. It is reported in its own step-summary section and as a
	# `::warning::` annotation, so an offline runner starving the queue is visible
	# on the run page without opening a job log. That is the shape #328 hit: a
	# registered-but-offline `fast` runner, invisible because the audit was blind.
	#
	# Recovery action (per cli/cli#9221 and the gh-run-rerun manual,
	# `gh run rerun --failed` cannot be used on a `status: queued` run because
	# the run never reached `failed`; the documented workaround is cancel +
	# redispatch / cancel + operator-managed re-run):
	# * `gh run cancel <id>` kills the stuck dispatch.
	# * If the run was triggered by push/schedule/workflow_dispatch on a
	# branch (not a tag) AND the workflow file declares `workflow_dispatch:`,
	# re-dispatch via REST API
	# (`POST repos/{owner}/{repo}/actions/workflows/{workflow_id}/dispatches`).
	# * Otherwise (pull_request, tag, no workflow_dispatch trigger), emit a
	# clear `GITHUB_STEP_SUMMARY` line instructing the operator to click
	# "Re-run all jobs" in the GitHub UI. Do NOT push a commit, comment on
	# the PR, or escalate any other automatic action.
	#
	# Cancel attempts are capped at 2 per run-id per 24h via a small state file
	# on the `watchdog-state` orphan branch. The state branch is materialized only
	# after a run is classified dispatcher-stuck, so a cycle that finds nothing to
	# cancel -- the overwhelming majority -- performs no clone and cannot be failed
	# by a state-branch problem it never needed to solve. State is pushed
	# immediately after each successful cancel, with a single rebase+retry on push
	# failure.
	#
	# Workflow-level concurrency guarantees only one watchdog instance ever runs;
	# newer schedules must not cancel an in-flight audit after it has cancelled a
	# stuck run but before it persists the state-branch counter.

	on:
	schedule:
	- cron: "/5 * * *"
	workflow_dispatch:

	concurrency:
	group: stuck-job-watchdog
	cancel-in-progress: false

	permissions:
	actions: write
	contents: write

	jobs:
	audit-queue:
	name: Audit queued runs and recover dispatcher-stuck ones
	runs-on: ubuntu-latest
	timeout-minutes: 10
	steps:
	# Fail closed by construction: this action errors when either secret is
	# empty, so a missing credential aborts the audit before it can report a
	# verdict it had no way to reach.
	- name: Mint the runner-inventory reader token
	id: reader_token
	timeout-minutes: 2
	uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3
	with:
	app-id: ${{ secrets.BUILD_LOCK_READER_APP_ID }}
	private-key: ${{ secrets.BUILD_LOCK_READER_APP_PRIVATE_KEY }}
	# Organization-scoped (no `repositories:`) because the runner-group
	# endpoints are organization endpoints, and narrowed to the single
	# permission this audit needs so the minted token can do nothing else.
	owner: ${{ github.repository_owner }}
	permission-organization-self-hosted-runners: read

	- name: Audit + cancel-and-redispatch
	shell: bash
	env:
	GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	RUNNER_INVENTORY_TOKEN: ${{ steps.reader_token.outputs.token }}
	REPO: ${{ github.repository }}
	REPO_ID: ${{ github.repository_id }}
	OWNER: ${{ github.repository_owner }}
	SELF_RUN_ID: ${{ github.run_id }}
	STATE_BRANCH: watchdog-state
	STATE_DIR: .watchdog-state
	MAX_CANCELS_PER_DAY: "2"
	MIN_QUEUE_AGE_SECONDS: "300"
	DEFAULT_EXCLUDED_WORKFLOWS: "release.yml"
	EXTRA_EXCLUDED_WORKFLOWS: ${{ vars.WATCHDOG_EXCLUDED_WORKFLOWS }}
	run: \|
	set -euo pipefail

	summary_file="$(mktemp)"
	: > "${summary_file}"
	log_summary() {
	printf '%s\n' "$1" \| tee -a "${summary_file}"
	}

	# Category buckets for the final step-summary table.
	healthy_runs_file="$(mktemp)"
	stuck_runs_file="$(mktemp)"
	excluded_runs_file="$(mktemp)"
	starved_runs_file="$(mktemp)"
	: > "${healthy_runs_file}"
	: > "${stuck_runs_file}"
	: > "${excluded_runs_file}"
	: > "${starved_runs_file}"

	summary_emitted=0
	deliberate_exit=0

	emit_summary() {
	if (( summary_emitted == 1 )); then
	return 0
	fi
	summary_emitted=1
	# `\|\| true`: this runs from an EXIT trap, so a failure here would
	# replace the status the job is actually exiting with.
	{
	echo "## Watchdog summary"
	cat "${summary_file}"
	echo ""
	echo "### Healthy queued (waiting on concurrency / matrix slot)"
	if [[ -s "${healthy_runs_file}" ]]; then cat "${healthy_runs_file}"; else echo "_(none)_"; fi
	echo ""
	echo "### Stuck (auto-cancelled)"
	if [[ -s "${stuck_runs_file}" ]]; then cat "${stuck_runs_file}"; else echo "_(none)_"; fi
	echo ""
	echo "### Starved (no online runner carries the required labels)"
	if [[ -s "${starved_runs_file}" ]]; then cat "${starved_runs_file}"; else echo "_(none)_"; fi
	echo ""
	echo "### Stuck but excluded (operator action needed)"
	if [[ -s "${excluded_runs_file}" ]]; then cat "${excluded_runs_file}"; else echo "_(none)_"; fi
	} >> "${GITHUB_STEP_SUMMARY}" \|\| true
	}

	finish() {
	deliberate_exit=1
	emit_summary
	exit "${1:-0}"
	}

	# Without this, ANY abort `set -e` catches outside the enumerated
	# paths -- and this is 500 lines of bash running 288 times a day --
	# produces a red run with a completely empty step summary and no
	# annotation, which is the worst signal an operator can be handed.
	# The trap makes every exit emit the buckets, and labels the ones
	# that did not come from `finish` so an unexpected abort is not
	# mistaken for a considered verdict.
	# Invoked indirectly by the EXIT trap below, so static analysis cannot
	# see the call. Both codes are needed because the version numbering
	# differs: 0.9 flags the body as unreachable (SC2317), while newer
	# builds -- including the one inside the actionlint container CI runs
	# -- flag the function as never invoked (SC2329). Note that a comment
	# line may not BEGIN with the analyzer's own name, or it is parsed as
	# a malformed directive and the whole script stops being checked.
	# shellcheck disable=SC2317,SC2329
	on_exit() {
	local code=$?
	if (( code != 0 )) && (( deliberate_exit == 0 )); then
	log_summary "ERROR: the audit aborted unexpectedly (exit ${code}); the queue was not fully evaluated."
	echo "::error::stuck-job watchdog aborted unexpectedly (exit ${code}); see the job log."
	fi
	emit_summary
	}
	trap on_exit EXIT

	# The audit could not answer. Never exit 0 from here: a green run has
	# to mean the queue was evaluated, not that evaluation was skipped.
	fail_closed() {
	log_summary "ERROR: ${1}"
	log_summary "The audit could not evaluate the queue, so this run fails rather than reporting a verdict it did not reach."
	echo "::error::stuck-job watchdog could not complete its audit: ${1}"
	finish 1
	}

	log_summary "## Stuck-job watchdog audit ($(date -u +'%Y-%m-%dT%H:%M:%SZ'))"
	log_summary "Repo: ${REPO}"
	log_summary "Owner: ${OWNER}"
	log_summary "Self run id (will skip): ${SELF_RUN_ID}"

	# Build the workflow-file exclusion list (default + repo variable).
	declare -A EXCLUDED_BY_FILE=()
	for wf in ${DEFAULT_EXCLUDED_WORKFLOWS} ${EXTRA_EXCLUDED_WORKFLOWS:-}; do
	[[ -z "${wf}" ]] && continue
	# Normalize: strip any leading .github/workflows/ if present.
	base="${wf##*/}"
	# An empty subscript is a hard bash error, and the strip above turns
	# any operator entry ending in `/` into one -- which the very next
	# comment invites by talking about directory prefixes. Guarding only
	# the read site left the write site able to kill the audit outright,
	# every five minutes, on a repo-variable typo.
	[[ -z "${base}" ]] && continue
	EXCLUDED_BY_FILE["${base}"]=1
	done
	excluded_list=""
	for k in "${!EXCLUDED_BY_FILE[@]}"; do
	excluded_list+="${k} "
	done
	log_summary "Excluded workflows: ${excluded_list:-<none>}"

	# ------------------------------------------------------------------
	# 1. Enumerate queued runs older than MIN_QUEUE_AGE_SECONDS.
	# ------------------------------------------------------------------
	now_epoch="$(date -u +%s)"
	queued_runs_json="$(mktemp)"
	# `gh api --paginate` over the runs endpoint returns one JSON object
	# per page; `jq -s '[.[] \| .workflow_runs[]?]'` flattens all pages
	# into a single array. Equivalent to using `--slurp` on newer gh
	# versions but works on all gh versions shipped with ubuntu-latest.
	if ! gh api --paginate "repos/${REPO}/actions/runs?status=queued&per_page=100" \
	\| jq -s '[.[] \| (.workflow_runs // [])[]]' > "${queued_runs_json}"; then
	fail_closed "failed to list queued runs for ${REPO}."
	fi

	# `mapfile < <(jq ...)` throws the process substitution's exit status
	# away, and `pipefail` does not reach inside one. A single unparseable
	# `created_at` would abort jq, yield an empty list, and print
	# "Queue is clean" over a genuinely stuck queue -- the exact fail-open
	# this rewrite exists to remove. Capture, check, then split.
	if ! queued_ids_raw="$(jq -r --argjson now "${now_epoch}" --argjson min "${MIN_QUEUE_AGE_SECONDS}" '
	.[]
	\| select(.created_at != null)
	\| (.created_at \| fromdateiso8601) as $created
	\| select(($now - $created) >= $min)
	\| .id
	' < "${queued_runs_json}" \| sort -u)"; then
	fail_closed "could not parse the queued-run listing for ${REPO}; the queue cannot be evaluated."
	fi
	queued_ids=()
	if [[ -n "${queued_ids_raw}" ]]; then
	mapfile -t queued_ids <<< "${queued_ids_raw}"
	fi

	log_summary "Queued runs older than ${MIN_QUEUE_AGE_SECONDS}s: ${#queued_ids[@]}"
	if [[ ${#queued_ids[@]} -eq 0 ]]; then
	log_summary "Queue is clean. No action."
	finish 0
	fi

	# ------------------------------------------------------------------
	# 2. Read the organization runner inventory through the reader App.
	# Fail closed on any failure:
	# an unreadable inventory cannot distinguish "nothing is stuck"
	# from "everything is stuck and we cannot see it".
	# ------------------------------------------------------------------
	gh_reader() {
	GH_TOKEN="${RUNNER_INVENTORY_TOKEN}" gh "$@"
	}

	# `unique_by(.id)`: `--paginate` concatenates page objects, and the
	# live organization returns "Default" twice through this walk. The
	# runner dedupe below already made that harmless for the COUNTS, but
	# it still cost a redundant API call per duplicate and printed a
	# group list that made an operator distrust the summary.
	runner_groups_json="$(mktemp)"
	if ! gh_reader api --paginate \
	"orgs/${OWNER}/actions/runner-groups?per_page=100" \
	\| jq -s '[.[] \| (.runner_groups // [])[] \| select(.id != null)] \| unique_by(.id)' > "${runner_groups_json}"; then
	fail_closed "could not read the organization runner groups."
	fi
	group_count="$(jq 'length' < "${runner_groups_json}")"
	if (( group_count == 0 )); then
	fail_closed "the organization reported no runner groups; the inventory cannot be trusted."
	fi
	log_summary "Organization runner groups: $(jq -r '[.[].name] \| join(", ")' < "${runner_groups_json}")"

	# Only groups THIS repository may actually use. Counting a runner that
	# cannot serve our jobs inflates capacity, and inflated capacity is
	# what turns a legitimately queued run into a wrongful cancel: the
	# audit would see an idle runner carrying the right labels, conclude
	# the dispatcher is stuck, and cancel a run that was correctly waiting.
	#
	# This reads each group's documented `visibility` field rather than the
	# `visible_to_repository` query parameter, which the REST docs type
	# only as "string" with no stated format. `all` needs no further call;
	# `selected` (and `private`) are resolved against the group's own
	# repositories endpoint, which returns repository objects with numeric
	# ids -- an unambiguous comparison against `github.repository_id`.
	#
	# The previous revision walked every group on the stated grounds that
	# the organization had exactly one. It has at least two -- `Default`
	# and `ambiguous-interactive-organization-builds` -- so that premise
	# was simply wrong, and the risk documented in #335 was live rather
	# than hypothetical.
	# `!= false`, NOT `// true`: jq's `//` treats an explicit `false` as
	# absent and yields the right-hand side, so a group that genuinely
	# refuses public repositories would read as allowing them -- the
	# exact inversion of the check. Caught by the fixture for it.
	if ! group_rows_raw="$(jq -r '.[] \| "\(.id)\t\(.visibility // "all")\t\(.allows_public_repositories != false)\t\(.name)"' < "${runner_groups_json}")"; then
	fail_closed "could not parse the organization runner-group inventory."
	fi
	group_rows=()
	if [[ -n "${group_rows_raw}" ]]; then
	mapfile -t group_rows <<< "${group_rows_raw}"
	fi
	# mapfile, not `while read < <(...)`: the loop body runs `gh`, and a
	# child that consumes the loop's stdin silently eats the remaining
	# group ids.
	runner_group_ids=()
	excluded_groups=()
	group_repos_json="$(mktemp)"
	for group_row in "${group_rows[@]}"; do
	IFS=$'\t' read -r group_id group_visibility group_public group_name <<< "${group_row}"
	# A group can be `visibility: all` and still refuse PUBLIC
	# repositories, and this repository is public -- its runners are
	# then unusable capacity, which is the wrongful cancel this walk
	# exists to prevent. (Cursor Bugbot.)
	#
	# Excluded unconditionally rather than only when we know we are
	# public, which would need another API call and another failure
	# mode. The cost of being wrong is asymmetric: for a PRIVATE
	# repository this under-counts capacity, and under-counting can only
	# produce a starvation report, never a cancel. Over-counting is what
	# cancels a live run.
	if [[ "${group_public}" != "true" ]]; then
	excluded_groups+=("${group_name} (refuses public repositories)")
	continue
	fi
	if [[ "${group_visibility}" == "all" ]]; then
	runner_group_ids+=("${group_id}")
	continue
	fi
	# `private` means "every PRIVATE repository in the organization" --
	# not a selected list. This repository is public, so such a group can
	# never serve it. Excluded here rather than fetching its repositories
	# endpoint, which would answer a question that does not apply and
	# could fail the whole audit closed on an error it did not need to
	# make. (Cursor Bugbot.)
	if [[ "${group_visibility}" == "private" ]]; then
	excluded_groups+=("${group_name} (private repositories only)")
	continue
	fi
	if ! gh_reader api --paginate \
	"orgs/${OWNER}/actions/runner-groups/${group_id}/repositories?per_page=100" \
	\| jq -s '[.[] \| (.repositories // [])[] \| .id \| tostring]' > "${group_repos_json}"; then
	fail_closed "could not read which repositories may use runner group ${group_name} (${group_id})."
	fi
	# `--arg`, not `--argjson`: an empty or non-numeric REPO_ID makes
	# `--argjson` fail to parse, which under `set -e` aborts the script
	# WITHOUT reaching `fail_closed` -- an audit that dies instead of
	# reporting. Both sides are compared as strings. (GitHub Copilot.)
	if [[ "$(jq --arg want "${REPO_ID}" 'any(.[]; . == $want)' < "${group_repos_json}")" == "true" ]]; then
	runner_group_ids+=("${group_id}")
	else
	excluded_groups+=("${group_name}")
	fi
	done
	if (( ${#excluded_groups[@]} > 0 )); then
	# NOT `IFS=', '` with `"${arr[*]}"`: bash joins with only the FIRST
	# character of IFS there, so that renders `a,b`. (GitHub Copilot.)
	log_summary "Runner groups not usable by ${REPO} (excluded from capacity): $(printf '%s, ' "${excluded_groups[@]}" \| sed 's/, $//')"
	fi
	if (( ${#runner_group_ids[@]} == 0 )); then
	fail_closed "no organization runner group is visible to ${REPO}; the inventory cannot be trusted."
	fi
	runners_json="$(mktemp)"
	: > "${runners_json}"
	group_runners_json="$(mktemp)"
	for group_id in "${runner_group_ids[@]}"; do
	if ! gh_reader api --paginate \
	"orgs/${OWNER}/actions/runner-groups/${group_id}/runners?per_page=100" \
	\| jq -s '[.[] \| (.runners // [])[]]' > "${group_runners_json}"; then
	fail_closed "could not read runners in organization runner group ${group_id}."
	fi
	cat "${group_runners_json}" >> "${runners_json}"
	done

	# Labels are compared case-insensitively in both directions: a job
	# declaring `windows` and a runner registered as `Windows` are the
	# same requirement, and an exact match would silently report every
	# such run as unmatched.
	inventory_json="$(mktemp)"
	jq -s '
	[ .[][]
	\| { id: .id,
	name: .name,
	online: (.status == "online"),
	busy: (.busy == true),
	labels: ([(.labels // [])[] \| (.name // "") \| ascii_downcase \| select(. != "")]) }
	]
	\| unique_by(.id)
	' < "${runners_json}" > "${inventory_json}"

	registered_count="$(jq 'length' < "${inventory_json}")"
	online_count="$(jq '[.[] \| select(.online)] \| length' < "${inventory_json}")"
	idle_count="$(jq '[.[] \| select(.online and (.busy \| not))] \| length' < "${inventory_json}")"
	log_summary "Runner inventory (groups visible to ${REPO}): ${registered_count} registered, ${online_count} online, ${idle_count} idle."

	# ------------------------------------------------------------------
	# 3. Classify every queued run. Nothing is cancelled in this pass, so
	# the state branch is not needed unless something lands in
	# `stuck_candidates`.
	# ------------------------------------------------------------------
	stuck_candidates="$(mktemp)"
	: > "${stuck_candidates}"

	# Reads the per-run loop variables directly. Called from FOUR branches:
	# healthy-queued and `busy` (the run can still proceed), not-idle (it
	# cannot), and dispatcher-stuck (it is about to be cancelled). Only the
	# last needs a disposition of its own. No-op when nothing starved.
	#
	# The disposition is the caller's to state, because it is the one thing
	# this function cannot know. It used to hard-code "Not dispatcher-stuck;
	# no action." for every caller, so a run with one starved label set and
	# another that IS dispatcher-stuck got two contradictory summary lines
	# under the same run id -- "no action" immediately above "queued for
	# cancel". A watchdog whose whole purpose is a verdict that matches what
	# it observed cannot print both. (Cursor Bugbot.)
	report_starvation() {
	[[ -z "${starved_kind}" ]] && return 0
	local detail
	local disposition="${1:-Not dispatcher-stuck; no action.}"
	if [[ "${starved_kind}" == "offline" ]]; then
	detail="every runner carrying [${starved_labels}] is registered but offline"
	else
	detail="no runner registered in any organization group carries [${starved_labels}]"
	fi
	log_summary "run ${run_id} (${run_path_base}, event=${run_event}): starved -- ${detail}. ${disposition}"
	echo "::warning::run ${run_id} (${run_path_base}) queued past ${MIN_QUEUE_AGE_SECONDS}s: ${detail}. Bring that runner online or fix the label set."
	printf '* run %s (%s, event=%s) -- %s -- %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${detail}" "${run_html_url}" >> "${starved_runs_file}"
	}

	for run_id in "${queued_ids[@]}"; do
	# Skip the watchdog's own run id (defense in depth - the watchdog
	# workflow file is excluded by name above, but if someone renames
	# the file or runs ad-hoc the id check still protects us).
	if [[ "${run_id}" == "${SELF_RUN_ID}" ]]; then
	log_summary "run ${run_id}: this is the watchdog's own run; skipping."
	continue
	fi

	# Pull the run metadata we need from the cached list (workflow
	# file path, event, head_branch).
	run_meta="$(
	jq -c --argjson id "${run_id}" '
	.[]
	\| select(.id == $id)
	\| {
	path: (.path // ""),
	event: (.event // ""),
	workflow_id: (.workflow_id // 0),
	head_branch: (.head_branch // ""),
	html_url: (.html_url // "")
	}
	' < "${queued_runs_json}" \| head -n 1
	)"
	if [[ -z "${run_meta}" ]]; then
	fail_closed "run ${run_id} appeared in the queued listing but carries no metadata; the queue cannot be evaluated."
	fi
	run_path="$(jq -r '.path' <<< "${run_meta}")"
	run_event="$(jq -r '.event' <<< "${run_meta}")"
	run_html_url="$(jq -r '.html_url' <<< "${run_meta}")"
	run_path_base="${run_path##*/}"

	# An empty subscript is a hard bash error, and `path: (.path // "")`
	# above already anticipates a null path. A run whose workflow file
	# cannot be named cannot be checked against the exclusion list, and
	# cancelling is the one irreversible thing this job does, so it is
	# reported and left alone rather than risked.
	if [[ -z "${run_path_base}" ]]; then
	log_summary "run ${run_id} (event=${run_event}): no workflow path reported; cannot check the exclusion list, so taking no action."
	echo "::warning::run ${run_id} reported no workflow path; the watchdog cannot classify it and will not cancel it."
	printf '* run %s (unknown workflow, event=%s) -- unclassifiable, left alone -- %s\n' "${run_id}" "${run_event}" "${run_html_url}" >> "${excluded_runs_file}"
	continue
	fi

	# Workflow-file exclusion (must NOT cancel release.yml).
	if [[ -n "${EXCLUDED_BY_FILE[${run_path_base}]+x}" ]]; then
	log_summary "run ${run_id} (${run_path_base}, event=${run_event}): workflow is excluded; operator action needed if stuck."
	printf '* run %s (%s, event=%s) -- %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${excluded_runs_file}"
	continue
	fi

	jobs_json="$(mktemp)"
	if ! gh api --paginate "repos/${REPO}/actions/runs/${run_id}/jobs?per_page=100" \
	\| jq -s '[.[] \| (.jobs // [])[]]' > "${jobs_json}"; then
	fail_closed "failed to list jobs for run ${run_id}; that run could not be evaluated."
	fi

	in_progress_count="$(jq '[ .[] \| select(.status == "in_progress") ] \| length' < "${jobs_json}")"
	queued_count="$(jq '[ .[] \| select(.status == "queued") ] \| length' < "${jobs_json}")"

	# Classification runs BEFORE the early exits, because starvation has
	# to be known on every path that reports one. The `in_progress`
	# exit used to `continue` before labels were evaluated at all, so
	# the moment a busy runner picked up the healthy cell the run became
	# in-progress and a co-resident starved job went silent again for
	# the rest of the matrix -- reopening the gap the `busy` fix had
	# just closed, in the state that lasts longest. (Cursor Bugbot.)
	#
	# The exits keep their original order and messages below; only the
	# point at which `starved_kind` becomes known has moved.
	#
	# Resolve each queued job's label set against the inventory. Only
	# `idle` is dispatcher-stuck; the other three verdicts are the
	# states a naive "no matching idle runner" test conflates, and
	# cancelling any of them would destroy work the fleet still
	# intends to run:
	# idle - an online, not-busy runner carries every label
	# busy - a matching runner is online but working
	# offline - a matching runner is registered but disconnected
	# unregistered - no registered runner carries the label set
	# `unregistered` splits again on whether the job asked for
	# `self-hosted`. The inventory covers only self-hosted runners, so
	# a GitHub-hosted job (`ubuntu-latest`) is unregistered by
	# definition and says nothing about our fleet; reporting it as
	# starved would fire a warning on every queued hosted run.
	# An empty label set would vacuously satisfy `all`, matching every
	# runner and manufacturing a dispatcher-stuck verdict, so a job
	# whose labels GitHub did not report is dropped rather than
	# trusted.
	if ! queued_job_labels_raw="$(jq -c '
	[ .[] \| select(.status == "queued") \| [ (.labels // [])[] \| ascii_downcase ] ]
	\| map(select(length > 0))
	\| unique
	\| .[]
	' < "${jobs_json}")"; then
	fail_closed "could not parse the job labels for run ${run_id}; that run could not be evaluated."
	fi
	queued_job_labels=()
	if [[ -n "${queued_job_labels_raw}" ]]; then
	mapfile -t queued_job_labels <<< "${queued_job_labels_raw}"
	fi
	# Deferred, not `continue`d: a run can have BOTH an in-progress job
	# and unlabelled queued jobs, and the in-progress verdict is the one
	# that describes it. Recording the condition here keeps the exit
	# order below identical to what it was before classification moved.
	labels_not_usable=0
	if (( queued_count == 0 )) \|\| [[ ${#queued_job_labels[@]} -eq 0 ]]; then
	labels_not_usable=1
	fi

	# `match_kind` is the best verdict across ALL of the run's label
	# sets and decides the action. `starved_*` is tracked SEPARATELY and
	# only from self-hosted sets, because those are the only ones this
	# inventory can speak to. Deriving the report from "the first set"
	# while deciding from "all sets" made both wrong: a GitHub-hosted
	# set sorting first suppressed a real starvation warning entirely,
	# and when it did warn it named the hosted label rather than the
	# self-hosted one, pointing the operator at a machine that does not
	# exist.
	match_kind="unregistered"
	starved_kind=""
	starved_labels=""
	for labels_csv in "${queued_job_labels[@]+"${queued_job_labels[@]}"}"; do
	kind="$(jq -r --argjson labels "${labels_csv}" '
	def satisfies($r): $labels \| all(. as $l \| $r.labels \| index($l) \| type == "number");
	if any(.[]; .online and (.busy \| not) and satisfies(.)) then "idle"
	elif any(.[]; .online and satisfies(.)) then "busy"
	elif any(.[]; satisfies(.)) then "offline"
	else "unregistered" end
	' < "${inventory_json}")"
	# Precedence idle > busy > offline > unregistered. The loop does
	# NOT break on `idle`: breaking meant the remaining label sets were
	# never scanned, so a run with one dispatchable cell and one
	# starved cell was cancelled with no starvation warning at all --
	# and because the stuck sibling keeps matching idle, that starved
	# cell could stay invisible across repeated cancel cycles. Scanning
	# every set costs one jq call per set and is what lets the verdict
	# and the starvation report be independent. (Cursor Bugbot.)
	if [[ "${kind}" == "idle" ]]; then
	match_kind="idle"
	elif [[ "${kind}" == "busy" && "${match_kind}" != "idle" ]]; then
	match_kind="busy"
	elif [[ "${kind}" == "offline" && "${match_kind}" == "unregistered" ]]; then
	match_kind="offline"
	fi
	# Only a self-hosted set can starve on OUR fleet. Prefer reporting
	# an `unregistered` set over an `offline` one: a label set nothing
	# carries needs a human, while an offline machine may come back.
	if jq -e 'index("self-hosted") \| type == "number"' <<< "${labels_csv}" > /dev/null; then
	if [[ "${kind}" == "offline" \|\| "${kind}" == "unregistered" ]]; then
	if [[ -z "${starved_kind}" ]] \
	\|\| { [[ "${starved_kind}" == "offline" ]] && [[ "${kind}" == "unregistered" ]]; }; then
	starved_kind="${kind}"
	starved_labels="$(jq -r 'join(", ")' <<< "${labels_csv}")"
	fi
	fi
	fi
	done

	if (( in_progress_count > 0 )); then
	# A run with even one in-progress job is by definition not
	# dispatcher-stuck. This covers matrix cells or later jobs
	# waiting while another job from the same run is active, and the
	# general case of a run that has at least one runner. It is still
	# not a reason to hide a sibling nothing can pick up.
	log_summary "run ${run_id} (${run_path_base}, event=${run_event}): healthy queued (${in_progress_count} in_progress, ${queued_count} queued -- waiting on concurrency/matrix slot)."
	printf '* run %s (%s, event=%s) -- %d in_progress, %d queued\n' "${run_id}" "${run_path_base}" "${run_event}" "${in_progress_count}" "${queued_count}" >> "${healthy_runs_file}"
	report_starvation
	continue
	fi

	if (( queued_count == 0 )); then
	# No queued jobs at all - the run is in some other transitional
	# state, not the dispatcher-stuck pattern.
	log_summary "run ${run_id} (${run_path_base}, event=${run_event}): no queued jobs yet (early state); skipping."
	continue
	fi

	if (( labels_not_usable == 1 )); then
	log_summary "run ${run_id} (${run_path_base}, event=${run_event}): queued jobs report no labels; not evaluable. No action."
	continue
	fi

	# A busy sibling does not make a starved sibling less starved. The
	# `busy` verdict used to `continue` without consulting `starved_kind`,
	# so a matrix run with one cell queued behind a busy runner and
	# another needing an offline or unregistered one reported only
	# "healthy backpressure" -- no warning, no Starved row -- until every
	# busy leg finished, which on Unity is hours. That is the #328
	# blindness in miniature: a starved job invisible because a sibling
	# looks fine. (Cursor Bugbot.)
	if [[ "${match_kind}" == "busy" ]]; then
	log_summary "run ${run_id} (${run_path_base}, event=${run_event}): every matching runner is online but busy; healthy backpressure. No action."
	printf '* run %s (%s, event=%s) -- waiting on a busy matching runner\n' "${run_id}" "${run_path_base}" "${run_event}" >> "${healthy_runs_file}"
	report_starvation
	continue
	fi

	# Reaching here implies no queued job asked for `self-hosted`:
	# a self-hosted set that is idle breaks out above, one that is busy
	# returns above, and one that is offline or unregistered sets
	# `starved_kind`. So the only surviving shape is a run this
	# inventory genuinely cannot speak to.
	#
	# The old wording claimed such a run "targets GitHub-hosted
	# capacity", which is not the same statement and can be false: a
	# registered runner may carry `ubuntu-latest` and simply be offline.
	# What is always true is that no queued job ASKED for self-hosted.
	if [[ "${match_kind}" != "idle" && -z "${starved_kind}" ]]; then
	log_summary "run ${run_id} (${run_path_base}): no queued job requests a self-hosted runner; not evaluable. No action."
	continue
	fi

	if [[ "${match_kind}" != "idle" ]]; then
	report_starvation
	continue
	fi

	# Log the verdict HERE, not only when the cancel succeeds. Classification
	# and action are separated by the state-branch materialization, which can
	# fail closed -- and then the summary said "a stuck run is pending" while
	# the Stuck section sat empty, so an operator could not tell WHICH run
	# needed attention. Every other verdict already announces itself at the
	# point it is reached; this one did not. (Cursor Bugbot.)
	# A cancellable sibling does not make a starved one less starved,
	# and this is the branch that acts, so it is the one most likely to
	# be read.
	report_starvation "Another label set in this run IS dispatcher-stuck; the run is queued for cancel below."
	log_summary "run ${run_id} (${run_path_base}, event=${run_event}): dispatcher-stuck; queued for cancel -- ${run_html_url}"
	printf '%s\n' "${run_meta}" \| jq -c --argjson id "${run_id}" '. + {id: $id}' >> "${stuck_candidates}"
	done

	if [[ ! -s "${stuck_candidates}" ]]; then
	log_summary "No dispatcher-stuck run found. No cancel issued."
	finish 0
	fi

	# ------------------------------------------------------------------
	# 4. Materialize the cancel-cap state branch. Only reached when a
	# cancel is actually about to be issued, so a state-branch problem
	# can no longer fail a cycle that had nothing to do -- and when it
	# IS needed, an unreadable cap means we must not cancel blind.
	# ------------------------------------------------------------------
	work_dir="$(mktemp -d)"
	pushd "${work_dir}" > /dev/null
	# Scope credentials to the exact remote commands that need them; the
	# cloned repo keeps a plain HTTPS origin with no tokenized remote.
	auth_header="$(printf 'x-access-token:%s' "${GH_TOKEN}" \| base64 \| tr -d '\n')"
	git_auth() {
	git -c "http.https://github.com/.extraheader=AUTHORIZATION: basic ${auth_header}" "$@"
	}
	remote_url="https://github.com/${REPO}.git"
	GIT_AUTHOR_ID=(-c "user.email=actions@github.com" -c "user.name=stuck-job-watchdog")

	# Never materialize the default branch. The watchdog reads and writes
	# only ${STATE_DIR} on ${STATE_BRANCH}, and checking the default branch
	# out just to switch away from it is what failed 13 runs straight on
	# 2026-07-29: one committed blob whose bytes disagreed with its
	# .gitattributes eol left the pristine clone content-dirty (git skips
	# the smudge when the blob already holds CRLF, so the file matched the
	# blob while the clean filter hashed to something else), and switching
	# trees aborted with "Your local changes to the following files would be
	# overwritten by checkout". Cloning the state branch directly retires
	# the whole class: no default-branch file is ever written, so none can
	# block the switch. ci.yml's `line-endings` job gates the drift itself.
	#
	# Decoupled probe + bootstrap: ls-remote distinguishes
	# "branch missing" (zero rows + exit 0) from "transient fetch
	# failure" (non-zero exit). We MUST NOT bootstrap on transient
	# failure - that would push-corrupt an existing branch by
	# rewriting it as a fresh orphan.
	state_branch_probe="$(mktemp)"
	if ! git_auth ls-remote --heads "${remote_url}" "${STATE_BRANCH}" > "${state_branch_probe}" 2>/dev/null; then
	fail_closed "'git ls-remote --heads ${STATE_BRANCH}' failed; the cancel cap is unreadable and a stuck run is pending."
	fi

	if [[ -s "${state_branch_probe}" ]]; then
	if ! git_auth clone --depth 1 --single-branch --branch "${STATE_BRANCH}" "${remote_url}" repo; then
	fail_closed "state branch '${STATE_BRANCH}' exists per ls-remote but the clone failed; the cancel cap is unreadable."
	fi
	cd repo
	log_summary "State branch '${STATE_BRANCH}' checked out."
	else
	log_summary "State branch '${STATE_BRANCH}' missing -- bootstrapping orphan branch."
	mkdir repo
	cd repo
	git init -q
	git checkout -q -b "${STATE_BRANCH}"
	git remote add origin "${remote_url}"
	mkdir -p "${STATE_DIR}"
	touch "${STATE_DIR}/.gitkeep"
	git add "${STATE_DIR}/.gitkeep"
	git "${GIT_AUTHOR_ID[@]}" commit -m "Initialize watchdog state" \|\| true
	if ! git_auth push origin "${STATE_BRANCH}"; then
	fail_closed "bootstrap push of '${STATE_BRANCH}' failed; the cancel cap cannot be persisted."
	fi
	fi
	mkdir -p "${STATE_DIR}"
	state_dirty=0

	persist_state_changes() {
	local reason="${1:-state update}"
	if (( state_dirty != 1 )); then
	return 0
	fi

	git add "${STATE_DIR}"
	if git diff-index --cached --quiet HEAD --; then
	if git rev-parse --verify "refs/remotes/origin/${STATE_BRANCH}" > /dev/null 2>&1 \
	&& [[ "$(git rev-parse HEAD)" == "$(git rev-parse "refs/remotes/origin/${STATE_BRANCH}")" ]]; then
	state_dirty=0
	return 0
	fi
	else
	if ! git "${GIT_AUTHOR_ID[@]}" commit -m "Watchdog: ${reason} at $(date -u +'%Y-%m-%dT%H:%M:%SZ')"; then
	log_summary "WARN: state commit failed; counters may double-count next run."
	return 1
	fi
	fi

	if git_auth push origin "${STATE_BRANCH}"; then
	log_summary "State branch updated (${reason})."
	state_dirty=0
	return 0
	fi

	log_summary "WARN: state push failed; attempting fetch + rebase + retry."
	if git_auth fetch origin "+${STATE_BRANCH}:refs/remotes/origin/${STATE_BRANCH}" \
	&& git rebase "refs/remotes/origin/${STATE_BRANCH}" \
	&& git_auth push origin "${STATE_BRANCH}"; then
	log_summary "State branch updated on second attempt (${reason}, after rebase)."
	state_dirty=0
	return 0
	fi

	log_summary "WARN: state push failed twice; counters may double-count next run."
	return 1
	}

	is_nonnegative_integer() {
	[[ "${1:-}" =~ ^[0-9]+$ ]]
	}

	# ------------------------------------------------------------------
	# 5. Cancel each dispatcher-stuck run, capped at MAX_CANCELS_PER_DAY
	# per run-id, then re-dispatch where a safe path exists.
	# ------------------------------------------------------------------
	# mapfile, not `while read < file`: `gh run cancel` inherits the
	# loop's stdin and would swallow the remaining candidates.
	mapfile -t stuck_candidate_lines < "${stuck_candidates}"
	for candidate in "${stuck_candidate_lines[@]}"; do
	run_id="$(jq -r '.id' <<< "${candidate}")"
	run_event="$(jq -r '.event' <<< "${candidate}")"
	run_workflow_id="$(jq -r '.workflow_id' <<< "${candidate}")"
	run_head_branch="$(jq -r '.head_branch' <<< "${candidate}")"
	run_html_url="$(jq -r '.html_url' <<< "${candidate}")"
	run_path_base="$(jq -r '.path' <<< "${candidate}")"
	run_path_base="${run_path_base##*/}"

	state_file="${STATE_DIR}/${run_id}.json"
	cancels=0
	last_cancel=0
	if [[ -f "${state_file}" ]]; then
	state_ok=1
	state_content="$(cat "${state_file}" 2>/dev/null)" \|\| state_ok=0
	if [[ ${state_ok} -eq 1 ]]; then
	parsed_cancels="$(jq -r '.cancels // .reruns // 0' <<< "${state_content}" 2>/dev/null)" \|\| state_ok=0
	parsed_last="$(jq -r '.last_cancel // .last_rerun // 0' <<< "${state_content}" 2>/dev/null)" \|\| state_ok=0
	fi
	if [[ ${state_ok} -eq 1 ]]; then
	if is_nonnegative_integer "${parsed_cancels}" && is_nonnegative_integer "${parsed_last}"; then
	cancels="${parsed_cancels}"
	last_cancel="${parsed_last}"
	else
	state_ok=0
	fi
	fi
	if [[ ${state_ok} -ne 1 ]]; then
	log_summary "run ${run_id}: state file corrupt; resetting."
	else
	log_summary "run ${run_id}: loaded cancel state (cancels=${cancels}, last_cancel=${last_cancel})."
	fi
	fi
	# Reset counter if last action was >24h ago.
	if (( now_epoch - last_cancel > 86400 )); then
	cancels=0
	fi

	if (( cancels >= MAX_CANCELS_PER_DAY )); then
	log_summary "run ${run_id} (${run_path_base}, event=${run_event}): cancel cap (${MAX_CANCELS_PER_DAY}/24h) reached; skipping. Manual intervention required."
	printf '* run %s (%s, event=%s) -- cap reached, operator action: %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${stuck_runs_file}"
	continue
	fi

	log_summary "run ${run_id} (${run_path_base}, event=${run_event}): dispatcher-stuck; cancelling (attempt $((cancels + 1))/${MAX_CANCELS_PER_DAY})."
	if ! gh run cancel "${run_id}" --repo "${REPO}" 2>&1 \| tee -a "${summary_file}"; then
	log_summary "run ${run_id}: 'gh run cancel' failed; will try again next cycle."
	continue
	fi

	cancels=$((cancels + 1))
	printf '{"cancels": %d, "last_cancel": %d}\n' "${cancels}" "${now_epoch}" > "${state_file}"
	git add "${state_file}"
	state_dirty=1
	persist_state_changes "record cancel for run ${run_id}" \|\| true

	# Decide redispatch path based on event + workflow_dispatch trigger.
	workflow_def="$(mktemp)"
	workflow_supports_dispatch=0
	if gh api "repos/${REPO}/actions/workflows/${run_workflow_id}" > "${workflow_def}" 2>/dev/null; then
	workflow_path="$(jq -r '.path // ""' < "${workflow_def}")"
	if [[ -n "${workflow_path}" ]]; then
	# Look at the contents of the workflow file to see if it
	# declares `workflow_dispatch:` (the trigger we need to use
	# the dispatches REST API).
	workflow_file_raw="$(mktemp)"
	if gh api "repos/${REPO}/contents/${workflow_path}" --jq '.content' 2>/dev/null \
	\| base64 -d > "${workflow_file_raw}" 2>/dev/null; then
	if grep -Eq '^[[:space:]]*workflow_dispatch:' "${workflow_file_raw}"; then
	workflow_supports_dispatch=1
	fi
	fi
	fi
	fi

	case "${run_event}" in
	push\|schedule\|workflow_dispatch)
	if [[ -n "${run_head_branch}" && ${workflow_supports_dispatch} -eq 1 ]]; then
	log_summary "run ${run_id}: re-dispatching workflow ${run_workflow_id} on ref '${run_head_branch}'."
	if gh api -X POST "repos/${REPO}/actions/workflows/${run_workflow_id}/dispatches" \
	-f "ref=${run_head_branch}" 2>&1 \| tee -a "${summary_file}"; then
	printf '* run %s (%s, event=%s) -- cancelled and re-dispatched on %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_head_branch}" >> "${stuck_runs_file}"
	else
	log_summary "run ${run_id}: re-dispatch failed; operator action: ${run_html_url}"
	printf '* run %s (%s, event=%s) -- cancelled; re-dispatch FAILED, operator action: %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${stuck_runs_file}"
	fi
	else
	log_summary "run ${run_id}: workflow does not support workflow_dispatch on ref '${run_head_branch}'; operator action: click 'Re-run all jobs' at ${run_html_url}"
	printf \
	'* run %s (%s, event=%s) -- cancelled; operator action: click "Re-run all jobs" at %s\n' \
	"${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${stuck_runs_file}"
	fi
	;;
	pull_request\|pull_request_target)
	# No safe API path: the dispatches endpoint cannot re-trigger
	# a pull_request run, and pushing a no-op commit to the head
	# ref would tamper with the PR. Operator-visible cancel +
	# explicit step-summary instruction is the supported path.
	log_summary "run ${run_id}: pull_request-triggered; operator action: click 'Re-run all jobs' at ${run_html_url}"
	printf '* run %s (%s, event=%s) -- cancelled; operator action: click "Re-run all jobs" at %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${stuck_runs_file}"
	;;
	*)
	log_summary "run ${run_id}: event '${run_event}' has no automatic recovery path; operator action: click 'Re-run all jobs' at ${run_html_url}"
	printf '* run %s (%s, event=%s) -- cancelled; operator action: click "Re-run all jobs" at %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${stuck_runs_file}"
	;;
	esac
	done

	# ------------------------------------------------------------------
	# 6. Commit + push state changes with a single rebase+retry.
	# ------------------------------------------------------------------
	persist_state_changes "final state sync" \|\| true

	popd > /dev/null

	finish 0

Navigation Menu

Uh oh!

Stuck Job Watchdog #1969

Workflow file

Stuck Job Watchdog #1969

Uh oh!

Workflow file for this run