stuck-job-watchdog.yml

name: Stuck Job Watchdog

# Recovery automation for the known GitHub Actions self-hosted dispatcher bug
# documented at https://github.com/orgs/community/discussions/186811:
# self-hosted runners report Online/Idle but `runner_id` stays at 0 for
# 7+ minutes, so a queued job whose label set is satisfied by an idle runner
# nonetheless waits indefinitely until manually re-run.
#
# FAIL CLOSED (#328). A green run means the audit ran to completion. Every
# input the verdict depends on -- the queued-run listing, the organization
# runner inventory, a per-run job listing, and the cancel-cap state branch --
# fails the job when it cannot be read, instead of logging and exiting 0.
# Before this rule the inventory read was unauthorized on both endpoints the
# watchdog tried and the job still reported success, so a `Performance Numbers`
# run sat queued for ten hours while the automation meant to notice it reported
# green every five minutes.
#
# Runner inventory is ORGANIZATION-scoped and is read with the build-lock
# reader App (`organization_self_hosted_runners: read`), the same credential
# `check-unity-runner-availability` already uses. The job's own GITHUB_TOKEN is
# repository-scoped and can never reach `/orgs/{org}/actions/runner-groups`.
# The repository-level fallback this workflow used to attempt was worse than
# no fallback: this repository registers zero repository-level runners, so the
# call would succeed and report an empty inventory, which reads identically to
# "no runner matches" and still issues no action.
#
# The inventory walk enumerates ALL organization runner groups rather than
# filtering with `visible_to_repository`. That parameter is documented only as a
# "string" with no stated format, and the two plausible readings -- repository
# name and numeric repository id -- are not distinguishable from the docs. Guess
# wrong and the call either 4xxs or returns an empty list, and BOTH land in
# `fail_closed` below, so this job would fail every one of its 288 daily runs
# while reporting a credential or outage problem it does not have. The
# organization registers a single group ("Default") today, so the filter would
# select exactly what the unfiltered call already returns: it is a live failure
# risk in exchange for nothing.
#
# The group names are logged for the day that stops being true. Enumerating all
# groups over-counts capacity if a second, restricted group is ever added, and
# over-counting capacity is what could turn a legitimately queued run into a
# wrongful cancel -- so if a second name ever appears in the summary, narrow this
# walk before trusting it, using an id confirmed against a live response rather
# than against the documentation.
#
# Detection requires ALL of the following to be true:
#   * The workflow run is `status: queued` AND older than MIN_QUEUE_AGE_SECONDS
#     (default 300s / 5 min; round-2 false-positive guards below -- zero
#     in-progress jobs, self-run exclusion, excluded-workflow list -- make
#     the previous conservative 600s buffer unnecessary).
#   * No job in the run is `status: in_progress` - a run with even one
#     in-progress job is by definition holding/using a runner, not
#     dispatcher-stuck. This is what prevents false positives for matrix cells
#     or jobs waiting while another job from the same run is actively running.
#   * At least one job in the run is `status: queued`.
#   * At least one ONLINE, NOT-BUSY runner's labels satisfy a queued job's
#     label requirements (superset match, case-insensitive on both sides).
#   * The run's workflow file is NOT in the exclusion list (`release.yml`
#     is hard-excluded by default; additional entries may be added via the
#     `WATCHDOG_EXCLUDED_WORKFLOWS` repo variable, whitespace-separated).
#   * The run id is NOT the watchdog's own run id.
#
# A queued run that no ONLINE runner can satisfy is NOT dispatcher-stuck --
# cancelling it would destroy work the fleet will pick up as soon as the
# machine reconnects. It is reported in its own step-summary section and as a
# `::warning::` annotation, so an offline runner starving the queue is visible
# on the run page without opening a job log. That is the shape #328 hit: a
# registered-but-offline `fast` runner, invisible because the audit was blind.
#
# Recovery action (per cli/cli#9221 and the gh-run-rerun manual,
# `gh run rerun --failed` cannot be used on a `status: queued` run because
# the run never reached `failed`; the documented workaround is cancel +
# redispatch / cancel + operator-managed re-run):
#   * `gh run cancel <id>` kills the stuck dispatch.
#   * If the run was triggered by push/schedule/workflow_dispatch on a
#     branch (not a tag) AND the workflow file declares `workflow_dispatch:`,
#     re-dispatch via REST API
#     (`POST repos/{owner}/{repo}/actions/workflows/{workflow_id}/dispatches`).
#   * Otherwise (pull_request, tag, no workflow_dispatch trigger), emit a
#     clear `GITHUB_STEP_SUMMARY` line instructing the operator to click
#     "Re-run all jobs" in the GitHub UI. Do NOT push a commit, comment on
#     the PR, or escalate any other automatic action.
#
# Cancel attempts are capped at 2 per run-id per 24h via a small state file
# on the `watchdog-state` orphan branch. The state branch is materialized only
# after a run is classified dispatcher-stuck, so a cycle that finds nothing to
# cancel -- the overwhelming majority -- performs no clone and cannot be failed
# by a state-branch problem it never needed to solve. State is pushed
# immediately after each successful cancel, with a single rebase+retry on push
# failure.
#
# Workflow-level concurrency guarantees only one watchdog instance ever runs;
# newer schedules must not cancel an in-flight audit after it has cancelled a
# stuck run but before it persists the state-branch counter.

on:
  schedule:
    - cron: "*/5 * * * *"
  workflow_dispatch:

concurrency:
  group: stuck-job-watchdog
  cancel-in-progress: false

permissions:
  actions: write
  contents: write

jobs:
  audit-queue:
    name: Audit queued runs and recover dispatcher-stuck ones
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      # Fail closed by construction: this action errors when either secret is
      # empty, so a missing credential aborts the audit before it can report a
      # verdict it had no way to reach.
      - name: Mint the runner-inventory reader token
        id: reader_token
        timeout-minutes: 2
        uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3
        with:
          app-id: ${{ secrets.BUILD_LOCK_READER_APP_ID }}
          private-key: ${{ secrets.BUILD_LOCK_READER_APP_PRIVATE_KEY }}
          # Organization-scoped (no `repositories:`) because the runner-group
          # endpoints are organization endpoints, and narrowed to the single
          # permission this audit needs so the minted token can do nothing else.
          owner: ${{ github.repository_owner }}
          permission-organization-self-hosted-runners: read

      - name: Audit + cancel-and-redispatch
        shell: bash
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          RUNNER_INVENTORY_TOKEN: ${{ steps.reader_token.outputs.token }}
          REPO: ${{ github.repository }}
          REPO_ID: ${{ github.repository_id }}
          OWNER: ${{ github.repository_owner }}
          SELF_RUN_ID: ${{ github.run_id }}
          STATE_BRANCH: watchdog-state
          STATE_DIR: .watchdog-state
          MAX_CANCELS_PER_DAY: "2"
          MIN_QUEUE_AGE_SECONDS: "300"
          DEFAULT_EXCLUDED_WORKFLOWS: "release.yml"
          EXTRA_EXCLUDED_WORKFLOWS: ${{ vars.WATCHDOG_EXCLUDED_WORKFLOWS }}
        run: |
          set -euo pipefail

          summary_file="$(mktemp)"
          : > "${summary_file}"
          log_summary() {
            printf '%s\n' "$1" | tee -a "${summary_file}"
          }

          # Category buckets for the final step-summary table.
          healthy_runs_file="$(mktemp)"
          stuck_runs_file="$(mktemp)"
          excluded_runs_file="$(mktemp)"
          starved_runs_file="$(mktemp)"
          : > "${healthy_runs_file}"
          : > "${stuck_runs_file}"
          : > "${excluded_runs_file}"
          : > "${starved_runs_file}"

          summary_emitted=0
          deliberate_exit=0

          emit_summary() {
            if (( summary_emitted == 1 )); then
              return 0
            fi
            summary_emitted=1
            # `|| true`: this runs from an EXIT trap, so a failure here would
            # replace the status the job is actually exiting with.
            {
              echo "## Watchdog summary"
              cat "${summary_file}"
              echo ""
              echo "### Healthy queued (waiting on concurrency / matrix slot)"
              if [[ -s "${healthy_runs_file}" ]]; then cat "${healthy_runs_file}"; else echo "_(none)_"; fi
              echo ""
              echo "### Stuck (auto-cancelled)"
              if [[ -s "${stuck_runs_file}" ]]; then cat "${stuck_runs_file}"; else echo "_(none)_"; fi
              echo ""
              echo "### Starved (no online runner carries the required labels)"
              if [[ -s "${starved_runs_file}" ]]; then cat "${starved_runs_file}"; else echo "_(none)_"; fi
              echo ""
              echo "### Stuck but excluded (operator action needed)"
              if [[ -s "${excluded_runs_file}" ]]; then cat "${excluded_runs_file}"; else echo "_(none)_"; fi
            } >> "${GITHUB_STEP_SUMMARY}" || true
          }

          finish() {
            deliberate_exit=1
            emit_summary
            exit "${1:-0}"
          }

          # Without this, ANY abort `set -e` catches outside the enumerated
          # paths -- and this is 500 lines of bash running 288 times a day --
          # produces a red run with a completely empty step summary and no
          # annotation, which is the worst signal an operator can be handed.
          # The trap makes every exit emit the buckets, and labels the ones
          # that did not come from `finish` so an unexpected abort is not
          # mistaken for a considered verdict.
          # Invoked indirectly by the EXIT trap below, so static analysis cannot
          # see the call. Both codes are needed because the version numbering
          # differs: 0.9 flags the body as unreachable (SC2317), while newer
          # builds -- including the one inside the actionlint container CI runs
          # -- flag the function as never invoked (SC2329). Note that a comment
          # line may not BEGIN with the analyzer's own name, or it is parsed as
          # a malformed directive and the whole script stops being checked.
          # shellcheck disable=SC2317,SC2329
          on_exit() {
            local code=$?
            if (( code != 0 )) && (( deliberate_exit == 0 )); then
              log_summary "ERROR: the audit aborted unexpectedly (exit ${code}); the queue was not fully evaluated."
              echo "::error::stuck-job watchdog aborted unexpectedly (exit ${code}); see the job log."
            fi
            emit_summary
          }
          trap on_exit EXIT

          # The audit could not answer. Never exit 0 from here: a green run has
          # to mean the queue was evaluated, not that evaluation was skipped.
          fail_closed() {
            log_summary "ERROR: ${1}"
            log_summary "The audit could not evaluate the queue, so this run fails rather than reporting a verdict it did not reach."
            echo "::error::stuck-job watchdog could not complete its audit: ${1}"
            finish 1
          }

          log_summary "## Stuck-job watchdog audit ($(date -u +'%Y-%m-%dT%H:%M:%SZ'))"
          log_summary "Repo: ${REPO}"
          log_summary "Owner: ${OWNER}"
          log_summary "Self run id (will skip): ${SELF_RUN_ID}"

          # Build the workflow-file exclusion list (default + repo variable).
          declare -A EXCLUDED_BY_FILE=()
          for wf in ${DEFAULT_EXCLUDED_WORKFLOWS} ${EXTRA_EXCLUDED_WORKFLOWS:-}; do
            [[ -z "${wf}" ]] && continue
            # Normalize: strip any leading .github/workflows/ if present.
            base="${wf##*/}"
            # An empty subscript is a hard bash error, and the strip above turns
            # any operator entry ending in `/` into one -- which the very next
            # comment invites by talking about directory prefixes. Guarding only
            # the read site left the write site able to kill the audit outright,
            # every five minutes, on a repo-variable typo.
            [[ -z "${base}" ]] && continue
            EXCLUDED_BY_FILE["${base}"]=1
          done
          excluded_list=""
          for k in "${!EXCLUDED_BY_FILE[@]}"; do
            excluded_list+="${k} "
          done
          log_summary "Excluded workflows: ${excluded_list:-<none>}"

          # ------------------------------------------------------------------
          # 1. Enumerate queued runs older than MIN_QUEUE_AGE_SECONDS.
          # ------------------------------------------------------------------
          now_epoch="$(date -u +%s)"
          queued_runs_json="$(mktemp)"
          # `gh api --paginate` over the runs endpoint returns one JSON object
          # per page; `jq -s '[.[] | .workflow_runs[]?]'` flattens all pages
          # into a single array. Equivalent to using `--slurp` on newer gh
          # versions but works on all gh versions shipped with ubuntu-latest.
          if ! gh api --paginate "repos/${REPO}/actions/runs?status=queued&per_page=100" \
              | jq -s '[.[] | (.workflow_runs // [])[]]' > "${queued_runs_json}"; then
            fail_closed "failed to list queued runs for ${REPO}."
          fi

          # `mapfile < <(jq ...)` throws the process substitution's exit status
          # away, and `pipefail` does not reach inside one. A single unparseable
          # `created_at` would abort jq, yield an empty list, and print
          # "Queue is clean" over a genuinely stuck queue -- the exact fail-open
          # this rewrite exists to remove. Capture, check, then split.
          if ! queued_ids_raw="$(jq -r --argjson now "${now_epoch}" --argjson min "${MIN_QUEUE_AGE_SECONDS}" '
            .[]
            | select(.created_at != null)
            | (.created_at | fromdateiso8601) as $created
            | select(($now - $created) >= $min)
            | .id
          ' < "${queued_runs_json}" | sort -u)"; then
            fail_closed "could not parse the queued-run listing for ${REPO}; the queue cannot be evaluated."
          fi
          queued_ids=()
          if [[ -n "${queued_ids_raw}" ]]; then
            mapfile -t queued_ids <<< "${queued_ids_raw}"
          fi

          log_summary "Queued runs older than ${MIN_QUEUE_AGE_SECONDS}s: ${#queued_ids[@]}"
          if [[ ${#queued_ids[@]} -eq 0 ]]; then
            log_summary "Queue is clean. No action."
            finish 0
          fi

          # ------------------------------------------------------------------
          # 2. Read the organization runner inventory through the reader App.
          #    Fail closed on any failure:
          #    an unreadable inventory cannot distinguish "nothing is stuck"
          #    from "everything is stuck and we cannot see it".
          # ------------------------------------------------------------------
          gh_reader() {
            GH_TOKEN="${RUNNER_INVENTORY_TOKEN}" gh "$@"
          }

          # `unique_by(.id)`: `--paginate` concatenates page objects, and the
          # live organization returns "Default" twice through this walk. The
          # runner dedupe below already made that harmless for the COUNTS, but
          # it still cost a redundant API call per duplicate and printed a
          # group list that made an operator distrust the summary.
          runner_groups_json="$(mktemp)"
          if ! gh_reader api --paginate \
              "orgs/${OWNER}/actions/runner-groups?per_page=100" \
              | jq -s '[.[] | (.runner_groups // [])[] | select(.id != null)] | unique_by(.id)' > "${runner_groups_json}"; then
            fail_closed "could not read the organization runner groups."
          fi
          group_count="$(jq 'length' < "${runner_groups_json}")"
          if (( group_count == 0 )); then
            fail_closed "the organization reported no runner groups; the inventory cannot be trusted."
          fi
          log_summary "Organization runner groups: $(jq -r '[.[].name] | join(", ")' < "${runner_groups_json}")"

          # Only groups THIS repository may actually use. Counting a runner that
          # cannot serve our jobs inflates capacity, and inflated capacity is
          # what turns a legitimately queued run into a wrongful cancel: the
          # audit would see an idle runner carrying the right labels, conclude
          # the dispatcher is stuck, and cancel a run that was correctly waiting.
          #
          # This reads each group's documented `visibility` field rather than the
          # `visible_to_repository` query parameter, which the REST docs type
          # only as "string" with no stated format. `all` needs no further call;
          # `selected` (and `private`) are resolved against the group's own
          # repositories endpoint, which returns repository objects with numeric
          # ids -- an unambiguous comparison against `github.repository_id`.
          #
          # The previous revision walked every group on the stated grounds that
          # the organization had exactly one. It has at least two -- `Default`
          # and `ambiguous-interactive-organization-builds` -- so that premise
          # was simply wrong, and the risk documented in #335 was live rather
          # than hypothetical.
          # `!= false`, NOT `// true`: jq's `//` treats an explicit `false` as
          # absent and yields the right-hand side, so a group that genuinely
          # refuses public repositories would read as allowing them -- the
          # exact inversion of the check. Caught by the fixture for it.
          if ! group_rows_raw="$(jq -r '.[] | "\(.id)\t\(.visibility // "all")\t\(.allows_public_repositories != false)\t\(.name)"' < "${runner_groups_json}")"; then
            fail_closed "could not parse the organization runner-group inventory."
          fi
          group_rows=()
          if [[ -n "${group_rows_raw}" ]]; then
            mapfile -t group_rows <<< "${group_rows_raw}"
          fi
          # mapfile, not `while read < <(...)`: the loop body runs `gh`, and a
          # child that consumes the loop's stdin silently eats the remaining
          # group ids.
          runner_group_ids=()
          excluded_groups=()
          group_repos_json="$(mktemp)"
          for group_row in "${group_rows[@]}"; do
            IFS=$'\t' read -r group_id group_visibility group_public group_name <<< "${group_row}"
            # A group can be `visibility: all` and still refuse PUBLIC
            # repositories, and this repository is public -- its runners are
            # then unusable capacity, which is the wrongful cancel this walk
            # exists to prevent. (Cursor Bugbot.)
            #
            # Excluded unconditionally rather than only when we know we are
            # public, which would need another API call and another failure
            # mode. The cost of being wrong is asymmetric: for a PRIVATE
            # repository this under-counts capacity, and under-counting can only
            # produce a starvation report, never a cancel. Over-counting is what
            # cancels a live run.
            if [[ "${group_public}" != "true" ]]; then
              excluded_groups+=("${group_name} (refuses public repositories)")
              continue
            fi
            if [[ "${group_visibility}" == "all" ]]; then
              runner_group_ids+=("${group_id}")
              continue
            fi
            # `private` means "every PRIVATE repository in the organization" --
            # not a selected list. This repository is public, so such a group can
            # never serve it. Excluded here rather than fetching its repositories
            # endpoint, which would answer a question that does not apply and
            # could fail the whole audit closed on an error it did not need to
            # make. (Cursor Bugbot.)
            if [[ "${group_visibility}" == "private" ]]; then
              excluded_groups+=("${group_name} (private repositories only)")
              continue
            fi
            if ! gh_reader api --paginate \
                "orgs/${OWNER}/actions/runner-groups/${group_id}/repositories?per_page=100" \
                | jq -s '[.[] | (.repositories // [])[] | .id | tostring]' > "${group_repos_json}"; then
              fail_closed "could not read which repositories may use runner group ${group_name} (${group_id})."
            fi
            # `--arg`, not `--argjson`: an empty or non-numeric REPO_ID makes
            # `--argjson` fail to parse, which under `set -e` aborts the script
            # WITHOUT reaching `fail_closed` -- an audit that dies instead of
            # reporting. Both sides are compared as strings. (GitHub Copilot.)
            if [[ "$(jq --arg want "${REPO_ID}" 'any(.[]; . == $want)' < "${group_repos_json}")" == "true" ]]; then
              runner_group_ids+=("${group_id}")
            else
              excluded_groups+=("${group_name}")
            fi
          done
          if (( ${#excluded_groups[@]} > 0 )); then
            # NOT `IFS=', '` with `"${arr[*]}"`: bash joins with only the FIRST
            # character of IFS there, so that renders `a,b`. (GitHub Copilot.)
            log_summary "Runner groups not usable by ${REPO} (excluded from capacity): $(printf '%s, ' "${excluded_groups[@]}" | sed 's/, $//')"
          fi
          if (( ${#runner_group_ids[@]} == 0 )); then
            fail_closed "no organization runner group is visible to ${REPO}; the inventory cannot be trusted."
          fi
          runners_json="$(mktemp)"
          : > "${runners_json}"
          group_runners_json="$(mktemp)"
          for group_id in "${runner_group_ids[@]}"; do
            if ! gh_reader api --paginate \
                "orgs/${OWNER}/actions/runner-groups/${group_id}/runners?per_page=100" \
                | jq -s '[.[] | (.runners // [])[]]' > "${group_runners_json}"; then
              fail_closed "could not read runners in organization runner group ${group_id}."
            fi
            cat "${group_runners_json}" >> "${runners_json}"
          done

          # Labels are compared case-insensitively in both directions: a job
          # declaring `windows` and a runner registered as `Windows` are the
          # same requirement, and an exact match would silently report every
          # such run as unmatched.
          inventory_json="$(mktemp)"
          jq -s '
            [ .[][]
              | { id: .id,
                  name: .name,
                  online: (.status == "online"),
                  busy: (.busy == true),
                  labels: ([(.labels // [])[] | (.name // "") | ascii_downcase | select(. != "")]) }
            ]
            | unique_by(.id)
          ' < "${runners_json}" > "${inventory_json}"

          registered_count="$(jq 'length' < "${inventory_json}")"
          online_count="$(jq '[.[] | select(.online)] | length' < "${inventory_json}")"
          idle_count="$(jq '[.[] | select(.online and (.busy | not))] | length' < "${inventory_json}")"
          log_summary "Runner inventory (groups visible to ${REPO}): ${registered_count} registered, ${online_count} online, ${idle_count} idle."

          # ------------------------------------------------------------------
          # 3. Classify every queued run. Nothing is cancelled in this pass, so
          #    the state branch is not needed unless something lands in
          #    `stuck_candidates`.
          # ------------------------------------------------------------------
          stuck_candidates="$(mktemp)"
          : > "${stuck_candidates}"

          # Reads the per-run loop variables directly. Called from FOUR branches:
          # healthy-queued and `busy` (the run can still proceed), not-idle (it
          # cannot), and dispatcher-stuck (it is about to be cancelled). Only the
          # last needs a disposition of its own. No-op when nothing starved.
          #
          # The disposition is the caller's to state, because it is the one thing
          # this function cannot know. It used to hard-code "Not dispatcher-stuck;
          # no action." for every caller, so a run with one starved label set and
          # another that IS dispatcher-stuck got two contradictory summary lines
          # under the same run id -- "no action" immediately above "queued for
          # cancel". A watchdog whose whole purpose is a verdict that matches what
          # it observed cannot print both. (Cursor Bugbot.)
          report_starvation() {
            [[ -z "${starved_kind}" ]] && return 0
            local detail
            local disposition="${1:-Not dispatcher-stuck; no action.}"
            if [[ "${starved_kind}" == "offline" ]]; then
              detail="every runner carrying [${starved_labels}] is registered but offline"
            else
              detail="no runner registered in any organization group carries [${starved_labels}]"
            fi
            log_summary "run ${run_id} (${run_path_base}, event=${run_event}): starved -- ${detail}. ${disposition}"
            echo "::warning::run ${run_id} (${run_path_base}) queued past ${MIN_QUEUE_AGE_SECONDS}s: ${detail}. Bring that runner online or fix the label set."
            printf '* run %s (%s, event=%s) -- %s -- %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${detail}" "${run_html_url}" >> "${starved_runs_file}"
          }

          for run_id in "${queued_ids[@]}"; do
            # Skip the watchdog's own run id (defense in depth - the watchdog
            # workflow file is excluded by name above, but if someone renames
            # the file or runs ad-hoc the id check still protects us).
            if [[ "${run_id}" == "${SELF_RUN_ID}" ]]; then
              log_summary "run ${run_id}: this is the watchdog's own run; skipping."
              continue
            fi

            # Pull the run metadata we need from the cached list (workflow
            # file path, event, head_branch).
            run_meta="$(
              jq -c --argjson id "${run_id}" '
                .[]
                | select(.id == $id)
                | {
                    path: (.path // ""),
                    event: (.event // ""),
                    workflow_id: (.workflow_id // 0),
                    head_branch: (.head_branch // ""),
                    html_url: (.html_url // "")
                  }
              ' < "${queued_runs_json}" | head -n 1
            )"
            if [[ -z "${run_meta}" ]]; then
              fail_closed "run ${run_id} appeared in the queued listing but carries no metadata; the queue cannot be evaluated."
            fi
            run_path="$(jq -r '.path' <<< "${run_meta}")"
            run_event="$(jq -r '.event' <<< "${run_meta}")"
            run_html_url="$(jq -r '.html_url' <<< "${run_meta}")"
            run_path_base="${run_path##*/}"

            # An empty subscript is a hard bash error, and `path: (.path // "")`
            # above already anticipates a null path. A run whose workflow file
            # cannot be named cannot be checked against the exclusion list, and
            # cancelling is the one irreversible thing this job does, so it is
            # reported and left alone rather than risked.
            if [[ -z "${run_path_base}" ]]; then
              log_summary "run ${run_id} (event=${run_event}): no workflow path reported; cannot check the exclusion list, so taking no action."
              echo "::warning::run ${run_id} reported no workflow path; the watchdog cannot classify it and will not cancel it."
              printf '* run %s (unknown workflow, event=%s) -- unclassifiable, left alone -- %s\n' "${run_id}" "${run_event}" "${run_html_url}" >> "${excluded_runs_file}"
              continue
            fi

            # Workflow-file exclusion (must NOT cancel release.yml).
            if [[ -n "${EXCLUDED_BY_FILE[${run_path_base}]+x}" ]]; then
              log_summary "run ${run_id} (${run_path_base}, event=${run_event}): workflow is excluded; operator action needed if stuck."
              printf '* run %s (%s, event=%s) -- %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${excluded_runs_file}"
              continue
            fi

            jobs_json="$(mktemp)"
            if ! gh api --paginate "repos/${REPO}/actions/runs/${run_id}/jobs?per_page=100" \
                | jq -s '[.[] | (.jobs // [])[]]' > "${jobs_json}"; then
              fail_closed "failed to list jobs for run ${run_id}; that run could not be evaluated."
            fi

            in_progress_count="$(jq '[ .[] | select(.status == "in_progress") ] | length' < "${jobs_json}")"
            queued_count="$(jq '[ .[] | select(.status == "queued") ] | length' < "${jobs_json}")"

            # Classification runs BEFORE the early exits, because starvation has
            # to be known on every path that reports one. The `in_progress`
            # exit used to `continue` before labels were evaluated at all, so
            # the moment a busy runner picked up the healthy cell the run became
            # in-progress and a co-resident starved job went silent again for
            # the rest of the matrix -- reopening the gap the `busy` fix had
            # just closed, in the state that lasts longest. (Cursor Bugbot.)
            #
            # The exits keep their original order and messages below; only the
            # point at which `starved_kind` becomes known has moved.
            #
            # Resolve each queued job's label set against the inventory. Only
            # `idle` is dispatcher-stuck; the other three verdicts are the
            # states a naive "no matching idle runner" test conflates, and
            # cancelling any of them would destroy work the fleet still
            # intends to run:
            #   idle         - an online, not-busy runner carries every label
            #   busy         - a matching runner is online but working
            #   offline      - a matching runner is registered but disconnected
            #   unregistered - no registered runner carries the label set
            # `unregistered` splits again on whether the job asked for
            # `self-hosted`. The inventory covers only self-hosted runners, so
            # a GitHub-hosted job (`ubuntu-latest`) is unregistered by
            # definition and says nothing about our fleet; reporting it as
            # starved would fire a warning on every queued hosted run.
            # An empty label set would vacuously satisfy `all`, matching every
            # runner and manufacturing a dispatcher-stuck verdict, so a job
            # whose labels GitHub did not report is dropped rather than
            # trusted.
            if ! queued_job_labels_raw="$(jq -c '
              [ .[] | select(.status == "queued") | [ (.labels // [])[] | ascii_downcase ] ]
              | map(select(length > 0))
              | unique
              | .[]
            ' < "${jobs_json}")"; then
              fail_closed "could not parse the job labels for run ${run_id}; that run could not be evaluated."
            fi
            queued_job_labels=()
            if [[ -n "${queued_job_labels_raw}" ]]; then
              mapfile -t queued_job_labels <<< "${queued_job_labels_raw}"
            fi
            # Deferred, not `continue`d: a run can have BOTH an in-progress job
            # and unlabelled queued jobs, and the in-progress verdict is the one
            # that describes it. Recording the condition here keeps the exit
            # order below identical to what it was before classification moved.
            labels_not_usable=0
            if (( queued_count == 0 )) || [[ ${#queued_job_labels[@]} -eq 0 ]]; then
              labels_not_usable=1
            fi

            # `match_kind` is the best verdict across ALL of the run's label
            # sets and decides the action. `starved_*` is tracked SEPARATELY and
            # only from self-hosted sets, because those are the only ones this
            # inventory can speak to. Deriving the report from "the first set"
            # while deciding from "all sets" made both wrong: a GitHub-hosted
            # set sorting first suppressed a real starvation warning entirely,
            # and when it did warn it named the hosted label rather than the
            # self-hosted one, pointing the operator at a machine that does not
            # exist.
            match_kind="unregistered"
            starved_kind=""
            starved_labels=""
            for labels_csv in "${queued_job_labels[@]+"${queued_job_labels[@]}"}"; do
              kind="$(jq -r --argjson labels "${labels_csv}" '
                def satisfies($r): $labels | all(. as $l | $r.labels | index($l) | type == "number");
                if   any(.[]; .online and (.busy | not) and satisfies(.)) then "idle"
                elif any(.[]; .online and satisfies(.))                   then "busy"
                elif any(.[]; satisfies(.))                               then "offline"
                else "unregistered" end
              ' < "${inventory_json}")"
              # Precedence idle > busy > offline > unregistered. The loop does
              # NOT break on `idle`: breaking meant the remaining label sets were
              # never scanned, so a run with one dispatchable cell and one
              # starved cell was cancelled with no starvation warning at all --
              # and because the stuck sibling keeps matching idle, that starved
              # cell could stay invisible across repeated cancel cycles. Scanning
              # every set costs one jq call per set and is what lets the verdict
              # and the starvation report be independent. (Cursor Bugbot.)
              if [[ "${kind}" == "idle" ]]; then
                match_kind="idle"
              elif [[ "${kind}" == "busy" && "${match_kind}" != "idle" ]]; then
                match_kind="busy"
              elif [[ "${kind}" == "offline" && "${match_kind}" == "unregistered" ]]; then
                match_kind="offline"
              fi
              # Only a self-hosted set can starve on OUR fleet. Prefer reporting
              # an `unregistered` set over an `offline` one: a label set nothing
              # carries needs a human, while an offline machine may come back.
              if jq -e 'index("self-hosted") | type == "number"' <<< "${labels_csv}" > /dev/null; then
                if [[ "${kind}" == "offline" || "${kind}" == "unregistered" ]]; then
                  if [[ -z "${starved_kind}" ]] \
                      || { [[ "${starved_kind}" == "offline" ]] && [[ "${kind}" == "unregistered" ]]; }; then
                    starved_kind="${kind}"
                    starved_labels="$(jq -r 'join(", ")' <<< "${labels_csv}")"
                  fi
                fi
              fi
            done

            if (( in_progress_count > 0 )); then
              # A run with even one in-progress job is by definition not
              # dispatcher-stuck. This covers matrix cells or later jobs
              # waiting while another job from the same run is active, and the
              # general case of a run that has at least one runner. It is still
              # not a reason to hide a sibling nothing can pick up.
              log_summary "run ${run_id} (${run_path_base}, event=${run_event}): healthy queued (${in_progress_count} in_progress, ${queued_count} queued -- waiting on concurrency/matrix slot)."
              printf '* run %s (%s, event=%s) -- %d in_progress, %d queued\n' "${run_id}" "${run_path_base}" "${run_event}" "${in_progress_count}" "${queued_count}" >> "${healthy_runs_file}"
              report_starvation
              continue
            fi

            if (( queued_count == 0 )); then
              # No queued jobs at all - the run is in some other transitional
              # state, not the dispatcher-stuck pattern.
              log_summary "run ${run_id} (${run_path_base}, event=${run_event}): no queued jobs yet (early state); skipping."
              continue
            fi

            if (( labels_not_usable == 1 )); then
              log_summary "run ${run_id} (${run_path_base}, event=${run_event}): queued jobs report no labels; not evaluable. No action."
              continue
            fi

            # A busy sibling does not make a starved sibling less starved. The
            # `busy` verdict used to `continue` without consulting `starved_kind`,
            # so a matrix run with one cell queued behind a busy runner and
            # another needing an offline or unregistered one reported only
            # "healthy backpressure" -- no warning, no Starved row -- until every
            # busy leg finished, which on Unity is hours. That is the #328
            # blindness in miniature: a starved job invisible because a sibling
            # looks fine. (Cursor Bugbot.)
            if [[ "${match_kind}" == "busy" ]]; then
              log_summary "run ${run_id} (${run_path_base}, event=${run_event}): every matching runner is online but busy; healthy backpressure. No action."
              printf '* run %s (%s, event=%s) -- waiting on a busy matching runner\n' "${run_id}" "${run_path_base}" "${run_event}" >> "${healthy_runs_file}"
              report_starvation
              continue
            fi

            # Reaching here implies no queued job asked for `self-hosted`:
            # a self-hosted set that is idle breaks out above, one that is busy
            # returns above, and one that is offline or unregistered sets
            # `starved_kind`. So the only surviving shape is a run this
            # inventory genuinely cannot speak to.
            #
            # The old wording claimed such a run "targets GitHub-hosted
            # capacity", which is not the same statement and can be false: a
            # registered runner may carry `ubuntu-latest` and simply be offline.
            # What is always true is that no queued job ASKED for self-hosted.
            if [[ "${match_kind}" != "idle" && -z "${starved_kind}" ]]; then
              log_summary "run ${run_id} (${run_path_base}): no queued job requests a self-hosted runner; not evaluable. No action."
              continue
            fi

            if [[ "${match_kind}" != "idle" ]]; then
              report_starvation
              continue
            fi

            # Log the verdict HERE, not only when the cancel succeeds. Classification
            # and action are separated by the state-branch materialization, which can
            # fail closed -- and then the summary said "a stuck run is pending" while
            # the Stuck section sat empty, so an operator could not tell WHICH run
            # needed attention. Every other verdict already announces itself at the
            # point it is reached; this one did not. (Cursor Bugbot.)
            # A cancellable sibling does not make a starved one less starved,
            # and this is the branch that acts, so it is the one most likely to
            # be read.
            report_starvation "Another label set in this run IS dispatcher-stuck; the run is queued for cancel below."
            log_summary "run ${run_id} (${run_path_base}, event=${run_event}): dispatcher-stuck; queued for cancel -- ${run_html_url}"
            printf '%s\n' "${run_meta}" | jq -c --argjson id "${run_id}" '. + {id: $id}' >> "${stuck_candidates}"
          done

          if [[ ! -s "${stuck_candidates}" ]]; then
            log_summary "No dispatcher-stuck run found. No cancel issued."
            finish 0
          fi

          # ------------------------------------------------------------------
          # 4. Materialize the cancel-cap state branch. Only reached when a
          #    cancel is actually about to be issued, so a state-branch problem
          #    can no longer fail a cycle that had nothing to do -- and when it
          #    IS needed, an unreadable cap means we must not cancel blind.
          # ------------------------------------------------------------------
          work_dir="$(mktemp -d)"
          pushd "${work_dir}" > /dev/null
          # Scope credentials to the exact remote commands that need them; the
          # cloned repo keeps a plain HTTPS origin with no tokenized remote.
          auth_header="$(printf 'x-access-token:%s' "${GH_TOKEN}" | base64 | tr -d '\n')"
          git_auth() {
            git -c "http.https://github.com/.extraheader=AUTHORIZATION: basic ${auth_header}" "$@"
          }
          remote_url="https://github.com/${REPO}.git"
          GIT_AUTHOR_ID=(-c "user.email=actions@github.com" -c "user.name=stuck-job-watchdog")

          # Never materialize the default branch. The watchdog reads and writes
          # only ${STATE_DIR} on ${STATE_BRANCH}, and checking the default branch
          # out just to switch away from it is what failed 13 runs straight on
          # 2026-07-29: one committed blob whose bytes disagreed with its
          # .gitattributes eol left the pristine clone content-dirty (git skips
          # the smudge when the blob already holds CRLF, so the file matched the
          # blob while the clean filter hashed to something else), and switching
          # trees aborted with "Your local changes to the following files would be
          # overwritten by checkout". Cloning the state branch directly retires
          # the whole class: no default-branch file is ever written, so none can
          # block the switch. ci.yml's `line-endings` job gates the drift itself.
          #
          # Decoupled probe + bootstrap: ls-remote distinguishes
          # "branch missing" (zero rows + exit 0) from "transient fetch
          # failure" (non-zero exit). We MUST NOT bootstrap on transient
          # failure - that would push-corrupt an existing branch by
          # rewriting it as a fresh orphan.
          state_branch_probe="$(mktemp)"
          if ! git_auth ls-remote --heads "${remote_url}" "${STATE_BRANCH}" > "${state_branch_probe}" 2>/dev/null; then
            fail_closed "'git ls-remote --heads ${STATE_BRANCH}' failed; the cancel cap is unreadable and a stuck run is pending."
          fi

          if [[ -s "${state_branch_probe}" ]]; then
            if ! git_auth clone --depth 1 --single-branch --branch "${STATE_BRANCH}" "${remote_url}" repo; then
              fail_closed "state branch '${STATE_BRANCH}' exists per ls-remote but the clone failed; the cancel cap is unreadable."
            fi
            cd repo
            log_summary "State branch '${STATE_BRANCH}' checked out."
          else
            log_summary "State branch '${STATE_BRANCH}' missing -- bootstrapping orphan branch."
            mkdir repo
            cd repo
            git init -q
            git checkout -q -b "${STATE_BRANCH}"
            git remote add origin "${remote_url}"
            mkdir -p "${STATE_DIR}"
            touch "${STATE_DIR}/.gitkeep"
            git add "${STATE_DIR}/.gitkeep"
            git "${GIT_AUTHOR_ID[@]}" commit -m "Initialize watchdog state" || true
            if ! git_auth push origin "${STATE_BRANCH}"; then
              fail_closed "bootstrap push of '${STATE_BRANCH}' failed; the cancel cap cannot be persisted."
            fi
          fi
          mkdir -p "${STATE_DIR}"
          state_dirty=0

          persist_state_changes() {
            local reason="${1:-state update}"
            if (( state_dirty != 1 )); then
              return 0
            fi

            git add "${STATE_DIR}"
            if git diff-index --cached --quiet HEAD --; then
              if git rev-parse --verify "refs/remotes/origin/${STATE_BRANCH}" > /dev/null 2>&1 \
                  && [[ "$(git rev-parse HEAD)" == "$(git rev-parse "refs/remotes/origin/${STATE_BRANCH}")" ]]; then
                state_dirty=0
                return 0
              fi
            else
              if ! git "${GIT_AUTHOR_ID[@]}" commit -m "Watchdog: ${reason} at $(date -u +'%Y-%m-%dT%H:%M:%SZ')"; then
                log_summary "WARN: state commit failed; counters may double-count next run."
                return 1
              fi
            fi

            if git_auth push origin "${STATE_BRANCH}"; then
              log_summary "State branch updated (${reason})."
              state_dirty=0
              return 0
            fi

            log_summary "WARN: state push failed; attempting fetch + rebase + retry."
            if git_auth fetch origin "+${STATE_BRANCH}:refs/remotes/origin/${STATE_BRANCH}" \
                && git rebase "refs/remotes/origin/${STATE_BRANCH}" \
                && git_auth push origin "${STATE_BRANCH}"; then
              log_summary "State branch updated on second attempt (${reason}, after rebase)."
              state_dirty=0
              return 0
            fi

            log_summary "WARN: state push failed twice; counters may double-count next run."
            return 1
          }

          is_nonnegative_integer() {
            [[ "${1:-}" =~ ^[0-9]+$ ]]
          }

          # ------------------------------------------------------------------
          # 5. Cancel each dispatcher-stuck run, capped at MAX_CANCELS_PER_DAY
          #    per run-id, then re-dispatch where a safe path exists.
          # ------------------------------------------------------------------
          # mapfile, not `while read < file`: `gh run cancel` inherits the
          # loop's stdin and would swallow the remaining candidates.
          mapfile -t stuck_candidate_lines < "${stuck_candidates}"
          for candidate in "${stuck_candidate_lines[@]}"; do
            run_id="$(jq -r '.id' <<< "${candidate}")"
            run_event="$(jq -r '.event' <<< "${candidate}")"
            run_workflow_id="$(jq -r '.workflow_id' <<< "${candidate}")"
            run_head_branch="$(jq -r '.head_branch' <<< "${candidate}")"
            run_html_url="$(jq -r '.html_url' <<< "${candidate}")"
            run_path_base="$(jq -r '.path' <<< "${candidate}")"
            run_path_base="${run_path_base##*/}"

            state_file="${STATE_DIR}/${run_id}.json"
            cancels=0
            last_cancel=0
            if [[ -f "${state_file}" ]]; then
              state_ok=1
              state_content="$(cat "${state_file}" 2>/dev/null)" || state_ok=0
              if [[ ${state_ok} -eq 1 ]]; then
                parsed_cancels="$(jq -r '.cancels // .reruns // 0' <<< "${state_content}" 2>/dev/null)" || state_ok=0
                parsed_last="$(jq -r '.last_cancel // .last_rerun // 0' <<< "${state_content}" 2>/dev/null)" || state_ok=0
              fi
              if [[ ${state_ok} -eq 1 ]]; then
                if is_nonnegative_integer "${parsed_cancels}" && is_nonnegative_integer "${parsed_last}"; then
                  cancels="${parsed_cancels}"
                  last_cancel="${parsed_last}"
                else
                  state_ok=0
                fi
              fi
              if [[ ${state_ok} -ne 1 ]]; then
                log_summary "run ${run_id}: state file corrupt; resetting."
              else
                log_summary "run ${run_id}: loaded cancel state (cancels=${cancels}, last_cancel=${last_cancel})."
              fi
            fi
            # Reset counter if last action was >24h ago.
            if (( now_epoch - last_cancel > 86400 )); then
              cancels=0
            fi

            if (( cancels >= MAX_CANCELS_PER_DAY )); then
              log_summary "run ${run_id} (${run_path_base}, event=${run_event}): cancel cap (${MAX_CANCELS_PER_DAY}/24h) reached; skipping. Manual intervention required."
              printf '* run %s (%s, event=%s) -- cap reached, operator action: %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${stuck_runs_file}"
              continue
            fi

            log_summary "run ${run_id} (${run_path_base}, event=${run_event}): dispatcher-stuck; cancelling (attempt $((cancels + 1))/${MAX_CANCELS_PER_DAY})."
            if ! gh run cancel "${run_id}" --repo "${REPO}" 2>&1 | tee -a "${summary_file}"; then
              log_summary "run ${run_id}: 'gh run cancel' failed; will try again next cycle."
              continue
            fi

            cancels=$((cancels + 1))
            printf '{"cancels": %d, "last_cancel": %d}\n' "${cancels}" "${now_epoch}" > "${state_file}"
            git add "${state_file}"
            state_dirty=1
            persist_state_changes "record cancel for run ${run_id}" || true

            # Decide redispatch path based on event + workflow_dispatch trigger.
            workflow_def="$(mktemp)"
            workflow_supports_dispatch=0
            if gh api "repos/${REPO}/actions/workflows/${run_workflow_id}" > "${workflow_def}" 2>/dev/null; then
              workflow_path="$(jq -r '.path // ""' < "${workflow_def}")"
              if [[ -n "${workflow_path}" ]]; then
                # Look at the contents of the workflow file to see if it
                # declares `workflow_dispatch:` (the trigger we need to use
                # the dispatches REST API).
                workflow_file_raw="$(mktemp)"
                if gh api "repos/${REPO}/contents/${workflow_path}" --jq '.content' 2>/dev/null \
                    | base64 -d > "${workflow_file_raw}" 2>/dev/null; then
                  if grep -Eq '^[[:space:]]*workflow_dispatch:' "${workflow_file_raw}"; then
                    workflow_supports_dispatch=1
                  fi
                fi
              fi
            fi

            case "${run_event}" in
              push|schedule|workflow_dispatch)
                if [[ -n "${run_head_branch}" && ${workflow_supports_dispatch} -eq 1 ]]; then
                  log_summary "run ${run_id}: re-dispatching workflow ${run_workflow_id} on ref '${run_head_branch}'."
                  if gh api -X POST "repos/${REPO}/actions/workflows/${run_workflow_id}/dispatches" \
                      -f "ref=${run_head_branch}" 2>&1 | tee -a "${summary_file}"; then
                    printf '* run %s (%s, event=%s) -- cancelled and re-dispatched on %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_head_branch}" >> "${stuck_runs_file}"
                  else
                    log_summary "run ${run_id}: re-dispatch failed; operator action: ${run_html_url}"
                    printf '* run %s (%s, event=%s) -- cancelled; re-dispatch FAILED, operator action: %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${stuck_runs_file}"
                  fi
                else
                  log_summary "run ${run_id}: workflow does not support workflow_dispatch on ref '${run_head_branch}'; operator action: click 'Re-run all jobs' at ${run_html_url}"
                  printf \
                    '* run %s (%s, event=%s) -- cancelled; operator action: click "Re-run all jobs" at %s\n' \
                    "${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${stuck_runs_file}"
                fi
                ;;
              pull_request|pull_request_target)
                # No safe API path: the dispatches endpoint cannot re-trigger
                # a pull_request run, and pushing a no-op commit to the head
                # ref would tamper with the PR. Operator-visible cancel +
                # explicit step-summary instruction is the supported path.
                log_summary "run ${run_id}: pull_request-triggered; operator action: click 'Re-run all jobs' at ${run_html_url}"
                printf '* run %s (%s, event=%s) -- cancelled; operator action: click "Re-run all jobs" at %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${stuck_runs_file}"
                ;;
              *)
                log_summary "run ${run_id}: event '${run_event}' has no automatic recovery path; operator action: click 'Re-run all jobs' at ${run_html_url}"
                printf '* run %s (%s, event=%s) -- cancelled; operator action: click "Re-run all jobs" at %s\n' "${run_id}" "${run_path_base}" "${run_event}" "${run_html_url}" >> "${stuck_runs_file}"
                ;;
            esac
          done

          # ------------------------------------------------------------------
          # 6. Commit + push state changes with a single rebase+retry.
          # ------------------------------------------------------------------
          persist_state_changes "final state sync" || true

          popd > /dev/null

          finish 0