runner tool capability probe cannot verify capabilities that are not CLI subcommands; local hot runs skip the preflight entirely

[chubes4]

Two connected gaps

A rig can declare requirements.runner_tools[].capabilities — named capabilities a runner tool must support. Two gaps make this declaration unenforceable in practice:

1. The probe assumes capability == CLI subcommand

tool_capability_probe_script (src/core/runner/capabilities.rs:425 @ a826874) verifies a declared capability by running:

"$tool" <capability> --help  ||  "$tool" <capability> --version

That contract only works for tools whose capabilities are literal CLI subcommands. For tools where declared capabilities are internal command/step identifiers (e.g. entries in a recipe-step registry executed via <tool> recipe-run <file>), the probe cannot distinguish a capable build from an incapable one.

Measured: two builds of the same declared tool — one whose command registry contains the declared capability, one 316 commits older without it — both exit 1 for <tool> <capability> --help and <tool> <capability> --version. The probe reports the capability missing on both, i.e. a false negative on a fully capable runner.

2. The preflight only runs on the runner exec/offload path

validate_runner_capability_preflight is invoked only from runner execution (src/core/runner/execution/daemon.rs:370, src/core/runner/execution/worker.rs:131 @ a826874). A --force-hot --allow-local-hot bench never passes through it. Locally, a stale tool build (a materialized source checkout hundreds of commits behind its upstream) is picked up from PATH silently; the declared capability requirement in the rig manifest is never evaluated, and the run fails mid-recipe at the tool's own schema validation — or the operator only avoids it by knowing to pass an explicit bin path.

Proposal

1. Declarable probe contract. Let the runner_tools requirement optionally declare how to probe, instead of hardcoding the subcommand assumption:

   {
     "tool": "<id>",
     "capabilities": ["x.y", "x.z"],
     "capability_probe": { "args": ["capabilities", "--json"], "match": "list" }
   }

   The probe runs the declared command once and checks the declared capability ids against its output. Default (no capability_probe) keeps today's subcommand behavior.
2. Run the same preflight for local/hot execution. The rig declared a requirement; placement (local vs runner) shouldn't decide whether it is checked. A failed local preflight should fail fast with the existing remediation envelope before any sandbox/recipe work starts.
3. Freshness metadata. When the resolved tool comes from a homeboy-materialized source checkout, include its upstream freshness (behind/ahead) in the probe result so a stale-tool root cause is visible in the failure envelope instead of requiring manual git archaeology.

Acceptance criteria

• A rig declaring tool capabilities fails fast, locally and remotely, when the resolved tool cannot prove those capabilities.
• A tool whose capabilities are not CLI subcommands can declare a probe that verifies them truthfully (no false negatives on capable builds).
• Probe results (per capability: present/missing, probe method, tool path, freshness when known) appear in plan/dry-run and failure output.

All proposal surfaces are generic; no tool- or product-specific identifiers enter homeboy core. Evidence refs against origin/main @ a826874.

AI assistance

• AI assistance: Yes
• Tool(s): Claude Opus 4.5 via opencode
• Used for: Traced the probe/preflight code paths at origin/main, empirically tested the probe contract against two tool builds, and drafted this issue. Reviewed by Chris Huber.

[chubes4]

Two live confirmations from today's fixture-matrix session, plus a concrete probe shape:

1. The stale-tool hazard bit again, twice. Mid-session the pinned fresh tool build (a workspace worktree) was reaped by hygiene; the bench spawned a nonexistent path and the failure surfaced as recipe-run failed with exit null with empty stdout/stderr — the spawn ENOENT was swallowed entirely. The surviving fallback build was then too old for the recipe schema (steps[1] must NOT have additional properties). Neither failure mode named the actual root cause; both would have been caught by a preflight that (a) stats the resolved tool path and (b) verifies schema/capability parity.

2. A natural generic probe contract exists in the wild. The tool in question exposes commands / runtime descriptor subcommands that print machine-readable capability/command metadata (<tool> commands --json lists every recipe command id). A declared capability_probe of that shape (run once, match declared capability ids against output) verifies truthfully where the current <tool> <capability> --help heuristic false-negatives on capable builds — measured again today: the fully capable current build still exits 1 on the subcommand probe.

Resolution that unblocked the session: refreshing the homeboy-materialized source cache to current upstream and rebuilding — exactly the freshness surface point 3 proposes to make visible.