feat: verification + eval harness + context budget + model-agnostic runner

.github/workflows/validate.yml

@@ -0,0 +1,55 @@
+name: validate
+
+# Deterministic guards that run on every PR — the contributor-facing safety net.
+# None of these need an LLM or paid tokens: they check structure, context budget,
+# and (on the fixture) live citation resolution. A PR that bloats SKILL.md, breaks
+# the output schema, or adds a dead control URL fails here.
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  structure-and-budget:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install deps
+        run: pip install -r scripts/requirements.txt
+
+      - name: Context budget (fail if SKILL.md / always-floor over budget)
+        run: python scripts/context_budget.py --ci
+
+      - name: Validate fixture run structure
+        run: python eval/validate_structure.py --research-dir eval/questions/_fixture --strict || true
+        # _fixture has only sources.csv (control URLs) — structure check is informational here;
+        # drop the `|| true` once a full fixture run (plan.md + report) is committed.
+
+  citation-fixture:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - run: pip install -r scripts/requirements.txt
+
+      - name: Citation checker resolves controls correctly
+        # _fixture/sources.csv has a live control (example.com) and a dead control
+        # (…/definitely-404…). This proves the checker still flags hallucinated URLs.
+        run: |
+          python eval/check_citations.py \
+            --research-dir eval/questions/_fixture \
+            --out eval/output/ci_fixture --json
+          python - <<'PY'
+          import json
+          r = json.load(open("eval/output/ci_fixture.json"))
+          flags = [x for x in r["results"] if x["red_flag"]]
+          assert any("404" in x["url"] or not x["alive"] for x in r["results"]), "dead control not detected"
+          assert flags, "expected at least one red flag from the dead control URL"
+          print(f"OK — {len(flags)} red flag(s) caught, integrity={r['citation_integrity']:.2f}")
+          PY

eval/BENCHMARK.md

@@ -0,0 +1,78 @@
+# Benchmark protocol
+
+The point of this suite is to replace claims with numbers. Right now the README
+asserts the skill verifies sources, triangulates, and runs adversarial passes — but
+ships zero evidence (`eval/output/` empty, `runs.csv` empty, no example runs). One
+published benchmark closes that gap and is the single highest-credibility artifact
+the project can have.
+
+**Nothing here fabricates results.** You run the skill on a fixed question set on
+your own machine; the scripts score what the runs produced.
+
+## The suite
+
+8 questions in `eval/questions/benchmark/`, one per file, spanning all 6 genres and
+all 3 depths:
+
+| slug | genre | depth |
+|---|---|---|
+| sqlite-vs-duckdb-analytics | decision | shallow |
+| postgres-replication-vs-cdc | decision | medium |
+| rag-chunking-strategies-2026 | qa | medium |
+| open-source-ai-licensing | qa | medium |
+| wasm-component-model | explainer | medium |
+| edge-inference-cost-model | explainer | medium |
+| vector-db-landscape-2026 | landscape | deep |
+| llms-plateau-claim | validation | deep |
+
+Genres/depths are spread on purpose: a benchmark that's all-medium-decision hides
+where the skill is weak.
+
+## Run it
+
+For each question, in a Claude Code session at the repo root:
+
+```
+/deep-research <paste the Question block>
+```
+
+Pin the depth stated in the file. To compare configs, run the same question under:
+- **A** — default routing
+- **B** — `... with all on opus`
+- **C** — `... with cheap mode`
+
+After each run, record the real cost (`/cost` in the session) into
+`eval/runs/runs.csv` under the matching `run_id` (`<slug>-A` etc.).
+
+## Score it
+
+```bash
+# deterministic axes + render the judge input
+python eval/score_run.py --research-dir research/<slug> --run-id <slug>-A
+
+# run eval/output/<slug>-A_judge_input.md through Opus, save JSON to
+# eval/output/<slug>-A_judge.json, then:
+python eval/score_run.py --research-dir research/<slug> --run-id <slug>-A \
+  --judge-json eval/output/<slug>-A_judge.json
+```
+
+Citation integrity is computed live by `check_citations.py` (it resolves every
+source URL). The judge handles the semantic axes. Floor rule: integrity < 0.70
+halves the final quality score.
+
+## Publish it
+
+```bash
+python eval/aggregate.py --out eval/BENCHMARK_RESULTS.md
+```
+
+Paste the resulting table into the README under a "Benchmarks" section, with a one-
+line method note and a link to `eval/rubric.md`. Re-run after catalog changes so the
+numbers stay honest.
+
+## What "good" looks like
+
+The headline metric is **quality-per-dollar**, not raw quality — a $8 all-opus run
+that scores 0.88 can lose to a $2 default run that scores 0.82. That comparison is
+the whole reason model routing exists, and publishing it is the proof the routing
+claim is real.

eval/aggregate.py

@@ -0,0 +1,102 @@
+#!/usr/bin/env python3
+"""
+Aggregate per-run scorecards into one benchmark table for the README.
+
+score_run.py writes eval/output/<run_id>_scorecard.md per run. This collects them
+all, joins with runs.csv (for question + config + real cost), and emits a single
+markdown table you paste into the README — the published benchmark that turns
+"trust me, it verifies sources" into numbers.
+
+It does NOT invent numbers. It only reads scorecards that real runs produced. If a
+run has no scorecard yet, it shows as PENDING.
+
+Usage:
+    python eval/aggregate.py                       # table to stdout
+    python eval/aggregate.py --out BENCHMARK_RESULTS.md
+"""
+
+import argparse
+import csv
+import re
+from pathlib import Path
+
+EVAL_DIR = Path(__file__).parent
+OUTPUT_DIR = EVAL_DIR / "output"
+RUNS_CSV = EVAL_DIR / "runs" / "runs.csv"
+
+QS_RE = re.compile(r"quality_score\s*=\s*([\d.]+)")
+QPD_RE = re.compile(r"quality_per_dollar\s*=\s*([\d.]+)")
+CITE_RE = re.compile(r"Citation integrity:\*\*\s*([\d.]+)")
+DIV_RE = re.compile(r"Source diversity:\*\*\s*([\d.]+)")
+
+
+def parse_scorecard(path: Path) -> dict:
+    t = path.read_text(encoding="utf-8")
+    g = lambda rx: (rx.search(t).group(1) if rx.search(t) else None)
+    return {
+        "citation": g(CITE_RE), "diversity": g(DIV_RE),
+        "quality": g(QS_RE), "qpd": g(QPD_RE),
+        "pending_judge": "Pending." in t,
+    }
+
+
+def load_runs() -> dict[str, dict]:
+    if not RUNS_CSV.is_file():
+        return {}
+    with RUNS_CSV.open(encoding="utf-8") as fh:
+        return {r["run_id"]: r for r in csv.DictReader(fh) if r.get("run_id")}
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--out", type=Path)
+    args = ap.parse_args()
+
+    runs = load_runs()
+    cards = {p.stem.replace("_scorecard", ""): parse_scorecard(p)
+             for p in sorted(OUTPUT_DIR.glob("*_scorecard.md"))}
+
+    # union of run_ids known from runs.csv and from scorecards
+    ids = sorted(set(runs) | set(cards))
+    rows = []
+    for rid in ids:
+        meta = runs.get(rid, {})
+        card = cards.get(rid)
+        if card is None:
+            rows.append((rid, meta.get("question_slug", "?"), meta.get("config", "?"),
+                         "—", "—", "—", "—", "PENDING (no scorecard)"))
+            continue
+        status = "judge pending" if card["pending_judge"] else "scored"
+        rows.append((rid, meta.get("question_slug", "?"), meta.get("config", "?"),
+                     card["citation"] or "—", card["diversity"] or "—",
+                     card["quality"] or "—", card["qpd"] or "—", status))
+
+    header = "| run_id | question | config | cite | diversity | quality | qual/$ | status |"
+    sep = "|---|---|---|---|---|---|---|---|"
+    lines = [
+        "# Benchmark results",
+        "",
+        f"{len(cards)} scored run(s) across {len({runs[r].get('question_slug') for r in runs}) or '?'} questions.",
+        "Generated by `eval/aggregate.py` from real scorecards — no synthetic numbers.",
+        "",
+        header, sep,
+    ]
+    for r in rows:
+        lines.append("| " + " | ".join(str(x) for x in r) + " |")
+    lines += [
+        "",
+        "_cite = citation integrity (0–1, ≥0.70 floor) · quality = weighted rubric score · "
+        "qual/$ = quality per dollar (the real verdict). See `eval/rubric.md`._",
+    ]
+    table = "\n".join(lines)
+
+    if args.out:
+        args.out.write_text(table, encoding="utf-8")
+        print(f"Wrote {args.out}")
+    else:
+        print(table)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

eval/questions/benchmark/edge-inference-cost-model.md

@@ -0,0 +1,19 @@
+# Benchmark — edge-inference-cost-model
+
+## Question
+How do you reason about the cost trade-off between edge/on-device inference and centralized API inference for a consumer app?
+
+## Genre (expected)
+explainer
+
+## Depth
+medium
+
+## Why this question (for the judge)
+Cost mental-model. Good = builds a real decision framework, names the variables, avoids a one-sided answer.
+
+## Configs to compare
+- A: default routing
+- B: all-opus
+- C: cheap-mode
+(run_ids: edge-inference-cost-model-A / edge-inference-cost-model-B / edge-inference-cost-model-C)

eval/questions/benchmark/llms-plateau-claim.md

@@ -0,0 +1,19 @@
+# Benchmark — llms-plateau-claim
+
+## Question
+Is the claim 'frontier LLM capabilities have plateaued since 2025' supported by evidence?
+
+## Genre (expected)
+validation
+
+## Depth
+deep
+
+## Why this question (for the judge)
+Hypothesis validation. Good = falsification criteria stated, benchmark + economic evidence on both sides, honest confidence.
+
+## Configs to compare
+- A: default routing
+- B: all-opus
+- C: cheap-mode
+(run_ids: llms-plateau-claim-A / llms-plateau-claim-B / llms-plateau-claim-C)

eval/questions/benchmark/open-source-ai-licensing.md

@@ -0,0 +1,19 @@
+# Benchmark — open-source-ai-licensing
+
+## Question
+What does 'open source' actually mean for AI models in 2026 (weights vs data vs license), and which licenses are genuinely OSI-compliant?
+
+## Genre (expected)
+qa
+
+## Depth
+medium
+
+## Why this question (for the judge)
+Definitional + landscape. Good = precise on the open-weight vs open-source distinction, cites actual license texts.
+
+## Configs to compare
+- A: default routing
+- B: all-opus
+- C: cheap-mode
+(run_ids: open-source-ai-licensing-A / open-source-ai-licensing-B / open-source-ai-licensing-C)

eval/questions/benchmark/postgres-replication-vs-cdc.md

@@ -0,0 +1,19 @@
+# Benchmark — postgres-replication-vs-cdc
+
+## Question
+Trade-offs between Postgres logical replication and dedicated CDC tooling (Debezium/Fivetran) for streaming changes to a warehouse — which to choose for a 50-table OLTP DB feeding analytics?
+
+## Genre (expected)
+decision
+
+## Depth
+medium
+
+## Why this question (for the judge)
+Supports an architecture decision. Good = clear conditional verdict, both options steel-manned, real failure modes (DDL changes, replication slot bloat) covered.
+
+## Configs to compare
+- A: default routing
+- B: all-opus
+- C: cheap-mode
+(run_ids: postgres-replication-vs-cdc-A / postgres-replication-vs-cdc-B / postgres-replication-vs-cdc-C)

eval/questions/benchmark/rag-chunking-strategies-2026.md

@@ -0,0 +1,19 @@
+# Benchmark — rag-chunking-strategies-2026
+
+## Question
+What chunking strategies for RAG actually improve retrieval quality in 2026, and where is the evidence weak vs strong?
+
+## Genre (expected)
+qa
+
+## Depth
+medium
+
+## Why this question (for the judge)
+Open meta-research. Good = separates hype from measured results, cites benchmarks, flags where claims are anecdotal.
+
+## Configs to compare
+- A: default routing
+- B: all-opus
+- C: cheap-mode
+(run_ids: rag-chunking-strategies-2026-A / rag-chunking-strategies-2026-B / rag-chunking-strategies-2026-C)

eval/questions/benchmark/sqlite-vs-duckdb-analytics.md

@@ -0,0 +1,19 @@
+# Benchmark — sqlite-vs-duckdb-analytics
+
+## Question
+SQLite vs DuckDB for embedded analytical workloads — when does each win?
+
+## Genre (expected)
+decision
+
+## Depth
+shallow
+
+## Why this question (for the judge)
+Low-stakes decision, shallow depth control case. Good = crisp, correct on OLAP vs OLTP, not over-researched.
+
+## Configs to compare
+- A: default routing
+- B: all-opus
+- C: cheap-mode
+(run_ids: sqlite-vs-duckdb-analytics-A / sqlite-vs-duckdb-analytics-B / sqlite-vs-duckdb-analytics-C)

eval/questions/benchmark/vector-db-landscape-2026.md

@@ -0,0 +1,19 @@
+# Benchmark — vector-db-landscape-2026
+
+## Question
+Who are the serious players in the vector database / hybrid-search space in 2026, and how are they differentiated?
+
+## Genre (expected)
+landscape
+
+## Depth
+deep
+
+## Why this question (for the judge)
+Market map. Good = no major omissions, positioning is accurate not marketing copy, distinguishes managed vs embedded.
+
+## Configs to compare
+- A: default routing
+- B: all-opus
+- C: cheap-mode
+(run_ids: vector-db-landscape-2026-A / vector-db-landscape-2026-B / vector-db-landscape-2026-C)

eval/questions/benchmark/wasm-component-model.md

@@ -0,0 +1,19 @@
+# Benchmark — wasm-component-model
+
+## Question
+How does the WebAssembly Component Model work under the hood, and what problem does it solve that core WASM modules don't?
+
+## Genre (expected)
+explainer
+
+## Depth
+medium
+
+## Why this question (for the judge)
+Mechanism explanation. Good = correct mental model, accurate on WIT/canonical ABI, no hand-waving.
+
+## Configs to compare
+- A: default routing
+- B: all-opus
+- C: cheap-mode
+(run_ids: wasm-component-model-A / wasm-component-model-B / wasm-component-model-C)