TESTING: review + expert-parallel protocol for the benchmark skill (@xuanguang-li)

[mmcky]

Testing and review protocol for @xuanguang-li — everything we'd like your eyes on, in order, with the machinery validation already done so your time goes to judgement, not reproduction.

Body updated 2026-08-03. PR #5 merged on 2026-07-27 and its branch was deleted, so the five links here 404'd; they now point at main. §3 has been replaced with the current install route, §1's last bullet corrected (#5 adopted the unconditional x64 cap), and §4 carries a status note about the PR it was written against. The comment below records the changes.

Context. Your evaluation system is packaged as the benchmark plugin, on main since #5 (shipped as v0.3.0 on 2026-07-27; current release 0.3.1): the deterministic engine (benchmark/scripts/scoring/), the framework (EVALUATION_FRAMEWORK.md), your ge_arrow (lecture-python.myst#717 → 2.85) and markov_asset (lecture-python.myst#654 → 2.25) evaluations as regression anchors, and a rubric-v2 revision from the 2026-07-21 adversarial design review (your measurement foundation was endorsed by both review tracks and is unchanged). We have already re-validated end-to-end from a fresh checkout of lecture-python.myst#717 on jax 0.10.1: exact reproduction of 2.85 / no-conversion / fragile with the same deciding flips — see reviews/validation-run-ge_arrow-2026-07-22.md. You do not need to repeat that exercise.

1. Review the rubric v2 changes (merged in #5 — read them on main)

The v2 changes touch only the verdict layer of your rubric — dimension scoring is untouched. Please push back on anything; rubric semantics stay yours.

• Verdict gates — correctness 1 caps the verdict at net regression, correctness 2 at mixed/wash, enforced in score_all (rubric.py); the logic&design bug-cap is now derived from the correctness evidence (builds / x64 divergence) rather than a hand-set boolean
• No-conversion verdict — fires when the baseline as-used total is under a 1 s materiality floor and the candidate is slower as-used; reconciles review mode with your triage bound (the polished-ge_arrow case). The 1 s floor is a labeled policy choice — is it the right one? A run since has produced a case that clears it while having nothing to sell; see TESTING: benchmark triage run on wald_friedman — six plugin gaps, and a floor that measures the wrong quantity #14, finding 4
• Sensitivity stamp — score.py perturbs every scored input one at a time (booleans flipped, counts ±1, floats ±10%) and stamps the scorecard robust/fragile with deciding flips listed
• K-repeat as-used — median of 3 fresh-process runs per side, with a contested-band annotation when runs span a band edge
• The flagged Copilot thread, resolved in Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5 and needing your confirmation: x64 divergence originally forced correctness 1 only when shipped drift also exceeded 1e-8 (your original semantics). Copilot argued for making it unconditional and Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5 adopted that, so matches_under_x64 now caps correctness on its own in score_correctness, with the reasoning in that function's docstring. benchmark/CHANGELOG.md 0.3.0 records the effect: the "wrong economics masked by low precision" case moves from correctness 5 / total 3.25 to correctness 1 / total 2.30, gated to net regression. Confirm or amend

2. Three design calls (comment on #7)

• Readability instrument v2 — docstring coverage inverts your own exemplars (odu.py, your LOW example, measures 0.86; flagship-aiyagari style ~0.41–0.55, capped at 2). Proposal: equation-traceability — per numbered equation, cite the single implementing expression; fraction traceable old vs new, same citation discipline as your checklists
• Extraction/replay verification — a mechanical diff of extracted code and replayed call sequence against the lecture's cells (our validation run caught an undisclosed whitespace normalisation this way); longer-term, whether executing the lecture itself at both refs should become the as-used source
• Confirm or amend the x64 guard decision from item 1 as a rubric-level statement

3. Optional — see the skill in action

Not required (the validation already covers the machinery), but if you want to feel out what reviewers will receive: from a lecture-python.myst checkout, in a Claude Code session, with no clone of this repo and no branch checkout —

/plugin marketplace add QuantEcon/skills
/plugin install benchmark@quantecon

Restart the session (plugins register at startup), then:

/benchmark:review-acceleration 717

/plugin is a terminal-CLI built-in; in the VS Code extension or the web app use the equivalent claude plugin … commands (using-skills § Setup).

Evaluation artifacts land in benchmark-eval/<lecture>/ in your workspace; lecture-python.myst#717 should land on exactly 2.85 / no-conversion / fragile (your numbers are the answer key). Any conversion PR works. The by-hand walkthrough of the same steps is docs/tutorial-run-an-evaluation.md. What we most want from this: critique the report — is it something you would sign?

4. The acceptance test — expert-parallel on a conversion PR

Status note (2026-08-03): this section was written against lecture-python.myst#654, which @jstac had already closed unmerged on 2026-07-21 — a couple of hours before this issue was opened — concluding that the lecture stays in NumPy, the same conclusion the skill's markov_asset evaluation reached (2.25/5, headline verdict no-conversion). Step 4 below therefore has no live PR to land on. Whether the acceptance test becomes a retrospective adjudication of lecture-python.myst#654 or retargets to an open conversion PR is being decided in the comments. Two open PRs qualify as independent targets — lecture-python.myst#657 (multi_hyper) and lecture-python.myst#624 (mccall_q). lecture-python.myst#644 (kalman_2) is open and unevaluated but is @xuanguang-li's own, so it cannot serve as the independent half of an expert-parallel test; lecture-python.myst#717 is out for the opposite reason — its 2.85 is published as the answer key and committed as a regression anchor.

The part only you can do (tracked on #4; the #8 work plan that originally carried it was closed on 2026-07-27). Protocol:

1. The skill runs a complete v2 evaluation of the PR under test from scratch (fresh extraction, measurement, evidence, score, report) — we drive this and post the report here, not on the PR
2. You evaluate the same PR independently (or, if you prefer, adjudicate the skill's report line by line — your choice; independent-first is the stronger test)
3. We reconcile: every discrepancy resolved by evidence, with the rubric amended where your judgement and the instrument disagree
4. The reconciled report becomes the single authoritative comment on the PR under test

• Target PR agreed (retrospective on lecture-python.myst#654, or an open conversion PR)
• Skill evaluation posted on this issue
• Independent expert evaluation (or line-by-line adjudication) by @xuanguang-li
• Reconciliation recorded; rubric amendments (if any) filed on PLAN: evaluation rubric v2 — enforce couplings, verdict vocabulary, instrument fixes (from the three-way design review) #7
• Authoritative comment posted to the PR under test

Comment here, on #7, or on #14 — whatever is easiest. Relates to #4 (skill wiring and the named validation runs), #7 (rubric v2), #14 (triage-run findings).

[xuanguang-li]

Hi @mmcky,

Thanks for this helpful summary!

I followed the Optional — see the skill in action to replicate the test, but there are some problems when I ran the commands in the terminal.

1. marketplace.json schema error
   After git clone https://github.com/QuantEcon/skills ~/skills, running /plugin marketplace add ~/skills failed with: Invalid schema: ... owner: Invalid input: expected object, received undefined.

The manifest was missing the required owner object (name + optional email). Adding it resolved the error.

2. Plugin clone goes over SSH, not HTTPS
   /plugin install benchmark@quantecon tried to clone over SSH (git@github.com) and failed with No ED25519 host key is known for github.com.

The cause is that the marketplace only indexes the plugin — the plugin source resolves to a separate repo that gets cloned fresh. That's reasonable, but it means a first-time user needs working GitHub SSH auth before the install can succeed. Getting through it required trusting GitHub's host key and registering an SSH key on my account. A note in the README about the auth prerequisite (and whether the plugin's source repo is public or access-restricted) would save a lot of time.

3. Skill invocation isn't discoverable
   After a successful install, /benchmark:review-acceleration 717 returned Unknown command: /benchmark:review-acceleration Args from unknown skill: 717, even though /plugin reported Status: Enabled, and /plugin enable benchmark@quantecon responded: "already enabled".

The plugin ships skills only — no commands/ directory — so it wasn't obvious from the outside how the documented invocation was meant to resolve.

Environment: Windows, PowerShell, Claude Code CLI, benchmark@quantecon v0.3.0.

The skill functions well after resolving these problems. The friction was all upstream of actually using it.

[mmcky]

Thanks @xuanguang-li — this is exactly the friction we needed surfaced, and all three were real. Each is now fixed at the root on the branch (PR #5). Here's what each one was and what changed.

1. Missing owner. You were right: owner is a required top-level object in the marketplace schema and ours omitted it, so /plugin marketplace add failed schema validation for everyone. Added "owner": { "name": "QuantEcon" }. The real gap was that our own scripts/validate.py never checked for owner, so it happily passed a manifest the installer rejects — it now requires it, so this can't slip through CI again.

2. SSH clone. Root cause found. Each plugin entry declared its source as the remote GitHub repo (source: { source: "github", repo: "QuantEcon/skills", … }). Because the plugins live inside the marketplace repo you'd just added, that told Claude Code to discard the local copy and re-clone QuantEcon/skills from GitHub — over SSH — purely to reach a subdirectory already on disk. That's your ED25519 failure. The fix is the documented pattern for a plugin co-located with its marketplace: a relative-path source ("source": "./benchmark"). Install now uses the copy already present, so there is no SSH step and no auth prerequisite at all — rather than document the prerequisite, we removed it. The validator now rejects any plugin whose source points back at this repo.

3. Unknown command. Two things were in play, and your read that it "wasn't obvious how the invocation resolves" was on target. First, the SSH-fought install may have left the plugin registered-but-not-populated — your error said unknown skill, not unknown plugin — which a clean install after fix #2 should settle. Second, and the more likely direct cause: the plugin-prefixed form /benchmark:review-acceleration is version-gated. Before Claude Code v2.1.216 the skill registered under the bare /review-acceleration and the prefixed form didn't resolve. What's your claude --version? If it's below 2.1.216 that alone explains it. Two fallbacks that work on any version: the bare /review-acceleration, and natural language — describe the task ("evaluate whether the JAX conversion in lecture-python.myst#654 is worth merging") and the skill triggers on its description. We deliberately did not add a commands/ shim for the prefixed form: skills already are slash commands on current Claude Code, and a second entry point would only be one more thing to drift.

To retry: pull the branch again and re-run /plugin marketplace add + /plugin install benchmark@quantecon — the install should be clean now, with no SSH. If /benchmark:review-acceleration 654 still doesn't resolve, send me your version and use /review-acceleration 654 or the natural-language form in the meantime.

None of this touched the evaluation machinery — it was all install surface, exactly as you called it, upstream of the skill actually running. The rubric-v2 review and the lecture-python.myst#654 acceptance test remain the parts we most want your eyes on.

[xuanguang-li]

Thanks @mmcky. This solves all the problems.

My Claude Code version is 2.1.218. I think it's the latest version.

After this update, the skill setup becomes very straightforward. I can use the /benchmark command after cloning the repo without setting /plugin marketplace add ~/skills and /plugin install benchmark@quantecon.

[mmcky]

Interesting @xuanguang-li -- I wonder if this is a new skill provided by Claude or if this is our custom benchmarking tool?

[mmcky]

PR #5 merged on 2026-07-27 and its branch land-evaluation-system was deleted, so all five links in the body above 404'd and §3 told you to check out a branch that no longer exists. Since this is a sheet you follow top-down, I have corrected the body rather than only commenting. What changed in it: every link now points at main; §3 is the current install route — /plugin marketplace add QuantEcon/skills, /plugin install benchmark@quantecon, restart the session, no clone and no checkout; §1's last bullet was wrong, because #5 adopted Copilot's unconditional form, so matches_under_x64 now caps correctness on its own in score_correctness, with the reasoning in that function's docstring; and §4's milestone pointer moved from #8 (closed completed on 2026-07-27) to #4. Confirming or amending the x64 call is still yours.

The install problems you hit on 2026-07-24 are now released rather than branch-only: benchmark 0.3.0 landed on main on 2026-07-27, current release 0.3.1. Root causes and fixes are written up in benchmark/CHANGELOG.md under 0.3.0, and scripts/validate.py now fails CI on a missing marketplace owner and on any plugin source pointing back at this repo, so neither can return.

§4 needs a decision. It was written against QuantEcon/lecture-python.myst#654, which @jstac had already closed unmerged on 2026-07-21 — a couple of hours before this issue was opened — concluding the lecture stays in NumPy, the same conclusion the skill's markov_asset evaluation reached (2.25/5, headline verdict no-conversion). So the "single authoritative comment" step has never had a live PR to land on. Two routes: a retrospective adjudication of QuantEcon/lecture-python.myst#654, or an open conversion PR this system has never touched. QuantEcon/lecture-python.myst#657 (multi_hyper) and QuantEcon/lecture-python.myst#624 (mccall_q) both qualify. QuantEcon/lecture-python.myst#644 (kalman_2) is open and unevaluated but is yours, so it cannot be the independent half of an expert-parallel test; QuantEcon/lecture-python.myst#717 is out for the opposite reason — its 2.85 is published as the answer key and committed as a regression anchor. @mmcky, that call is yours.

Also new since you last looked: the first triage-mode run is on main at reviews/triage-run-wald_friedman-2026-07-27.md, and it puts two questions to you on #14 — whether the 1 s materiality floor should key on addressable time rather than the total (finding 4), and whether triage should ship a scaffold (finding 2).

[xuanguang-li]

Hi @mmcky,

Thanks for the latest update, and sorry for falling so far behind.

I wonder if this is a new skill provided by Claude or if this is our custom benchmarking tool?

It’s our custom benchmarking tool. It works by adding \benchmark as an interpretable command for Claude. Specifically, /plugin marketplace add QuantEcon/skills tells Claude to find the skill based on the .\claude-plugin\marketplace.json file in the QuantEcon/skills repo. Then, /plugin install benchmark@quantecon tells Claude to add \benchmark as a plugin command.

I read and compared the evaluation reports generated by the local command and the skill, and found them to be nearly identical. The only differences I found are that the skill version includes a sensitivity test and a gate. I think it makes sense to give a low score when basic requirements, such as correctness, are not satisfied.

However, the sensitivity test that perturbs correctness.builds from True to False and correctness.matches_under_x64 from True to False confused me. It caused the score to drop to 2.3 rather than 1. According to rubric.py, however, both perturbations should make correctness equal to 1:

def score_correctness(builds, max_delta_shipped, matches_under_x64):
    """max|Δ| vs the original *as the lecture ships* (float32 unless x64 set)."""
    if not builds:
        return 1, "does not build as shipped → 1 (overrides Δ bands)"
    d = max_delta_shipped
    if not matches_under_x64 and d > 1e-8:
        return 1, f"logic diverges under x64 (max|Δ|={d:.1e}) → wrong economics → 1"

Then, the correctness gate should set the final score to 0 according to the following code:

if scores["correctness"] == 1 and base_idx > 0:
    final_idx = 0

So I’m not quite sure why the sensitivity test produces a score of 2.3 instead of 1.

[xuanguang-li]

Hi @mmcky,

I experimented with /benchmark:review-acceleration. First, I modified the ge_arrow lecture based on the suggestions from the previous evaluation. Then, I used the benchmark skill to evaluate it again.

The scores changed as expected following my changes to the lecture. In this sense, I think the skill is self-consistent. This is good news to us!

I've also pasted a summary of the evaluation below in case you're interested.

Changes: vectorised kernel, x64 enabled, s0_idx dropped from static args, docstrings restored, and lax.cond removed.

Results:

Dimension	Wt	Score	Weighted	How the score arises	vs #717 anchor
Correctness & numerical fidelity	0.20	5	1.00	max|Δ|=1.4e-14 ≤ 1e-10 (ships x64)	3 → 5 ▲
Readability & pedagogical clarity	0.25	2	0.50	Δprereq +6→2; docstrings 1.00→5; worse-of-two	2 → 2 =
Computational efficiency (as used)	0.15	2	0.30	as-used 0.053× < 0.8 (slower), correct/fixable	2 → 2 =
Logic & design	0.15	5	0.75	4/4 criteria met	4 → 5 ▲
Coding style & idiom	0.10	5	0.50	4/4 criteria met	2 → 5 ▲
API ergonomics & reusability	0.10	5	0.50	1 statement per result	5 → 5 =
Maintainability & robustness	0.05	5	0.25	4/4 criteria met	3 → 5 ▲
Total	1.00		3.80		2.85 → 3.80 ▲