Skip to content

Commit 0b63f2e

Browse files
mmckyclaude
andcommitted
Docs: bring the audit tutorial and CATALOG up to what audit 0.2.0 actually does
The tutorial is the procedure for run 2, and run 2 is the run that tests resume — so it was the one document that could not be left describing the pre-#34 behaviour. Step 4 taught the single-block resume rule that #34 replaced: read findings.md, resume at the lowest number with no entry. It now describes partitioning issues.json by state and resuming each side independently, and names the new failure mode — resuming the open set correctly while re-doing the closed set from scratch. The same step quoted the three-form citation list that doctrine §2 has outgrown, without the reachability rule that is the whole point of defect 1. Two review checks are strengthened where run 1 showed them to be weak. Check 3 passed a citation that pointed at an unmerged branch, so it now says to resolve each sampled citation on the ref the header names. Check 5 held in the report and broke in the checkpoint, so it now looks at findings.md as well as the output. CATALOG records the two severity-1 fixes. Its central caveat is unchanged and still true: resumability is untested, because run 1 was never interrupted. Repo-level files only — no plugin directory touched, so no version bump. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent d4e8df8 commit 0b63f2e

2 files changed

Lines changed: 7 additions & 7 deletions

File tree

CATALOG.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Merged-but-not-yet-working scaffolding is deliberately absent. It registers in t
88
|---|---|---|---|
99
| **`qe`** — author-facing skills, from drafting a lecture to merging its PR | `/qe:copilot-review` | Operational, and validated from an installed plugin on 2026-08-03: plugin-root path resolution, cross-repo mode, and running from outside a working tree. The style skills (`check-style` and six per-category siblings) are merged as scaffolding and report that they are not yet operational, so they are not listed here until the rule snapshot and deterministic preflight land. | [#3](https://github.com/QuantEcon/skills/issues/3) |
1010
| **`benchmark`** — evaluating accelerated lecture implementations | `/benchmark:review-acceleration` | Operational for workspace runs: rubric v2, a deterministic scoring engine, and two complete worked evaluations as regression baselines. | [#4](https://github.com/QuantEcon/skills/issues/4) |
11-
| **`audit`** — bulk, read-only repository audits | `/audit:issues` | Run once **as a skill** — a 230-item tracker on 2026-07-28, 22 minutes, seven plugin defects found and recorded ([record](https://github.com/QuantEcon/skills/blob/main/reviews/audit-run-action-translation-2026-07-28.md)). That validates the method as far as one run goes and no further: its central claim, resumability, is still untested, because the run was never interrupted. Further runs: [#16](https://github.com/QuantEcon/skills/issues/16). | [#12](https://github.com/QuantEcon/skills/issues/12), [#16](https://github.com/QuantEcon/skills/issues/16) |
11+
| **`audit`** — bulk, read-only repository audits | `/audit:issues` | Run once **as a skill** — a 230-item tracker on 2026-07-28, 22 minutes, seven plugin defects found and recorded ([record](https://github.com/QuantEcon/skills/blob/main/reviews/audit-run-action-translation-2026-07-28.md)). Both severity-1 defects are fixed as of 0.2.0 ([#34](https://github.com/QuantEcon/skills/pull/34)): a `[verified]` citation must now resolve on the ref the audit named, and phase 2 checkpoints its closed pass as well as its open one. That validates the method as far as one run goes and no further: its central claim, resumability, is still untested, because the run was never interrupted. Further runs: [#16](https://github.com/QuantEcon/skills/issues/16). | [#12](https://github.com/QuantEcon/skills/issues/12), [#16](https://github.com/QuantEcon/skills/issues/16) |
1212

1313
Installation and setup are in [README.md](README.md); what it is like to run one is in [docs/using-skills.md](docs/using-skills.md).
1414

docs/tutorial-run-an-audit.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -70,13 +70,13 @@ Your counts will differ from the ones above — the tracker moves (#11 measured
7070

7171
## Step 4 — phase 2, verify — and interrupt it
7272

73-
The long phase: 116 items checked against the default branch rather than against what their threads claim. Findings are appended to `findings.md` **one entry per item, as each is verified**.
73+
The long phase: 116 items checked against the default branch rather than against what their threads claim. Findings are appended to `findings.md` **one entry per item, as each is verified** — both passes, the open issues under `## Open` and the closed ones under `## Closed`.
7474

75-
**This is the test.** Once 20–30 entries exist, interrupt the session — close it, or press Esc twice. Then open a new session in the same directory and re-invoke the same command. What should happen: it reads `findings.md`, resumes at the lowest issue number with no entry, and re-verifies only the last entry (which may have been half-written). What would be a failure: restarting from item 1, skipping the item it died on, or duplicating entries.
75+
**This is the test.** Once 20–30 entries exist, interrupt the session — close it, or press Esc twice. Then open a new session in the same directory and re-invoke the same command. What should happen: it partitions `issues.json` by state and, for each side independently, resumes at the lowest number with no entry under the matching heading, re-verifying only the last entry in each (which may have been half-written). What would be a failure: restarting from item 1, skipping the item it died on, duplicating entries, or resuming the open set correctly while re-doing the closed set from scratch.
7676

77-
Resumability is asserted in three separate files and has never been tested. Until this run, phases 2 and 3 named no artifacts at all, so a resumed session could only work if it happened to invent the same filename — the fix is [#17](https://github.com/QuantEcon/skills/pull/17), and this is what checks it.
77+
**Interrupt during the closed pass too, if you get the chance** — that is the half that has never been checkpointed at all. Resumability is asserted in three separate files and has never been tested. Two fixes have gone in ahead of this run and neither has been exercised: [#17](https://github.com/QuantEcon/skills/pull/17) named the artifacts, since before it phases 2 and 3 named none and a resumed session could only work by inventing the same filename; and [#34](https://github.com/QuantEcon/skills/pull/34) made the checkpoint cover both passes, since run 1 wrote only the open set and sent 62 closed issues straight to the catalog. This run is what checks both.
7878

79-
While it runs, `tail findings.md` occasionally. Every status claim should carry `[verified]`, `[stated]` or `[inferred]`, and a `[verified]` should cite `file:line`, a merged PR, or a tag — never a comment.
79+
While it runs, `tail findings.md` occasionally. Every status claim should carry `[verified]`, `[stated]` or `[inferred]`, and a `[verified]` should cite `file:line`, a merged PR, a tag, or a commit**and whatever it cites must resolve on the ref the audit named**, never a comment. A citation that only resolves in the author's working tree or on an unmerged branch is the defect [doctrine §2](https://github.com/QuantEcon/skills/blob/main/audit/references/doctrine.md#2-evidence-classes) now rules out; run 1's headline finding had exactly that shape.
8080

8181
## Step 5 — phases 3 to 5
8282

@@ -92,9 +92,9 @@ The run cannot check any of this about itself. Ten checks, the last two of which
9292
|---|---|---|---|
9393
| 1 | Method section is complete | report §1 | The plan anchor, label policy and prior-audit search are not all named |
9494
| 2 | Every claim is tagged | catalog entries | An untagged status claim — a defect by the doctrine's own rule |
95-
| 3 | `[verified]` means verified | sample 5 "fixed" calls | Citation is a thread comment rather than code, a merged PR, or a tag |
95+
| 3 | `[verified]` means verified | sample 5 "fixed" calls, and **resolve each on the ref named in the header** | Citation is a thread comment rather than code — or a commit, `file:line` or PR that looks right and does not resolve on that ref. `git merge-base --is-ancestor <sha> <ref>` on any commit cited. This is the check run 1 passed and should not have |
9696
| 4 | Code beat the thread | sample 5 more | The report repeats "fixed in #204" without saying it checked the branch |
97-
| 5 | The closed side was read | closed-set verification | Closed issues summarised from title and `stateReason` only; no deferred remainders surfaced |
97+
| 5 | The closed side was read | closed-set verification, **and `findings.md`** | Closed issues summarised from title and `stateReason` only; no deferred remainders surfaced. Also check the checkpoint, not just the output: closed entries reaching the catalog with no matching `## Closed` block is how run 1 passed this check in its report while failing it in its log |
9898
| 6 | Coverage is honest | `README.md` index | Unaccounted numbers waved at collectively; residue (inline review comments, GraphQL-only data) not stated |
9999
| 7 | Siblings were checked | external cross-link registry | No sibling considered, in an org where the same fix lands in several repos as `SYNC:` PRs |
100100
| 8 | Drafted comments are safe | GC tier | A closing keyword immediately before an `owner/repo#N` reference — that closes the *upstream* item when it lands |

0 commit comments

Comments
 (0)