You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Docs: bring the audit tutorial and CATALOG up to what audit 0.2.0 actually does
The tutorial is the procedure for run 2, and run 2 is the run that tests
resume — so it was the one document that could not be left describing the
pre-#34 behaviour.
Step 4 taught the single-block resume rule that #34 replaced: read findings.md,
resume at the lowest number with no entry. It now describes partitioning
issues.json by state and resuming each side independently, and names the new
failure mode — resuming the open set correctly while re-doing the closed set
from scratch. The same step quoted the three-form citation list that doctrine §2
has outgrown, without the reachability rule that is the whole point of defect 1.
Two review checks are strengthened where run 1 showed them to be weak. Check 3
passed a citation that pointed at an unmerged branch, so it now says to resolve
each sampled citation on the ref the header names. Check 5 held in the report
and broke in the checkpoint, so it now looks at findings.md as well as the
output.
CATALOG records the two severity-1 fixes. Its central caveat is unchanged and
still true: resumability is untested, because run 1 was never interrupted.
Repo-level files only — no plugin directory touched, so no version bump.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: CATALOG.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@ Merged-but-not-yet-working scaffolding is deliberately absent. It registers in t
8
8
|---|---|---|---|
9
9
|**`qe`** — author-facing skills, from drafting a lecture to merging its PR |`/qe:copilot-review`| Operational, and validated from an installed plugin on 2026-08-03: plugin-root path resolution, cross-repo mode, and running from outside a working tree. The style skills (`check-style` and six per-category siblings) are merged as scaffolding and report that they are not yet operational, so they are not listed here until the rule snapshot and deterministic preflight land. |[#3](https://github.com/QuantEcon/skills/issues/3)|
10
10
|**`benchmark`** — evaluating accelerated lecture implementations |`/benchmark:review-acceleration`| Operational for workspace runs: rubric v2, a deterministic scoring engine, and two complete worked evaluations as regression baselines. |[#4](https://github.com/QuantEcon/skills/issues/4)|
11
-
|**`audit`** — bulk, read-only repository audits |`/audit:issues`| Run once **as a skill** — a 230-item tracker on 2026-07-28, 22 minutes, seven plugin defects found and recorded ([record](https://github.com/QuantEcon/skills/blob/main/reviews/audit-run-action-translation-2026-07-28.md)). That validates the method as far as one run goes and no further: its central claim, resumability, is still untested, because the run was never interrupted. Further runs: [#16](https://github.com/QuantEcon/skills/issues/16). |[#12](https://github.com/QuantEcon/skills/issues/12), [#16](https://github.com/QuantEcon/skills/issues/16)|
11
+
|**`audit`** — bulk, read-only repository audits |`/audit:issues`| Run once **as a skill** — a 230-item tracker on 2026-07-28, 22 minutes, seven plugin defects found and recorded ([record](https://github.com/QuantEcon/skills/blob/main/reviews/audit-run-action-translation-2026-07-28.md)). Both severity-1 defects are fixed as of 0.2.0 ([#34](https://github.com/QuantEcon/skills/pull/34)): a `[verified]` citation must now resolve on the ref the audit named, and phase 2 checkpoints its closed pass as well as its open one. That validates the method as far as one run goes and no further: its central claim, resumability, is still untested, because the run was never interrupted. Further runs: [#16](https://github.com/QuantEcon/skills/issues/16). |[#12](https://github.com/QuantEcon/skills/issues/12), [#16](https://github.com/QuantEcon/skills/issues/16)|
12
12
13
13
Installation and setup are in [README.md](README.md); what it is like to run one is in [docs/using-skills.md](docs/using-skills.md).
Copy file name to clipboardExpand all lines: docs/tutorial-run-an-audit.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -70,13 +70,13 @@ Your counts will differ from the ones above — the tracker moves (#11 measured
70
70
71
71
## Step 4 — phase 2, verify — and interrupt it
72
72
73
-
The long phase: 116 items checked against the default branch rather than against what their threads claim. Findings are appended to `findings.md`**one entry per item, as each is verified**.
73
+
The long phase: 116 items checked against the default branch rather than against what their threads claim. Findings are appended to `findings.md`**one entry per item, as each is verified** — both passes, the open issues under `## Open` and the closed ones under `## Closed`.
74
74
75
-
**This is the test.** Once 20–30 entries exist, interrupt the session — close it, or press Esc twice. Then open a new session in the same directory and re-invoke the same command. What should happen: it reads `findings.md`, resumes at the lowest issue number with no entry, and re-verifies only the last entry (which may have been half-written). What would be a failure: restarting from item 1, skipping the item it died on, or duplicating entries.
75
+
**This is the test.** Once 20–30 entries exist, interrupt the session — close it, or press Esc twice. Then open a new session in the same directory and re-invoke the same command. What should happen: it partitions `issues.json` by state and, for each side independently, resumes at the lowest number with no entry under the matching heading, re-verifying only the last entry in each (which may have been half-written). What would be a failure: restarting from item 1, skipping the item it died on, duplicating entries, or resuming the open set correctly while re-doing the closed set from scratch.
76
76
77
-
Resumability is asserted in three separate files and has never been tested. Until this run, phases 2 and 3 named no artifacts at all, so a resumed session could only work if it happened to invent the same filename — the fix is [#17](https://github.com/QuantEcon/skills/pull/17), and this is what checks it.
77
+
**Interrupt during the closed pass too, if you get the chance** — that is the half that has never been checkpointed at all. Resumability is asserted in three separate files and has never been tested. Two fixes have gone in ahead of this run and neither has been exercised: [#17](https://github.com/QuantEcon/skills/pull/17) named the artifacts, since before it phases 2 and 3 named none and a resumed session could only work by inventing the same filename; and [#34](https://github.com/QuantEcon/skills/pull/34) made the checkpoint cover both passes, since run 1 wrote only the open set and sent 62 closed issues straight to the catalog. This run is what checks both.
78
78
79
-
While it runs, `tail findings.md` occasionally. Every status claim should carry `[verified]`, `[stated]` or `[inferred]`, and a `[verified]` should cite `file:line`, a merged PR, or a tag — never a comment.
79
+
While it runs, `tail findings.md` occasionally. Every status claim should carry `[verified]`, `[stated]` or `[inferred]`, and a `[verified]` should cite `file:line`, a merged PR, a tag, or a commit — **and whatever it cites must resolve on the ref the audit named**, never a comment. A citation that only resolves in the author's working tree or on an unmerged branch is the defect [doctrine §2](https://github.com/QuantEcon/skills/blob/main/audit/references/doctrine.md#2-evidence-classes) now rules out; run 1's headline finding had exactly that shape.
80
80
81
81
## Step 5 — phases 3 to 5
82
82
@@ -92,9 +92,9 @@ The run cannot check any of this about itself. Ten checks, the last two of which
92
92
|---|---|---|---|
93
93
| 1 | Method section is complete | report §1 | The plan anchor, label policy and prior-audit search are not all named |
94
94
| 2 | Every claim is tagged | catalog entries | An untagged status claim — a defect by the doctrine's own rule |
95
-
| 3 |`[verified]` means verified | sample 5 "fixed" calls| Citation is a thread comment rather than code, a merged PR, or a tag|
95
+
| 3 |`[verified]` means verified | sample 5 "fixed" calls, and **resolve each on the ref named in the header**| Citation is a thread comment rather than code — or a commit, `file:line`or PR that looks right and does not resolve on that ref. `git merge-base --is-ancestor <sha> <ref>` on any commit cited. This is the check run 1 passed and should not have|
96
96
| 4 | Code beat the thread | sample 5 more | The report repeats "fixed in #204" without saying it checked the branch |
97
-
| 5 | The closed side was read | closed-set verification| Closed issues summarised from title and `stateReason` only; no deferred remainders surfaced |
97
+
| 5 | The closed side was read | closed-set verification, **and `findings.md`**| Closed issues summarised from title and `stateReason` only; no deferred remainders surfaced. Also check the checkpoint, not just the output: closed entries reaching the catalog with no matching `## Closed` block is how run 1 passed this check in its report while failing it in its log|
98
98
| 6 | Coverage is honest |`README.md` index | Unaccounted numbers waved at collectively; residue (inline review comments, GraphQL-only data) not stated |
99
99
| 7 | Siblings were checked | external cross-link registry | No sibling considered, in an org where the same fix lands in several repos as `SYNC:` PRs |
100
100
| 8 | Drafted comments are safe | GC tier | A closing keyword immediately before an `owner/repo#N` reference — that closes the *upstream* item when it lands |
0 commit comments