feat: add doc-accuracy reviewer to catch harmful documentation changes - #714
feat: add doc-accuracy reviewer to catch harmful documentation changes#714Iheanacho-ai wants to merge 1 commit into
Conversation
ada792b to
2dac14c
Compare
| args = append(args, | ||
| "--output-format", "stream-json", "--verbose", | ||
| "--permission-mode", "bypassPermissions", | ||
| "--allowedTools", "Read", "Grep", "Glob", "WebFetch", |
There was a problem hiding this comment.
From Claude:
--allowedTools is ignored under --permission-mode bypassPermissions (claude-code#12232). The whitelist here isn't actually a whitelist, so Bash is unrestricted and the read-only claim doesn't hold. --disallowedTools Edit,Write should still block those two. --permission-mode dontAsk with this allowedTools list, or --tools as a real whitelist, would do what this is aiming for.
| that previously made the procedure safe. Compare against the diff and the | ||
| upstream source; a deletion that makes a command *more* dangerous is a | ||
| top-priority finding even if what remains is valid. | ||
| - **Security downgrades / exposure** — disabling TLS or auth, `--insecure`, |
There was a problem hiding this comment.
--insecure as CRITICAL will fail half the Talos bootstrap corpus (talosctl apply-config --insecure). Worth carving out the documented bootstrap case so this doesn't train us to ignore the tool.
| DOC_ACCURACY_MODEL ?= | ||
|
|
||
| .PHONY: check-doc-accuracy | ||
| check-doc-accuracy: ## AI-review docs for accuracy/harm. Scope one file with DOC=public/path; base with DOC_ACCURACY_BASE |
There was a problem hiding this comment.
The default base is HEAD, so after a local commit this won't actually review anything (DOC_ACCURACY_BASE=origin/main is in the README but easy to miss). Defaulting to origin/main (with a fallback) would match how this will actually get run.
2dac14c to
bbf5052
Compare
Signed-off-by: Amarachi Iheanacho <amarachi.iheanacho@siderolabs.com>
bbf5052 to
1ad23e7
Compare
What this adds
A new local
maketarget,check-doc-accuracy, that reviews documentation for changes that would harm a reader who follows them — not just typos, but commands that silently lose data, destroy things, or weaken security — before they ship.Why
We recently had a near-miss where an edit dropped a volume-mount flag (
-v $HOME/etcd:/_out/etcd) from adocker runcommand in the self-hosted Omni guide. The command still ran with no error, but etcd data was no longer persisted — so recreating the container would have wiped a self-hosted customer's entire Omni state. Nothing "broke" on copy-paste; it worked, and silently destroyed data.A flag validator scoped to
talosctl/omnictlwould never have caught it — it was a plaindockercommand, and the command was valid. The only thing that catches that class of bug is a reviewer that reads the snippet and reasons about its blast radius. This tool does exactly that.How it works
tools/doc-accuracyis a small Go program that drives theclaudeCLI headless as a read-only documentation reviewer (Edit/Write are disallowed — it reports, it never changes docs). It reviews the.mdxfiles you changed and, when a snippet or claim concerns Talos, Omni, extensions, or the discovery service, cross-checks it against the upstream source.It prioritizes harm over mere breakage:
rm -rf,docker volume rm,kubectl delete,git push --force, disk wipes…)--dry-run/backup/confirmation)chmod 777, binding to0.0.0.0,curl … | sudo sh, leaked secrets)…applied to every command in a snippet, not only the Sidero CLIs.
Usage