Skip to content

feat: add doc-accuracy reviewer to catch harmful documentation changes - #714

Open
Iheanacho-ai wants to merge 1 commit into
siderolabs:mainfrom
Iheanacho-ai:check-doc-code
Open

feat: add doc-accuracy reviewer to catch harmful documentation changes#714
Iheanacho-ai wants to merge 1 commit into
siderolabs:mainfrom
Iheanacho-ai:check-doc-code

Conversation

@Iheanacho-ai

Copy link
Copy Markdown
Member

What this adds

A new local make target, check-doc-accuracy, that reviews documentation for changes that would harm a reader who follows them — not just typos, but commands that silently lose data, destroy things, or weaken security — before they ship.

Why

We recently had a near-miss where an edit dropped a volume-mount flag (-v $HOME/etcd:/_out/etcd) from a docker run command in the self-hosted Omni guide. The command still ran with no error, but etcd data was no longer persisted — so recreating the container would have wiped a self-hosted customer's entire Omni state. Nothing "broke" on copy-paste; it worked, and silently destroyed data.

A flag validator scoped to talosctl/omnictl would never have caught it — it was a plain docker command, and the command was valid. The only thing that catches that class of bug is a reviewer that reads the snippet and reasons about its blast radius. This tool does exactly that.

How it works

tools/doc-accuracy is a small Go program that drives the claude CLI headless as a read-only documentation reviewer (Edit/Write are disallowed — it reports, it never changes docs). It reviews the .mdx files you changed and, when a snippet or claim concerns Talos, Omni, extensions, or the discovery service, cross-checks it against the upstream source.

It prioritizes harm over mere breakage:

  • silent data loss (a stateful service run without its persistence mount)
  • destructive/irreversible commands (rm -rf, docker volume rm, kubectl delete, git push --force, disk wipes…)
  • removed safeguards (a deleted flag/mount/--dry-run/backup/confirmation)
  • security downgrades (disabling TLS/auth, chmod 777, binding to 0.0.0.0, curl … | sudo sh, leaked secrets)
  • plus ordinary wrong-flag/wrong-value/false-prose mistakes

…applied to every command in a snippet, not only the Sidero CLIs.

Usage

make check-doc-accuracy                                   # files you changed vs HEAD
make check-doc-accuracy DOC=public/omni/....mdx           # one specific file
make check-doc-accuracy DOC_ACCURACY_BASE=origin/main     # PR-style, vs a branch
make check-doc-accuracy-all                               # the whole public/ tree (slow)

@github-project-automation github-project-automation Bot moved this to To Do in Planning Aug 20, 2026
@talos-bot talos-bot moved this from To Do to In Review in Planning Aug 20, 2026
@smira smira removed this from Planning Aug 20, 2026
Comment thread tools/doc-accuracy/main.go Outdated
args = append(args,
"--output-format", "stream-json", "--verbose",
"--permission-mode", "bypassPermissions",
"--allowedTools", "Read", "Grep", "Glob", "WebFetch",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Claude:
--allowedTools is ignored under --permission-mode bypassPermissions (claude-code#12232). The whitelist here isn't actually a whitelist, so Bash is unrestricted and the read-only claim doesn't hold. --disallowedTools Edit,Write should still block those two. --permission-mode dontAsk with this allowedTools list, or --tools as a real whitelist, would do what this is aiming for.

that previously made the procedure safe. Compare against the diff and the
upstream source; a deletion that makes a command *more* dangerous is a
top-priority finding even if what remains is valid.
- **Security downgrades / exposure** — disabling TLS or auth, `--insecure`,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--insecure as CRITICAL will fail half the Talos bootstrap corpus (talosctl apply-config --insecure). Worth carving out the documented bootstrap case so this doesn't train us to ignore the tool.

Comment thread Makefile Outdated
DOC_ACCURACY_MODEL ?=

.PHONY: check-doc-accuracy
check-doc-accuracy: ## AI-review docs for accuracy/harm. Scope one file with DOC=public/path; base with DOC_ACCURACY_BASE

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default base is HEAD, so after a local commit this won't actually review anything (DOC_ACCURACY_BASE=origin/main is in the README but easy to miss). Defaulting to origin/main (with a fallback) would match how this will actually get run.

Signed-off-by: Amarachi Iheanacho <amarachi.iheanacho@siderolabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants