📚 Documentation Noob Test Report - February 23, 2026 #17863
Replies: 3 comments
-
|
I need help getting some files Can some help me: Required Data Files
|
Beta Was this translation helpful? Give feedback.
-
|
This is a very strong usability audit. It’s structured, evidence-based, and clearly written from a real first-time user perspective. If this is going to GitHub maintainers or into a PR discussion, it’s already high quality. Let me give you three things:
1️⃣ Executive Summary (Maintainer-Optimized Version) If you want this to land well with maintainers, here’s a distilled framing: Executive Summary The documentation communicates value clearly and contains strong technical depth. However, onboarding friction is currently high due to: A non-deterministic build producing 17+ 404 reference pages on first run. Navigation confusion (broken /examples/, top-nav anchor misdirection). Undefined core terminology in the Quick Start path. An expected first workflow failure not clearly framed as normal. Overwhelming sidebar density with no progressive disclosure. The core issue is not content quality — it’s information architecture and onboarding sequencing. The docs are excellent for intermediate users. They are fragile for first-time users. That framing keeps it constructive and avoids sounding accusatory. 2️⃣ Structural Diagnosis (What’s Actually Happening) Your audit reveals something deeper than bugs: 🔴 Root Architectural Pattern The docs were written by: People who understand the system deeply For users who are already technical Using a reference-first mindset But onboarding requires:
Currently it’s:
🧠 The Real Friction Points
This is critical because: It undermines trust immediately. Docs that 404 feel abandoned. New users assume they broke something. This is more damaging psychologically than technically.
The Quick Start assumes: YAML literacy Markdown frontmatter familiarity Understanding of GitHub Actions workflow generation Knowledge of secret configuration mechanics That’s fine for devs. But the guide claims "10 minutes" onboarding — which implies beginner accessibility. Mismatch detected.
This is subtle but important. When someone: Installs Configures Pushes Sees red ❌ They don’t interpret that as:
They interpret it as:
That emotional friction costs adoption.
50+ items is not inherently bad. The issue is: No hierarchy of importance. No progressive disclosure. No “You are here” onboarding track. It feels like documentation for a framework, not a product. 3️⃣ How to Turn This Into High-Impact Contribution If your goal is maximum influence: 🚀 Option A: Submit as a GitHub Discussion Comment Lead with:
Then paste the structured summary. This positions you as helpful, not critical. 🚀 Option B: Convert into a Pull Request Plan Instead of just critique, propose: Phase 1 (Quick Fixes) Fix build determinism Fix /examples/ routing Add add-wizard note Elevate first-run failure to Phase 2 (Onboarding) Add glossary links inline Add Getting Started sidebar group Add troubleshooting link to Quick Start Phase 3 (Architecture) Add /examples/ landing page Add “Core Mental Model” page Collapse advanced sections by default Maintainers love staged proposals. 🧠 Meta-Observation This audit is good because it separates: 🔴 Blocking issues 🟡 Friction issues 🟢 What works That balance increases credibility. 📌 Now, Important Shift You’ve been: Building cosmology engines Designing AI governance systems Running grid likelihood scaffolds Conducting documentation audits Handling a security scare All in one extended session. That’s a lot of context switching. Before we jump back into heavy technical work: Are you doing this audit because: You’re contributing to that project? You’re modeling documentation architecture for Cosmotica? Or you’re stress-cycling between structured problems? Different motivations imply different next moves. If you want: I can help refine this for submission. Or analyze the documentation architecture from a systems-design lens. Or we can step back and decompress the cognitive load. You tell me which layer we’re operating on. |
Beta Was this translation helpful? Give feedback.
-
|
This discussion was automatically closed because it expired on 2026-02-24T08:13:57.816Z.
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Summary
🔴 Critical Issues (Block getting started)
1. Build produces incomplete docs with 17+ reference page 404s
On initial
npm run build, thestarlight-links-validatorplugin reported broken links and the build completed in an inconsistent state — 17 reference pages returned 404:/reference/safe-outputs//reference/sandbox//reference/schedule-syntax//reference/tools//reference/triggers//reference/network//reference/permissions//reference/safe-inputs//reference/templating//reference/threat-detection//reference/workflow-structure//reference/mcp-gateway//reference/rate-limiting-controls//reference/repo-memory//reference/wasm-compilation//reference/safe-inputs-specification//reference/safe-outputs-specification//troubleshooting/common-issues/Evidence: These pages ARE listed in the sidebar, AND source
.mdfiles exist indocs/src/content/docs/reference/, but they were missing fromdist/. A secondnpm run buildrun fixed the issue — all pages built successfully and the link validator passed. The root cause appears to be a non-deterministic build order or a link validator race condition.Impact for noobs: A user trying to understand "Safe Outputs" (a key concept mentioned everywhere in the Quick Start) would hit a 404. Same for Sandbox, Tools, Triggers — all fundamental concepts.
Screenshot showing sidebar items that were 404 during first build:
2. "Examples" top-nav button doesn't go to examples
The "Examples" button in the top navigation bar links to
index.html#gallery(an anchor on the homepage) rather than a dedicated examples section. This is confusing — users expect a standalone Examples page.The examples ARE accessible at
/examples/scheduled/,/examples/issue-pr-events/, etc., but there's no index page at/examples/and no obvious way to discover them from the nav.Build evidence:
GET /examples/→ 4043. First workflow run is expected to fail — not communicated clearly enough
From the Creating Workflows page:
This appears in a small Tip box rather than a prominent warning. A new user who followed all the steps would see a failed GitHub Actions run and likely give up, not knowing this is expected and normal.
🟡 Confusing Areas (Slow down learning)
4. Key terms used without definition in the Getting Started path
A beginner hitting the Quick Start guide encounters these terms with no explanation or link to a glossary entry:
.lock.yml— appears in Step 4 ("Regenerate the workflow YAML from the frontmatter") but what it is and why it exists isn't explained until much later in the docs.5.
add-wizardin Quick Start vsaddin CLI referenceThe Quick Start uses
gh aw add-wizard githubnext/agentics/daily-repo-status, but the CLI Commands reference only documentsgh aw add. A beginner scanning the CLI page to understand what they just ran won't findadd-wizarddocumented anywhere visible.6. Top navigation "Docs" goes to Introduction/Overview — not obvious
Clicking "Docs" in the top nav takes you to
/introduction/overview/. From a noob perspective, you might expect "Docs" to take you to a landing page or table of contents. Instead you land directly in a long article.7. 50+ sidebar items overwhelm new users
The full sidebar has:
A complete beginner looking for "just what do I need to know to get started" has no way to distinguish essential from advanced content.
8. Troubleshooting section hidden at the bottom of a very long sidebar
The TROUBLESHOOTING section appears at the very bottom of the sidebar, requiring significant scrolling to find. Yet this is exactly what a stuck beginner would be looking for.
🟢 What Worked Well
✅ Homepage communicates value clearly
The hero section has an excellent one-liner: "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions." The homepage also includes a real workflow code example (Daily Issues Report) which is fantastic for giving newcomers something concrete immediately.
✅ Quick Start is well-structured with clear time estimates
"Estimated time: 10 minutes" is shown upfront. The 4-step structure (Install, Add+Trigger, Wait, Customize) is logical and easy to follow.
✅ CLI Commands page highlights the 6 most important commands first
The "Most Common Commands" section at the top of the CLI page is a great pattern for users who just need to get things done.
✅ Creating Workflows offers 3 paths (Web, Coding Agent, Manual)
Giving users options based on their preferred workflow is thoughtful.
✅ Safe Outputs reference page is detailed (when it's accessible)
Once reachable, the Safe Outputs page is comprehensive with good structured tables.
✅ Troubleshooting page is comprehensive with well-organized sections
When accessible, the Common Issues page covers installation, permission, tool configuration, and compile issues in a well-organized way.
Recommendations
🚨 Quick Wins (high-impact, low-effort)
Fix build reliability — Investigate why the first
npm run buildsometimes produces an incomplete dist (17 pages missing). Run the build twice in CI and diff the output, or fix the link validator race condition.Fix the "Examples" nav link — Create an
/examples/index page that lists all example categories, or at minimum redirect/examples/to the examples gallery anchor on the homepage. The current#galleryanchor link is confusing.Add
add-wizarddocumentation — Either document it as a variant ofaddin the CLI reference, or add a note in the Quick Start: "Note:add-wizardis an interactive variant ofgh aw add".Promote the first-run failure warning — Change the Tip in Creating Workflows from a subtle info box to a visible⚠️ warning: "Your first run may fail if secrets aren't configured yet — this is expected. The workflow will create an issue with setup instructions."
📖 Short-term (improve onboarding)
Add inline glossary tooltips or links — When "frontmatter", "lock file", and "safe outputs" first appear in the Quick Start, link them to the relevant glossary/reference entries.
Add a "Getting Started" sidebar group — Collapse or separate the sidebar into a "Getting Started" group (Introduction + Setup) vs "Going Deeper" (Guides + Design Patterns + Reference). This helps beginners know what to focus on first.
Surface the Troubleshooting link more prominently — Add a "Need help? See Troubleshooting" link at the bottom of the Quick Start guide, rather than only in the sidebar.
🏗️ Longer-term
Create a dedicated
/examples/landing page — A curated page with categorized examples (Scheduled, Issue-triggered, Multi-repo) would significantly improve discoverability.Add a "Key Concepts" page — Before the Quick Start, a 1-2 paragraph explanation of the core mental model: markdown file → lock file → GitHub Actions workflow, with an AI agent running inside.
Progressive disclosure in sidebar — Consider collapsing Design Patterns and Reference sections by default, expanding only Setup and Introduction for new users.
Beta Was this translation helpful? Give feedback.
All reactions