[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-05-31

[github-actions[bot]]

Summary

• Date of test: 2026-05-31
• Pages visited: Home (/gh-aw/), Quick Start (/gh-aw/setup/quick-start/), CLI Commands (/gh-aw/setup/cli/)
• Overall impression: The docs have a polished look and the Quick Start guide is well-structured with a clear 4-step flow. However, as a total beginner I ran into several confusing moments — particularly around new concepts like "frontmatter", "lock files", and setting up AI tokens.

---

🔴 Critical Issues

1. No definition of "frontmatter" on first mention

In Step 4 of the Quick Start, "frontmatter" is suddenly introduced: "If you changed the frontmatter (the configuration block between the --- markers...)". This is a parenthetical aside, not a proper intro. A beginner with no YAML experience won't know what this means and the parenthetical is easy to miss.

Recommendation: Add a callout box or inline tooltip: "Frontmatter is the YAML configuration block at the top of your .md file, surrounded by --- lines. It controls engine selection, triggers, and other workflow settings."

2. Lock file concept is confusing without context

The Quick Start introduces .lock.yml in Step 2 with a dense paragraph explaining it. For a beginner, seeing two files for one workflow (.md + .lock.yml) is unexpected and adds cognitive load. The instruction "Do not edit .lock.yml directly" can trigger anxiety — what happens if I accidentally do?

Recommendation: Simplify the Step 2 callout to one sentence and link to a dedicated "What is a lock file?" explanation in the reference docs.

3. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN distinction is buried

The note about COPILOT_GITHUB_TOKEN being "a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN" is in a collapsed note box. Many beginners will miss this and then wonder why their workflow fails. This is a likely #1 cause of first-run failures.

Recommendation: Surface this warning more prominently — perhaps a ⚠️ callout before Step 2, not buried inside a note box.

---

🟡 Confusing Areas

4. Home page tagline is technical

The hero subtitle says: "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions." A true beginner may not know what "guardrails" or "coding agents" means. The secondary paragraph is more helpful but appears below the fold.

Recommendation: Add a single plain-English sentence above the fold: "Automate your GitHub repo using AI — triage issues, update docs, run tests — all from a simple text file."

5. add-wizard command format is unclear

Step 2 uses gh aw add-wizard githubnext/agentics/daily-repo-status. A beginner will wonder: is githubnext/agentics a real public repo? Do I need access to it? Is this a URL or a special format? The format is explained inline but the explanation comes after the command.

Recommendation: Move the format explanation (<owner>/<repo>/<workflow-name>) before the command block, not after.

6. CLI page: command table lacks examples

The "Most Common Commands" table lists 10 commands with one-line descriptions, but no usage examples. A beginner seeing gh aw audit won't know what argument it needs or when to use it.

Recommendation: Add a second column with a minimal usage example for each command, e.g. gh aw audit 12345.

7. Video auto-plays but has no captions or fallback description

The Quick Start page has a video (<video> element). The browser fallback text says "Your browser doesn't support HTML5 video." There's no text description of what the video shows, so users in environments without video (CI, screen readers, slow connections) get nothing.

Recommendation: Add a short text description below the video summarizing what it demonstrates.

---

🟢 What Worked Well

• Estimated time upfront: "Estimated time: 10 minutes" is immediately reassuring and sets expectations.
• Prerequisites checklist: Clear, actionable list with version numbers and verification commands (gh --version, gh auth status).
• Step 3 outcome description: Explaining exactly what the created issue will look like (with a screenshot) is excellent — it gives beginners a concrete success criterion.
• Navigation bar: "Quick Start" is the first nav item — exactly right.
• add-wizard interactive flow: The bullet list explaining what the wizard does (check prerequisites → select engine → set up secret → add workflow → trigger run) is a great mental model.
• Troubleshooting link in Step 2: The tip linking to FAQ and Common Issues is well-placed.

---

Recommendations (Prioritized)

1. Quick win: Add inline definition of "frontmatter" on first use in Step 4.
2. Quick win: Move the COPILOT_GITHUB_TOKEN warning to a prominent ⚠️ callout before Step 2.
3. Quick win: Add usage examples to the CLI commands table.
4. Medium effort: Rewrite the home page hero with plain-English benefit statement.
5. Medium effort: Add a text description/transcript for the Quick Start video.
6. Longer term: Add a glossary page (frontmatter, lock file, engine, safe outputs, activation) linked from the Quick Start.

---

Screenshots

📎 [home.png] — Home page overview

📎 [quick-start.png] — Quick Start guide

📎 [cli.png] — CLI Commands page

---

References: §26703943973

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · sonnet46 4M · ◷

• expires on Jun 1, 2026, 5:21 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #36186.