[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-30

[github-actions[bot]]

Summary

• Date of test: 2026-06-30
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/ (Quick Start)
  • https://github.github.com/gh-aw/setup/cli/ (CLI Commands)
• Overall impression: The docs have a polished look and a clear "Quick Start" path, but a true beginner will hit a wall when setting up authentication (COPILOT_GITHUB_TOKEN / PATs) without much hand-holding. The volume of upfront jargon ("frontmatter", "lock file", "fine-grained PAT") risks losing newcomers before they complete step 2.

---

🔴 Critical Issues Found

1. add-wizard command syntax is non-obvious

Page: Quick Start → Step 2

gh aw add-wizard githubnext/agentics/daily-repo-status

For a beginner, add-wizard is not an intuitive command name — it sounds like it adds a wizard, not that it is an interactive wizard. More importantly, the owner/repo/workflow-name three-part format is not explained before it's used; the explanation comes after the command. First-time users may copy-paste without understanding the format and get confused if they mistype.

📎 [quick-start.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e245c43451a87dff077234fbf3fee417a79bdc405273c192cf41e3c11d29f5b1.png?raw=true

2. COPILOT_GITHUB_TOKEN setup is complex and buried

Page: Quick Start → Step 2

The guide tells the user they need to create a fine-grained Personal Access Token (PAT) with specific Copilot Requests: Read permissions. For a beginner:

• "Fine-grained PAT" is GitHub-internal jargon with no linked explanation of what it is at the point of use.
• The token needs a specific, non-obvious account permission ("Copilot Requests") that is hard to find in the GitHub UI.
• The note boxes for COPILOT_GITHUB_TOKEN and ANTHROPIC_API_KEY appear after running add-wizard, but the token is needed during add-wizard — the prerequisite comes after the step that requires it.

📎 [confusing-pat-setup.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/7d8650bc7c37a80f43cdd2b22c786d56e6ba7b6a387e3776a27c52961a3d199a.png?raw=true

---

🟡 Confusing Areas

3. "Frontmatter" used without definition

Page: Quick Start → Step 4

"If you changed the frontmatter (the configuration block between the --- markers at the top of the file)..."

The word "frontmatter" appears in the text with a partial clarification in parentheses — but only in the optional customization step. It then links to the frontmatter reference. For a beginner who doesn't know YAML, even "the configuration block between --- markers" may not be clear. A small visual annotation showing a real frontmatter block at the beginning of the guide would help.

📎 [quick-start-prerequisites.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/ef710fbd6eb5b4da73399eb1dbfb35b7b75c68247756462fa071e308edbeee47.png?raw=true

4. .lock.yml concept introduced without clear "why"

Page: Quick Start → Step 2

"The .lock.yml is the compiled GitHub Actions workflow generated from your markdown — it is what actually runs, and it is regenerated automatically, so you never edit it by hand."

This is good! But it appears as a note block after the command that creates it. A newcomer who sees a second file appear in .github/workflows/ may be confused before they reach this explanation. Moving this note to before the command, or adding an inline inline comment in the command output preview, would reduce that confusion.

5. CLI Commands page — GHES content appears before "hello world" commands

Page: CLI Commands

The page begins with "Most Common Commands" (great!), then jumps into GitHub Enterprise Server setup, pinning versions, corporate firewall scenarios, etc. For a new user who just wants to run their first workflow, the GHES content between the installation snippet and first real examples is overwhelming and makes it feel like the tool is aimed at enterprise power users.

📎 [cli-top.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/52dab7acc61a935a1d3f9daddaa6be4e6a64f8ecf89949f43cbdb05d65e297ce.png?raw=true

📎 [cli-commands.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/c902f57c38b524b20def9d4b272394e9df4c674d097a20ba8be95c1c728b3073.png?raw=true

6. "Peli's Agent Factory" link in main nav

Page: Home / all pages (nav bar)

The primary navigation bar includes a link called "Peli's Agent Factory" — which points to a blog post. For a first-time visitor this is confusing: it looks like a feature or product name rather than a blog series. It takes prime nav real estate that might be better used for "Getting Started" or "Examples."

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/8384556e2a918576e66c6ba5cd5435682bca3a31d52a32ccfa8d5d09cbd4ac25.png?raw=true

7. "Public Preview" notice is not prominently positioned

Page: Home

The Public Preview notice is buried in the middle of the homepage body as a blockquote. For a product in preview that "may change significantly", this should appear near the top of every page (or at least the home page) so new users set their expectations correctly from the start.

---

🟢 What Worked Well

• Estimated time on Quick Start (⏱️ 10 minutes): Excellent — immediately tells the user what to expect.
• Prerequisites list: Very clear, with links to all required installs. The checklist format is easy to scan.
• Video at the top of Quick Start: Including a walkthrough video right at the top is perfect for visual learners and gives a great orientation before reading.
• Most Common Commands table on the CLI page: Fantastic quick reference. Shows the 10 most important commands at a glance.
• Step-by-step numbered structure in Quick Start: Easy to follow, each step is clearly scoped.
• Tip/Note callout boxes: Well-used throughout the Quick Start to surface helpful info without cluttering the main flow.
• gh aw status and gh run watch mentioned in Step 3: Great UX tip that new users would never think to look for on their own.

---

Recommendations

Quick Wins 🚀

1. Move the auth token setup before Step 2. Add a "Before you run add-wizard: make sure you have your AI secret ready" callout with the token setup instructions. This removes the ordering confusion.

2. Add a glossary tooltip or first-use footnote for "frontmatter". The first time the word appears anywhere in the docs, add a brief inline definition or a link to a glossary entry.

3. Rename or subtitle the nav entry "Peli's Agent Factory" → "Blog / Agent Factory" to clarify it's a blog series, not a product.

4. Add a visible "Public Preview" badge at the top of the homepage (above the fold) and/or in the site-wide header banner.

5. Explain add-wizard format before the command: Add one sentence like "The format is owner/repo/workflow-name, where owner/repo is the GitHub repository hosting the workflow" before showing the command.

Longer-term Improvements 🏗️

6. Move GHES content on the CLI page into a collapsible <details> block or a separate "Enterprise Setup" page. Keep the standard installation flow clean for beginners.

7. Add a brief visual explainer of .lock.yml with a small diagram showing: daily-repo-status.md → compile → .lock.yml → GitHub Actions runs .lock.yml. This mental model takes seconds to explain visually but currently requires reading several paragraphs.

8. Consider a "Beginner" vs "Advanced" track at the top of the docs. Even a simple "🟢 New here? Start with Quick Start" / "🔵 Already using gh-aw? See CLI Reference" toggle at the top of the sidebar would help users self-select.

---

Screenshots

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/8384556e2a918576e66c6ba5cd5435682bca3a31d52a32ccfa8d5d09cbd4ac25.png?raw=true

📎 [quick-start.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e245c43451a87dff077234fbf3fee417a79bdc405273c192cf41e3c11d29f5b1.png?raw=true

📎 [quick-start-prerequisites.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/ef710fbd6eb5b4da73399eb1dbfb35b7b75c68247756462fa071e308edbeee47.png?raw=true

📎 [confusing-pat-setup.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/7d8650bc7c37a80f43cdd2b22c786d56e6ba7b6a387e3776a27c52961a3d199a.png?raw=true

📎 [cli-top.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/52dab7acc61a935a1d3f9daddaa6be4e6a64f8ecf89949f43cbdb05d65e297ce.png?raw=true

📎 [cli-commands.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/c902f57c38b524b20def9d4b272394e9df4c674d097a20ba8be95c1c728b3073.png?raw=true

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 67 AIC · ⌖ 29.3 AIC · ⊞ 1.8K · ◷

• expires on Jun 30, 2026, 9:17 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #42648.