[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-13

[github-actions[bot]]

Summary

• Date of test: 2026-04-13
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/
  • https://github.github.com/gh-aw/setup/cli/
• Overall impression: The home page is polished and communicates a strong value proposition. The Quick Start guide is well-structured, but several terms and decisions are unexplained for a true first-time user — most notably the unexplained registry namespace, the lock file concept, and the mysterious COPILOT_GITHUB_TOKEN.

⚠️ Note: Visual screenshots were unavailable — Playwright could not reach the dev server's container bridge IP (network isolation constraint). All analysis was performed using curl + HTML text extraction.

---

🔴 Critical Issues

1. COPILOT_GITHUB_TOKEN — Why a second token?

In Quick Start Step 2, the wizard mentions setting up COPILOT_GITHUB_TOKEN described as "a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN." A complete beginner will not understand:

• Why does Copilot need a separate token from GITHUB_TOKEN?
• What scopes/permissions does this token need?
• Is this a Personal Access Token (classic) or a Fine-Grained PAT? Or an OAuth token?

The text says "See Authentication for setup instructions" with a link to the auth page, but for a quick-start experience this creates an immediate roadblock with no inline guidance. A first-time user may abandon setup here.

Recommendation: Add a 2-sentence inline callout box explaining why a second token is needed (Copilot API access requires a user-level token, not the repo-scoped GITHUB_TOKEN) and what type it should be, before sending them to the Auth page.

2. gh aw add-wizard githubnext/agentics/daily-repo-status — Unexplained registry path

Quick Start Step 2 tells users to run:

gh aw add-wizard githubnext/agentics/daily-repo-status

Without any explanation of what githubnext/agentics is. A new user will ask:

• What is githubnext/agentics? A GitHub repo? A workflow registry? My repo?
• Can I use any GitHub repo here? Only certain ones?
• Why is the format owner/repo/workflow-name and not just daily-repo-status?

Recommendation: Add one sentence: "This pulls a pre-built workflow from the githubnext/agentics workflow library on GitHub."

---

🟡 Confusing Areas

3. "lock file" — Undefined term at first use

In Step 2, the wizard output is described as: "Adds the workflow and lock file to .github/workflows/."

A new user will not know what a "lock file" is in this context (it's not a typical package-lock.json). The term appears without definition.

Recommendation: Add a tooltip or parenthetical: "lock file (the compiled GitHub Actions YAML)", or link to the Compilation Process reference.

4. Search unavailable in dev mode

The docs site prominently displays: "Search is only available in production builds. Try building and previewing the site to test it out locally."

A user following the docs who starts the dev server to test the docs will see this and be confused — they are running locally, and the message suggests a different command ("preview"). The relationship between npm run dev and npm run preview is not explained.

Recommendation: Clarify the message: "Search requires a production build (npm run build && npm run preview)."

5. "Peli's Agent Factory" in the top navigation

The main navigation bar includes "Peli's Agent Factory" without any explanation. A first-time user has no context for this — it sounds like a personal project or demo page, not official documentation. It's also in the same navigation level as "Quick Start", "Docs", and "FAQ".

Recommendation: Either rename it to something more descriptive like "Examples Gallery" or add a tooltip/subtitle explaining what it is.

6. Step 3 — Missing screenshot of expected output

The Quick Start Step 3 says: "The report will look something like this:" but no image description is visible. If the image fails to load (e.g., slow connection, browser extension blocking), a user would have no visual reference for whether their workflow ran correctly.

Recommendation: Add alt text to the screenshot, or add a short text description of what the report contains immediately before/after the image.

7. CLI page — Overwhelming sidebar and deep reference complexity

The CLI Commands page sidebar lists 60+ reference items. For a beginner landing here from the Quick Start to understand basic commands, this is immediately overwhelming. The "Most Common Commands" table at the top partially addresses this, but the GHES (GitHub Enterprise Server) content appears very early and isn't relevant to most beginners.

Recommendation: Consider hiding or collapsing the enterprise/advanced sections by default on the CLI page, or add a "You are here: beginner" banner that links to only the relevant subset.

8. Additional minor confusions

• "frontmatter" (YAML config block) — the Quick Start Step 4 does provide a good inline definition ("the YAML configuration block between --- markers"), but frontmatter is used earlier in the page without this definition.
• "lock file" in Step 4 — same issue as Add workflow: githubnext/agentics/weekly-research #3 above, appears twice.
• gh aw compile — Step 4 mentions this command but a beginner may not understand when it is required vs optional. The condition ("if you have changed the frontmatter") uses the term before it's well-understood.
• Prerequisites: "GitHub Actions enabled" — The linked docs page is the official GitHub docs but requires several clicks to actually check the setting. Consider showing the specific UI path: "GitHub.com → Your Repo → Settings → Actions → General → Allow all actions".

---

🟢 What Worked Well

1. Clear value proposition on homepage — The headline "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" immediately answers "what is this?"
2. Estimated time — "Estimated time: 10 minutes" on the Quick Start sets clear expectations.
3. Four-step structure — The numbered steps with clear section headings make the Quick Start easy to scan.
4. Prerequisites checklist — Listing GitHub Copilot/Claude/Codex as separate options, plus GitHub CLI version requirement, is thorough.
5. Tip boxes — The authentication issues tip and troubleshooting tips in Step 2 proactively address common failure points.
6. "Most Common Commands" table — The CLI page leads with the 7 most important commands, which is exactly what a beginner needs.
7. Security architecture — The homepage's 5 guardrail sections (read-only tokens, zero secrets, firewall, safe outputs, threat detection) are clear and build trust.
8. Embedded video in Quick Start — Having a demo video embedded inline is excellent for new users who prefer visual walkthroughs.
9. frontmatter is linked — Step 4 links the word "frontmatter" to the reference docs, good progressive disclosure.

---

Recommendations

Quick wins (high impact, low effort):

1. Add a 2-sentence inline box before the COPILOT_GITHUB_TOKEN mention explaining why it's needed and what type to create.
2. Add one sentence explaining githubnext/agentics is the official workflow library.
3. Add a parenthetical after "lock file" defining it.
4. Update the search-unavailable message to mention npm run build && npm run preview.

Medium-term improvements:
5. Rename "Peli's Agent Factory" in the nav to "Examples" or "Gallery" with a subtitle.
6. Add alt text / fallback text description to the Step 3 report screenshot.
7. Consider a "beginner-friendly view" of the CLI page that hides enterprise sections.

---

References:

• §24345362325 — Workflow run that produced this report

Generated by Documentation Noob Tester · ● 2.1M · ◷

• expires on Apr 14, 2026, 1:31 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #26224.