[docs-noob-tester] 📚 Documentation Noob Test Report - June 13, 2026

[github-actions[bot]]

Summary

• Date of test: June 13, 2026
• Pages visited: Home (/gh-aw/), Quick Start (/gh-aw/setup/quick-start/), CLI Commands (/gh-aw/setup/cli/)
• Tester persona: Complete beginner — never used GitHub Agentic Workflows before
• Overall impression: Strong home page and clear step-by-step Quick Start, but several spots will trip up a true beginner — especially the secrets/token setup and unexplained jargon.

References: §27457440763

---

🔴 Critical Issues

1. PDF loading error on the home page

The WorkflowHero component throws a JS console error on first load:

Failed to load PDF slides InvalidPDFException

If the PDF was meant to show a demo or walkthrough, new users are missing it silently.

2. Secret setup instruction is confusing

Quick Start Step 2 shows:

gh secret set COPILOT_GITHUB_TOKEN < /path/to/token.txt

A beginner who just generated a token in their browser has no idea what /path/to/token.txt means. The docs do not explain that you must save the token to a file first, or that < is a shell redirect. This is the most likely abandonment point.

📎 [quick-start-prerequisites.png] —

---

🟡 Confusing Areas

3. "Peli's Agent Factory" in the primary navigation

Top nav shows: Quick Start | Create | Examples | Docs | FAQ | Blog | Peli's Agent Factory. A personal name in the main nav is jarring and confusing for new users.

📎 [home.png] —

4. "frontmatter" used without inline definition

Quick Start Step 4 says "If you changed the frontmatter..." without a concise inline explanation. The link to the reference page helps, but most beginners won't click it before being confused.

5. gh aw add-wizard format not obvious

The owner/repo/workflow-name format is explained below the command, but beginners may wonder: is this a GitHub URL path? A folder? An explicit example linking to the GitHub file URL would help.

📎 [quick-start.png] —

6. CLI Commands page goes advanced too quickly

After the excellent "Most Common Commands" table, the page immediately covers version pinning, standalone installer, GitHub Actions setup action, and GHES — all edge cases. Beginners have to scroll past a lot that doesn't apply to them to find basic usage.

📎 [cli-commands.png] —

7. No brief "what are you installing" intro at top of Quick Start

The Quick Start jumps straight into prerequisites. One sentence — "You are about to set up your first AI-powered GitHub Action, written in plain markdown" — would orient first-time users.

8. "What's next?" only links to a blog post

The Quick Start ends by linking to "Peli's Agent Factory" (a personal blog post). A better next step for beginners would be: try creating your own workflow, or explore the examples gallery.

---

🟢 What Worked Well

• ✅ Clear CTAs on home page — "Quick Start with CLI" is prominent and above the fold
• ✅ Estimated time shown — "⏱️ Estimated time: 10 minutes" sets expectations
• ✅ Prerequisites thoroughly listed — All dependencies with install links
• ✅ Auth troubleshooting TIP callout — Well-placed and helpful
• ✅ Most Common Commands table — Excellent top-of-page orientation on CLI Commands
• ✅ Guardrails Mermaid diagram — Renders well; communicates the security model visually
• ✅ NOTE callouts for token setup — Colocated with the step that needs them

---

💡 Recommendations

Quick wins:

1. Fix /path/to/token.txt — add echo "<your-token>" | gh secret set COPILOT_GITHUB_TOKEN as an alternative, or explain the file-save step explicitly
2. Move "Peli's Agent Factory" out of primary navigation (into Blog sidebar or submenu)
3. Add a one-line inline definition of "frontmatter" on first use
4. Add a one-sentence intro at the top of Quick Start explaining what the user is installing

Medium-term:
5. Restructure CLI Commands — move basic usage examples up; collapse version-pinning and GHES into an "Advanced" section
6. Improve "What's next?" — link to a beginner-friendly second tutorial or the examples gallery
7. Fix or remove the broken PDF in WorkflowHero

Longer-term:
8. Consider a 30-second explainer animation or video on the home page
9. Tag CLI Commands entries as beginner vs. advanced

---

🤖 Automated noob test — workflow run §27457440763

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 861.5 AIC · ⌖ 14.5 AIC · ⊞ 19.1K · ◷

• expires on Jun 13, 2026, 9:30 PM UTC-08:00