diff --git a/.agents/skills/end-to-end-doctesting/SKILL.md b/.agents/skills/end-to-end-doctesting/SKILL.md new file mode 100644 index 00000000..cb8c342f --- /dev/null +++ b/.agents/skills/end-to-end-doctesting/SKILL.md @@ -0,0 +1,431 @@ +--- +name: end-to-end-doctesting +description: Blank-slate end-to-end smoketest of General Translation's setup docs. Use when asked to test, QA, or smoketest a docs branch or docs PR, or on any message starting with /end-to-end-doctesting (e.g. "/end-to-end-doctesting", "/end-to-end-doctesting 3427", "/end-to-end-doctesting content#354 all", "doctest the refactor branch"). Resolves the docs branch, follows the chosen quickstart verbatim in a throwaway project like a first-time user, logs every deviation needed to succeed, and replies with a glanceable verdict plus a full .md report when there are findings. +--- + +# End to End Doctesting + +You QA documentation by **being the user**: set up General Translation from a +blank slate, following the docs on a given branch to the letter, and report +where they break. Most GT users now point an agent at the docs and say +"install this" — an agent following the docs verbatim IS the representative +first-run experience. Where you get stuck, users get stuck. + +Prime directive: **the docs are the system under test, not you.** The job is +never "get GT working by any means" — it is "discover whether the docs alone +get a newcomer to a working result." Every time you need knowledge that is +not on the page, that is a finding, even if you already know the answer. + +Run fully non-interactively. Never ask the user questions mid-run. The only +acceptable pre-run reply is a one-line refusal when a hard precondition fails +(no target on a first-ever run, `gh` unauthenticated, no network, no node). + +## Command grammar + + /end-to-end-doctesting [target] [scope] [keep] + +- `target` — which docs to test: + - omitted → `last_target` from state (see § State). No state either → reply + with one line asking for a branch or PR; stop. + - gt-cloud PR: `3427`, `gt-cloud#3427`, or a full PR URL + - content PR: `content#354`, or a full PR URL + - a branch name: tried on `generaltranslation/content` first, then on + `generaltranslation/gt-cloud` + - `main` → content repo `main` (the live docs) +- `scope` — how much of the tree to test. **Default: everything — the full + sweep (§3)**: every actionable instruction on every page under + `docs/en-US/**` at the resolved SHA. Narrowing values: a flow (`next` | + `react` | `node` | `python` | `cli` | `rn` | `tanstack`) walks just that + quickstart; `+` adds a follow-on guide; a value containing `/` + or ending in `.mdx` targets that page or directory; `all` = every + quickstart but nothing else (the old mode). Expect: one flow 10–30 min; + the full sweep runs for hours — it's built to run unattended, resumes + from its ledger if interrupted, and always reports its own coverage. +- `keep` — keep the sandboxes after the run and include their paths in the + reply. + +Utility subcommands (no sandbox, answer inline, ≤15 lines): + +- `/end-to-end-doctesting help` — the grammar above, current defaults, and a one-line + summary of the last run from state. +- `/end-to-end-doctesting ls [target] [path]` — resolve the target and show what's there: + by default the top-level dirs under `docs/en-US/` plus the spine pages the + flow globs matched on that branch; with a `path`, list that subtree. This + is how you see what's available to point at, per branch, without cloning + anything. +- `/end-to-end-doctesting doctor` — host preflight: git/node/npm versions, `gh` or + `GITHUB_TOKEN` present and both repos reachable, `python3` (python flow), + `DOCTEST_GT_*` credentials present (report yes/no, never values), + `$DOCTEST_HOME` writable, free disk (warn under ~10 GB — sandboxes hold + node_modules), and the state summary. Run this first on a fresh host. + +## Plain-English steering + +The grammar is shorthand, not a parser — the agent reading this is a full +agent. Anything else in the message is an instruction: "use pnpm", "keep the +sandbox", "only steps 1–3", "walk the react flow on yesterday's target", +"skip the dev-server check". Apply it, and record every deviation from the +default protocol in the report header as `Run variant:` so repeat-tracking +across runs stays honest. Questions about a past run ("why did step 6 +fail?", "show finding 2") are answered from the saved report and state — +don't re-run for a question. Anything that posts externally (`file it` → PR +review comments) still happens only on that explicit ask. + +## 1 · Resolve the target to a content SHA + +Docs text lives in `generaltranslation/content` under `docs/en-US/**`. That +repo is a git submodule of `generaltranslation/gt-cloud` at +`apps/landing/content`, so docs PRs come in pairs (e.g. gt-cloud#3427 ⇄ +content#354). **The content SHA is the thing you test.** Resolve: + +- **gt-cloud PR or branch** → head branch, then read the submodule gitlink: + + gh pr view --repo generaltranslation/gt-cloud --json headRefName + gh api "repos/generaltranslation/gt-cloud/contents/apps/landing/content?ref=" --jq .sha + + (Fallback if that returns nothing: recursive tree, take the + `apps/landing/content` entry with `"type":"commit"`.) +- **content PR** → `gh pr view --repo generaltranslation/content --json + headRefName` → CONTENT_SHA = head of that branch. +- **Bare number** → try as gt-cloud PR first, then content PR. If both exist, + prefer gt-cloud and say which you picked in the report header. +- Record CONTENT_SHA (short), its branch/PR provenance, and the paired PR if + one is linked. Always re-resolve — never reuse a SHA from a previous run. + +Fetch the docs tree once and cache it for the run: + + gh api "repos/generaltranslation/content/git/trees/?recursive=1" \ + --jq '.tree[] | select(.type=="blob") | .path' + +Fetch each page you follow with: + + gh api "repos/generaltranslation/content/contents/?ref=" --jq .content | base64 -d + +URL ↔ file mapping (fumadocs): `/docs/react/nextjs/config` → +`docs/en-US/react/nextjs/config.mdx`; parenthesized route groups like +`(additional-frameworks)` appear in file paths but are elided from URLs. + +**No `gh` on the host?** Every `gh api ` here is equivalently +`curl -s -H "Authorization: Bearer $GITHUB_TOKEN" https://api.github.com/` +(fine-grained PAT, read-only on the two repos), and +`gh pr view --repo --json headRefName` is +`curl … https://api.github.com/repos//pulls/` → `.head.ref`. +curl ships with macOS; parse JSON with `node -e` or by reading it — don't +assume `jq`. If neither `gh` nor `GITHUB_TOKEN` is available, stop with a +one-line reply saying which to provide. + +## 2 · Pick the docs to walk + +Find the flow's **spine page** by globbing the fetched tree — paths move +between branches, so never hardcode them. Current refactor-era globs, under +`docs/en-US/`: + +| flow | spine glob | blank-slate substrate | +|----------|------------------------------------------|-----------------------------------------------| +| next | `react/*nextjs*quickstart*.mdx` | `create-next-app` (App Router, TS, defaults) | +| react | `react/react-quickstart*.mdx` | Vite react-ts template | +| node | `node/quickstart.mdx` | `npm init -y`, empty dir | +| python | `python/quickstart.mdx` | fresh venv | +| cli | `cli/quickstart.mdx` | whatever the page itself assumes | +| rn | `react/*react-native*.mdx` (quickstart) | Expo default template | +| tanstack | `react/*tanstack*quickstart*.mdx` | TanStack Start scaffold | + +Older-IA branches (pre-refactor) keep quickstarts under `next/`, `react/`, +etc. — the globs are hints, not contracts; search the tree and pick the page +a newcomer would land on. If a flow's spine page doesn't exist on this +branch, mark the flow `not present` in the report and move on (only a finding +if the branch claims to support it, e.g. a nav entry pointing nowhere). + +The **tested set** = the spine page plus every page a step forces you through +(e.g. the CLI auth page when a step says "run `npx gt auth`... see the +Dashboard docs"). List all tested pages in the report. Do not crawl beyond +need — the content repo has its own link CI; only report a dead link when a +step actually depended on it. + +The substrate is the precondition the doc states ("assumes Next.js 13 or +later" → current `create-next-app` with defaults). Substrate commands are +environment prep, not doc steps — log them separately in the transcript. If +the doc never states what it assumes you're starting from, that's a 🟡 +`missing-step` finding. + +**Reachability:** confirm the spine (and every pulled-in page) is reachable +from the branch's nav manifest (meta.json / filetree manifest). A page a +reader can only reach by URL-guessing is a 🟡 `nav-orphan` finding — correct +content nobody can find has a long history here. + +**Extensions — the user's second session.** `+` walks a follow-on +guide on top of the finished spine app, because real adopters don't stop at +the quickstart and nobody has ever executed the follow-on guides end-to-end. +Known exts (same glob-discovery rules as flows): `routing` (URL +routing/middleware guide — historically demands relocating every page and was +never tested as written) · `ssg` (static rendering guide, if the branch has +one). Extension findings get ids `+/-`. Only run an +extension after its spine passes. + +## 3 · The sweep — default scope + +With no narrowing scope you test **everything the tree tells a reader to +do**: every page under `docs/en-US/**` at the resolved SHA (blog, devlog, +authors, and templates are out), every actionable item — commands to run, +code to add, files to create, options a page says can or should be set. The +unit of coverage is the actionable item, not the page. + +1. **Inventory.** Walk the tree once and extract each page's actionable + items (imperatives, fenced commands, config/code the reader is told to + apply). Stable id per item: `#-`. +2. **Ledger.** Persist to + `$DOCTEST_HOME/sweeps/-/ledger.json`: every item, its + disposition, its status. A ledger already existing for this target+SHA + means RESUME — never redo finished items. The ledger is the coverage + proof; the report is generated from it. +3. **Dispositions** — every item gets exactly one: + - `executed` — run here, verdict attached + - `needs-creds` — ⚪ unless `DOCTEST_GT_*` present (then execute) + - `needs-external-service` — dashboard/browser/CMS-account steps + (Sanity/Storyblok/Mintlify accounts, OAuth handoffs): execute the local + half, ⚪ the remainder with one line stating exactly what a human must + click + - `destructive-platform` — never run against platform state; probe the + local-only semantics instead (verification probe 7) + - `not-runnable-here` — needs hardware/OS the box lacks (device builds) + - `duplicate-of ` — same instruction on multiple pages: execute once, + link the rest, and log substantive differences between the copies as + `drift` +4. **Order and context.** Quickstart flows first (stateful walks per §2/§5, + one sandbox each), then their extensions, then everything else grouped + into per-section work units (`react/`, `cli/`, `node/`, `python/`, + `platform/`, `integrations/`). Run each item in the cheapest valid + context: on top of the finished flow app when the page assumes one (most + guides do), in a fresh sandbox when it assumes none; config options by + actually setting them and observing the documented effect. +5. **Fan-out.** If the harness offers subagents, give each work unit its own + subagent and sandbox; each returns findings + ledger updates as + structured data; merge centrally (dedup by finding id, keep the + worst-severity copy). No subagents → run units sequentially; the ledger + makes dying and resuming cheap. +6. **Timeboxes.** ~20 min soft cap per work unit; ~4 h per sweep + (`DOCTEST_MAX_HOURS` overrides). Hitting a cap is reportable coverage + ("stopped at 71% — the ledger lists what remains"), never a silent stop. +7. The static snippet pass (§5) covers the WHOLE tree in a sweep, not just + changed pages. + +## 4 · Sandbox + +- `WORKDIR=$(mktemp -d /tmp/doctest--XXXXXX)` — everything happens in + there. Never touch any existing checkout, and never run package installs + outside WORKDIR. +- Capture the environment first: `node -v`, `npm -v`, `git --version`, OS, + date. After install steps, capture resolved GT package versions from the + lockfile. All of this goes in the report header. +- Non-interactive discipline: set `CI=1`. Where the SUBSTRATE needs flags to + avoid prompts (e.g. `create-next-app --yes`), use them. Where a **doc + command** prompts interactively with no documented flags, answer with the + most default choice, and log a 🟡 `agent-blocker` finding — agents (and CI) + cannot follow that step, which is a doc defect by this skill's premise. +- Credentials: if the host provides `DOCTEST_GT_API_KEY` / + `DOCTEST_GT_PROJECT_ID` (a dedicated throwaway GT project), export them + inside the sandbox under whatever names the docs specify, at the step where + the docs say to. If absent, execute key-gated steps anyway to capture their + failure mode, then mark them ⚪ `skipped-credentials` (not blockers) and + continue with whatever remains testable. Never print key values; redact + them from every transcript and report. +- Timeboxes for narrowed runs: ~15 min soft cap per flow, 25 min per run + (`all`: 60). Sweeps use §3's caps. A cap overrun is itself a signal — + report it rather than silently grinding. + +## 5 · Walk the docs — newcomer mode + +Read the spine page top to bottom first; enumerate its steps (the refactor +uses numbered `### N.` headings — but count whatever the page's structure +implies). Then execute in order: + +- Run every command **exactly as written**, in the order written. Typo in the + doc → run the typo. Pick ONE package-manager tab per run (npm unless the + page defaults otherwise) and note the choice — but statically compare the + other tabs: packages, flags, or steps that differ in substance between tabs + is a 🟡 `drift` finding (don't run all four; read them). +- While reading, collect the page's explicit falsifiable claims ("no other + changes to your config are needed", "works in both server and client + components", "X is created automatically"). After verification, spot-check + the cheap ones — ≤4 per run, ≤2 min each. Observed ≠ claimed → 🟡 + `claim-mismatch` (a doc that overpromises burns more trust than one that + errors). +- Code blocks with `title="path"` → write exactly that file at exactly that + path. A code block with no stated destination → 🟡 `ambiguous`, then use + the most literal placement. +- Placeholders (`YOUR_API_KEY`, project IDs) → substitute sandbox + credentials. If the page never says where the value comes from → 🟡 + `missing-step`. +- Ambiguous instruction → take the most literal reading, log 🟡 `ambiguous` + with the reading you took. +- Optional/collapsed steps: follow the main path; note untested branches in + the report ("not walked"). +- **No outside knowledge.** If a step only works because you knew something + the page doesn't say, stop pretending it passed — that's a finding. + +### When a step fails — expert mode (logged) + +1. Capture the failure verbatim: command, exit code, the error text (trim to + the ~20 relevant lines). Capture exit codes without piping the command + (`cmd > out 2>&1; echo $?`) — piping through `head`/`tee` reports the + pipe's status, and zsh doesn't honor bash's `PIPESTATUS`. +2. Diagnose and try the **smallest** fix that unblocks. Up to 3 distinct + approaches, then stop trying. +3. Fixed → log a 🟠 `detour`: root cause, the exact fix applied, and a + concrete suggested doc edit (see § Findings). Resume newcomer mode at the + next instruction. +4. Not fixed → 🔴 `blocker`. Continue with later steps only if they're + independent of the wreckage; otherwise the flow ends here. + +### Verification — the value moment + +Docs "completing" is not the bar; a working result is. After the last step +(every probe below earned its place by catching a real shipped bug): + +1. Boot the app the way the flow's framework does (`npm run dev` etc.), wait + for ready, and request the page — expect HTTP 200 and no framework error + overlay in the HTML. +2. Actuate the locale-switch mechanism the docs provided — with or without + credentials (e.g. set the documented cookie and re-request; no browser + needed). With translations: translated text must actually render — + silent default-locale = 🔴 on that step even if every command + "succeeded". Keyless: record what the switch does, and whether anything + (UI, console, server log) explains a no-op — an unexplained no-op is the + 🟡 silent-failure class. +3. Check the served ``: it must exist and match the rendered + locale — the one bug an i18n product can't afford. +4. Request `/` (e.g. `/es`). A 404 is fine only if the + flow discussed routing; an undiscussed 404 = 🟡 (a week-one decision the + docs never surface). +5. If credentials present, run the production build the docs configured — + and read the route table, don't just pass/fail it: quickstart-scaffold + routes flipping from `○ (Static)` to `ƒ (Dynamic)` with no doc mention of + the trade-off = 🟡. Keyless: skip, and say so. +6. If the page itself never tells the reader how to see it working — no + "run this and you should see X" — log 🟡 `no-verify`. Silent success is + the top historical failure mode of these docs. +7. If the flow ran a file-generating command (`generate`, `translate`, + `init`, `setup`), run it once more, unmodified, and diff the project: + silently clobbered edits or new stray files = 🟡 `destructive` (the + merge-vs-clobber semantics of these commands are undocumented and have + already cost one user their manual overrides — never probe this against + platform state, local files only). + +### Snippet pass — static + +The walk covers pages a user executes; this covers pages a user *copies +from*. Scope: in a sweep, every page in the tree; in a narrowed run, the +pages the PR changes (bare-branch target → just the tested set; cap at 40 +pages and say so if you truncate). Extract fenced code blocks and parse — +don't run, don't type-check: + +- `json` → `JSON.parse` (this alone has caught shipped invalid examples) +- `js` → `node --check`; `ts`/`tsx` → parse-only via the sandbox's + `typescript` (`transpileModule`, report syntactic diagnostics only) +- `bash` → `bash -n` +- Skip blocks that are deliberate fragments: containing `...`/`…` + placeholders, `[!code` annotations, or diff markers. + +Hard parse failures only, reported as 🟡 `broken-code` with file:line — a +snippet that can't parse can't ever have been tested. No style opinions, no +type errors on fragments; false positives kill this check's credibility. + +## 6 · Findings + +Each finding gets a stable id `/-` (e.g. +`next/6-translate-needs-prod-key`), a severity, and a category: + +- Severities: 🔴 `blocker` — no way through, even with fixes · 🟠 `detour` — + docs-as-written fail; you found a fix a newcomer (or a weaker agent) + wouldn't · 🟡 `friction` — passable but ambiguous, misleading, or + agent-hostile · ⚪ `skipped` — untestable here (credentials, platform). +- Categories: `wrong-command`, `broken-code`, `missing-step`, `ambiguous`, + `env-gap`, `agent-blocker`, `no-verify`, `dead-link`, `claim-mismatch`, + `nav-orphan`, `drift`, `destructive`. +- Every 🔴/🟠 carries: what the doc says (quoted), what actually happened + (error excerpt), root cause, fix applied (if any), and a **suggested edit** + — concrete replacement text in a ```suggestion block against the exact file + and line, ready to paste into a PR review. "Clarify this section" is not a + suggested edit. Link every finding to its source line: + `https://github.com/generaltranslation/content/blob//#L`. +- Repeats: compare finding ids (fuzzy-match titles too) against `history` in + state for the same target family. Tag `(repeat ×N)` or `(new)`; findings + from the previous run that no longer reproduce get one "fixed since last + run" line in the report. Repeats are the tech-debt signal this skill + exists to kill — surface them first. + +Flow verdict: ✅ CLEAN (no findings) · 🟡 FRICTION (worst is 🟡) · 🟠 +DETOURED (worst is 🟠) · 🔴 BLOCKED. Run verdict = worst flow. + +## 7 · Report back + +**Summary — always, ≤6 lines, glanceable.** Shape: + + 🩺 end-to-end-doctesting · next · content@077b1f9 (via gt-cloud#3427 · jackie-content-docs-refactor) + verdict: 🟠 DETOURED — setup finished, but 2 steps needed fixes the docs don't give + 🔴 0 · 🟠 2 · 🟡 3 · ⚪ 1 (no creds) · 9 steps · 12 min · covers 3 of the PR's 14 docs pages + worst: step 6 — `npx gt translate` rejects the key the quickstart had you create (new) → + report: + + The coverage clause is mandatory — partial coverage must never read as + "checked everything". Narrowed run on a PR: pages, as above. Sweep: items + from the ledger, e.g. `412 executed · 57 ⚪ (reasons in report) · 469 + items · 148 pages`. The `verdict:` line is stable-format on purpose + (`verdict: 🔴|🟠|🟡|✅ …`) so machines can parse it. + + Clean run — two lines, no report file: + + 🩺 end-to-end-doctesting · next · content@077b1f9 (gt-cloud#3427) — ✅ CLEAN + 9/9 steps verbatim · dev 200 · es renders translated · prod build ok · 11 min · no report needed + +**Full report — when the verdict isn't CLEAN** (or on explicit request). +Build it from `references/report-template.md` (worked example from a real +run: `references/example-report.md`), save to +`$DOCTEST_HOME/reports/doctest---.md`, +and put its path on the summary's `report:` line (a harness that can attach +files should attach it too). The report holds the full findings, the tested +page list, environment, and the complete command transcript with exit codes. + +Never post to the PR, file issues, or push anything on your own. End the +report line with: `say "file it" to turn these into PR review comments` — +and only do that when asked in a follow-up. + +**Headless runs** (`claude -p "/end-to-end-doctesting …"` — CI, cron, a bot relaying to +chat): your final printed message is the entire interface — whatever +launched you relays or parses it verbatim. End the run with exactly the +summary shape above and nothing after it, and still write the report file. +You still never ask questions; a failed hard precondition is a one-line +reply, not a prompt. + +## 8 · State + +`$DOCTEST_HOME/state.json` (default `DOCTEST_HOME=~/.doctest`). Read it +before resolving; write it after every run, pass or fail: + + { + "last_target": "gt-cloud#3427", + "last_flow": "next", + "resolved": { "gt_cloud_branch": "...", "content_ref": "...", "content_sha": "..." }, + "last_run": { "at": "", "verdict": "DETOURED", "report": "", "counts": {"blocker":0,"detour":2,"friction":3,"skipped":1} }, + "history": [ { "at": "...", "target": "...", "flow": "...", "verdict": "...", "findings": [{"id":"...","sev":"...","title":"..."}] } ] + } + +Keep the last 20 runs in `history` (that's what powers repeat detection). +First run ever: no state file means no default target — the user names a +branch or PR once, and state remembers it from then on. + +## 9 · Cleanup + +Delete WORKDIR after the report is written (`keep` → leave it and print the +path). Reports and state stay. Leave nothing else on the machine — no global +installs, no config outside WORKDIR and $DOCTEST_HOME. + +## Reference + +- `references/report-template.md` — full-report skeleton and summary shapes + (§7 builds from this). +- `references/example-report.md` — a real narrowed run's report, the worked + example (including `Since last run` repeat-tagging). +- `references/operations.md` — operator notes: host requirements, + headless/CI recipes, tester-model choice, the naive first-timer probe, + state layout. Nothing in it changes agent behavior. diff --git a/.agents/skills/end-to-end-doctesting/agents/openai.yaml b/.agents/skills/end-to-end-doctesting/agents/openai.yaml new file mode 100644 index 00000000..d2f3627a --- /dev/null +++ b/.agents/skills/end-to-end-doctesting/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "End to End Doctesting" + short_description: "Smoketest GT setup docs from a blank slate." + default_prompt: "Use $end-to-end-doctesting to smoketest a General Translation docs branch or PR by following it verbatim from a blank slate." diff --git a/.agents/skills/end-to-end-doctesting/references/example-report.md b/.agents/skills/end-to-end-doctesting/references/example-report.md new file mode 100644 index 00000000..f63ab43e --- /dev/null +++ b/.agents/skills/end-to-end-doctesting/references/example-report.md @@ -0,0 +1,183 @@ +# end-to-end-doctesting · next · gt-cloud#3427 + +> Real output of a narrowed run (2026-07-16, spawned headless instance, keyless +> host), kept as the worked example of the report format — including the +> `Since last run` repeat-tagging that a second run over the same SHA produces. + +**Verdict: 🟡 FRICTION** — all 6 quickstart steps run verbatim and the app boots clean, but the reader is never told how to see it working, the page ships unstyled with no `` and no ``, and nothing visibly translates or explains why. + +| | | +|---|---| +| Docs under test | `generaltranslation/content@077b1f9` (branch `docs-refactor`) | +| Via | gt-cloud#3427 `jackie-content-docs-refactor` (submodule `apps/landing/content`) | +| Flow / substrate | next · create-next-app --yes (Next 16.2.10, App Router, TS, Tailwind), npm tab | +| Environment | macOS 26.4 · node 24.18.0 · npm 11.16.0 · gt-next 11.0.8 · gt 2.14.62 | +| Run | 2026-07-16T00:18Z · ~7 min · 6 steps · 9 commands | +| Findings | 🔴 0 · 🟠 0 · 🟡 7 · ⚪ 1 | + +## Since last run (2026-07-15, gt-cloud#3427) + +Content SHA is unchanged (`077b1f9`) — nothing could have been fixed, and nothing was. + +- Still present: `next/6-ssg-warning-dead-link` (repeat ×2) +- Still present: `next/4-layout-snippet-clobbers-scaffold` (repeat ×2) +- Still present: `next/4-html-lang-never-set` (repeat ×2) +- Still present: `next/6-build-wired-before-keys` (repeat ×2) +- New this run (page unchanged — surfaced by fuller probe coverage, not doc regressions): `next/7-no-verify`, `next/7-es-404-undiscussed`, `next/5-duplicate-page-snippet` + +## Findings + +### 🟡 next/4-layout-snippet-clobbers-scaffold · Full-file layout snippet silently drops globals.css, fonts, and metadata (repeat ×2) + +- **Where:** step 4 "Add the provider to your root layout" — [nextjs-quickstart.mdx:91](https://github.com/generaltranslation/content/blob/077b1f99ce278a8909b782bf4f2d49db5b716bd9/docs/en-US/react/nextjs-quickstart.mdx#L91) +- **Category:** broken-code +- **Doc says:** + > Wrap your app in `GTProvider` at the root layout. — followed by a complete `app/layout.tsx` file with bare `<html>`/`<body>` and no other imports. +- **What happened:** written verbatim over the scaffold's layout, the served page has no stylesheet link, no font variables, no `<title>`, and no meta description (`grep '<title>' → nothing`). Every create-next-app user "wraps their root layout" by destroying it. +- **Root cause:** the snippet is a replacement file styled as an addition. The scaffold layout imports `./globals.css`, two Geist fonts, and exports `metadata` — all silently discarded. +- **Fix applied:** none — kept the doc's file verbatim per protocol; app still boots, just unstyled and untitled. +- **Suggested edit:** nextjs-quickstart.mdx:91–107 + ```suggestion + ```tsx title="app/layout.tsx" + import { GTProvider } from 'gt-next'; + // ...keep your existing imports (globals.css, fonts, metadata)... + + export default function RootLayout({ + children, + }: { + children: React.ReactNode; + }) { + return ( + <html lang="en"> + <body> + <GTProvider>{children}</GTProvider> + </body> + </html> + ); + } + ``` + Only the `GTProvider` import and wrapper are new — keep everything else your layout already has. + ``` + +### 🟡 next/4-html-lang-never-set · Rendered page ships no `<html lang>` at all (repeat ×2) + +- **Where:** step 4 — [nextjs-quickstart.mdx:100](https://github.com/generaltranslation/content/blob/077b1f99ce278a8909b782bf4f2d49db5b716bd9/docs/en-US/react/nextjs-quickstart.mdx#L100) +- **Category:** broken-code +- **Doc says:** the layout snippet renders `<html>` with no `lang` attribute. +- **What happened:** served HTML is literally `<html>` — the snippet *removed* the scaffold's `lang="en"`, and neither `GTProvider` nor the plugin injects one. An i18n quickstart ends with a page less locale-correct than the blank scaffold it started from. +- **Root cause:** snippet omits `lang`; gt-next 11.0.8 does not set it automatically in this configuration. +- **Fix applied:** none (verbatim run). +- **Suggested edit:** covered by the step-4 suggestion above (`<html lang="en">`), ideally with a sentence pointing at the dynamic-locale approach (`getLocale()`) for when routing/middleware is added. + +### 🟡 next/6-build-wired-before-keys · Quickstart wires a key-requiring command into `build` before keys exist (repeat ×2) + +- **Where:** step 6 "Generate translations" — [nextjs-quickstart.mdx:150–164](https://github.com/generaltranslation/content/blob/077b1f99ce278a8909b782bf4f2d49db5b716bd9/docs/en-US/react/nextjs-quickstart.mdx#L150) +- **Category:** missing-step +- **Doc says:** + > Run the CLI to translate your project… `npx gt translate` … Add the command to your build script… — with the credentials note only *after* both instructions. +- **What happened:** + ```text + $ npx gt translate + ■ No API key was found. Pass --api-key or set the GT_API_KEY environment variable. + → exit 1 + ``` + A verbatim reader runs a failing command, then wires that same failing command into `npm run build`, and only afterwards reads the footnote about `GT_PROJECT_ID`/`GT_API_KEY`. +- **Root cause:** ordering — credentials setup is a trailing italic note instead of a prerequisite sub-step. +- **Fix applied:** none needed to proceed (later steps independent); build left failing by construction on a keyless machine. +- **Suggested edit:** nextjs-quickstart.mdx:148 — move the note up as the step's first paragraph: + ```suggestion + `npx gt translate` needs a Project ID and a production API key. Run `npx gt auth` or visit the [Dashboard](/docs/platform/dashboard/get-started) to get them, and set them as `GT_PROJECT_ID` and `GT_API_KEY` in your environment (never prefixed with `NEXT_PUBLIC_`). Then run the CLI to translate your project: + ``` + +### 🟡 next/6-ssg-warning-dead-link · SDK per-request warning deep-links a page this branch deletes (repeat ×2) + +- **Where:** runtime, during verification — every uncached request logs: + ```text + gt-next: No locale could be determined for this request. If you use SSG, configure locale + resolution, or gt-next will fall back to the default locale. + Learn more: https://generaltranslation.com/en/docs/next/guides/ssg#ssg-custom-get-locale + ``` +- **Category:** dead-link +- **What happened:** `docs/en-US/next/guides/ssg.mdx` does not exist in this branch's tree (the entire `docs/en-US/next/` directory is removed by the refactor), and a gt-cloud code search finds no redirect for `guides/ssg`. The link works on prod today only because the old IA is still live; the moment this PR ships, the SDK's most-printed remedy link 404s. +- **Root cause:** IA refactor moved/removed the SSG guide without a redirect; SDK pins the old URL. +- **Fix applied:** n/a. +- **Suggested edit:** add a redirect from `/docs/next/guides/ssg` → its new-IA equivalent (or restore a stub page) in the gt-cloud landing app before merge; longer-term, the SDK should link a stable shortlink. + +### 🟡 next/5-duplicate-page-snippet · Two conflicting code blocks both titled `app/page.tsx` (new) + +- **Where:** step 5 "Mark content for translation" — [nextjs-quickstart.mdx:137](https://github.com/generaltranslation/content/blob/077b1f99ce278a8909b782bf4f2d49db5b716bd9/docs/en-US/react/nextjs-quickstart.mdx#L137) +- **Category:** ambiguous +- **Doc says:** + > To translate strings in an async server component, use `getGT` instead: — followed by a second block titled `app/page.tsx` containing a different default export. +- **What happened:** nothing broke here because I read "instead" as an illustrative alternative and kept the first block. But `title="app/page.tsx"` is a write-this-file instruction; an agent applying every titled block verbatim overwrites the step-5 demo (LocaleSelector, `<T>`, `useGT`) with a bare `getGT` page and the quickstart's visible result vanishes. +- **Fix applied:** took the most literal main-path reading (first block kept); second block not walked. +- **Suggested edit:** nextjs-quickstart.mdx:137 + ```suggestion + ```tsx title="app/about/page.tsx" + ``` + (or drop the `title=` entirely so the block reads as an example, not a file to write) + +### 🟡 next/7-no-verify · The quickstart never tells the reader how to see it working (new) + +- **Where:** whole page — no step boots the app; `npm run dev`, `localhost`, and "you should see…" appear nowhere on the page. +- **Category:** no-verify +- **What happened:** the flow "completes" at step 6 with a failing CLI command and no moment where the reader confirms anything renders, switches locale, or translates. Everything the verification section below found (no styles, no lang, silent no-op switch) is invisible to a reader who follows the page and stops. +- **Suggested edit:** add a closing "### 7. See it work" step: `npm run dev`, open `http://localhost:3000`, "you should see the locale selector; pick Español and the heading becomes *Bienvenido a mi aplicación*" — plus one sentence on what keyless dev mode shows instead. + +### 🟡 next/7-es-404-undiscussed · `/es` 404s and routing is never surfaced (new) + +- **Where:** verification probe — `GET /es → 404`; the spine never mentions locale paths, middleware, or routing (middleware exists only as a sidebar "related" link). +- **Category:** missing-step +- **What happened:** target locales are declared in step 3 (`es`, `fr`, `ja`) but no locale is reachable at a URL, and the LocaleSelector doesn't produce one. The reader ends the quickstart never knowing locale-path routing is a decision they must make. +- **Suggested edit:** one sentence in step 3 or a closing step: "Locales are served on the same URL by default; to give each locale its own path (`/es`, `/fr`), add the [middleware](/docs/react/nextjs/middleware)." + +### ⚪ next/6-translate-skipped-credentials · `gt translate` untested end-to-end (repeat ×2) + +- **Category:** env-gap — no `DOCTEST_GT_*` credentials on this host. Failure mode captured above (clean exit-1 with a clear message, no stack trace). Translated rendering, CDN loading, and the production build (which now begins with `npx gt translate`) are all untested this run. + +## Verification + +- Dev server: ✅ 200 on `/`, ready in <1 s, no error overlay. (Bound to port 3002 — 3000 busy on this host; not a doc issue.) +- Rendered page: `<T>` content and both `gt()` strings render in English; **no `<html lang>`, no `<title>`, no stylesheet** (see findings). +- Locale switch (LocaleSelector, the mechanism the docs provide): selecting Español triggers a reload, then the selector snaps back to English and content is unchanged — a keyless no-op. It **is** explained, but only in the terminal: `I18nCache: Loading translations from a remote store needs a projectId. No translations will be loaded.` plus the per-request SSG warning whose "learn more" link is the dead page from the ssg-warning finding above. Nothing on the page itself explains it. (Expert-mode extra, not a doc step: forcing `Cookie: generaltranslation.locale=es` also still renders English.) +- `/es`: 404 (finding above). +- Production build: skipped — keyless, and the docs-authored build script starts with the key-requiring `npx gt translate` (exit 1 captured at step 6). + +## Pages tested + +- `docs/en-US/react/nextjs-quickstart.mdx` (spine — the only page any step forced through; CLI-auth and dashboard pages referenced at step 6 were not needed keyless) + +## Not walked + +- yarn / bun / pnpm tabs (npm chosen) +- Step 5's alternative `getGT` block (see finding `next/5-duplicate-page-snippet`) +- Sidebar "related" pages: server-functions, middleware, config, link +- `npx gt auth` / Dashboard key setup (no credentials) + +## Transcript + +<details><summary>Substrate (environment prep — not doc steps)</summary> + +```text +$ mktemp -d /tmp/doctest-next-XXXXXX → /tmp/doctest-next-Y00x8R +$ CI=1 npx create-next-app@latest app --yes → exit 0 (Next 16.2.10, TS, Tailwind, App Router) +``` +</details> + +<details><summary>Doc steps as executed</summary> + +```text +[step 1] $ npm install gt-next && npm install gt --save-dev → exit 0 (gt-next 11.0.8, gt 2.14.62) +[step 2] wrote next.config.ts (verbatim from doc; replaced scaffold config) +[step 3] wrote gt.config.json (verbatim) +[step 4] wrote app/layout.tsx (verbatim; replaced scaffold layout — dropped globals.css/fonts/metadata/lang) +[step 5] wrote app/page.tsx (first block verbatim; second same-titled block treated as alternative) +[step 6] $ npx gt translate → exit 1 ("No API key was found. Pass --api-key or set the GT_API_KEY environment variable.") +[step 6] package.json scripts.build = "npx gt translate && next build" +[verify] $ npm run dev → Ready in 186ms on :3002 +[verify] $ curl -s http://localhost:3002/ → 200; <html> (no lang); no <title>; "Welcome to my app" present +[verify] browser: LocaleSelector → Español → reload, selector back to English, content unchanged +[verify] server console: "I18nCache: ... needs a projectId. No translations will be loaded." +[verify] $ curl -s http://localhost:3002/es → 404 +``` +</details> diff --git a/.agents/skills/end-to-end-doctesting/references/operations.md b/.agents/skills/end-to-end-doctesting/references/operations.md new file mode 100644 index 00000000..e97a321a --- /dev/null +++ b/.agents/skills/end-to-end-doctesting/references/operations.md @@ -0,0 +1,71 @@ +# Operations + +For the humans running or wiring the skill. The protocol itself is +`SKILL.md`; nothing here changes agent behavior. + +## Getting the command + +There is nothing to install. Open Claude Code anywhere in a checkout of +this repo and `/end-to-end-doctesting` is available — the skill loads +automatically from `.claude/skills/` (a symlink into `.agents/skills/`). +To use it outside this repo, copy the `end-to-end-doctesting/` folder into +another repo's `.claude/skills/`, or into `~/.claude/skills/` for a +personal install that works everywhere. + +## Why this exists + +The QS-01 quickstart crash (500 on every route of a fresh project) sat live +in the docs until an *agent* following them hit it. Most users now install +these frameworks with agents, so an agent walking the docs from scratch is +the representative first run — and the cheapest possible CI for +documentation. Repeats are tagged `(repeat ×N)` run over run, so doc debt +can't accumulate silently. Maintenance for docs, so the next refactor isn't +a rescue. + +## Host requirements + +- `git`, `node`/`npm`; `python3` for the python flow. +- GitHub read access to `generaltranslation/gt-cloud` and + `generaltranslation/content`: `gh auth login`, or `GITHUB_TOKEN` + (fine-grained PAT, read-only on the two repos — the skill falls back to + curl). +- Optional but recommended: `DOCTEST_GT_API_KEY` + `DOCTEST_GT_PROJECT_ID` + from a **dedicated throwaway GT project** — without them, key-gated steps + (`npx gt translate`, prod builds) are exercised for their failure mode, + then marked ⚪ skipped. Never point these at a real project. +- Check everything at once: `/end-to-end-doctesting doctor`. + +## Headless / CI + +The prompt is the command — run from a checkout that has the skill: + + claude -p "/end-to-end-doctesting 3427 next" --model opus --effort xhigh \ + --dangerously-skip-permissions + +The final printed message is the summary; its `verdict:` line is +stable-format (`verdict: 🔴|🟠|🟡|✅ …`) so a workflow can parse it — e.g. a +GitHub Action on docs PRs posting the summary as a PR comment, plus a +nightly run against `main`. + +Keep the tester model fixed across runs you want to compare: a frontier +model can quietly out-muscle friction a typical setup would die on, so a +stronger or weaker tester probes a different user population. `opus` +mirrors the strongest agent a typical new user actually runs. + +Cost/time datum: one narrowed flow ≈ 8 min / ~$4 at xhigh effort. The full +sweep runs for hours — run the first one attended. + +## The naive probe + +For the truest first-timer signal, skip the skill entirely: run Claude Code +in an empty directory (where no skill is visible) and prompt "You are a +developer trying General Translation for the first time — install gt-next +by following the docs." What it gets stuck on, users get stuck on. + +## State + +Run-to-run memory lives in `~/.doctest/` (`DOCTEST_HOME` to move it), +deliberately outside any session: `state.json` (last target + last-20-run +history — powers repeat tagging), `reports/`, and `sweeps/<target>/` +ledgers. Bare `/end-to-end-doctesting` re-tests the last target; first run +ever names a branch or PR once. diff --git a/.agents/skills/end-to-end-doctesting/references/report-template.md b/.agents/skills/end-to-end-doctesting/references/report-template.md new file mode 100644 index 00000000..f0583e8f --- /dev/null +++ b/.agents/skills/end-to-end-doctesting/references/report-template.md @@ -0,0 +1,117 @@ +# end-to-end-doctesting report template + +Copy this shape. Drop sections that are empty rather than leaving "none" +stubs — the report should read like a QA engineer's writeup, not a form. +Everything in `<angle brackets>` is a placeholder. Redact credentials +everywhere, including inside quoted error output. + +```markdown +# end-to-end-doctesting · <flow> · <target> + +**Verdict: <emoji> <VERDICT>** — <one-sentence plain-language summary of the run> + +| | | +|---|---| +| Docs under test | `generaltranslation/content@<CONTENT_SHA7>` (branch `<content branch>`, PR <content#N>) | +| Via | gt-cloud#<N> `<gt-cloud branch>` (submodule `apps/landing/content`) | +| Flow / substrate | <flow> · <substrate, e.g. create-next-app 16.2.10 --yes, npm tab> | +| Environment | <OS> · node <v> · npm <v> · gt-next <resolved v> · gt <resolved v> | +| Run | <UTC ISO datetime> · <duration> · <n> steps · <n> commands · slowest step <n> (<time>) | +| Run variant | <only when steered off-default: "pnpm tab", "steps 1–3 only", …; omit row otherwise> | +| Coverage | <narrowed: "spine + 2 pulled pages · PR touches 14 docs pages (11 unexercised)"> <sweep: "412 executed · 57 ⚪ · 469 items · 148 pages · ledger: sweeps/<slug>/ledger.json"> | +| Findings | 🔴 <n> · 🟠 <n> · 🟡 <n> · ⚪ <n> | + +## Since last run (<date>, <target>) + +- Fixed: <finding id — one line> ← findings that no longer reproduce +- Still present: <finding id> (repeat ×<N>) ← lead with these; they are the debt + +## Findings + +### 🔴 <id> · <title> (new | repeat ×N) + +- **Where:** step <n> "<step heading>" — [<file>:<line>](https://github.com/generaltranslation/content/blob/<CONTENT_SHA>/<path>#L<n>) +- **Category:** <wrong-command | broken-code | missing-step | ambiguous | env-gap | agent-blocker | no-verify | dead-link> +- **Doc says:** + > <exact quote from the page> +- **What happened:** + ```text + $ <command as run> + <trimmed error output, ≤20 lines> + ``` +- **Root cause:** <one or two sentences> +- **Fix applied:** <exact deviation taken, or "none found — 3 approaches exhausted: <a>, <b>, <c>"> +- **Suggested edit:** <file>:<line> + ```suggestion + <exact replacement text for that line/block> + ``` + +<!-- repeat per finding, 🔴 first, then 🟠, 🟡, ⚪ --> + +## Verification + +- Dev server: <200 / error — detail> +- Locale switch (<mechanism the docs gave>): <translated text rendered? quote a translated string> +- Production build: <ok / failed — detail / skipped (no credentials)> + +## Coverage (sweep runs) + +- Executed: <n> items · ⚪ by disposition: needs-creds <n> · needs-external-service <n> · destructive-platform <n> · not-runnable-here <n> · duplicates <n> +- <one line per ⚪ disposition group worth a human's attention — e.g. "9 dashboard items: local halves executed; the click-path from 'Create project' onward needs a human once"> + +## Pages tested + +- <docs/en-US/...> (spine) +- <docs/en-US/...> (pulled in by step <n>) + +## Not walked + +- <optional/collapsed branches, other package-manager tabs, flows not present on this branch> + +## Transcript + +<details><summary>Substrate (environment prep — not doc steps)</summary> + + ```text + $ <command> → exit <code> + ``` +</details> + +<details><summary>Doc steps as executed</summary> + + ```text + [step 1] $ <command> → exit <code> + [step 2] wrote next.config.ts (verbatim from doc) + ... + ``` +</details> +``` + +## Summary shapes (for reference) + +Findings run — ≤6 lines, worst thing named in plain words, deep link, report path: + +```text +🩺 end-to-end-doctesting · next · content@077b1f9 (via gt-cloud#3427 · jackie-content-docs-refactor) +verdict: 🟠 DETOURED — setup finished, but 2 steps needed fixes the docs don't give +🔴 0 · 🟠 2 · 🟡 3 · ⚪ 1 (no creds) · 9 steps · 12 min · node 24.13.0 +worst: step 6 — `npx gt translate` rejects the key the quickstart had you create (repeat ×2) → <link> +report: <path> · say "file it" to turn these into PR review comments +``` + +Blocked run — say where it died and what a user would see: + +```text +🩺 end-to-end-doctesting · next · content@077b1f9 (gt-cloud#3427) +verdict: 🔴 BLOCKED at step 4/9 — fresh app 500s on first `npm run dev`, no fix found in 3 attempts +🔴 1 · 🟠 1 · 🟡 2 · 8 min · node 24.13.0 +a newcomer following this page verbatim cannot reach a running app → <link> +report: <path> · say "file it" to turn these into PR review comments +``` + +Clean run — two lines, no report file: + +```text +🩺 end-to-end-doctesting · next · content@077b1f9 (gt-cloud#3427) — ✅ CLEAN +9/9 steps verbatim · dev 200 · es renders translated · prod build ok · 11 min · no report needed +``` diff --git a/.claude/skills/end-to-end-doctesting b/.claude/skills/end-to-end-doctesting new file mode 120000 index 00000000..cfe8f643 --- /dev/null +++ b/.claude/skills/end-to-end-doctesting @@ -0,0 +1 @@ +../../.agents/skills/end-to-end-doctesting \ No newline at end of file