feat(seo): serve prerendered pages to AI crawlers, state the crawler policy in robots.txt#9898
Merged
Merged
Conversation
…n robots.txt Claude, ChatGPT and Perplexity fetch pages either on a user's explicit request or to build a citation index, and none of them execute JavaScript — so they were being handed the empty SPA shell, the same failure mode the search engines had. Map them onto the seo-proxy path and guard it in the daily bot-serving check (origin-level, so it holds regardless of the Cloudflare edge policy). robots.txt now carries the full policy itself — content signals incl. the Art. 4 EU-DSM training reservation, the welcomed retrieval agents, the declined training collectors — so it survives with Cloudflare's managed block turned off. docs/reference/seo.md records the decision, the measured edge behaviour (hard 403 for every AI UA, llms.txt included) and the dashboard steps that lift it. Refs #9633
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves anyplot.ai’s SEO serving behavior for AI crawlers by routing additional AI-related user agents to the prerendered seo-proxy path, and by explicitly documenting/encoding the project’s AI crawler policy in robots.txt and the SEO reference docs. It also extends the daily synthetic “bot serving” workflow to cover an AI UA and llms.txt under an AI UA.
Changes:
- Route additional AI crawler/user-directed UAs (Claude*, OpenAI*, Perplexity*) to prerendered pages via the nginx bot map.
- Expand
robots.txtinto a full, explicit AI crawler policy (content signals + allowed vs declined groups) and document the rationale/Cloudflare steps indocs/reference/seo.md. - Extend the
bot-serving-checkworkflow with AI-UA coverage and adjust timeout accordingly; add a changelog entry.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
docs/reference/seo.md |
Updates bot inventory/documentation and adds an explicit “AI crawler policy” section with verification steps. |
CHANGELOG.md |
Adds an Unreleased entry describing the new AI crawler policy + serving behavior. |
app/public/robots.txt |
Encodes the crawler policy (content signals + allow/deny groups for AI UAs). |
app/nginx.conf |
Extends $is_bot mapping to include AI assistant/search UAs so they receive prerendered HTML. |
.github/workflows/bot-serving-check.yml |
Adds AI-UA checks (spec page + llms.txt under AI UA) and increases the workflow timeout. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
A crawler obeys the single robots.txt group that matches it most specifically, so ClaudeBot & co. would never have seen the ai-train=no reservation declared under User-agent: * (Copilot #9898).
Spec-compliant crawlers pick the most specific matching group, but simpler parsers take the first match — with User-agent: * on top those would read Allow: / and never reach the declining groups (Copilot #9898).
With 'Allow: /' first, a first-match parser (Python's urllib.robotparser, for one) hands out /debug and /interactive; the longest-match parsers are order-independent, so leading with the Disallow lines is correct for both.
4 tasks
The declining groups carried no Content-Signal, so the very agents the ai-train=no reservation is aimed at would never have read it — a crawler only reads the group that matches it. Also replaces the seo.md snippet, which showed invented '…' lines and the old Allow-before-Disallow order, with the first group verbatim (Copilot #9898).
MarkusNeusinger
added a commit
that referenced
this pull request
Jul 25, 2026
…9899) ## Summary - `pyproject.toml` pins `ruff>=0.15.21` with no upper bound. CI resolved **0.16.0** today, and that version formats Python code blocks **inside Markdown** — 20 tracked `.md` files immediately counted as unformatted and `Run Linting` went red on every PR (first seen on #9898, which touches none of those files). - Fix: `*.md` joins ruff's exclude list. Ruff owns this repo's Python, not its prose. - Deliberately **not** fixed by reformatting the 20 files: nine are `prompts/library/*.md`, whose snippets steer code generation, so a quote-style rewrite there is a pipeline change wearing a formatting costume. If the docs should be ruff-formatted, that is its own decision with its own review. ## Plan N/A — unblocks CI. ## Test plan - [x] `uvx ruff@0.16.0 format --check .` → `148 files already formatted` (was: `20 files would be reformatted`) - [x] `uvx ruff@0.16.0 check .` → `All checks passed!` - [x] Same two commands on `ruff@0.15.21` (the version CI resolved while green) → identical results, so the change is not version-specific - [ ] `Run Linting` green on this PR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
app/nginx.conf):ClaudeBot,Claude-User,Claude-SearchBot,GPTBot,OAI-SearchBot,ChatGPT-User,PerplexityBot,Perplexity-User. None of them execute JavaScript, so they were being served the empty SPA shell — the same failure mode the search engines had before they were added. The map decides what a crawler is served, never whether it may fetch.app/public/robots.txtcarries the whole policy itself — content signals (incl.ai-train=no, an express reservation of rights under Art. 4 of EU Directive 2019/790), the welcomed retrieval/citation agents, the declined training collectors. Previously the file was three lines plus a comment noting that Cloudflare contradicted it; now the policy survives with Cloudflare's managed block turned off, which is the configuration this repo wants.docs/reference/seo.mddocuments the decision for Decision: robots.txt AI-crawler policy (Cloudflare-managed) #9633, the measured edge behaviour, and the exact Cloudflare steps + verification commands. Also corrects the bot inventory (27 → 37 patterns;google-inspectiontool/googleotherwere missing from the table) and addsllms.txt+ the bot-serving check to the file map.bot-serving-check.ymlgained two checks (an AI UA on a spec page,llms.txtunder an AI UA) and a matching timeout bump (10 → 14 min for now 7 checks). It targets the Cloud Run origin, so it verifies the nginx map regardless of the zone-level AI policy.Measured before this change (2026-07-25, edge ZRH)
ClaudeBot/Claude-User/Claude-SearchBotonanyplot.ai/403 Your request was blocked.GPTBot/OAI-SearchBot/ChatGPT-User/PerplexityBotonanyplot.ai/403anyplot.ai/llms.txt403Googlebot200ClaudeBotonapi.anyplot.ai/libraries200(subdomain not covered by the block)So
llms.txtand the MCP pitch were unreachable to exactly the agents they are written for, and user-directed fetches were blocked along with the scrapers.This PR does not unblock anything — lifting the 403 is a Cloudflare dashboard action on zone
anyplot.ai(AI Crawl Control), documented indocs/reference/seo.md. This PR makes the repo say what the policy is and makes the serving correct for the moment the edge lets the agents through.Plan
N/A — follow-up to #9633 (maintainer decision requested there).
Test plan
app/nginx.confparses cleanly:crossplane parseof the fragment wrapped inevents{}/http{}with the include resolved →status: ok, errors: []~*= case-insensitive regex): all 8 new patterns match their agent,Googlebot/Twitterbotstill match, and Chrome/Safari/Firefox UAs still take the human pathbot-serving-check.ymlis valid YAML; therunblock passesbash -n; 7checkcalls match the updated timeout commentyarn fm:check,yarn lint,yarn type-check,yarn test(609 tests) greencurl -A "…ClaudeBot…" https://anyplot.ai/scatter-basic | grep '<title>'returns the per-route title, andcurl -A "…CCBot…" https://anyplot.ai/still returns 403