feat(seo): serve prerendered pages to AI crawlers, state the crawler policy in robots.txt

.github/workflows/bot-serving-check.yml

@@ -28,9 +28,9 @@ permissions:
 jobs:
   bot-serving:
     runs-on: ubuntu-latest
-    # 5 checks x (--retry 2 -> up to 3 attempts x --max-time 30) can reach
-    # ~7.5 min worst-case; 10 leaves room to report a clean failure.
-    timeout-minutes: 10
+    # 7 checks x (--retry 2 -> up to 3 attempts x --max-time 30) can reach
+    # ~10.5 min worst-case; 14 leaves room to report a clean failure.
+    timeout-minutes: 14
     steps:
       - name: Crawler UAs must get 200 + per-route titles
         run: |
@@ -40,6 +40,8 @@ jobs:
           ORIGIN="https://anyplot-app-r3tvmejsmq-ez.a.run.app"
           GOOGLEBOT="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
           TWITTERBOT="Twitterbot/1.0"
+          CLAUDEBOT="Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
+          CHATGPTUSER="Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)"
           HUMAN="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36"
           fail=0
 
@@ -65,8 +67,17 @@ jobs:
           check "$GOOGLEBOT"  "$ORIGIN/scatter-basic"                   "<title>Basic Scatter Plot | anyplot.ai</title>"
           check "$TWITTERBOT" "$ORIGIN/scatter-basic/python/matplotlib" "<title>Basic Scatter Plot - Matplotlib | anyplot.ai</title>"
 
-          # llms.txt must be served directly, never proxied to the seo backend
-          check "$GOOGLEBOT" "$ORIGIN/llms.txt" "# anyplot"
+          # AI assistants take the same prerendered path (nginx $is_bot). These
+          # checks hit the ORIGIN, so they verify the nginx map independently of
+          # whether Cloudflare's AI Crawl Control currently 403s these UAs at
+          # the edge — an edge-level policy change needs no change here.
+          check "$CLAUDEBOT" "$ORIGIN/scatter-basic" "<title>Basic Scatter Plot | anyplot.ai</title>"
+
+          # llms.txt must be served directly, never proxied to the seo backend —
+          # including for a mapped crawler UA, which is the whole point of the
+          # `location = /llms.txt` bypass.
+          check "$GOOGLEBOT"   "$ORIGIN/llms.txt" "# anyplot"
+          check "$CHATGPTUSER" "$ORIGIN/llms.txt" "# anyplot"
 
           # Control: humans must still get the SPA shell
           check "$HUMAN" "$ORIGIN/" '<div id="root">'

CHANGELOG.md

@@ -18,6 +18,15 @@ aggregate instead: an italic *Catalog* line at the end of the version section an
 
 ### Added
 
+- **AI crawler policy, written down and served** — `app/public/robots.txt` now states the whole
+  policy itself (content signals incl. the Art. 4 EU-DSM training reservation, the welcomed
+  retrieval/citation agents, the declined training collectors), so it holds with or without
+  Cloudflare's managed block; `app/nginx.conf` routes `Claude*`, `GPTBot`, `OAI-SearchBot`,
+  `ChatGPT-User` and `Perplexity*` to the prerendered seo-proxy pages like the search engines
+  (they run no JavaScript and were being served the empty SPA shell); the daily
+  `bot-serving-check` now covers an AI UA and `llms.txt` under an AI UA. `docs/reference/seo.md`
+  documents the decision, the measured edge behaviour and the Cloudflare steps that lift the
+  403 (#9898, decision #9633).
 - **`CODE_OF_CONDUCT.md`** — Contributor Covenant 2.1, linked from `docs/contributing.md`
   (closes the last repo-health gap from audit 2026-07-15 Medium#29) (#9644).
 - **Bot-served pages got a real SEO surface** — `seo_home()` now emits the site-level JSON-LD

app/nginx.conf

@@ -27,6 +27,24 @@ map $http_user_agent $is_bot {
     ~*yandexbot 1;
     ~*baiduspider 1;
     ~*applebot 1;
+    # AI assistants, AI search and citation crawlers. Same failure mode as the
+    # search engines: none of them execute JavaScript, so without these entries
+    # an AI answer citing anyplot describes the empty SPA shell. Claude-User /
+    # ChatGPT-User / Perplexity-User are USER-directed fetches — a human asked
+    # their assistant to open the page — and matter most.
+    # This map only decides WHAT a crawler gets once it arrives, never whether
+    # it may: the allow/deny policy is robots.txt plus the Cloudflare AI Crawl
+    # Control zone setting (docs/reference/seo.md). GPTBot is listed even
+    # though robots.txt declines it, so the rendering is right if that call is
+    # ever reversed.
+    ~*claudebot 1;
+    ~*claude-user 1;
+    ~*claude-searchbot 1;
+    ~*gptbot 1;
+    ~*oai-searchbot 1;
+    ~*chatgpt-user 1;
+    ~*perplexitybot 1;
+    ~*perplexity-user 1;
     # Social Media
     ~*twitterbot 1;
     ~*facebookexternalhit 1;

app/public/robots.txt

@@ -1,16 +1,75 @@
-# Intended crawler policy: AI crawlers are WELCOME (see /llms.txt and the MCP
-# server — anyplot's strategy is to be maximally consumable by AI agents).
+# Crawler policy — AI crawlers that RETRIEVE and CITE are welcome: anyplot is
+# built to be consumable by AI agents (see /llms.txt and the MCP server at
+# https://api.anyplot.ai/mcp/). Training-only scrapers are declined below.
 #
-# NOTE: Cloudflare's "block AI bots" zone setting currently PREPENDS a managed
-# block (Disallow: / for ClaudeBot, GPTBot, CCBot, Google-Extended, ...) to the
-# live https://anyplot.ai/robots.txt, contradicting this file and llms.txt.
-# Fix lives in the Cloudflare dashboard (zone setting), not in this repo:
-# disable the managed AI-bot block (or switch it to Content-Signal-only) so
-# the live file matches this one.
+# Cloudflare's AI Crawl Control can PREPEND its own managed block to the live
+# file and additionally answer the blocked user agents with HTTP 403 at the
+# edge, so https://anyplot.ai/robots.txt may be longer and stricter than this
+# file. The policy is therefore spelled out here in full — including the
+# content signals Cloudflare would otherwise supply — so it survives with the
+# managed block turned off. Rationale, dashboard steps and how to verify:
+# docs/reference/seo.md ("AI crawler policy").
+#
+# Two ordering rules in here are load-bearing, both for the same reason —
+# a compliant crawler resolves by specificity, simpler parsers by first match:
+#   1. Named groups come FIRST, `User-agent: *` last. With the wildcard on top,
+#      a first-match parser would read `Allow: /` and never reach the declining
+#      groups below.
+#   2. Inside a group, `Disallow:` lines come BEFORE `Allow: /` — otherwise a
+#      first-match parser lets /debug and /interactive through (verified with
+#      Python's urllib.robotparser, which is exactly such a parser).
 
-User-agent: *
+# Content signals per contentsignals.org. ANY RESTRICTION EXPRESSED HERE IS AN
+# EXPRESS RESERVATION OF RIGHTS UNDER ARTICLE 4 OF EU DIRECTIVE 2019/790.
+# search:   indexing and returning links/excerpts — yes
+# ai-input: retrieval for AI answers, grounding, citation — yes
+# ai-train: training or fine-tuning models — no
+#
+# The signal is repeated in EVERY group below, because a crawler reads only the
+# group that matches it: a reservation declared once under `User-agent: *` would
+# never reach a named agent — least of all the training collectors it is aimed
+# at.
+#
+# AI assistants, AI search and citation crawlers. These send readers back and
+# are the reason /llms.txt exists; the *-User agents fetch only because a human
+# asked their assistant to open the page.
+User-agent: ClaudeBot
+User-agent: Claude-User
+User-agent: Claude-SearchBot
+User-agent: OAI-SearchBot
+User-agent: ChatGPT-User
+User-agent: PerplexityBot
+User-agent: Perplexity-User
+Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
+Disallow: /debug
+Disallow: /interactive
 Allow: /
+
+# Training-only collectors: they take the catalogue without ever returning a
+# reader, which is what ai-train=no says in prose. GPTBot sits here because it
+# is OpenAI's TRAINING crawler — ChatGPT's retrieval path is
+# OAI-SearchBot/ChatGPT-User above and stays open. Flip this one group if the
+# training stance ever changes.
+User-agent: GPTBot
+User-agent: CCBot
+User-agent: Bytespider
+User-agent: Amazonbot
+User-agent: meta-externalagent
+Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
+Disallow: /
+
+# Opt-out tokens for vendors that crawl under a different user agent
+# (Googlebot / Applebot fetch for search and stay allowed).
+User-agent: Google-Extended
+User-agent: Applebot-Extended
+Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
+Disallow: /
+
+# Everyone else: search engines, social/link previews, feed readers.
+User-agent: *
+Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
 Disallow: /debug
 Disallow: /interactive
+Allow: /
 
 Sitemap: https://anyplot.ai/sitemap.xml

docs/reference/seo.md

@@ -52,7 +52,7 @@ Our solution uses **nginx-based bot detection** to serve pre-rendered HTML with
 
 ### Detected Bots
 
-nginx detects 27 bots via User-Agent matching, organized by category:
+nginx detects 37 User-Agent patterns, organized by category:
 
 **Social Media:**
 | Bot | User-Agent Pattern |
@@ -82,12 +82,30 @@ nginx detects 27 bots via User-Agent matching, organized by category:
 | Bot | User-Agent Pattern |
 |-----|-------------------|
 | Google | `googlebot` |
+| Google URL inspection | `google-inspectiontool` |
+| Google misc crawler | `googleother` |
 | Bing | `bingbot` |
 | Yandex | `yandexbot` |
 | DuckDuckGo | `duckduckbot` |
 | Baidu | `baiduspider` |
 | Apple | `applebot` |
 
+**AI Assistants & AI Search:**
+| Bot | User-Agent Pattern | Role |
+|-----|-------------------|------|
+| Anthropic crawler | `claudebot` | index/crawl |
+| Claude user fetch | `claude-user` | a human asked Claude to open the page |
+| Claude search | `claude-searchbot` | citation index |
+| OpenAI crawler | `gptbot` | training (declined in robots.txt, mapped anyway — see below) |
+| ChatGPT search | `oai-searchbot` | citation index |
+| ChatGPT user fetch | `chatgpt-user` | a human asked ChatGPT to open the page |
+| Perplexity | `perplexitybot`, `perplexity-user` | citation index / user fetch |
+
+None of these execute JavaScript, so without the map entries an AI answer
+citing anyplot would describe the empty SPA shell. The map decides **what** a
+crawler is served, never **whether** it may fetch — that is the robots.txt +
+Cloudflare question in [AI crawler policy](#ai-crawler-policy).
+
 **Link Preview Services:**
 | Bot | User-Agent Pattern |
 |-----|-------------------|
@@ -236,16 +254,45 @@ Uses **MonoLisa** variable font (commercial, not in repo):
 
 ### Frontend (anyplot.ai)
 
-Static file at `app/public/robots.txt`:
+Static file at `app/public/robots.txt`. It carries the full policy — content
+signals, the welcomed AI agents, the declined training collectors — so it holds
+regardless of what Cloudflare does or does not prepend (see
+[AI crawler policy](#ai-crawler-policy)):
+
+Four groups in this order — welcomed AI agents (`ClaudeBot`, `Claude-User`,
+`Claude-SearchBot`, `OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`,
+`Perplexity-User`), declined training collectors (`GPTBot`, `CCBot`,
+`Bytespider`, `Amazonbot`, `meta-externalagent`), opt-out tokens
+(`Google-Extended`, `Applebot-Extended`), and finally the wildcard. The first
+group verbatim; read the file for the rest:
 
 ```txt
-User-agent: *
-Allow: /
+User-agent: ClaudeBot
+User-agent: Claude-User
+User-agent: Claude-SearchBot
+User-agent: OAI-SearchBot
+User-agent: ChatGPT-User
+User-agent: PerplexityBot
+User-agent: Perplexity-User
+Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
 Disallow: /debug
-
-Sitemap: https://anyplot.ai/sitemap.xml
+Disallow: /interactive
+Allow: /
 ```
 
+Three properties of that file are deliberate and should survive any cleanup:
+
+- The `Content-Signal` line is repeated in **every** group, declining ones
+  included. A crawler reads only the group that matches it, so a reservation
+  declared once under `User-agent: *` never reaches a named agent — least of all
+  the training collectors it is aimed at.
+- The named groups come **before** the wildcard group. A spec-compliant crawler
+  picks the most specific match regardless of order, but simpler parsers take
+  the first match and would read `Allow: /` and stop.
+- Inside each group, `Disallow:` comes **before** `Allow: /` — same reason: with
+  the broad allow first, a first-match parser (Python's `urllib.robotparser`,
+  for one) hands out `/debug` and `/interactive`.
+
 ### Backend (api.anyplot.ai)
 
 Dynamic endpoint at `GET /robots.txt`:
@@ -260,6 +307,61 @@ Disallow: /
 - Prevents crawling of debug endpoints, docs, and API responses
 - Social media bots (WhatsApp, Twitter, etc.) are unaffected - they fetch og:images directly
 
+### AI crawler policy
+
+**Decision (issue #9633):** AI agents that *retrieve and cite* are welcome;
+agents that only *collect for training* are declined. anyplot's entire strategy
+is to be consumable by AI agents — `/llms.txt`, the MCP server, MIT-licensed
+code on every page — so blocking the retrieval side works against the product.
+The training reservation stays expressed, and it is the legally load-bearing
+part: `Content-Signal: ai-train=no` is an express reservation of rights under
+Article 4 of EU Directive 2019/790.
+
+| Group | Agents | Policy |
+|---|---|---|
+| Retrieval / citation / user-directed | `ClaudeBot`, `Claude-User`, `Claude-SearchBot`, `OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`, `Perplexity-User` | allowed |
+| Training collectors | `GPTBot`, `CCBot`, `Bytespider`, `Amazonbot`, `meta-externalagent` | declined |
+| Opt-out tokens (vendor crawls under another UA) | `Google-Extended`, `Applebot-Extended` | declined |
+
+`GPTBot` is the deliberate borderline call: it is OpenAI's *training* crawler,
+so it sits with the declined group, while ChatGPT's retrieval path
+(`OAI-SearchBot`, `ChatGPT-User`) stays open. Reversing that is a one-group
+edit in `app/public/robots.txt`. Vendor UA roles shift — re-check the current
+role of each agent against Cloudflare's AI Crawl Control categories before
+changing the policy.
+
+#### Cloudflare is the enforcement layer
+
+Measured 2026-07-25 on the live zone: Cloudflare prepends a **managed
+robots.txt** block (`Disallow: /` for ClaudeBot, GPTBot, CCBot, Google-Extended,
+Amazonbot, Applebot-Extended, Bytespider, meta-externalagent,
+CloudflareBrowserRenderingCrawler) *and* answers those user agents with a hard
+`HTTP 403 Your request was blocked.` at the edge — including `Claude-User` and
+`ChatGPT-User`, and including `/llms.txt` itself. The file written for AI agents
+was unreachable to every agent it was written for. Googlebot passes (200).
+
+`api.anyplot.ai` is **not** covered by the block (ClaudeBot gets 200 there), so
+the MCP server stayed reachable.
+
+Aligning the edge with this policy is a dashboard action (zone `anyplot.ai` →
+**AI Crawl Control** / Bots): allow the retrieval group, keep the training
+group blocked, and either turn off the managed robots.txt (this repo's file
+already carries the content signals) or leave it on and accept that the live
+file is stricter than the repo's.
+
+Verify afterwards:
+
+```bash
+curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://anyplot.ai/llms.txt   # expect 200
+curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; CCBot/2.0; +https://commoncrawl.org/faq/)"   https://anyplot.ai/          # expect 403
+curl -sA "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://anyplot.ai/scatter-basic | grep -o '<title>[^<]*</title>'   # per-route title, not the SPA shell
+```
+
+The last command is the part this repo owns: `app/nginx.conf` maps the AI UAs
+onto the seo-proxy path, and `.github/workflows/bot-serving-check.yml` guards it
+daily against the Cloud Run origin (origin, not edge — so it reports on the
+nginx map no matter what the zone policy is).
+
 ## Sitemap
 
 Dynamic XML sitemap for search engine indexing.
@@ -335,10 +437,12 @@ curl -o test.png https://api.anyplot.ai/og/scatter-basic.png
 | File | Purpose |
 |------|---------|
 | `app/nginx.conf` | Bot detection, SPA routing, sitemap proxy |
-| `app/public/robots.txt` | Frontend robots.txt (blocks /debug) |
+| `app/public/robots.txt` | Frontend robots.txt (content signals, AI crawler policy, blocks /debug) |
+| `app/public/llms.txt` | Agent-facing site summary; served directly, never via the seo-proxy |
 | `api/routers/seo.py` | SEO proxy endpoints, robots.txt, sitemap generation |
 | `api/routers/og_images.py` | Branded og:image endpoints |
 | `core/images.py` | Image processing, branding functions |
+| `.github/workflows/bot-serving-check.yml` | Daily synthetic check of the bot → seo-proxy path |
 
 ## Multi-Language URL Strategy