diff --git a/.github/workflows/bot-serving-check.yml b/.github/workflows/bot-serving-check.yml
index 6c337de384..b39b6edbca 100644
--- a/.github/workflows/bot-serving-check.yml
+++ b/.github/workflows/bot-serving-check.yml
@@ -28,9 +28,9 @@ permissions:
jobs:
bot-serving:
runs-on: ubuntu-latest
- # 5 checks x (--retry 2 -> up to 3 attempts x --max-time 30) can reach
- # ~7.5 min worst-case; 10 leaves room to report a clean failure.
- timeout-minutes: 10
+ # 7 checks x (--retry 2 -> up to 3 attempts x --max-time 30) can reach
+ # ~10.5 min worst-case; 14 leaves room to report a clean failure.
+ timeout-minutes: 14
steps:
- name: Crawler UAs must get 200 + per-route titles
run: |
@@ -40,6 +40,8 @@ jobs:
ORIGIN="https://anyplot-app-r3tvmejsmq-ez.a.run.app"
GOOGLEBOT="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
TWITTERBOT="Twitterbot/1.0"
+ CLAUDEBOT="Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
+ CHATGPTUSER="Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)"
HUMAN="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36"
fail=0
@@ -65,8 +67,17 @@ jobs:
check "$GOOGLEBOT" "$ORIGIN/scatter-basic" "
Basic Scatter Plot | anyplot.ai"
check "$TWITTERBOT" "$ORIGIN/scatter-basic/python/matplotlib" "Basic Scatter Plot - Matplotlib | anyplot.ai"
- # llms.txt must be served directly, never proxied to the seo backend
- check "$GOOGLEBOT" "$ORIGIN/llms.txt" "# anyplot"
+ # AI assistants take the same prerendered path (nginx $is_bot). These
+ # checks hit the ORIGIN, so they verify the nginx map independently of
+ # whether Cloudflare's AI Crawl Control currently 403s these UAs at
+ # the edge — an edge-level policy change needs no change here.
+ check "$CLAUDEBOT" "$ORIGIN/scatter-basic" "Basic Scatter Plot | anyplot.ai"
+
+ # llms.txt must be served directly, never proxied to the seo backend —
+ # including for a mapped crawler UA, which is the whole point of the
+ # `location = /llms.txt` bypass.
+ check "$GOOGLEBOT" "$ORIGIN/llms.txt" "# anyplot"
+ check "$CHATGPTUSER" "$ORIGIN/llms.txt" "# anyplot"
# Control: humans must still get the SPA shell
check "$HUMAN" "$ORIGIN/" ''
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4dc728dc3a..209eb8cf7b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,6 +18,15 @@ aggregate instead: an italic *Catalog* line at the end of the version section an
### Added
+- **AI crawler policy, written down and served** — `app/public/robots.txt` now states the whole
+ policy itself (content signals incl. the Art. 4 EU-DSM training reservation, the welcomed
+ retrieval/citation agents, the declined training collectors), so it holds with or without
+ Cloudflare's managed block; `app/nginx.conf` routes `Claude*`, `GPTBot`, `OAI-SearchBot`,
+ `ChatGPT-User` and `Perplexity*` to the prerendered seo-proxy pages like the search engines
+ (they run no JavaScript and were being served the empty SPA shell); the daily
+ `bot-serving-check` now covers an AI UA and `llms.txt` under an AI UA. `docs/reference/seo.md`
+ documents the decision, the measured edge behaviour and the Cloudflare steps that lift the
+ 403 (#9898, decision #9633).
- **`CODE_OF_CONDUCT.md`** — Contributor Covenant 2.1, linked from `docs/contributing.md`
(closes the last repo-health gap from audit 2026-07-15 Medium#29) (#9644).
- **Bot-served pages got a real SEO surface** — `seo_home()` now emits the site-level JSON-LD
diff --git a/app/nginx.conf b/app/nginx.conf
index 2aef813bc9..b9997fc453 100644
--- a/app/nginx.conf
+++ b/app/nginx.conf
@@ -27,6 +27,24 @@ map $http_user_agent $is_bot {
~*yandexbot 1;
~*baiduspider 1;
~*applebot 1;
+ # AI assistants, AI search and citation crawlers. Same failure mode as the
+ # search engines: none of them execute JavaScript, so without these entries
+ # an AI answer citing anyplot describes the empty SPA shell. Claude-User /
+ # ChatGPT-User / Perplexity-User are USER-directed fetches — a human asked
+ # their assistant to open the page — and matter most.
+ # This map only decides WHAT a crawler gets once it arrives, never whether
+ # it may: the allow/deny policy is robots.txt plus the Cloudflare AI Crawl
+ # Control zone setting (docs/reference/seo.md). GPTBot is listed even
+ # though robots.txt declines it, so the rendering is right if that call is
+ # ever reversed.
+ ~*claudebot 1;
+ ~*claude-user 1;
+ ~*claude-searchbot 1;
+ ~*gptbot 1;
+ ~*oai-searchbot 1;
+ ~*chatgpt-user 1;
+ ~*perplexitybot 1;
+ ~*perplexity-user 1;
# Social Media
~*twitterbot 1;
~*facebookexternalhit 1;
diff --git a/app/public/robots.txt b/app/public/robots.txt
index 4785cbb835..210f603294 100644
--- a/app/public/robots.txt
+++ b/app/public/robots.txt
@@ -1,16 +1,75 @@
-# Intended crawler policy: AI crawlers are WELCOME (see /llms.txt and the MCP
-# server — anyplot's strategy is to be maximally consumable by AI agents).
+# Crawler policy — AI crawlers that RETRIEVE and CITE are welcome: anyplot is
+# built to be consumable by AI agents (see /llms.txt and the MCP server at
+# https://api.anyplot.ai/mcp/). Training-only scrapers are declined below.
#
-# NOTE: Cloudflare's "block AI bots" zone setting currently PREPENDS a managed
-# block (Disallow: / for ClaudeBot, GPTBot, CCBot, Google-Extended, ...) to the
-# live https://anyplot.ai/robots.txt, contradicting this file and llms.txt.
-# Fix lives in the Cloudflare dashboard (zone setting), not in this repo:
-# disable the managed AI-bot block (or switch it to Content-Signal-only) so
-# the live file matches this one.
+# Cloudflare's AI Crawl Control can PREPEND its own managed block to the live
+# file and additionally answer the blocked user agents with HTTP 403 at the
+# edge, so https://anyplot.ai/robots.txt may be longer and stricter than this
+# file. The policy is therefore spelled out here in full — including the
+# content signals Cloudflare would otherwise supply — so it survives with the
+# managed block turned off. Rationale, dashboard steps and how to verify:
+# docs/reference/seo.md ("AI crawler policy").
+#
+# Two ordering rules in here are load-bearing, both for the same reason —
+# a compliant crawler resolves by specificity, simpler parsers by first match:
+# 1. Named groups come FIRST, `User-agent: *` last. With the wildcard on top,
+# a first-match parser would read `Allow: /` and never reach the declining
+# groups below.
+# 2. Inside a group, `Disallow:` lines come BEFORE `Allow: /` — otherwise a
+# first-match parser lets /debug and /interactive through (verified with
+# Python's urllib.robotparser, which is exactly such a parser).
-User-agent: *
+# Content signals per contentsignals.org. ANY RESTRICTION EXPRESSED HERE IS AN
+# EXPRESS RESERVATION OF RIGHTS UNDER ARTICLE 4 OF EU DIRECTIVE 2019/790.
+# search: indexing and returning links/excerpts — yes
+# ai-input: retrieval for AI answers, grounding, citation — yes
+# ai-train: training or fine-tuning models — no
+#
+# The signal is repeated in EVERY group below, because a crawler reads only the
+# group that matches it: a reservation declared once under `User-agent: *` would
+# never reach a named agent — least of all the training collectors it is aimed
+# at.
+#
+# AI assistants, AI search and citation crawlers. These send readers back and
+# are the reason /llms.txt exists; the *-User agents fetch only because a human
+# asked their assistant to open the page.
+User-agent: ClaudeBot
+User-agent: Claude-User
+User-agent: Claude-SearchBot
+User-agent: OAI-SearchBot
+User-agent: ChatGPT-User
+User-agent: PerplexityBot
+User-agent: Perplexity-User
+Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
+Disallow: /debug
+Disallow: /interactive
Allow: /
+
+# Training-only collectors: they take the catalogue without ever returning a
+# reader, which is what ai-train=no says in prose. GPTBot sits here because it
+# is OpenAI's TRAINING crawler — ChatGPT's retrieval path is
+# OAI-SearchBot/ChatGPT-User above and stays open. Flip this one group if the
+# training stance ever changes.
+User-agent: GPTBot
+User-agent: CCBot
+User-agent: Bytespider
+User-agent: Amazonbot
+User-agent: meta-externalagent
+Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
+Disallow: /
+
+# Opt-out tokens for vendors that crawl under a different user agent
+# (Googlebot / Applebot fetch for search and stay allowed).
+User-agent: Google-Extended
+User-agent: Applebot-Extended
+Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
+Disallow: /
+
+# Everyone else: search engines, social/link previews, feed readers.
+User-agent: *
+Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
Disallow: /debug
Disallow: /interactive
+Allow: /
Sitemap: https://anyplot.ai/sitemap.xml
diff --git a/docs/reference/seo.md b/docs/reference/seo.md
index 741cfb5a88..45659baba6 100644
--- a/docs/reference/seo.md
+++ b/docs/reference/seo.md
@@ -52,7 +52,7 @@ Our solution uses **nginx-based bot detection** to serve pre-rendered HTML with
### Detected Bots
-nginx detects 27 bots via User-Agent matching, organized by category:
+nginx detects 37 User-Agent patterns, organized by category:
**Social Media:**
| Bot | User-Agent Pattern |
@@ -82,12 +82,30 @@ nginx detects 27 bots via User-Agent matching, organized by category:
| Bot | User-Agent Pattern |
|-----|-------------------|
| Google | `googlebot` |
+| Google URL inspection | `google-inspectiontool` |
+| Google misc crawler | `googleother` |
| Bing | `bingbot` |
| Yandex | `yandexbot` |
| DuckDuckGo | `duckduckbot` |
| Baidu | `baiduspider` |
| Apple | `applebot` |
+**AI Assistants & AI Search:**
+| Bot | User-Agent Pattern | Role |
+|-----|-------------------|------|
+| Anthropic crawler | `claudebot` | index/crawl |
+| Claude user fetch | `claude-user` | a human asked Claude to open the page |
+| Claude search | `claude-searchbot` | citation index |
+| OpenAI crawler | `gptbot` | training (declined in robots.txt, mapped anyway — see below) |
+| ChatGPT search | `oai-searchbot` | citation index |
+| ChatGPT user fetch | `chatgpt-user` | a human asked ChatGPT to open the page |
+| Perplexity | `perplexitybot`, `perplexity-user` | citation index / user fetch |
+
+None of these execute JavaScript, so without the map entries an AI answer
+citing anyplot would describe the empty SPA shell. The map decides **what** a
+crawler is served, never **whether** it may fetch — that is the robots.txt +
+Cloudflare question in [AI crawler policy](#ai-crawler-policy).
+
**Link Preview Services:**
| Bot | User-Agent Pattern |
|-----|-------------------|
@@ -236,16 +254,45 @@ Uses **MonoLisa** variable font (commercial, not in repo):
### Frontend (anyplot.ai)
-Static file at `app/public/robots.txt`:
+Static file at `app/public/robots.txt`. It carries the full policy — content
+signals, the welcomed AI agents, the declined training collectors — so it holds
+regardless of what Cloudflare does or does not prepend (see
+[AI crawler policy](#ai-crawler-policy)):
+
+Four groups in this order — welcomed AI agents (`ClaudeBot`, `Claude-User`,
+`Claude-SearchBot`, `OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`,
+`Perplexity-User`), declined training collectors (`GPTBot`, `CCBot`,
+`Bytespider`, `Amazonbot`, `meta-externalagent`), opt-out tokens
+(`Google-Extended`, `Applebot-Extended`), and finally the wildcard. The first
+group verbatim; read the file for the rest:
```txt
-User-agent: *
-Allow: /
+User-agent: ClaudeBot
+User-agent: Claude-User
+User-agent: Claude-SearchBot
+User-agent: OAI-SearchBot
+User-agent: ChatGPT-User
+User-agent: PerplexityBot
+User-agent: Perplexity-User
+Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
Disallow: /debug
-
-Sitemap: https://anyplot.ai/sitemap.xml
+Disallow: /interactive
+Allow: /
```
+Three properties of that file are deliberate and should survive any cleanup:
+
+- The `Content-Signal` line is repeated in **every** group, declining ones
+ included. A crawler reads only the group that matches it, so a reservation
+ declared once under `User-agent: *` never reaches a named agent — least of all
+ the training collectors it is aimed at.
+- The named groups come **before** the wildcard group. A spec-compliant crawler
+ picks the most specific match regardless of order, but simpler parsers take
+ the first match and would read `Allow: /` and stop.
+- Inside each group, `Disallow:` comes **before** `Allow: /` — same reason: with
+ the broad allow first, a first-match parser (Python's `urllib.robotparser`,
+ for one) hands out `/debug` and `/interactive`.
+
### Backend (api.anyplot.ai)
Dynamic endpoint at `GET /robots.txt`:
@@ -260,6 +307,61 @@ Disallow: /
- Prevents crawling of debug endpoints, docs, and API responses
- Social media bots (WhatsApp, Twitter, etc.) are unaffected - they fetch og:images directly
+### AI crawler policy
+
+**Decision (issue #9633):** AI agents that *retrieve and cite* are welcome;
+agents that only *collect for training* are declined. anyplot's entire strategy
+is to be consumable by AI agents — `/llms.txt`, the MCP server, MIT-licensed
+code on every page — so blocking the retrieval side works against the product.
+The training reservation stays expressed, and it is the legally load-bearing
+part: `Content-Signal: ai-train=no` is an express reservation of rights under
+Article 4 of EU Directive 2019/790.
+
+| Group | Agents | Policy |
+|---|---|---|
+| Retrieval / citation / user-directed | `ClaudeBot`, `Claude-User`, `Claude-SearchBot`, `OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`, `Perplexity-User` | allowed |
+| Training collectors | `GPTBot`, `CCBot`, `Bytespider`, `Amazonbot`, `meta-externalagent` | declined |
+| Opt-out tokens (vendor crawls under another UA) | `Google-Extended`, `Applebot-Extended` | declined |
+
+`GPTBot` is the deliberate borderline call: it is OpenAI's *training* crawler,
+so it sits with the declined group, while ChatGPT's retrieval path
+(`OAI-SearchBot`, `ChatGPT-User`) stays open. Reversing that is a one-group
+edit in `app/public/robots.txt`. Vendor UA roles shift — re-check the current
+role of each agent against Cloudflare's AI Crawl Control categories before
+changing the policy.
+
+#### Cloudflare is the enforcement layer
+
+Measured 2026-07-25 on the live zone: Cloudflare prepends a **managed
+robots.txt** block (`Disallow: /` for ClaudeBot, GPTBot, CCBot, Google-Extended,
+Amazonbot, Applebot-Extended, Bytespider, meta-externalagent,
+CloudflareBrowserRenderingCrawler) *and* answers those user agents with a hard
+`HTTP 403 Your request was blocked.` at the edge — including `Claude-User` and
+`ChatGPT-User`, and including `/llms.txt` itself. The file written for AI agents
+was unreachable to every agent it was written for. Googlebot passes (200).
+
+`api.anyplot.ai` is **not** covered by the block (ClaudeBot gets 200 there), so
+the MCP server stayed reachable.
+
+Aligning the edge with this policy is a dashboard action (zone `anyplot.ai` →
+**AI Crawl Control** / Bots): allow the retrieval group, keep the training
+group blocked, and either turn off the managed robots.txt (this repo's file
+already carries the content signals) or leave it on and accept that the live
+file is stricter than the repo's.
+
+Verify afterwards:
+
+```bash
+curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://anyplot.ai/llms.txt # expect 200
+curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; CCBot/2.0; +https://commoncrawl.org/faq/)" https://anyplot.ai/ # expect 403
+curl -sA "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://anyplot.ai/scatter-basic | grep -o '
[^<]*' # per-route title, not the SPA shell
+```
+
+The last command is the part this repo owns: `app/nginx.conf` maps the AI UAs
+onto the seo-proxy path, and `.github/workflows/bot-serving-check.yml` guards it
+daily against the Cloud Run origin (origin, not edge — so it reports on the
+nginx map no matter what the zone policy is).
+
## Sitemap
Dynamic XML sitemap for search engine indexing.
@@ -335,10 +437,12 @@ curl -o test.png https://api.anyplot.ai/og/scatter-basic.png
| File | Purpose |
|------|---------|
| `app/nginx.conf` | Bot detection, SPA routing, sitemap proxy |
-| `app/public/robots.txt` | Frontend robots.txt (blocks /debug) |
+| `app/public/robots.txt` | Frontend robots.txt (content signals, AI crawler policy, blocks /debug) |
+| `app/public/llms.txt` | Agent-facing site summary; served directly, never via the seo-proxy |
| `api/routers/seo.py` | SEO proxy endpoints, robots.txt, sitemap generation |
| `api/routers/og_images.py` | Branded og:image endpoints |
| `core/images.py` | Image processing, branding functions |
+| `.github/workflows/bot-serving-check.yml` | Daily synthetic check of the bot → seo-proxy path |
## Multi-Language URL Strategy