From 62af6bf1ea62430a3ee461959adf4c5e7dcfe560 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Sat, 25 Jul 2026 23:37:15 +0200 Subject: [PATCH 1/6] feat(seo): serve prerendered pages to AI crawlers, state the policy in robots.txt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude, ChatGPT and Perplexity fetch pages either on a user's explicit request or to build a citation index, and none of them execute JavaScript — so they were being handed the empty SPA shell, the same failure mode the search engines had. Map them onto the seo-proxy path and guard it in the daily bot-serving check (origin-level, so it holds regardless of the Cloudflare edge policy). robots.txt now carries the full policy itself — content signals incl. the Art. 4 EU-DSM training reservation, the welcomed retrieval agents, the declined training collectors — so it survives with Cloudflare's managed block turned off. docs/reference/seo.md records the decision, the measured edge behaviour (hard 403 for every AI UA, llms.txt included) and the dashboard steps that lift it. Refs #9633 --- .github/workflows/bot-serving-check.yml | 21 ++++-- CHANGELOG.md | 9 +++ app/nginx.conf | 18 +++++ app/public/robots.txt | 56 +++++++++++++--- docs/reference/seo.md | 89 ++++++++++++++++++++++++- 5 files changed, 177 insertions(+), 16 deletions(-) diff --git a/.github/workflows/bot-serving-check.yml b/.github/workflows/bot-serving-check.yml index 6c337de384..b39b6edbca 100644 --- a/.github/workflows/bot-serving-check.yml +++ b/.github/workflows/bot-serving-check.yml @@ -28,9 +28,9 @@ permissions: jobs: bot-serving: runs-on: ubuntu-latest - # 5 checks x (--retry 2 -> up to 3 attempts x --max-time 30) can reach - # ~7.5 min worst-case; 10 leaves room to report a clean failure. - timeout-minutes: 10 + # 7 checks x (--retry 2 -> up to 3 attempts x --max-time 30) can reach + # ~10.5 min worst-case; 14 leaves room to report a clean failure. + timeout-minutes: 14 steps: - name: Crawler UAs must get 200 + per-route titles run: | @@ -40,6 +40,8 @@ jobs: ORIGIN="https://anyplot-app-r3tvmejsmq-ez.a.run.app" GOOGLEBOT="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" TWITTERBOT="Twitterbot/1.0" + CLAUDEBOT="Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" + CHATGPTUSER="Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)" HUMAN="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36" fail=0 @@ -65,8 +67,17 @@ jobs: check "$GOOGLEBOT" "$ORIGIN/scatter-basic" "Basic Scatter Plot | anyplot.ai" check "$TWITTERBOT" "$ORIGIN/scatter-basic/python/matplotlib" "Basic Scatter Plot - Matplotlib | anyplot.ai" - # llms.txt must be served directly, never proxied to the seo backend - check "$GOOGLEBOT" "$ORIGIN/llms.txt" "# anyplot" + # AI assistants take the same prerendered path (nginx $is_bot). These + # checks hit the ORIGIN, so they verify the nginx map independently of + # whether Cloudflare's AI Crawl Control currently 403s these UAs at + # the edge — an edge-level policy change needs no change here. + check "$CLAUDEBOT" "$ORIGIN/scatter-basic" "Basic Scatter Plot | anyplot.ai" + + # llms.txt must be served directly, never proxied to the seo backend — + # including for a mapped crawler UA, which is the whole point of the + # `location = /llms.txt` bypass. + check "$GOOGLEBOT" "$ORIGIN/llms.txt" "# anyplot" + check "$CHATGPTUSER" "$ORIGIN/llms.txt" "# anyplot" # Control: humans must still get the SPA shell check "$HUMAN" "$ORIGIN/" '
' diff --git a/CHANGELOG.md b/CHANGELOG.md index 1177c6b133..3492b15335 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,15 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Added +- **AI crawler policy, written down and served** — `app/public/robots.txt` now states the whole + policy itself (content signals incl. the Art. 4 EU-DSM training reservation, the welcomed + retrieval/citation agents, the declined training collectors), so it holds with or without + Cloudflare's managed block; `app/nginx.conf` routes `Claude*`, `GPTBot`, `OAI-SearchBot`, + `ChatGPT-User` and `Perplexity*` to the prerendered seo-proxy pages like the search engines + (they run no JavaScript and were being served the empty SPA shell); the daily + `bot-serving-check` now covers an AI UA and `llms.txt` under an AI UA. `docs/reference/seo.md` + documents the decision, the measured edge behaviour and the Cloudflare steps that lift the + 403 (#9633). - **`CODE_OF_CONDUCT.md`** — Contributor Covenant 2.1, linked from `docs/contributing.md` (closes the last repo-health gap from audit 2026-07-15 Medium#29) (#9644). - **Bot-served pages got a real SEO surface** — `seo_home()` now emits the site-level JSON-LD diff --git a/app/nginx.conf b/app/nginx.conf index 2aef813bc9..b9997fc453 100644 --- a/app/nginx.conf +++ b/app/nginx.conf @@ -27,6 +27,24 @@ map $http_user_agent $is_bot { ~*yandexbot 1; ~*baiduspider 1; ~*applebot 1; + # AI assistants, AI search and citation crawlers. Same failure mode as the + # search engines: none of them execute JavaScript, so without these entries + # an AI answer citing anyplot describes the empty SPA shell. Claude-User / + # ChatGPT-User / Perplexity-User are USER-directed fetches — a human asked + # their assistant to open the page — and matter most. + # This map only decides WHAT a crawler gets once it arrives, never whether + # it may: the allow/deny policy is robots.txt plus the Cloudflare AI Crawl + # Control zone setting (docs/reference/seo.md). GPTBot is listed even + # though robots.txt declines it, so the rendering is right if that call is + # ever reversed. + ~*claudebot 1; + ~*claude-user 1; + ~*claude-searchbot 1; + ~*gptbot 1; + ~*oai-searchbot 1; + ~*chatgpt-user 1; + ~*perplexitybot 1; + ~*perplexity-user 1; # Social Media ~*twitterbot 1; ~*facebookexternalhit 1; diff --git a/app/public/robots.txt b/app/public/robots.txt index 4785cbb835..3916d18d2e 100644 --- a/app/public/robots.txt +++ b/app/public/robots.txt @@ -1,16 +1,56 @@ -# Intended crawler policy: AI crawlers are WELCOME (see /llms.txt and the MCP -# server — anyplot's strategy is to be maximally consumable by AI agents). +# Crawler policy — AI crawlers that RETRIEVE and CITE are welcome: anyplot is +# built to be consumable by AI agents (see /llms.txt and the MCP server at +# https://api.anyplot.ai/mcp/). Training-only scrapers are declined below. # -# NOTE: Cloudflare's "block AI bots" zone setting currently PREPENDS a managed -# block (Disallow: / for ClaudeBot, GPTBot, CCBot, Google-Extended, ...) to the -# live https://anyplot.ai/robots.txt, contradicting this file and llms.txt. -# Fix lives in the Cloudflare dashboard (zone setting), not in this repo: -# disable the managed AI-bot block (or switch it to Content-Signal-only) so -# the live file matches this one. +# Cloudflare's AI Crawl Control can PREPEND its own managed block to the live +# file and additionally answer the blocked user agents with HTTP 403 at the +# edge, so https://anyplot.ai/robots.txt may be longer and stricter than this +# file. The policy is therefore spelled out here in full — including the +# content signals Cloudflare would otherwise supply — so it survives with the +# managed block turned off. Rationale, dashboard steps and how to verify: +# docs/reference/seo.md ("AI crawler policy"). +# Content signals per contentsignals.org. ANY RESTRICTION EXPRESSED HERE IS AN +# EXPRESS RESERVATION OF RIGHTS UNDER ARTICLE 4 OF EU DIRECTIVE 2019/790. +# search: indexing and returning links/excerpts — yes +# ai-input: retrieval for AI answers, grounding, citation — yes +# ai-train: training or fine-tuning models — no User-agent: * +Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference Allow: / Disallow: /debug Disallow: /interactive +# AI assistants, AI search and citation crawlers. These send readers back and +# are the reason /llms.txt exists; the *-User agents fetch only because a human +# asked their assistant to open the page. +User-agent: ClaudeBot +User-agent: Claude-User +User-agent: Claude-SearchBot +User-agent: OAI-SearchBot +User-agent: ChatGPT-User +User-agent: PerplexityBot +User-agent: Perplexity-User +Allow: / +Disallow: /debug +Disallow: /interactive + +# Training-only collectors: they take the catalogue without ever returning a +# reader, which is what ai-train=no above says in prose. GPTBot sits here +# because it is OpenAI's TRAINING crawler — ChatGPT's retrieval path is +# OAI-SearchBot/ChatGPT-User above and stays open. Flip this one group if the +# training stance ever changes. +User-agent: GPTBot +User-agent: CCBot +User-agent: Bytespider +User-agent: Amazonbot +User-agent: meta-externalagent +Disallow: / + +# Opt-out tokens for vendors that crawl under a different user agent +# (Googlebot / Applebot fetch for search and stay allowed above). +User-agent: Google-Extended +User-agent: Applebot-Extended +Disallow: / + Sitemap: https://anyplot.ai/sitemap.xml diff --git a/docs/reference/seo.md b/docs/reference/seo.md index 741cfb5a88..3f002bcce4 100644 --- a/docs/reference/seo.md +++ b/docs/reference/seo.md @@ -52,7 +52,7 @@ Our solution uses **nginx-based bot detection** to serve pre-rendered HTML with ### Detected Bots -nginx detects 27 bots via User-Agent matching, organized by category: +nginx detects 37 User-Agent patterns, organized by category: **Social Media:** | Bot | User-Agent Pattern | @@ -82,12 +82,30 @@ nginx detects 27 bots via User-Agent matching, organized by category: | Bot | User-Agent Pattern | |-----|-------------------| | Google | `googlebot` | +| Google URL inspection | `google-inspectiontool` | +| Google misc crawler | `googleother` | | Bing | `bingbot` | | Yandex | `yandexbot` | | DuckDuckGo | `duckduckbot` | | Baidu | `baiduspider` | | Apple | `applebot` | +**AI Assistants & AI Search:** +| Bot | User-Agent Pattern | Role | +|-----|-------------------|------| +| Anthropic crawler | `claudebot` | index/crawl | +| Claude user fetch | `claude-user` | a human asked Claude to open the page | +| Claude search | `claude-searchbot` | citation index | +| OpenAI crawler | `gptbot` | training (declined in robots.txt, mapped anyway — see below) | +| ChatGPT search | `oai-searchbot` | citation index | +| ChatGPT user fetch | `chatgpt-user` | a human asked ChatGPT to open the page | +| Perplexity | `perplexitybot`, `perplexity-user` | citation index / user fetch | + +None of these execute JavaScript, so without the map entries an AI answer +citing anyplot would describe the empty SPA shell. The map decides **what** a +crawler is served, never **whether** it may fetch — that is the robots.txt + +Cloudflare question in [AI crawler policy](#ai-crawler-policy). + **Link Preview Services:** | Bot | User-Agent Pattern | |-----|-------------------| @@ -236,12 +254,20 @@ Uses **MonoLisa** variable font (commercial, not in repo): ### Frontend (anyplot.ai) -Static file at `app/public/robots.txt`: +Static file at `app/public/robots.txt`. It carries the full policy — content +signals, the welcomed AI agents, the declined training collectors — so it holds +regardless of what Cloudflare does or does not prepend (see +[AI crawler policy](#ai-crawler-policy)): ```txt User-agent: * +Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference Allow: / Disallow: /debug +Disallow: /interactive +… welcomed AI agents (Claude*, OAI-SearchBot, ChatGPT-User, Perplexity*) +… declined training collectors (GPTBot, CCBot, Bytespider, Amazonbot, meta-externalagent) +… opt-out tokens (Google-Extended, Applebot-Extended) Sitemap: https://anyplot.ai/sitemap.xml ``` @@ -260,6 +286,61 @@ Disallow: / - Prevents crawling of debug endpoints, docs, and API responses - Social media bots (WhatsApp, Twitter, etc.) are unaffected - they fetch og:images directly +### AI crawler policy + +**Decision (issue #9633):** AI agents that *retrieve and cite* are welcome; +agents that only *collect for training* are declined. anyplot's entire strategy +is to be consumable by AI agents — `/llms.txt`, the MCP server, MIT-licensed +code on every page — so blocking the retrieval side works against the product. +The training reservation stays expressed, and it is the legally load-bearing +part: `Content-Signal: ai-train=no` is an express reservation of rights under +Article 4 of EU Directive 2019/790. + +| Group | Agents | Policy | +|---|---|---| +| Retrieval / citation / user-directed | `ClaudeBot`, `Claude-User`, `Claude-SearchBot`, `OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`, `Perplexity-User` | allowed | +| Training collectors | `GPTBot`, `CCBot`, `Bytespider`, `Amazonbot`, `meta-externalagent` | declined | +| Opt-out tokens (vendor crawls under another UA) | `Google-Extended`, `Applebot-Extended` | declined | + +`GPTBot` is the deliberate borderline call: it is OpenAI's *training* crawler, +so it sits with the declined group, while ChatGPT's retrieval path +(`OAI-SearchBot`, `ChatGPT-User`) stays open. Reversing that is a one-group +edit in `app/public/robots.txt`. Vendor UA roles shift — re-check the current +role of each agent against Cloudflare's AI Crawl Control categories before +changing the policy. + +#### Cloudflare is the enforcement layer + +Measured 2026-07-25 on the live zone: Cloudflare prepends a **managed +robots.txt** block (`Disallow: /` for ClaudeBot, GPTBot, CCBot, Google-Extended, +Amazonbot, Applebot-Extended, Bytespider, meta-externalagent, +CloudflareBrowserRenderingCrawler) *and* answers those user agents with a hard +`HTTP 403 Your request was blocked.` at the edge — including `Claude-User` and +`ChatGPT-User`, and including `/llms.txt` itself. The file written for AI agents +was unreachable to every agent it was written for. Googlebot passes (200). + +`api.anyplot.ai` is **not** covered by the block (ClaudeBot gets 200 there), so +the MCP server stayed reachable. + +Aligning the edge with this policy is a dashboard action (zone `anyplot.ai` → +**AI Crawl Control** / Bots): allow the retrieval group, keep the training +group blocked, and either turn off the managed robots.txt (this repo's file +already carries the content signals) or leave it on and accept that the live +file is stricter than the repo's. + +Verify afterwards: + +```bash +curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://anyplot.ai/llms.txt # expect 200 +curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; CCBot/2.0; +https://commoncrawl.org/faq/)" https://anyplot.ai/ # expect 403 +curl -sA "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://anyplot.ai/scatter-basic | grep -o '[^<]*' # per-route title, not the SPA shell +``` + +The last command is the part this repo owns: `app/nginx.conf` maps the AI UAs +onto the seo-proxy path, and `.github/workflows/bot-serving-check.yml` guards it +daily against the Cloud Run origin (origin, not edge — so it reports on the +nginx map no matter what the zone policy is). + ## Sitemap Dynamic XML sitemap for search engine indexing. @@ -335,10 +416,12 @@ curl -o test.png https://api.anyplot.ai/og/scatter-basic.png | File | Purpose | |------|---------| | `app/nginx.conf` | Bot detection, SPA routing, sitemap proxy | -| `app/public/robots.txt` | Frontend robots.txt (blocks /debug) | +| `app/public/robots.txt` | Frontend robots.txt (content signals, AI crawler policy, blocks /debug) | +| `app/public/llms.txt` | Agent-facing site summary; served directly, never via the seo-proxy | | `api/routers/seo.py` | SEO proxy endpoints, robots.txt, sitemap generation | | `api/routers/og_images.py` | Branded og:image endpoints | | `core/images.py` | Image processing, branding functions | +| `.github/workflows/bot-serving-check.yml` | Daily synthetic check of the bot → seo-proxy path | ## Multi-Language URL Strategy From 83d15fb77f55b19c943813ba1717f847d8addb88 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Sat, 25 Jul 2026 23:41:31 +0200 Subject: [PATCH 2/6] docs: reference the PR in the changelog entry --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3492b15335..fb4031edfb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,7 +26,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an (they run no JavaScript and were being served the empty SPA shell); the daily `bot-serving-check` now covers an AI UA and `llms.txt` under an AI UA. `docs/reference/seo.md` documents the decision, the measured edge behaviour and the Cloudflare steps that lift the - 403 (#9633). + 403 (#9898, decision #9633). - **`CODE_OF_CONDUCT.md`** — Contributor Covenant 2.1, linked from `docs/contributing.md` (closes the last repo-health gap from audit 2026-07-15 Medium#29) (#9644). - **Bot-served pages got a real SEO surface** — `seo_home()` now emits the site-level JSON-LD From 206fd20603402f92543e47a81d80779bdef82fd4 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Sat, 25 Jul 2026 23:45:54 +0200 Subject: [PATCH 3/6] fix(seo): repeat the content signals in the AI user-agent group A crawler obeys the single robots.txt group that matches it most specifically, so ClaudeBot & co. would never have seen the ai-train=no reservation declared under User-agent: * (Copilot #9898). --- app/public/robots.txt | 5 +++++ docs/reference/seo.md | 5 +++++ 2 files changed, 10 insertions(+) diff --git a/app/public/robots.txt b/app/public/robots.txt index 3916d18d2e..f0ca63be62 100644 --- a/app/public/robots.txt +++ b/app/public/robots.txt @@ -24,6 +24,10 @@ Disallow: /interactive # AI assistants, AI search and citation crawlers. These send readers back and # are the reason /llms.txt exists; the *-User agents fetch only because a human # asked their assistant to open the page. +# The Content-Signal line is REPEATED here on purpose: a crawler obeys the one +# group that matches it most specifically, so an agent named below would never +# see the signal declared in the `User-agent: *` group above — and this is +# exactly the group the training reservation has to reach. User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot @@ -31,6 +35,7 @@ User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User +Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference Allow: / Disallow: /debug Disallow: /interactive diff --git a/docs/reference/seo.md b/docs/reference/seo.md index 3f002bcce4..de9749c28d 100644 --- a/docs/reference/seo.md +++ b/docs/reference/seo.md @@ -272,6 +272,11 @@ Disallow: /interactive Sitemap: https://anyplot.ai/sitemap.xml ``` +The `Content-Signal` line is repeated inside the welcomed-AI group: a crawler +obeys the single group that matches it most specifically, so an agent named in +its own group never sees the signal declared under `User-agent: *` — and that +group is precisely where the training reservation has to land. + ### Backend (api.anyplot.ai) Dynamic endpoint at `GET /robots.txt`: From 410e123b30e27adaeb958a94de5424195dca588e Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Sat, 25 Jul 2026 23:47:40 +0200 Subject: [PATCH 4/6] fix(seo): put the named robots.txt groups before the wildcard group MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec-compliant crawlers pick the most specific matching group, but simpler parsers take the first match — with User-agent: * on top those would read Allow: / and never reach the declining groups (Copilot #9898). --- app/public/robots.txt | 33 +++++++++++++++++++-------------- docs/reference/seo.md | 21 ++++++++++++++------- 2 files changed, 33 insertions(+), 21 deletions(-) diff --git a/app/public/robots.txt b/app/public/robots.txt index f0ca63be62..ef6e15c536 100644 --- a/app/public/robots.txt +++ b/app/public/robots.txt @@ -9,25 +9,23 @@ # content signals Cloudflare would otherwise supply — so it survives with the # managed block turned off. Rationale, dashboard steps and how to verify: # docs/reference/seo.md ("AI crawler policy"). +# +# Named groups come FIRST and the `User-agent: *` group last: a compliant +# crawler picks the most specific matching group regardless of order, but +# simpler parsers take the first match — and with the wildcard on top those +# would read `Allow: /` and never reach the declining groups below. # Content signals per contentsignals.org. ANY RESTRICTION EXPRESSED HERE IS AN # EXPRESS RESERVATION OF RIGHTS UNDER ARTICLE 4 OF EU DIRECTIVE 2019/790. # search: indexing and returning links/excerpts — yes # ai-input: retrieval for AI answers, grounding, citation — yes # ai-train: training or fine-tuning models — no -User-agent: * -Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference -Allow: / -Disallow: /debug -Disallow: /interactive - +# # AI assistants, AI search and citation crawlers. These send readers back and # are the reason /llms.txt exists; the *-User agents fetch only because a human -# asked their assistant to open the page. -# The Content-Signal line is REPEATED here on purpose: a crawler obeys the one -# group that matches it most specifically, so an agent named below would never -# see the signal declared in the `User-agent: *` group above — and this is -# exactly the group the training reservation has to reach. +# asked their assistant to open the page. The signal is declared per group +# because a crawler obeys only the group that matches it — the agents named +# here are precisely the ones the training reservation has to reach. User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot @@ -41,8 +39,8 @@ Disallow: /debug Disallow: /interactive # Training-only collectors: they take the catalogue without ever returning a -# reader, which is what ai-train=no above says in prose. GPTBot sits here -# because it is OpenAI's TRAINING crawler — ChatGPT's retrieval path is +# reader, which is what ai-train=no says in prose. GPTBot sits here because it +# is OpenAI's TRAINING crawler — ChatGPT's retrieval path is # OAI-SearchBot/ChatGPT-User above and stays open. Flip this one group if the # training stance ever changes. User-agent: GPTBot @@ -53,9 +51,16 @@ User-agent: meta-externalagent Disallow: / # Opt-out tokens for vendors that crawl under a different user agent -# (Googlebot / Applebot fetch for search and stay allowed above). +# (Googlebot / Applebot fetch for search and stay allowed). User-agent: Google-Extended User-agent: Applebot-Extended Disallow: / +# Everyone else: search engines, social/link previews, feed readers. +User-agent: * +Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference +Allow: / +Disallow: /debug +Disallow: /interactive + Sitemap: https://anyplot.ai/sitemap.xml diff --git a/docs/reference/seo.md b/docs/reference/seo.md index de9749c28d..f2e6f4f738 100644 --- a/docs/reference/seo.md +++ b/docs/reference/seo.md @@ -260,22 +260,29 @@ regardless of what Cloudflare does or does not prepend (see [AI crawler policy](#ai-crawler-policy)): ```txt +… welcomed AI agents (Claude*, OAI-SearchBot, ChatGPT-User, Perplexity*) + + Content-Signal, Allow: /, Disallow: /debug, /interactive +… declined training collectors (GPTBot, CCBot, Bytespider, Amazonbot, meta-externalagent) +… opt-out tokens (Google-Extended, Applebot-Extended) + User-agent: * Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference Allow: / Disallow: /debug Disallow: /interactive -… welcomed AI agents (Claude*, OAI-SearchBot, ChatGPT-User, Perplexity*) -… declined training collectors (GPTBot, CCBot, Bytespider, Amazonbot, meta-externalagent) -… opt-out tokens (Google-Extended, Applebot-Extended) Sitemap: https://anyplot.ai/sitemap.xml ``` -The `Content-Signal` line is repeated inside the welcomed-AI group: a crawler -obeys the single group that matches it most specifically, so an agent named in -its own group never sees the signal declared under `User-agent: *` — and that -group is precisely where the training reservation has to land. +Two properties of that file are deliberate: + +- The `Content-Signal` line is **repeated** in the welcomed-AI group. A crawler + obeys the single group that matches it, so an agent named in its own group + never sees the signal declared under `User-agent: *` — and that group is + precisely where the training reservation has to land. +- The named groups come **before** the wildcard group. A spec-compliant crawler + picks the most specific match regardless of order, but simpler parsers take + the first match and would read `Allow: /` and stop. ### Backend (api.anyplot.ai) From c34c44ff08d7b0991231dce9ca09fe8279b25d9f Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Sat, 25 Jul 2026 23:53:22 +0200 Subject: [PATCH 5/6] fix(seo): put Disallow before the broad Allow in every robots.txt group With 'Allow: /' first, a first-match parser (Python's urllib.robotparser, for one) hands out /debug and /interactive; the longest-match parsers are order-independent, so leading with the Disallow lines is correct for both. --- app/public/robots.txt | 16 ++++++++++------ docs/reference/seo.md | 3 +++ 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/app/public/robots.txt b/app/public/robots.txt index ef6e15c536..2c34a08ae1 100644 --- a/app/public/robots.txt +++ b/app/public/robots.txt @@ -10,10 +10,14 @@ # managed block turned off. Rationale, dashboard steps and how to verify: # docs/reference/seo.md ("AI crawler policy"). # -# Named groups come FIRST and the `User-agent: *` group last: a compliant -# crawler picks the most specific matching group regardless of order, but -# simpler parsers take the first match — and with the wildcard on top those -# would read `Allow: /` and never reach the declining groups below. +# Two ordering rules in here are load-bearing, both for the same reason — +# a compliant crawler resolves by specificity, simpler parsers by first match: +# 1. Named groups come FIRST, `User-agent: *` last. With the wildcard on top, +# a first-match parser would read `Allow: /` and never reach the declining +# groups below. +# 2. Inside a group, `Disallow:` lines come BEFORE `Allow: /` — otherwise a +# first-match parser lets /debug and /interactive through (verified with +# Python's urllib.robotparser, which is exactly such a parser). # Content signals per contentsignals.org. ANY RESTRICTION EXPRESSED HERE IS AN # EXPRESS RESERVATION OF RIGHTS UNDER ARTICLE 4 OF EU DIRECTIVE 2019/790. @@ -34,9 +38,9 @@ User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference -Allow: / Disallow: /debug Disallow: /interactive +Allow: / # Training-only collectors: they take the catalogue without ever returning a # reader, which is what ai-train=no says in prose. GPTBot sits here because it @@ -59,8 +63,8 @@ Disallow: / # Everyone else: search engines, social/link previews, feed readers. User-agent: * Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference -Allow: / Disallow: /debug Disallow: /interactive +Allow: / Sitemap: https://anyplot.ai/sitemap.xml diff --git a/docs/reference/seo.md b/docs/reference/seo.md index f2e6f4f738..bbfbbe095d 100644 --- a/docs/reference/seo.md +++ b/docs/reference/seo.md @@ -283,6 +283,9 @@ Two properties of that file are deliberate: - The named groups come **before** the wildcard group. A spec-compliant crawler picks the most specific match regardless of order, but simpler parsers take the first match and would read `Allow: /` and stop. +- Inside each group, `Disallow:` comes **before** `Allow: /` — same reason: with + the broad allow first, a first-match parser (Python's `urllib.robotparser`, + for one) hands out `/debug` and `/interactive`. ### Backend (api.anyplot.ai) From 6556e3bbac32456c4c670b68538c46a6d34a8a4d Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Sun, 26 Jul 2026 00:02:12 +0200 Subject: [PATCH 6/6] fix(seo): declare the content signals in every robots.txt group MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The declining groups carried no Content-Signal, so the very agents the ai-train=no reservation is aimed at would never have read it — a crawler only reads the group that matches it. Also replaces the seo.md snippet, which showed invented '…' lines and the old Allow-before-Disallow order, with the first group verbatim (Copilot #9898). --- app/public/robots.txt | 11 ++++++++--- docs/reference/seo.md | 34 ++++++++++++++++++++-------------- 2 files changed, 28 insertions(+), 17 deletions(-) diff --git a/app/public/robots.txt b/app/public/robots.txt index 2c34a08ae1..210f603294 100644 --- a/app/public/robots.txt +++ b/app/public/robots.txt @@ -25,11 +25,14 @@ # ai-input: retrieval for AI answers, grounding, citation — yes # ai-train: training or fine-tuning models — no # +# The signal is repeated in EVERY group below, because a crawler reads only the +# group that matches it: a reservation declared once under `User-agent: *` would +# never reach a named agent — least of all the training collectors it is aimed +# at. +# # AI assistants, AI search and citation crawlers. These send readers back and # are the reason /llms.txt exists; the *-User agents fetch only because a human -# asked their assistant to open the page. The signal is declared per group -# because a crawler obeys only the group that matches it — the agents named -# here are precisely the ones the training reservation has to reach. +# asked their assistant to open the page. User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot @@ -52,12 +55,14 @@ User-agent: CCBot User-agent: Bytespider User-agent: Amazonbot User-agent: meta-externalagent +Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference Disallow: / # Opt-out tokens for vendors that crawl under a different user agent # (Googlebot / Applebot fetch for search and stay allowed). User-agent: Google-Extended User-agent: Applebot-Extended +Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference Disallow: / # Everyone else: search engines, social/link previews, feed readers. diff --git a/docs/reference/seo.md b/docs/reference/seo.md index bbfbbe095d..45659baba6 100644 --- a/docs/reference/seo.md +++ b/docs/reference/seo.md @@ -259,27 +259,33 @@ signals, the welcomed AI agents, the declined training collectors — so it hold regardless of what Cloudflare does or does not prepend (see [AI crawler policy](#ai-crawler-policy)): -```txt -… welcomed AI agents (Claude*, OAI-SearchBot, ChatGPT-User, Perplexity*) - + Content-Signal, Allow: /, Disallow: /debug, /interactive -… declined training collectors (GPTBot, CCBot, Bytespider, Amazonbot, meta-externalagent) -… opt-out tokens (Google-Extended, Applebot-Extended) +Four groups in this order — welcomed AI agents (`ClaudeBot`, `Claude-User`, +`Claude-SearchBot`, `OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`, +`Perplexity-User`), declined training collectors (`GPTBot`, `CCBot`, +`Bytespider`, `Amazonbot`, `meta-externalagent`), opt-out tokens +(`Google-Extended`, `Applebot-Extended`), and finally the wildcard. The first +group verbatim; read the file for the rest: -User-agent: * +```txt +User-agent: ClaudeBot +User-agent: Claude-User +User-agent: Claude-SearchBot +User-agent: OAI-SearchBot +User-agent: ChatGPT-User +User-agent: PerplexityBot +User-agent: Perplexity-User Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference -Allow: / Disallow: /debug Disallow: /interactive - -Sitemap: https://anyplot.ai/sitemap.xml +Allow: / ``` -Two properties of that file are deliberate: +Three properties of that file are deliberate and should survive any cleanup: -- The `Content-Signal` line is **repeated** in the welcomed-AI group. A crawler - obeys the single group that matches it, so an agent named in its own group - never sees the signal declared under `User-agent: *` — and that group is - precisely where the training reservation has to land. +- The `Content-Signal` line is repeated in **every** group, declining ones + included. A crawler reads only the group that matches it, so a reservation + declared once under `User-agent: *` never reaches a named agent — least of all + the training collectors it is aimed at. - The named groups come **before** the wildcard group. A spec-compliant crawler picks the most specific match regardless of order, but simpler parsers take the first match and would read `Allow: /` and stop.