Docs URL Discovery Bench

Benchmarks coding agents on URL discovery: given a question, how efficiently does an agent find the right page on a docs site? It compares four serving formats to measure whether content negotiation (markdown for agents) and llms.txt reduce wasted fetches and 404s.

Finding

Across 20 Mintlify docs sites and two agents (Claude Code on Sonnet 5, Codex on GPT-5.5), serving an llms.txt index eliminates ~90%+ of 404s versus raw HTML. The main finding is that it's the index, not the markdown, that does it. It replicates on three further models (Opus 4.8, Fable 5, and GPT-5.6), same direction on 19–20 of 20 sites per model. Full numbers and significance are in RESULTS.md.

Arms

Arm	Serving behavior
html	HTML
md	Markdown
md-link	Markdown with the llms.txt pointer
md-inline	Markdown with the full llms.txt content inlined into every page

All arms serve identical content through a local proxy that rewrites the upstream origin to itself, so every agent fetch is observable and 404s are counted from proxy logs.

Metrics

Per arm × agent: 404s per task, fetches-to-success, tokens, wall time, and, when the dataset provides expected answers, URL accuracy.

Setup

git clone https://github.com/mintlify/url-discovery-bench
cd url-discovery-bench
pip install -r requirements.txt   # requests, numpy, scipy

Requires Python 3.9+. That's everything Tier 0 needs, but Tiers 1 and 2 additionally need the claude (Claude Code) and/or codex (OpenAI Codex CLI) installed and authenticated. macOS/Linux only.

Reproduce

Tier 0: verify the findings (no API keys, instant)

The raw results of the full run ship in the repo, so every table and p-value is regenerable without re-running anything.

python analysis/stats.py data/full/results.jsonl   # aggregate table + site-clustered significance
python -m url_discovery_bench.report data/full      # same aggregate table
python analysis/breakdown_404.py data/full dataset/full.json --per-agent \
  --inventory data/full/page-inventory.json          # 404 composition (proxy logs + frozen inventory, both committed)
python analysis/multimodel.py                       # per-model table across all five models
python analysis/stats.py data/full/results.jsonl data/multimodel/results.jsonl --by-model  # per-model significance

Tier 1: run it on your own docs (live, minutes)

Point the benchmark at any docs site and watch the 404/fetch/token difference across formats. No ground truth required. expected answers are optional; omit them and you get the efficiency metrics (accuracy shows n/a), or add them to also grade correctness.

cp dataset/example.json dataset/mysite.json
# edit mysite.json: set target, basePath, rewriteOrigins, and a few questions
python -m url_discovery_bench.probe https://docs.mysite.com https://mysite.com/docs   # confirm it's a Mintlify site
python -m url_discovery_bench.run --dataset dataset/mysite.json --agents claude --snapshot-mode record --job mysite
python -m url_discovery_bench.report jobs/mysite

--snapshot-mode record measures against the live site while freezing every response to snapshots/ so the run can later be replayed deterministically (--snapshot-mode replay).

Tier 2: the full study (hours, ~$100, both CLIs)

20 sites × 5 questions × 4 arms × 2 agents × 3 attempts = 2,400 attempts, and re-crawls ~3 GB of snapshots. Exits non-zero while attempts remain, so wrap it in a retry loop:

until python -m url_discovery_bench.run --dataset dataset/full.json --attempts 3 --job full \
  --agents claude,codex --concurrency 8 --snapshot-mode record; do sleep 120; done

Contamination gate

Every attempt's transcript is audited for commands containing URLs whose host is not the proxy (url_discovery_bench/audit.py). Any hit hard-fails the attempt: it is recorded as an error: "contaminated" row with the offending URLs, excluded from the report, and retried on resume. Claude is additionally tool-restricted to Bash(curl:*) with WebFetch/WebSearch disallowed and MCP servers disabled; Codex has no per-tool allowlist (its sandbox needs network to reach the proxy), so the audit gate is the enforcement there. Commands whose URL can't be statically resolved (e.g. curl "$u" in a loop) are counted in the report's unaudited column rather than silently passed; the audit's scope and limits are spelled out in RESULTS.md.

Snapshots

python -m url_discovery_bench.snapshot_crawl --dataset dataset/<file>.json crawls every page each site declares (llms.txt + sitemap, both variants) into snapshots/. Three proxy modes:

• live (default): fetch upstream every request.
• record: fetch upstream and persist each response, used for report-quality runs, so published numbers reflect real sites while still producing a frozen artifact.
• replay: serve purely from disk; no upstream traffic. Deterministic reproduction mode. Unknown paths return 404 and are flagged snapshotMiss in proxy logs and a snapshotMisses count per attempt.

Snapshots (~3 GB) and transcripts are gitignored; re-crawl with python -m url_discovery_bench.snapshot_crawl.

Layout

url_discovery_bench/proxy.py          # serving proxy: 4 modes, origin rewrite, snapshot record/replay, JSONL log
url_discovery_bench/runners.py        # headless claude (curl-only) + codex exec runners
url_discovery_bench/audit.py          # contamination audit: external-host fetch detection
url_discovery_bench/grade.py          # URL normalization + answer grading (expected optional)
url_discovery_bench/run.py            # orchestrator: task x arm x agent x attempt, one proxy per attempt
url_discovery_bench/report.py         # per-arm aggregate table
url_discovery_bench/snapshot.py       # snapshot read/write + live fetch
url_discovery_bench/snapshot_crawl.py # crawl a site's pages into snapshots/
url_discovery_bench/probe.py          # check whether a site negotiates markdown + serves llms.txt
analysis/stats.py     # aggregate + site-clustered paired significance (Wilcoxon + paired t); --by-model to group by model
analysis/multimodel.py # per-model per-arm table across the core + replication runs
analysis/breakdown_404.py # classify each 404 against the site's real page list (reads <job>/logs)
dataset/              # sites + question -> expected-path ground truth; example.json is a template
data/full/            # committed core-run artifacts (sonnet-5 + gpt-5.5, record mode): results.jsonl, proxy logs, frozen page inventory
data/multimodel/      # committed replication results (results.jsonl): opus-4.8, fable-5, gpt-5.6, replay mode
jobs/                 # run outputs (gitignored): results.jsonl, proxy logs, transcripts

Notes

• Web search and non-curl tools are disabled; the prompt confines the agent to the proxy origin and transcripts are audited for external fetches. Answers pointing at any host other than the attempt's proxy origin grade incorrect (offProxy); a memorized production URL never counts as discovery.
• The proxy origin (127.0.0.1:<port>) prevents reuse of memorized URLs, but questions should still avoid leaking the page slug, and a dataset should span many sites.
• Upstream content can change between runs; run all arms of a comparison in the same session, or use record/replay to freeze content before a report-quality run.
• All fetches go through curl, so every request is one observable, countable hit through the proxy. That makes the absolute 404 counts method-dependent (a fetch tool that converts HTML to markdown client-side would show a smaller absolute gap), but the relative index win (html → md-link) and its direction are unaffected, and curl-only if anything understates HTML's disadvantage rather than inflating it.
• Grading is exact normalized-path match against expected. If a question has more than one valid answer page, list every acceptable path in expected or correct answers score wrong.

License

MIT, see LICENSE.