crawler: enforce Crawl-delay, back off on 429/503, surface politeness counters

[andreimarinescu]

Two field incidents motivated this: a temporary IP-level ban from news.ycombinator.com (robots.txt asks Crawl-delay: 30; the crawler fetched at ~1/s regardless) and a 429 storm from huggingface.co (an 823-item feed crawled with no backoff — 424 throttle responses in one pass, roughly half the feed lost to terminal errored entries).

Crawl-delay was parsed but effectively unenforced. The delay was slept per-worker after the host-gate slot was reserved, so concurrent workers still received slots at per_host_delay_ms spacing and the effective per-host rate ignored robots.txt entirely. robots.txt is now consulted before the gate and the delay widens the slot reservation itself (hostGate.WaitFor). Fractional values (Crawl-delay: 0.5) parse, and a new crawler.max_crawl_delay_ms (default 2 min) clamps hostile robots.txt values.

429 (and 503 with Retry-After) no longer error the frontier entry. Errored entries are terminal, so every throttled URL was lost until a manual recrawl. Now the host's next slot is deferred by Retry-After (or an exponential fallback, capped at 5 min) and the URL is requeued up to 3 times before failing. Rate-limited responses also stay out of hostStats so the success-ratio blacklist can't be poisoned by a healthy-but-busy server.

Politeness observability. Claim-time allowlist drops were silent — imports report success while every item is discarded. They're now counted, logged once per host, and exposed together with deferral counts via /stats (crawl_dropped_disallowed, crawl_rate_limited_deferrals) and /metrics.

/admin/crawl-enqueue accepts an optional lane (empty keeps the historical discovered default) so bulk feeds don't have to compete with link discovery.

Replaying the huggingface feed on a fresh index: 429s received 424 → 85 (each one deferring the host), URLs lost ~half → 1 of 823, no blacklisting. Tests cover the gate timing under concurrency (the HN failure mode), robots parsing, the previously untested fetchRSS paths, both 429 flows (recover and give-up), and lane routing; the suite is green with -race.

[codecov]

Codecov Report

❌ Patch coverage is 92.72727% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/crawler/crawler.go	92.10%	3 Missing and 3 partials ⚠️
cmd/cosift/serve_setup.go	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!