Skip to content

crawler: enforce Crawl-delay, back off on 429/503, surface politeness counters#30

Open
andreimarinescu wants to merge 3 commits into
mainfrom
fix/crawler-politeness
Open

crawler: enforce Crawl-delay, back off on 429/503, surface politeness counters#30
andreimarinescu wants to merge 3 commits into
mainfrom
fix/crawler-politeness

comments: trim to essentials

81e71f3
Select commit
Loading
Failed to load commit list.

Select a check to view from the sidebar