is_blocked() (antibot_detector.py) is applied unconditionally in arun()'s post-processing and can mark successfully-fetched content as failed. Your own code already special-cases it twice in async_webcrawler.py — the comments note it "would misread 0 bytes html as a block" for binary downloads, and that real pages containing anti-bot script markers "trigger false positives" after fallback fetches. We hit a third case: small legitimate pages (a 378-byte file:// document) fail the minimal_text structural tier. Downstream consumers with their own quality pipelines currently have no way to receive the fetched content plus the verdict instead of a hard failure. Would you accept a PR adding a CrawlerRunConfig flag (e.g. check_blocked: bool = True) — or alternatively surfacing the verdict on a successful CrawlResult field — so callers can opt into making that judgment themselves?
is_blocked()(antibot_detector.py) is applied unconditionally inarun()'s post-processing and can mark successfully-fetched content as failed. Your own code already special-cases it twice in async_webcrawler.py — the comments note it "would misread 0 bytes html as a block" for binary downloads, and that real pages containing anti-bot script markers "trigger false positives" after fallback fetches. We hit a third case: small legitimate pages (a 378-byte file:// document) fail theminimal_textstructural tier. Downstream consumers with their own quality pipelines currently have no way to receive the fetched content plus the verdict instead of a hard failure. Would you accept a PR adding aCrawlerRunConfigflag (e.g.check_blocked: bool = True) — or alternatively surfacing the verdict on a successfulCrawlResultfield — so callers can opt into making that judgment themselves?