Skip to content

Resilience: Add retry/backoff for idempotent HTTP GET requests #59

Description

@MozzamShahid

File / lines: src/gmaps_scraper/scraper.py, lines 213-277; src/gmaps_scraper/place_scraper.py, lines 2736-2778

Problem: Both collect_http_artifacts and _collect_preview_place_enrichment create a single session.get(...) for the primary request and any preload request. There is no retry loop, no jitter, and no handling of Retry-After.

Why it matters: Transient Google rate-limiting, TLS handshake failures, or short network blips immediately surface as ScrapeError or degraded place output. Since these are read-only GET requests, retrying is safe and expected for a production scraper.

Suggested fix:

  1. Introduce a small shared helper that wraps idempotent GET calls with configurable retries (e.g., 3 attempts), exponential backoff with jitter, and explicit handling of 429 / 5xx / timeout.
  2. Respect Retry-After when present.
  3. Add tests for retry behavior using mocked responses.

Difficulty: medium
Impact: High — reduces transient failures and improves batch reliability.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions