fix: Gracefully close sitemap stream on SitemapRequestLoader abort

[Mantisus]

Description

Use contextlib.aclosing to let the
stream request finishes gracefully and avoids errors when closing generators.

Example error:

an error occurred during closing of asynchronous generator <async_generator object HttpxHttpClient.stream at 0x7bad2c3fbe10>
asyncgen: <async_generator object HttpxHttpClient.stream at 0x7bad2c3fbe10>
RuntimeError: aclose(): asynchronous generator is already running

[vdusek]

Isn't this just a non-optimal workaround? 🙂

CC suggests utilizing this pattern:

  from contextlib import aclosing, suppress  # already importing suppress here

  # in _load_sitemaps, replace the bare `async for item in parse_sitemap(...)`:
  async with aclosing(
      parse_sitemap(
          [SitemapSource(type='url', url=sitemap_url)],
          self._http_client,
          proxy_info=self._proxy_info,
          options=parse_options,
      )
  ) as sitemap_items:
      async for item in sitemap_items:
          ...

[Mantisus]

Well, asyncio.sleep(0) is pretty common when working with HTTP clients. But yeah, contextlib.aclosing is better in this case.

[vdusek]

LGTM