Skip to content

Add Pyodide/Emscripten client support via FetchConnector (continuation of #7803)#13057

Draft
bendichter wants to merge 4 commits into
aio-libs:masterfrom
bendichter:pyodide-support-v2
Draft

Add Pyodide/Emscripten client support via FetchConnector (continuation of #7803)#13057
bendichter wants to merge 4 commits into
aio-libs:masterfrom
bendichter:pyodide-support-v2

Conversation

@bendichter

Copy link
Copy Markdown

What do these changes do?

Adds client support for running under Pyodide / Emscripten (WebAssembly in browsers and Node.js), picking up where #7803 left off. I'd really like to be able to use aiohttp from Pyodide (e.g. pyodide/pyodide#4998), and since #7803 invited someone to continue the work, I took a stab at it.

Full disclosure up front: AI (Claude) was used heavily to write this. I've reviewed it and tested it end-to-end and it looks reasonable to me, but I'm very aware this could be more noise than help for maintainers — if it isn't useful, please feel free to close it, no hard feelings. If the overall direction is right, I'm happy to keep working on it based on your feedback.

Design

Following the suggestion in #7803 (comment) (encapsulate the logic in a custom connector so _request() needs minimal changes), everything lives in a new aiohttp/pyodide.py module providing FetchConnector, which ClientSession selects as the default connector when sys.platform == "emscripten" — that default selection is the only change to the client flow.

Rather than subclassing ClientRequest/ClientResponse (as #7803 did against the 3.x codebase), the connector uses the HTTP/1.1 wire format as its interface:

  • a fake transport captures the serialized request and decodes it with aiohttp's own HttpRequestParser (so cookies, auto-added headers, and chunked bodies are all handled by existing code),
  • the request is dispatched through the JavaScript fetch() API,
  • the response is re-serialized into HTTP/1.1 bytes and fed through the standard ResponseHandler/HttpResponseParser.

The fetch-specific details handled: framing/hop-by-hop headers are stripped in both directions (fetch delivers bodies already decompressed/unframed, so e.g. a stale Content-Encoding would corrupt parsing), repeated Set-Cookie headers are recovered via Headers.getSetCookie(), cancellation/timeouts abort the fetch via AbortController, Expect: 100-continue is answered locally, and proxies / CONNECT / upgrades (WebSockets) raise a clear ClientConnectionError.

Two side discoveries:

  • FetchConnector(fetch=...) accepts an injected fetch implementation, so nearly the whole pipeline is unit-testable on regular CPython without a WebAssembly runtime (tests/test_fetch_connector.py) — this should help with the coverage concerns from ENH Add Pyodide support #7803.
  • aiohttp's C extensions could not load under Pyodide at all: llhttp's api.c references JavaScript-provided wasm_on_* callbacks whenever __wasm__ is defined, leaving undefined symbols in the extension. aiohttp/_llhttp_wasm_shim.c provides no-op definitions (aiohttp uses llhttp_init() with its own callbacks and never touches that path).

Testing

  • tests/test_fetch_connector.py: 18 unit tests with a stubbed fetch() (GET/POST/chunked async-generator bodies, header filtering both directions, multi Set-Cookie → cookie jar, 204/HEAD, error propagation, expect100, fetch_options, timeout-cancels-fetch, concurrency). These run in the normal test matrix.
  • tests/test_pyodide.py: pytest-pyodide integration tests that run the built wasm wheel (with compiled C extensions) in Node.js against a live local HTTP server — real POSTs, redirects, cookies, 404s, concurrency, and connection errors. Run by the new test-pyodide CI job (cibuildwheel --platform pyodide, modeled on the test-mobile job).
  • Locally I also ran a 13-scenario end-to-end script on Pyodide 314.0.2 (Python 3.14) under Node, including gzip responses and raise_for_status — plain ClientSession() + session.get(...) works unmodified.

Are there changes in behavior for the user?

None on existing platforms. Under Emscripten/Pyodide (previously unusable — getaddrinfo raises NotImplementedError), ClientSession now works out of the box via fetch(). Platform limitations are documented in docs/client_reference.rst: no proxies or WebSockets, redirects are followed transparently by fetch() before aiohttp sees the response, ssl arguments are ignored, and browsers enforce their own cookie/CORS policies.

Is it a substantial burden for the maintainers to support this?

I've tried to minimize it: the feature is a single self-contained module plus a 2-line default-connector selection; no existing code paths change. Because it talks to the rest of the client through the HTTP wire format and the public parser/protocol interfaces, it should be fairly insulated from internal refactors. The main ongoing costs are the extra CI job (one Ubuntu runner building a cp313 wasm wheel) and keeping the pinned Pyodide/cibuildwheel versions fresh. That said, you're the best judges of this — and if the answer is "yes, too much", closing this is a completely fair outcome.

Related issue number

Related to #7253 and continues #7803 (original work by @hoodmane, who is credited in the CHANGES fragment). Also relevant to pyodide/pyodide#4998.

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes
  • If you provide code modification, please add yourself to CONTRIBUTORS.txt
  • Add a new news fragment into the CHANGES/ folder

🤖 Generated with Claude Code

bendichter and others added 2 commits July 4, 2026 19:03
FetchConnector dispatches requests through the JavaScript fetch() API,
decoding the serialized request with HttpRequestParser and re-serializing
the fetch() response into HTTP/1.1 bytes for the standard ResponseHandler,
so ClientSession needs no platform-specific changes beyond connector
selection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- tests/test_fetch_connector.py: unit tests for the whole fetch pipeline
  using an injected stub fetch(), runnable on any platform.
- tests/test_pyodide.py: pytest-pyodide integration tests running the
  built wasm wheel in Node.js against a local HTTP server.
- CI: test-pyodide job building the cp313 Emscripten wheel with
  cibuildwheel and running the integration tests with --rt node.
- aiohttp/_llhttp_wasm_shim.c: no-op definitions for llhttp's
  JavaScript-oriented wasm_on_* callbacks so the C extensions load
  under Emscripten.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@psf-chronographer psf-chronographer Bot added the bot:chronographer:provided There is a change note present in this PR label Jul 5, 2026
Comment thread tests/test_pyodide.py Fixed
@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.68153% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 98.96%. Comparing base (e3774b4) to head (72a65e3).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
aiohttp/pyodide.py 99.07% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           master   #13057    +/-   ##
========================================
  Coverage   98.96%   98.96%            
========================================
  Files         131      133     +2     
  Lines       48156    48469   +313     
  Branches     2499     2514    +15     
========================================
+ Hits        47656    47968   +312     
  Misses        376      376            
- Partials      124      125     +1     
Flag Coverage Δ
Autobahn 22.25% <28.02%> (+0.03%) ⬆️
CI-GHA 98.91% <99.68%> (+<0.01%) ⬆️
OS-Linux 98.67% <99.68%> (+<0.01%) ⬆️
OS-Windows 97.06% <99.68%> (+0.01%) ⬆️
OS-macOS 97.95% <99.68%> (+<0.01%) ⬆️
Py-3.10 98.15% <99.68%> (+<0.01%) ⬆️
Py-3.11 98.42% <99.68%> (+<0.01%) ⬆️
Py-3.12 98.51% <99.68%> (+<0.01%) ⬆️
Py-3.13 98.48% <99.68%> (+<0.01%) ⬆️
Py-3.14 98.50% <99.68%> (+<0.01%) ⬆️
Py-3.14t 97.60% <99.68%> (+<0.01%) ⬆️
Py-pypy-3.11 97.45% <99.68%> (+<0.01%) ⬆️
VM-macos 97.95% <99.68%> (+<0.01%) ⬆️
VM-ubuntu 98.67% <99.68%> (+<0.01%) ⬆️
VM-windows 97.06% <99.68%> (+0.01%) ⬆️
cython-coverage 37.94% <12.42%> (-0.17%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@codspeed-hq

codspeed-hq Bot commented Jul 5, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 83 untouched benchmarks
⏩ 83 skipped benchmarks1


Comparing bendichter:pyodide-support-v2 (72a65e3) with master (e3774b4)

Open in CodSpeed

Footnotes

  1. 83 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

- Add 'js' (from Node.js) to the docs spelling wordlist.
- Exclude tests/test_pyodide.py from codecov (it only executes inside
  the Pyodide runtime, where coverage is not collected).
- Add unit tests covering the remaining FetchConnector lines: platform
  default-connector selection, large-body flow control, transports
  writelines/abort/double-close, fetch abort on close, and responses
  without Headers.getSetCookie().
- Sanitize the echoed Content-Type in the test HTTP server (CodeQL
  response-splitting warning).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided There is a change note present in this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants