Skip to content

Commit bcce8a7

Browse files
authored
refactor(internal): extract body-cap module out of client.py
Extracts the max_response_body_bytes body-capping subsystem and the two httpx2-exception-mapper context managers out of client.py into _internal/body_cap.py (new) and _internal/exception_mapping.py — pure move, no behavior change. Fixes a doc-location claim in architecture/errors.md/client.md that this move made stale. Design: planning/changes/2026-07-13.01-body-cap-module-extraction.md
1 parent 5c05f1d commit bcce8a7

10 files changed

Lines changed: 359 additions & 184 deletions

File tree

architecture/client.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
## The internal terminal
66

7-
The bottom of the middleware chain (the "terminal") is internal. It calls `self._httpx2_client.send(request)`, maps `httpx2` errors to `httpware` errors, and raises a `StatusError` subclass on 4xx/5xx. The error-mapping table (what `httpx2` exception maps to which `httpware` exception) lives at the terminal in `src/httpware/client.py`; status-keyed exceptions are looked up via the `STATUS_TO_EXCEPTION` table in `src/httpware/errors.py`. The same terminal lifecycle holds in both worlds: `Client.send` / `AsyncClient.send` enter the middleware chain first, and it is the internal terminal — `Client._terminal` / `AsyncClient._terminal` — that calls `httpx2.Client.send` / `httpx2.AsyncClient.send`.
7+
The bottom of the middleware chain (the "terminal") is internal. It calls `self._httpx2_client.send(request)`, maps `httpx2` errors to `httpware` errors, and raises a `StatusError` subclass on 4xx/5xx. The error-mapping table (what `httpx2` exception maps to which `httpware` exception) lives in `src/httpware/_internal/exception_mapping.py` and fires at the terminal in `src/httpware/client.py`; status-keyed exceptions are looked up via the `STATUS_TO_EXCEPTION` table in `src/httpware/errors.py`. The same terminal lifecycle holds in both worlds: `Client.send` / `AsyncClient.send` enter the middleware chain first, and it is the internal terminal — `Client._terminal` / `AsyncClient._terminal` — that calls `httpx2.Client.send` / `httpx2.AsyncClient.send`.
88

99
## Sync/async parity
1010

architecture/errors.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ exc.response.request.url # URL of the failed request
1010

1111
`__repr__` and the `str()` summary redact URL userinfo (`user:pass@`) and mask the values of known-sensitive query and fragment parameters (e.g. `token`, `api_key`, `secret`) to avoid leaking credentials in tracebacks.
1212

13-
The error-mapping table (what `httpx2` exception maps to which `httpware` exception) lives at the terminal in `src/httpware/client.py`. Status-keyed exceptions are looked up via the `STATUS_TO_EXCEPTION` table in `src/httpware/errors.py`. Unknown 4xx falls back to `ClientStatusError`; unknown 5xx falls back to `ServerStatusError`.
13+
The error-mapping table (what `httpx2` exception maps to which `httpware` exception) lives in `src/httpware/_internal/exception_mapping.py` and fires at the terminal in `src/httpware/client.py`. Status-keyed exceptions are looked up via the `STATUS_TO_EXCEPTION` table in `src/httpware/errors.py`. Unknown 4xx falls back to `ClientStatusError`; unknown 5xx falls back to `ServerStatusError`.
1414

1515
`TimeoutError` inherits from both `httpware.ClientError` and `builtins.TimeoutError` so `except builtins.TimeoutError` (the form `asyncio.wait_for` uses) also catches httpware-raised timeouts.
1616

architecture/overview.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,9 @@ src/httpware/
3535
│ └── _backoff.py # full-jitter helper (shared)
3636
├── decoders/ # shared (ResponseDecoder + adapters)
3737
└── _internal/
38-
├── exception_mapping.py # map_httpx2_exception (shared)
39-
├── import_checker.py # is_*_installed flags
40-
├── observability.py # _emit_event
41-
└── status.py # _raise_on_status_error, _is_streaming_body_*, STREAMING_BODY_MARKER
38+
├── body_cap.py # max_response_body_bytes: validate, read-capped (sync+async)
39+
├── exception_mapping.py # map_httpx2_exception + context-manager wrappers (shared)
40+
├── import_checker.py # is_*_installed flags
41+
├── observability.py # _emit_event
42+
└── status.py # _raise_on_status_error, _is_streaming_body_*, STREAMING_BODY_MARKER
4243
```
Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
---
2+
summary: Extract the body-cap subsystem and the httpx2-exception-mapper context managers out of client.py into dedicated _internal/ modules.
3+
---
4+
5+
# Design: Extract the body-cap subsystem into `_internal/body_cap.py`
6+
7+
## Summary
8+
9+
`client.py:38-212` holds ~180 lines of pure, self-contained code with zero
10+
`self`/`Client`/`AsyncClient` coupling: the `max_response_body_bytes`
11+
body-capping subsystem (`_validate_max_response_body_bytes`,
12+
`_parse_content_length`, `_CapExceeded`, `_accumulate_capped`,
13+
`_safe_extensions`, `_buffered_headers`, `_response_has_body`,
14+
`_read_capped`, `_read_capped_async`) and two httpx2-exception-mapper context
15+
managers (`_httpx2_exception_mapper`, `_httpx2_exception_mapper_sync`). This
16+
change moves the body-cap subsystem into a new `_internal/body_cap.py`, and
17+
folds the two context managers into the existing `_internal/exception_mapping.py`
18+
next to the `map_httpx2_exception` function they wrap. No behavior change.
19+
20+
## Motivation
21+
22+
- `client.py` is 2,154 lines, by far the largest module in the package.
23+
Everything genuinely coupled to `Client`/`AsyncClient` (`_terminal`,
24+
`_prepare_request`, `stream()`) has to stay there; ~200 lines of this file
25+
currently don't.
26+
- The package already has a home for exactly this shape of code:
27+
`_internal/status.py` holds `_raise_on_status_error` and the
28+
streaming-body predicates, `_internal/exception_mapping.py` holds
29+
`map_httpx2_exception` — both pure, self-contained, shared by both clients.
30+
The body-cap subsystem and the exception-mapper context managers are the
31+
same shape and never received the same treatment.
32+
- Same-day precedent: `2026-06-23.01-retry-policy-extraction.md` and
33+
`2026-06-23.02-decoder-resolver-extraction.md` pulled comparable
34+
self-contained logic out of their hosting files into dedicated modules.
35+
`2026-06-23.03-response-body-cap.md` landed the body-cap subsystem directly
36+
in `client.py` the same day and never got the follow-up move.
37+
- **Depth:** the body-cap subsystem's own tests (`tests/test_capped_read.py`,
38+
`tests/test_capped_read_props.py`) already exercise it as a standalone
39+
unit, importing the functions directly rather than through `Client`. The
40+
seam already exists in the tests; the source hasn't caught up.
41+
- **Deletion test:** delete `_internal/body_cap.py` after this change and the
42+
~150 lines of cap-accumulation/header-rebuild logic reappear somewhere —
43+
it's real, load-bearing complexity, not a pass-through. It earns the move.
44+
45+
## Design
46+
47+
### 1. `_internal/body_cap.py` — new module
48+
49+
Moves verbatim (no logic change):
50+
51+
- `_MAX_RESPONSE_BODY_BYTES_INVALID`, `_validate_max_response_body_bytes`
52+
- `_parse_content_length`
53+
- `_CapExceeded`, `_accumulate_capped`
54+
- `_safe_extensions`
55+
- `_WIRE_BODY_HEADERS`, `_BODILESS_STATUS`, `_buffered_headers`
56+
- `_response_has_body`
57+
- `_read_capped`, `_read_capped_async`
58+
59+
Needs its own imports: `httpx2`, `HTTPStatus` (from `http`), `Mapping` (from
60+
`collections.abc`), and `ResponseTooLargeError` (from `httpware.errors`) —
61+
all currently imported by `client.py` for this code's sake and dropped from
62+
`client.py`'s import block once it moves (`Mapping` and
63+
`ResponseTooLargeError` are used nowhere else in `client.py`; `HTTPStatus`
64+
stays imported in `client.py` because `stream()` uses it independently at
65+
`client.py:1152` and `:2146`).
66+
67+
`_build_default_decoders` (`client.py:172-190`) stays in `client.py`.
68+
`architecture/decoders.md` documents its location explicitly ("`decoders=None`
69+
resolves via `client.py:_build_default_decoders()`"); it is the same shape of
70+
pure, self-contained code but moving it isn't part of this change's motivation
71+
and would mean editing a Seam B contract that this change has no reason to
72+
touch.
73+
74+
### 2. `_internal/exception_mapping.py` — gains two context managers
75+
76+
`_httpx2_exception_mapper` (async) and `_httpx2_exception_mapper_sync` (sync)
77+
move in next to `map_httpx2_exception`, which they already wrap. Same
78+
rationale as body-cap — pure, self-contained, sibling concern already has a
79+
module — bundled into this change rather than a second near-identical PR.
80+
81+
### 3. `client.py` shrinks to three imported names
82+
83+
Only `_validate_max_response_body_bytes` (called once per `__init__`),
84+
`_read_capped`, and `_read_capped_async` (called from `_terminal` and
85+
`stream()`, both worlds) are actually referenced from inside `Client`/
86+
`AsyncClient`. Every other moved name (`_parse_content_length`,
87+
`_CapExceeded`, `_accumulate_capped`, `_safe_extensions`, `_buffered_headers`,
88+
`_response_has_body`) is internal to `body_cap.py` and never imported by
89+
`client.py`. `_httpx2_exception_mapper`/`_httpx2_exception_mapper_sync` are
90+
imported by name, same call sites as today (`client.py:283`, `:1151`,
91+
`:1255`, `:2145`).
92+
93+
```python
94+
from httpware._internal.body_cap import _read_capped, _read_capped_async, _validate_max_response_body_bytes
95+
from httpware._internal.exception_mapping import _httpx2_exception_mapper, _httpx2_exception_mapper_sync, map_httpx2_exception
96+
```
97+
98+
Nine functions/one class collapse behind a three-name import in `client.py`
99+
for body-cap, plus the two context managers for exception mapping.
100+
101+
### 4. `architecture/overview.md` — module-layout table
102+
103+
Add a `body_cap.py` row under `_internal/`; update `exception_mapping.py`'s
104+
description to mention the context managers:
105+
106+
```text
107+
└── _internal/
108+
├── body_cap.py # max_response_body_bytes: validate, read-capped (sync+async)
109+
├── exception_mapping.py # map_httpx2_exception + context-manager wrappers (shared)
110+
├── import_checker.py # is_*_installed flags
111+
├── observability.py # _emit_event
112+
└── status.py # _raise_on_status_error, _is_streaming_body_*, STREAMING_BODY_MARKER
113+
```
114+
115+
## Non-goals
116+
117+
- No behavior change. Cap validation, accumulation, header rebuild,
118+
`ResponseTooLargeError` reasons, and exception-mapping order stay
119+
byte-identical.
120+
- Not moving `_build_default_decoders` (see Design §1).
121+
- Not touching the public `max_response_body_bytes` parameter, its
122+
validation error message, or `ResponseTooLargeError`'s shape.
123+
- Not addressing the larger `Client`/`AsyncClient` sync/async duplication
124+
(the per-verb methods, `_prepare_request`, `_request_with_body`) — a
125+
separate, much larger candidate surfaced in the same architecture review,
126+
deliberately deferred pending its own design call.
127+
128+
## Testing
129+
130+
- **Parity net:** rename `tests/test_capped_read.py`
131+
`tests/test_body_cap.py` and `tests/test_capped_read_props.py`
132+
`tests/test_body_cap_props.py`, updating their imports from
133+
`httpware.client` to `httpware._internal.body_cap`. All existing
134+
assertions stay unchanged — byte-identical behavior is the bar.
135+
- Move `test_parse_content_length` (currently `tests/test_client_stream.py:446-447`,
136+
testing a `body_cap.py` function directly rather than `stream()` behavior)
137+
into `tests/test_body_cap.py`, updating its import.
138+
- No new tests — this is a pure move with an existing, adequate net
139+
(`test_body_cap.py`, `test_body_cap_props.py`, `test_client_body_cap.py`
140+
for cap wiring through the client, `test_error_mapping_terminal.py` for
141+
exception-mapping-at-terminal behavior).
142+
- `just lint && just test` green; 100% coverage maintained.
143+
144+
## Risk
145+
146+
- **Import cycle** (unlikely × medium): `_internal/body_cap.py` imports
147+
`httpware.errors.ResponseTooLargeError`. *Mitigation:* `_internal/status.py`
148+
already imports from `httpware.errors` today with no cycle — same
149+
direction, same precedent.
150+
- **Missed call site during the move** (unlikely × low): a stale import left
151+
in `client.py`, or a name imported but now unused. *Mitigation:* `just
152+
lint` catches unused imports; `just test` exercises every moved function
153+
through its existing suite.
154+
- **Test-file rename loses git history readability** (certain × low):
155+
`git mv` preserves blame across the rename; low impact either way.

src/httpware/_internal/body_cap.py

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
"""Response-body cap enforcement: validate, read-capped (sync + async)."""
2+
3+
import typing
4+
from collections.abc import Mapping
5+
from http import HTTPStatus
6+
7+
import httpx2
8+
9+
from httpware.errors import ResponseTooLargeError
10+
11+
12+
_MAX_RESPONSE_BODY_BYTES_INVALID = "max_response_body_bytes must be >= 1"
13+
14+
15+
def _validate_max_response_body_bytes(cap: int | None) -> None:
16+
"""Reject a non-None cap below 1. None means unbounded (the default)."""
17+
if cap is not None and cap < 1:
18+
raise ValueError(_MAX_RESPONSE_BODY_BYTES_INVALID)
19+
20+
21+
def _parse_content_length(raw: str | None) -> int | None:
22+
"""Return a non-negative int Content-Length, or None for missing/garbage. Never raises."""
23+
if raw is None:
24+
return None
25+
try:
26+
value = int(raw)
27+
except ValueError:
28+
return None
29+
return value if value >= 0 else None
30+
31+
32+
class _CapExceeded(Exception): # noqa: N818 — internal control-flow signal, not a user-facing error
33+
"""Internal signal: decoded bytes crossed the cap mid-read. Carries bytes read so far."""
34+
35+
def __init__(self, *, read: int) -> None:
36+
self.read = read
37+
super().__init__(f"decoded body exceeded cap after {read} bytes")
38+
39+
40+
def _accumulate_capped(chunks: typing.Iterable[bytes], cap: int) -> bytes:
41+
"""Concatenate `chunks`, raising `_CapExceeded` the moment the running total exceeds `cap`.
42+
43+
Counts decoded bytes (the in-memory footprint). Grown in a single bytearray
44+
so there is no transient list-plus-join double allocation.
45+
"""
46+
buf = bytearray()
47+
for chunk in chunks:
48+
buf += chunk
49+
if len(buf) > cap:
50+
raise _CapExceeded(read=len(buf))
51+
return bytes(buf)
52+
53+
54+
def _safe_extensions(extensions: Mapping[str, typing.Any]) -> dict[str, typing.Any]:
55+
"""Copy response extensions, dropping the now-stale `network_stream`.
56+
57+
The rebuilt buffered Response never touches its network stream, so carrying a
58+
consumed/closed one wholesale is sloppy. `http_version`/`reason_phrase` and
59+
any other keys are preserved.
60+
"""
61+
return {key: value for key, value in extensions.items() if key != "network_stream"}
62+
63+
64+
# Headers describing the wire encoding of the body. The accumulator yields the
65+
# DECODED body, so these no longer apply; httpx2 recomputes content-length from
66+
# the buffered content. Carrying content-encoding forward makes httpx2 try to
67+
# re-decode already-decoded bytes and raise.
68+
_WIRE_BODY_HEADERS = ("content-encoding", "content-length", "transfer-encoding")
69+
_BODILESS_STATUS = frozenset({HTTPStatus.NO_CONTENT, HTTPStatus.NOT_MODIFIED}) # 204, 304
70+
71+
72+
def _buffered_headers(headers: httpx2.Headers) -> httpx2.Headers:
73+
"""Copy `headers`, stripping wire-encoding headers stale after decoding+buffering."""
74+
out = httpx2.Headers(headers)
75+
for name in _WIRE_BODY_HEADERS:
76+
if name in out:
77+
del out[name]
78+
return out
79+
80+
81+
def _response_has_body(method: str, status_code: int) -> bool:
82+
"""Whether a response carries a message body (RFC 9110 §6.4.1).
83+
84+
HEAD responses and 204/304 never have a body regardless of a declared
85+
Content-Length, so they must never trip the cap.
86+
"""
87+
return method.upper() != "HEAD" and status_code not in _BODILESS_STATUS
88+
89+
90+
def _read_capped(response: httpx2.Response, cap: int, request: httpx2.Request) -> httpx2.Response:
91+
"""Buffer a streaming sync `response` under `cap` decoded bytes; return a buffered Response.
92+
93+
Raises `ResponseTooLargeError` (reason="declared") if the declared
94+
Content-Length already exceeds `cap` — before any byte is read — and
95+
(reason="streamed") if the decoded body crosses `cap` mid-read. Does not
96+
close `response`; the caller owns the stream lifecycle.
97+
"""
98+
if not _response_has_body(request.method, response.status_code):
99+
response.read() # empty body; preserve the original response (and its headers)
100+
return response
101+
content_length = _parse_content_length(response.headers.get("content-length"))
102+
if content_length is not None and content_length > cap:
103+
raise ResponseTooLargeError(
104+
status_code=response.status_code, limit=cap, content_length=content_length, reason="declared"
105+
)
106+
try:
107+
content = _accumulate_capped(response.iter_bytes(), cap)
108+
except _CapExceeded:
109+
raise ResponseTooLargeError(
110+
status_code=response.status_code, limit=cap, content_length=content_length, reason="streamed"
111+
) from None
112+
return httpx2.Response(
113+
status_code=response.status_code,
114+
headers=_buffered_headers(response.headers),
115+
content=content,
116+
request=request,
117+
extensions=_safe_extensions(response.extensions),
118+
history=response.history,
119+
)
120+
121+
122+
async def _read_capped_async(response: httpx2.Response, cap: int, request: httpx2.Request) -> httpx2.Response:
123+
"""Async mirror of `_read_capped` (counts decoded bytes from `aiter_bytes`)."""
124+
if not _response_has_body(request.method, response.status_code):
125+
await response.aread() # empty body; preserve the original response (and its headers)
126+
return response
127+
content_length = _parse_content_length(response.headers.get("content-length"))
128+
if content_length is not None and content_length > cap:
129+
raise ResponseTooLargeError(
130+
status_code=response.status_code, limit=cap, content_length=content_length, reason="declared"
131+
)
132+
buf = bytearray()
133+
async for chunk in response.aiter_bytes():
134+
buf += chunk
135+
if len(buf) > cap:
136+
raise ResponseTooLargeError(
137+
status_code=response.status_code, limit=cap, content_length=content_length, reason="streamed"
138+
)
139+
return httpx2.Response(
140+
status_code=response.status_code,
141+
headers=_buffered_headers(response.headers),
142+
content=bytes(buf),
143+
request=request,
144+
extensions=_safe_extensions(response.extensions),
145+
history=response.history,
146+
)

src/httpware/_internal/exception_mapping.py

Lines changed: 31 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,15 @@
1-
"""httpx2 -> httpware exception mapping.
1+
"""httpx2 -> httpware exception mapping + context-manager wrappers (shared).
22
3-
Pure function used by both Client._terminal and AsyncClient._terminal,
4-
and by both stream() methods. Clause ordering: TimeoutException ->
5-
InvalidURL/CookieConflict -> NetworkError -> HTTPError (subclass before
6-
parent so the right type wins).
3+
map_httpx2_exception is a pure function used by both Client._terminal and
4+
AsyncClient._terminal, and by both stream() methods. Clause ordering:
5+
TimeoutException -> InvalidURL/CookieConflict -> NetworkError -> HTTPError
6+
(subclass before parent so the right type wins). The two context managers
7+
below wrap it for use as `with`/`async with` blocks around the httpx2 call.
78
"""
89

10+
import contextlib
11+
from collections.abc import AsyncIterator, Iterator
12+
913
import httpx2
1014

1115
from httpware.errors import NetworkError, TimeoutError, TransportError # noqa: A004
@@ -26,3 +30,25 @@ def map_httpx2_exception(exc: BaseException) -> NetworkError | TimeoutError | Tr
2630
if isinstance(exc, httpx2.HTTPError):
2731
return TransportError(str(exc))
2832
return TransportError(str(exc)) # pragma: no cover — defensive default; httpx2.HTTPError is the root
33+
34+
35+
@contextlib.asynccontextmanager
36+
async def _httpx2_exception_mapper() -> AsyncIterator[None]:
37+
"""Map httpx2 exceptions to httpware exceptions. Shared by AsyncClient._terminal and stream()."""
38+
try:
39+
yield
40+
except httpx2.HTTPError as exc:
41+
raise map_httpx2_exception(exc) from exc
42+
except (httpx2.InvalidURL, httpx2.CookieConflict) as exc:
43+
raise map_httpx2_exception(exc) from exc
44+
45+
46+
@contextlib.contextmanager
47+
def _httpx2_exception_mapper_sync() -> Iterator[None]:
48+
"""Map httpx2 exceptions to httpware exceptions. Sync sibling of _httpx2_exception_mapper."""
49+
try:
50+
yield
51+
except httpx2.HTTPError as exc:
52+
raise map_httpx2_exception(exc) from exc
53+
except (httpx2.InvalidURL, httpx2.CookieConflict) as exc:
54+
raise map_httpx2_exception(exc) from exc

0 commit comments

Comments
 (0)