Skip to content

Commit 294030f

Browse files
lesnik512claude
andcommitted
docs(resilience): document CircuitBreaker + AsyncTimeout (0.10.0)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent c59b301 commit 294030f

3 files changed

Lines changed: 184 additions & 11 deletions

File tree

README.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77

88
**A Python HTTP client framework with sync and async clients for building resilient service clients.**
99

10-
`httpware` is a thin opinionated wrapper around `httpx2`. It re-exports `httpx2.Request`/`httpx2.Response`, adds a middleware chain composed at client construction, supports opt-in typed response decoding (pydantic and msgspec are both extras), and raises a status-keyed exception tree automatically on 4xx/5xx. It also ships a small resilience suite — `AsyncRetry`/`Retry` middleware with a Finagle-style `RetryBudget`, plus an `AsyncBulkhead`/`Bulkhead` concurrency limiter — under `httpware.middleware.resilience`.
10+
`httpware` is a thin opinionated wrapper around `httpx2`. It re-exports `httpx2.Request`/`httpx2.Response`, adds a middleware chain composed at client construction, supports opt-in typed response decoding (pydantic and msgspec are both extras), and raises a status-keyed exception tree automatically on 4xx/5xx. It also ships a resilience suite under `httpware.middleware.resilience` `AsyncRetry`/`Retry` with a Finagle-style `RetryBudget`, `AsyncBulkhead`/`Bulkhead` concurrency limiter, `AsyncCircuitBreaker`/`CircuitBreaker` consecutive-failure breaker, and `AsyncTimeout` for overall-operation wall-clock bounds.
1111

1212
> **Status:** Pre-1.0. Public API is subject to change between minor releases until v1.0.
1313
@@ -116,16 +116,18 @@ All 4xx/5xx responses raise typed exceptions automatically: `NotFoundError`, `Se
116116

117117
## Observability
118118

119-
`AsyncRetry`/`Retry` and `AsyncBulkhead`/`Bulkhead` emit operational events via two channels — stdlib `logging` records (always on) and OpenTelemetry span events (when `opentelemetry-api` is installed). Event names and payloads are identical across sync and async; dashboards built against one class apply unchanged to the other.
119+
All resilience middleware emit operational events via two channels — stdlib `logging` records (always on) and OpenTelemetry span events (when `opentelemetry-api` is installed). Event names and payloads are identical across sync and async; dashboards built against one class apply unchanged to the other.
120120

121-
Logger names (`httpware.retry`, `httpware.bulkhead`) and event names (`retry.giving_up`, `retry.budget_refused`, `retry.streaming_refused`, `bulkhead.rejected`) are the stable public contract.
121+
Logger names and event names are the stable public contract: `httpware.retry` (`retry.giving_up`, `retry.budget_refused`, `retry.streaming_refused`), `httpware.bulkhead` (`bulkhead.rejected`), `httpware.circuit_breaker` (`circuit.opened`, `circuit.rejected`, `circuit.half_open`, `circuit.closed`), and `httpware.timeout` (`timeout.exceeded`).
122122

123123
```python
124124
import logging
125125

126-
# Enable visibility into retry / bulkhead operational events
126+
# Enable visibility into resilience operational events
127127
logging.getLogger("httpware.retry").setLevel(logging.WARNING)
128128
logging.getLogger("httpware.bulkhead").setLevel(logging.WARNING)
129+
logging.getLogger("httpware.circuit_breaker").setLevel(logging.WARNING)
130+
logging.getLogger("httpware.timeout").setLevel(logging.WARNING)
129131
```
130132

131133
For OTel attribute enrichment on the active span — install the extra:

docs/resilience.md

Lines changed: 134 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -144,28 +144,155 @@ async with (
144144

145145
When `acquire_timeout` elapses without a slot opening, `AsyncBulkhead` raises `BulkheadFullError` (carries the configured `max_concurrent` and `acquire_timeout` for caller logging). See the [Errors reference](errors.md). The `httpware.bulkhead` `bulkhead.rejected` observability event fires at the same site — see [Observability](index.md#observability).
146146

147+
## `AsyncCircuitBreaker` / `CircuitBreaker`
148+
149+
```python
150+
from httpware.middleware.resilience import AsyncCircuitBreaker # async
151+
from httpware.middleware.resilience import CircuitBreaker # sync
152+
```
153+
154+
Classic consecutive-failure circuit breaker. Counts failures and prevents requests from reaching a downstream that is known to be broken.
155+
156+
### States
157+
158+
- **CLOSED** — normal operation. Each counted failure increments the consecutive-failure counter. Once `failure_threshold` consecutive counted failures accumulate, the circuit opens.
159+
- **OPEN** — fast-fail. All requests are rejected immediately with `CircuitOpenError` (carrying `retry_after` seconds until the next probe window). After `reset_timeout` seconds the circuit moves to HALF_OPEN.
160+
- **HALF_OPEN** — exactly one probe is admitted. If `success_threshold` consecutive probe successes are observed, the circuit closes. A single probe failure re-opens the circuit.
161+
162+
### Constructor
163+
164+
| Parameter | Default | Effect |
165+
|---|---|---|
166+
| `failure_threshold` | `5` | Consecutive counted failures required to open. `<1` raises `ValueError`. |
167+
| `reset_timeout` | `30.0` (s) | Seconds to stay OPEN before admitting a probe. `<0` raises `ValueError`. |
168+
| `success_threshold` | `1` | Consecutive probe successes required to close. `<1` raises `ValueError`. |
169+
| `failure_status_codes` | `None` | Which status codes count as failures. `None` → all 5xx (`500``599`). |
170+
171+
### Failure classification
172+
173+
A **counted failure** is a `NetworkError`, an httpware `TimeoutError`, or a `StatusError` whose status code is in `failure_status_codes`. All other exceptions propagate without affecting circuit state.
174+
175+
**4xx responses — including 429 — count as successes.** A 429 means the service is healthy but throttling; tripping the circuit on it would amplify an incident by adding circuit-open rejections on top of the throttle.
176+
177+
### `CircuitOpenError`
178+
179+
Raised when the circuit is OPEN (with a positive `retry_after: float`) or when HALF_OPEN with a probe already in flight (`retry_after=None`). Inherits `httpware.ClientError`. See the [Errors reference](errors.md).
180+
181+
### Observability
182+
183+
Emitted on logger `httpware.circuit_breaker`:
184+
185+
| Event | When |
186+
|---|---|
187+
| `circuit.opened` | Failure threshold reached; circuit transitions CLOSED → OPEN |
188+
| `circuit.rejected` | Request fast-failed (OPEN or HALF_OPEN probe slot taken) |
189+
| `circuit.half_open` | Reset timeout elapsed; circuit transitions OPEN → HALF_OPEN |
190+
| `circuit.closed` | Success threshold reached; circuit transitions HALF_OPEN → CLOSED |
191+
192+
### Sharing
193+
194+
Pass the same instance to multiple clients to enforce one shared circuit across them. A `CircuitBreaker` (sync) cannot be shared with an `AsyncCircuitBreaker` — they use different concurrency primitives.
195+
196+
### Async example
197+
198+
```python
199+
from httpware import AsyncClient
200+
from httpware.middleware.resilience import AsyncCircuitBreaker
201+
202+
203+
breaker = AsyncCircuitBreaker(failure_threshold=3, reset_timeout=60.0)
204+
205+
async with AsyncClient(
206+
base_url="https://api.example.com",
207+
middleware=[breaker],
208+
) as client:
209+
response = await client.get("/users/1")
210+
```
211+
212+
### Sync example
213+
214+
```python
215+
from httpware import Client
216+
from httpware.middleware.resilience import CircuitBreaker
217+
218+
219+
breaker = CircuitBreaker(failure_threshold=3, reset_timeout=60.0)
220+
221+
with Client(
222+
base_url="https://api.example.com",
223+
middleware=[breaker],
224+
) as client:
225+
client.get("/users/1")
226+
```
227+
228+
## `AsyncTimeout`
229+
230+
```python
231+
from httpware.middleware.resilience import AsyncTimeout
232+
```
233+
234+
Bounds total wall-clock time across the entire inner pipeline. Place it outermost to enforce "this whole operation must finish within `timeout` seconds, even across retries and backoff sleeps." On expiry it raises `httpware.TimeoutError`.
235+
236+
| Parameter | Default | Effect |
237+
|---|---|---|
238+
| `timeout` | **REQUIRED** | Overall deadline in seconds. Must be `> 0`; `≤0` raises `ValueError`. |
239+
240+
**This is not a per-call timeout.** httpx2's connect/read/write/pool timeouts are the right tool for bounding a single outbound call; `AsyncTimeout` doesn't duplicate them. What httpx2 cannot bound is the total wall-clock across a whole retry sequence — `AsyncTimeout` fills that gap.
241+
242+
**No sync `Timeout` exists.** Sync Python has no cancellation primitive that can interrupt a blocking httpx2 call mid-flight. For sync per-call bounds, configure `httpx2.Timeout` on the wrapped client or pass `timeout=` per request.
243+
244+
Observability event: `timeout.exceeded` on logger `httpware.timeout`.
245+
246+
```python
247+
from httpware import AsyncClient
248+
from httpware.middleware.resilience import AsyncCircuitBreaker, AsyncRetry, AsyncTimeout
249+
250+
251+
async with AsyncClient(
252+
base_url="https://api.example.com",
253+
middleware=[
254+
AsyncTimeout(timeout=10.0), # overall deadline across the whole chain
255+
AsyncRetry(max_attempts=3),
256+
],
257+
) as client:
258+
response = await client.get("/users/1")
259+
```
260+
147261
## Composition
148262

149-
The canonical ordering is `middleware=[AsyncBulkhead, AsyncRetry]``AsyncBulkhead` outermost so one slot covers all retry attempts of a single call:
263+
The recommended ordering (not enforced, but each position has a reason):
264+
265+
```
266+
AsyncTimeout → AsyncCircuitBreaker → AsyncRetry → AsyncBulkhead → terminal
267+
```
268+
269+
- `AsyncTimeout` outermost so the overall deadline covers the entire sequence including retries and backoff.
270+
- `AsyncCircuitBreaker` outside `AsyncRetry` so an open circuit short-circuits the whole retry loop without attempting any calls. This also means the breaker counts one outcome per fully-exhausted retry sequence rather than one per individual attempt.
271+
- `AsyncBulkhead` inside `AsyncRetry` so one slot covers all retry attempts of a single call. Flip it (`[AsyncRetry, AsyncBulkhead]`) and each retry grabs a fresh slot — defeating the bulkhead under load.
150272

151273
```python
152274
from httpware import AsyncClient
153-
from httpware.middleware.resilience import AsyncBulkhead, AsyncRetry
275+
from httpware.middleware.resilience import (
276+
AsyncBulkhead,
277+
AsyncCircuitBreaker,
278+
AsyncRetry,
279+
AsyncTimeout,
280+
)
154281

155282

156283
async def main() -> None:
157284
async with AsyncClient(
158285
base_url="https://api.example.com",
159286
middleware=[
160-
AsyncBulkhead(max_concurrent=10),
287+
AsyncTimeout(timeout=30.0),
288+
AsyncCircuitBreaker(),
161289
AsyncRetry(),
290+
AsyncBulkhead(max_concurrent=10),
162291
],
163292
) as client:
164293
await client.get("/users/1")
165294
```
166295

167-
Flipping the order (`[AsyncRetry, AsyncBulkhead]`) means each retry attempt grabs a fresh slot — defeating the bulkhead under load. Don't do that.
168-
169296
Cross-cutting middleware that emit per-call state (e.g., the Request-ID middleware in the [Middleware guide](middleware.md)) should sit outside `AsyncRetry` for the same reason — so all attempts of one call share one ID rather than getting a fresh ID per attempt.
170297

171298
## Sync Retry and Bulkhead
@@ -227,6 +354,6 @@ with Client(
227354
## See also
228355

229356
- **[Middleware guide](middleware.md)** — write your own resilience middleware against the same protocol `AsyncRetry` and `AsyncBulkhead` use.
230-
- **[Errors reference](errors.md)**`RetryBudgetExhaustedError`, `BulkheadFullError`, and the broader exception tree.
231-
- **[Observability](index.md#observability)** — the four operational events these middleware emit.
357+
- **[Errors reference](errors.md)**`RetryBudgetExhaustedError`, `BulkheadFullError`, `CircuitOpenError`, and the broader exception tree.
358+
- **[Observability](index.md#observability)** — the operational events these middleware emit.
232359
- **`planning/engineering.md` §3** — the formal Middleware/Seam-A contract.

planning/releases/0.10.0.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# httpware 0.10.0 — circuit breaker + async timeout
2+
3+
**Minor release. Additive only — no breaking changes.**
4+
5+
## New public names
6+
7+
```python
8+
from httpware.middleware.resilience import AsyncCircuitBreaker # async
9+
from httpware.middleware.resilience import CircuitBreaker # sync
10+
from httpware.middleware.resilience import AsyncTimeout
11+
from httpware import CircuitOpenError
12+
```
13+
14+
## `AsyncCircuitBreaker` / `CircuitBreaker`
15+
16+
Classic consecutive-failure circuit breaker. Counts counted failures (5xx, `NetworkError`, `TimeoutError`) and fast-fails with `CircuitOpenError` once `failure_threshold` consecutive failures are observed. Recovers via a HALF_OPEN probe after `reset_timeout` seconds; closes when `success_threshold` consecutive probe successes are seen.
17+
18+
4xx responses — including 429 — count as successes. A 429 means healthy-but-throttling; tripping the circuit on it would amplify incidents.
19+
20+
`CircuitOpenError` (a `ClientError` subclass) carries `retry_after: float | None` — the seconds until the next probe window (`None` when HALF_OPEN with a probe already in flight).
21+
22+
Sharable across multiple clients (one shared circuit). A sync `CircuitBreaker` cannot be shared with an `AsyncCircuitBreaker`.
23+
24+
## `AsyncTimeout`
25+
26+
Bounds total wall-clock across the inner pipeline — including retries and backoff sleeps. Raises `httpware.TimeoutError` on expiry. Async-only: sync Python has no cancellation primitive that can interrupt a blocking call mid-flight.
27+
28+
## New observability events
29+
30+
| Logger | Event | When |
31+
|---|---|---|
32+
| `httpware.circuit_breaker` | `circuit.opened` | Failure threshold reached |
33+
| `httpware.circuit_breaker` | `circuit.rejected` | Request fast-failed (OPEN or HALF_OPEN probe taken) |
34+
| `httpware.circuit_breaker` | `circuit.half_open` | Reset timeout elapsed; probe admitted |
35+
| `httpware.circuit_breaker` | `circuit.closed` | Success threshold reached; service recovered |
36+
| `httpware.timeout` | `timeout.exceeded` | Overall timeout expired |
37+
38+
## Recommended chain ordering
39+
40+
```
41+
AsyncTimeout → AsyncCircuitBreaker → AsyncRetry → AsyncBulkhead → terminal
42+
```
43+
44+
Breaker outside retry: an open circuit short-circuits the whole retry loop; the breaker counts one outcome per fully-exhausted retry sequence.

0 commit comments

Comments
 (0)