Skip to content

Commit abb7ff7

Browse files
lesnik512claude
andauthored
test(circuit-breaker): close 2026-06-16 delta-audit Low findings (#71)
* test(circuit-breaker): close 2026-06-16 delta-audit Low findings Delta audit of the 0.12-0.14 cluster came back clean (0 Blocker/High/Medium). Close the two Low findings: add a rate-mode HALF_OPEN probe-failure re-open test (async + sync) and document that such a re-open emits the classic circuit.opened attribute shape in both modes (intentional — a probe failure is a distinct event from a rate trip). Adds the audit report + change bundle. No source behavior change; no release required. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(planning): archive the delta-audit-followups bundle (#71) Ship bookkeeping folded into PR #71: mark shipped (pr: 71), move the bundle active -> archive, flip its Index line to Archived. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 7938984 commit abb7ff7

6 files changed

Lines changed: 188 additions & 1 deletion

File tree

architecture/resilience.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616

1717
Both `AsyncCircuitBreaker` and `CircuitBreaker` expose a read-only `state` property that returns a public `CircuitState` enum (`CLOSED`/`OPEN`/`HALF_OPEN`), importable from `httpware`, for health checks and introspection. The property is a raw read of the stored state: because the OPEN→HALF_OPEN transition is lazy (it fires on the next request after `reset_timeout` elapses, not on a clock tick), `state` continues to report `OPEN` until a request is actually admitted as the probe — reading the property never triggers the transition.
1818

19-
The classic consecutive-failure mode is the default and unchanged. An opt-in time-based failure-rate mode is available: set `failure_rate_threshold` (a float in `(0, 1]`) to switch. In rate mode the circuit opens when the observed failure rate over a rolling `window_seconds` window (default `30.0` s) meets or exceeds the threshold, but only once `minimum_calls` outcomes have been observed in that window (default `20`). The presence of `failure_rate_threshold` is the sole mode switch: when it is set, the breaker is in rate mode and `failure_threshold` is ignored (setting both is not an error — rate mode wins). `window_seconds` and `minimum_calls` are validated at construction in both modes even though they are inert in classic mode, so an invalid value is rejected eagerly regardless of mode. Half-open recovery (`reset_timeout`, `success_threshold`, the single-probe admission) is identical to classic mode. The event names (`circuit.opened`, `circuit.rejected`, `circuit.half_open`, `circuit.closed`) are the same in both modes; in rate mode the `circuit.opened` event carries extra attributes — `failure_rate`, `failure_rate_threshold`, `window_seconds`, `observed_calls` — and its message is `"circuit opened — failure rate threshold reached"`.
19+
The classic consecutive-failure mode is the default and unchanged. An opt-in time-based failure-rate mode is available: set `failure_rate_threshold` (a float in `(0, 1]`) to switch. In rate mode the circuit opens when the observed failure rate over a rolling `window_seconds` window (default `30.0` s) meets or exceeds the threshold, but only once `minimum_calls` outcomes have been observed in that window (default `20`). The presence of `failure_rate_threshold` is the sole mode switch: when it is set, the breaker is in rate mode and `failure_threshold` is ignored (setting both is not an error — rate mode wins). `window_seconds` and `minimum_calls` are validated at construction in both modes even though they are inert in classic mode, so an invalid value is rejected eagerly regardless of mode. Half-open recovery (`reset_timeout`, `success_threshold`, the single-probe admission) is identical to classic mode. The event names (`circuit.opened`, `circuit.rejected`, `circuit.half_open`, `circuit.closed`) are the same in both modes; in rate mode the `circuit.opened` event carries extra attributes — `failure_rate`, `failure_rate_threshold`, `window_seconds`, `observed_calls` — and its message is `"circuit opened — failure rate threshold reached"`. The rate-flavored `circuit.opened` is emitted only on the *initial* trip from CLOSED: a HALF_OPEN **probe failure** re-opens the circuit through the shared half-open path (a probe failure is a distinct event from a rate/threshold trip), so that re-open's `circuit.opened` carries the classic `failure_threshold` / `failures` attributes (with `failures = 1`) and the `"circuit opened — failure threshold reached"` message in *both* modes. This is intentional; the event name is the stable contract, and a consumer keys off the present attribute set rather than assuming a fixed shape.
2020

2121
`AsyncTimeout` is an async-only middleware that bounds the total wall-clock for the whole inner pipeline (most importantly across an `AsyncRetry` loop, whose attempts and backoff sleeps `httpx2` cannot bound). It is not a per-call timeout — `httpx2`'s connect/read/write/pool timeouts are the right tool for a single outbound call, and `AsyncTimeout` does not duplicate them. It rejects a non-finite or non-positive `timeout` at construction, and on expiry raises httpware `TimeoutError`. There is no sync `Timeout`: a sync total-deadline cannot interrupt a blocking call mid-flight, and `httpx2` already covers sync per-call timeouts. Sync callers configure `httpx2`'s timeouts directly.
2222

planning/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,8 @@ _None._
7474

7575
### Archived (shipped)
7676

77+
- **[delta-audit-followups](changes/archive/2026-06-16.04-delta-audit-followups/change.md)** (#71, 2026-06-16) — Closed the [2026-06-16 delta audit](audits/2026-06-16-delta-audit.md) Low findings: rate-mode HALF_OPEN probe-failure re-open test + document-as-intended note. Tests + doc only, no release.
78+
7779
- **[circuit-breaker-state](changes/archive/2026-06-16.03-circuit-breaker-state/design.md)** (#70, 2026-06-16) — Read-only `state` property + public `CircuitState` enum on the circuit breaker. Shipped 0.14.0; closed the read-only-state half of the deferred CircuitBreaker introspection item.
7880

7981
- **[circuit-breaker-rate-mode](changes/archive/2026-06-16.02-circuit-breaker-rate-mode/design.md)** (#69, 2026-06-16) — Added an opt-in time-based failure-rate trip mode to the circuit breaker (classic stays default). Shipped 0.13.0; closed deferred item "CircuitBreaker v2 (a)".
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# httpware delta audit — 2026-06-16
2+
3+
**Status:** complete
4+
**Scope:** the four changes shipped since the 2026-06-14 deep audit (baseline `9297ced`): the custom-decoder guide (#67), per-verb `*_with_response` siblings + the `_prepare_request` refactor (#68, 0.12.0), the time-based rate-mode circuit breaker (#69, 0.13.0), and read-only circuit-breaker `state` + the public `CircuitState` enum (#70, 0.14.0).
5+
**Method:** five adversarial finders fanned out across dimensions — correctness/edge-cases, concurrency, public-API/consistency, docs-accuracy, test-quality — each instructed to refute its own candidates and report only survivors. Single synthesis pass (this report); the controller re-verified the one substantive finding against source.
6+
7+
## Summary
8+
9+
A clean delta. Four of five dimensions produced **zero** confirmed findings: every correctness, concurrency, public-API, and docs-accuracy candidate was refuted against the actual code. The only residue is in test coverage.
10+
11+
- Blockers: 0
12+
- High: 0
13+
- Medium: 0
14+
- Low: 2
15+
- Nits: 1
16+
17+
**Headline:** there is no headline defect. The accumulated 0.12–0.14 surface (per-verb siblings, rate-mode breaker, `state` introspection) is correct, concurrency-safe under the documented model, API-consistent and sync/async-symmetric, and its docs match the code — including the rate-mode defaults, the `(0,1]` range, the `failure_threshold`-ignored / both-set-precedence notes, and the raw-read `state` caveat. The two Low findings are a missing rate-mode test path and a minor observability-attribute inconsistency on one re-open path.
18+
19+
**Not covered:** static analysis only (source/test/doc reading + reproducer reasoning + targeted `ty`/import checks). No fuzzing, live-network, or dependency-CVE scan. The full suite (704 tests, 100% line coverage) and `ty`/`ruff` were green throughout — line coverage is total, so the test-quality finder targeted *semantic* gaps.
20+
21+
## Findings
22+
23+
### Low
24+
25+
#### 1. Rate-mode HALF_OPEN probe-failure re-open emits classic `circuit.opened` attributes
26+
*(consistency / observability — verified against source)*
27+
28+
`src/httpware/middleware/resilience/circuit_breaker.py``on_failure`, HALF_OPEN branch (`self._open(request, failures=1)`)
29+
30+
When a circuit is in HALF_OPEN, a single probe failure re-opens it via `_open(request, failures=1)` — the *classic* opener, which emits `circuit.opened` with `{failure_threshold, failures}` and the message `"circuit opened — failure threshold reached"`. This path is shared across modes, so a **rate-mode** breaker that re-opens from a failed probe emits classic-shaped attributes rather than the rate shape (`failure_rate`/`failure_rate_threshold`/`window_seconds`/`observed_calls`) it emits when it first trips from CLOSED. An observability consumer keying off `circuit.opened` attributes sees an inconsistent attribute set from the same breaker depending on which transition opened it, and `failure_threshold` — documented as *ignored* in rate mode — appears in the event.
31+
32+
Capped at Low: reachable only under the non-default rate knob, affects event *attributes* only (no control-flow effect — the re-open itself is correct), and half-open recovery is documented as mode-independent, so this is arguably acceptable. **Suggested direction:** either (a) document that a probe-failure re-open always carries the probe-failure attribute shape regardless of mode, or (b) emit a transition-specific attribute set for the half-open re-open (e.g. a `reason: "probe_failed"` attribute) so the event is self-describing in both modes. Not a functional bug.
33+
34+
Note: the window is *not* cleared on this re-open (only on HALF_OPEN→CLOSED). That is harmless — while OPEN/HALF_OPEN no outcomes are recorded, and the window is cleared when the circuit eventually closes (confirmed by the correctness finder).
35+
36+
#### 2. No rate-mode probe-failure re-open test
37+
*(test coverage — verified)*
38+
39+
`tests/test_circuit_breaker.py` / `tests/test_circuit_breaker_sync.py` (rate-mode section)
40+
41+
Every rate-mode recovery test drives HALF_OPEN→CLOSED through a *successful* probe. No rate-mode test exercises a probe *failure* re-opening the circuit. The classic equivalent (`test_probe_failure_reopens_circuit`) exists but runs in consecutive-failure mode, so it never touches the rate-mode opener selection or the window. A regression in the rate-mode re-open path (the very behavior in finding 1) would pass the suite. Line coverage is 100% because the classic tests already cover the shared `_open` line — this is a *semantic* gap. **Suggested direction:** add a rate-mode test (async + sync) that opens via rate, advances past `reset_timeout`, fails the probe, and asserts the circuit re-opens; assert whatever attribute behavior finding 1 settles on.
42+
43+
### Nits
44+
45+
#### 3. `_RollingWindow` Hypothesis oracle shares the slot formula with the implementation
46+
*(test quality)*
47+
48+
`tests/test_rolling_window.py``test_totals_match_live_events`
49+
50+
The property's `expected_live` oracle recomputes `int(t // bucket_width)` and the live-cutoff with the same arithmetic as `_RollingWindow`, so a systematic off-by-one in the window-boundary formula would be replicated in the oracle and the property would still pass. Largely neutralized: the example tests (`test_counts_within_window`, `test_stale_buckets_evicted_by_time`) pin the boundary with hand-computed literals the implementation can't co-vary with, and the property still exercises bucket-collision/eviction interleavings. Nit only.
51+
52+
## Refuted (recorded so the reasoning is auditable)
53+
54+
- **Correctness:** `_prepare_request` is byte-equivalent to the old inline logic (kwargs order, marker, sync/async streaming-detector split all preserved); all 12 verb delegations use the correct verb string and kwarg subset; rate trip math has no division-by-zero (guarded by `minimum_calls >= 1`); `_RollingWindow` slot aliasing is sound (live window spans exactly `_BUCKET_COUNT` slots → no live-bucket collision mod 10); `state` is a pure read.
55+
- **Concurrency:** no `await` anywhere in the async transition chain (`_record_outcome``_open_rate``_enter_open``_emit``_emit_event` all sync); `_now()` read once inside the critical section; sync rate path fully under `self._lock`; lock-free `state` read is a single atomic enum-reference load (no tear).
56+
- **Public API:** `*_with_response` correctly overload-free (single return shape), `response_model` required, exact sync/async parity; new breaker params identical on both wrappers; `CircuitState` is one object exported from both `__all__`s, alphabetical; `ty` clean.
57+
- **Docs accuracy:** every load-bearing claim across `docs/decoders.md`, `docs/resilience.md`, the pagination recipe, and the three `architecture/` files verified against source — defaults, ranges, precedence, the raw-read caveat, import paths, and the CSV example all match and run.
58+
59+
## Remediation
60+
61+
Findings 1 + 2 are naturally one small change: add the rate-mode probe-failure re-open test (sync + async) and, in the same pass, settle finding 1 (document-as-intended or add a self-describing re-open attribute). Lightweight lane. Finding 3 is a Nit — optional.
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
---
2+
status: shipped
3+
date: 2026-06-16
4+
slug: delta-audit-followups
5+
supersedes: null
6+
superseded_by: null
7+
pr: 71
8+
outcome: Closed the 2026-06-16 delta-audit Low findings — rate-mode probe-failure re-open test (async+sync) + document-as-intended note. No source behavior change.
9+
---
10+
11+
# Change: Close the 2026-06-16 delta-audit Low findings
12+
13+
**Lane:** lightweight — tests + a one-paragraph doc clarification, no source
14+
behavior change, no public-API change.
15+
16+
Closes the actionable findings from the
17+
[2026-06-16 delta audit](../../../audits/2026-06-16-delta-audit.md), which gave
18+
the 0.12–0.14 change cluster a clean bill (0 Blocker/High/Medium) with two Low
19+
items and a Nit.
20+
21+
## Goal
22+
23+
Lock and document the one behavioral subtlety the audit surfaced: a rate-mode
24+
circuit that re-opens from a **failed HALF_OPEN probe** emits `circuit.opened`
25+
with the *classic* `failure_threshold`/`failures` attributes (not the rate
26+
shape), because the half-open re-open path is shared and calls the classic
27+
`_open`. Per the audit's resolution (Option 1 — **document as intended**): a
28+
probe-failure re-open is a distinct event from a trip, so the classic shape is
29+
correct in both modes. No code behavior changes.
30+
31+
## Approach
32+
33+
- **Low #1 (observability, document-as-intended):** append a sentence to the
34+
CircuitBreaker section of `architecture/resilience.md` stating that the
35+
rate-flavored `circuit.opened` fires only on the initial trip from CLOSED, and
36+
that a HALF_OPEN probe-failure re-open carries the classic
37+
`failure_threshold`/`failures` (`failures = 1`) attributes and message in
38+
*both* modes — intentional, event-name is the stable contract.
39+
- **Low #2 (test gap):** add `test_rate_mode_probe_failure_reopens_with_classic_attributes`
40+
(async + sync mirror) — trip via rate, advance past `reset_timeout`, fail the
41+
probe, assert the circuit re-opens AND the re-open `circuit.opened` carries the
42+
classic shape (`failures == 1`, has `failure_threshold`, NOT `failure_rate`).
43+
This locks the Low #1 behavior so a regression can't pass silently.
44+
- **Nit #3 (Hypothesis oracle coupling):** not actioned — already neutralized by
45+
the hand-computed example tests; recorded in the audit as a Nit.
46+
47+
No `pyproject.toml` change; no release required (tests + doc clarification only,
48+
no user-facing functionality or API change). The audit report itself is added
49+
under `planning/audits/`.
50+
51+
## Files
52+
53+
- `planning/audits/2026-06-16-delta-audit.md` — the findings report (added).
54+
- `architecture/resilience.md` — the document-as-intended paragraph.
55+
- `tests/test_circuit_breaker.py` — async probe-failure re-open test.
56+
- `tests/test_circuit_breaker_sync.py` — sync mirror.
57+
58+
## Verification
59+
60+
- [x] Both new tests fail-safe (assert the actual emitted attribute set) and pass:
61+
`just test ...::test_rate_mode_probe_failure_reopens_with_classic_attributes`.
62+
- [x] `just test` — 706 passed, 100% coverage.
63+
- [x] `just lint` — clean.
64+
- [x] No source behavior change (only tests + docs).

tests/test_circuit_breaker.py

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -669,3 +669,34 @@ async def test_state_half_open_while_probing() -> None:
669669
assert breaker.state is CircuitState.HALF_OPEN
670670
await ok.get("https://example.test/x") # 2nd consecutive success → CLOSED
671671
assert breaker.state is CircuitState.CLOSED
672+
673+
674+
async def test_rate_mode_probe_failure_reopens_with_classic_attributes(caplog: pytest.LogCaptureFixture) -> None:
675+
"""A failed HALF_OPEN probe re-opens a rate-mode circuit via the shared classic path.
676+
677+
Documented as intended (2026-06-16 delta audit, Low #1): a probe-failure re-open is a
678+
distinct event from a rate trip, so circuit.opened carries the classic
679+
failure_threshold/failures shape (failures=1) in both modes; the rate attributes appear
680+
only on the initial rate trip from CLOSED.
681+
"""
682+
clock = _Clock()
683+
breaker = AsyncCircuitBreaker(
684+
failure_rate_threshold=0.5, window_seconds=100.0, minimum_calls=2, reset_timeout=5.0, _now=clock
685+
)
686+
fail = _client(_StatusSequence([500, 500]), breaker=breaker) # total=2, rate=1.0 → trip via rate
687+
for _ in range(2):
688+
with pytest.raises(InternalServerError):
689+
await fail.get("https://example.test/x")
690+
assert breaker.state is CircuitState.OPEN
691+
clock.advance(5.0) # past reset_timeout → next request is admitted as the probe
692+
with caplog.at_level(logging.WARNING, logger="httpware.circuit_breaker"):
693+
probe = _client(_StatusSequence([500]), breaker=breaker)
694+
with pytest.raises(InternalServerError):
695+
await probe.get("https://example.test/x")
696+
assert breaker.state is CircuitState.OPEN # the failed probe re-opened it
697+
opened = [r for r in caplog.records if r.event == "circuit.opened"] # ty: ignore[unresolved-attribute]
698+
assert opened
699+
rec = opened[-1]
700+
assert rec.failures == 1 # ty: ignore[unresolved-attribute] # classic probe-failure shape
701+
assert hasattr(rec, "failure_threshold")
702+
assert not hasattr(rec, "failure_rate") # NOT the rate shape

tests/test_circuit_breaker_sync.py

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -560,3 +560,32 @@ def test_state_half_open_while_probing() -> None:
560560
assert breaker.state is CircuitState.HALF_OPEN
561561
ok.get("https://example.test/x") # 2nd consecutive success → CLOSED
562562
assert breaker.state is CircuitState.CLOSED
563+
564+
565+
def test_rate_mode_probe_failure_reopens_with_classic_attributes(caplog: pytest.LogCaptureFixture) -> None:
566+
"""Sync mirror: a failed HALF_OPEN probe re-opens a rate-mode circuit via the shared classic path.
567+
568+
Documented as intended (2026-06-16 delta audit, Low #1): the re-open's circuit.opened carries
569+
the classic failure_threshold/failures shape (failures=1) in both modes.
570+
"""
571+
clock = _Clock()
572+
breaker = CircuitBreaker(
573+
failure_rate_threshold=0.5, window_seconds=100.0, minimum_calls=2, reset_timeout=5.0, _now=clock
574+
)
575+
fail = _client(_StatusSequence([500, 500]), breaker=breaker) # total=2, rate=1.0 → trip via rate
576+
for _ in range(2):
577+
with pytest.raises(InternalServerError):
578+
fail.get("https://example.test/x")
579+
assert breaker.state is CircuitState.OPEN
580+
clock.advance(5.0) # past reset_timeout → next request is admitted as the probe
581+
with caplog.at_level(logging.WARNING, logger="httpware.circuit_breaker"):
582+
probe = _client(_StatusSequence([500]), breaker=breaker)
583+
with pytest.raises(InternalServerError):
584+
probe.get("https://example.test/x")
585+
assert breaker.state is CircuitState.OPEN # the failed probe re-opened it
586+
opened = [r for r in caplog.records if r.event == "circuit.opened"] # ty: ignore[unresolved-attribute]
587+
assert opened
588+
rec = opened[-1]
589+
assert rec.failures == 1 # ty: ignore[unresolved-attribute] # classic probe-failure shape
590+
assert hasattr(rec, "failure_threshold")
591+
assert not hasattr(rec, "failure_rate") # NOT the rate shape

0 commit comments

Comments
 (0)