Skip to content

fix(enrichment): single-flight the half-open circuit-breaker probe (#65)#77

Merged
JumpTechCode merged 1 commit into
mainfrom
fix/breaker-half-open-single-flight
Jun 16, 2026
Merged

fix(enrichment): single-flight the half-open circuit-breaker probe (#65)#77
JumpTechCode merged 1 commit into
mainfrom
fix/breaker-half-open-single-flight

Conversation

@JumpTechCode

Copy link
Copy Markdown
Collaborator

Summary

Closes #65.

In the half_open state the breaker released its lock before the trial call, so every caller arriving after cooldown saw state == half_open, skipped the cooldown branch, and ran its own trial — N concurrent callers all probing the unhealthy dependency instead of one trial deciding for the rest. The breaker is an exported, reusable primitive advertised as correct, and nothing covered concurrent call().

Fix

  • Gate the half-open trial behind a _half_open_in_flight flag set under the lock: only the first post-cooldown caller probes; concurrent callers short-circuit with CircuitOpenError.
  • The flag is cleared on every exit path: success (closes), failure (still counts → re-opens), and cancellation.
  • The cancellation clear is an off-lock bare assignment (no await point ⇒ race-free) so a cancelled cleanup can't itself be cancelled while awaiting the lock and leak the slot — which would wedge the breaker half-open forever.

CircuitOpenError from a rejected concurrent probe is consumed downstream as a degraded FetcherResult (invariant 1), never raised to the user — so rejecting (fail-fast) is correct and avoids coupling N callers' latency to one slow probe.

Tests (all verified to fail without the fix via mutation)

  • test_half_open_single_flights_concurrent_probes — only one caller runs fn; the rest get CircuitOpenError.
  • test_half_open_probe_reopens_then_allows_next_probe_after_cooldown — failure-path flag release.
  • test_half_open_cancelled_probe_releases_slot — cancellation-path flag release (anti-wedge).

Closed-state path is untouched (probing stays False); all 491 unit tests pass, ruff + mypy --strict clean.

Note

The wait_for 5s-timeout path raises TimeoutError (an Exception, not CancelledError) to call, so it lands in the failure branch — flag cleared, failure counted, breaker re-opened. No leak on the timeout path.

In half_open the breaker released its lock before the trial call, so every
caller arriving after cooldown saw `state == half_open`, skipped the cooldown
branch, and ran its own trial — N concurrent callers all probing the unhealthy
dependency instead of one trial deciding for the rest.

Gate the trial behind an in-flight flag set under the lock: only the first
post-cooldown caller probes; concurrent callers short-circuit with
CircuitOpenError (consumed downstream as a degraded FetcherResult, never
raised). The flag is cleared on every exit — success, failure (which still
counts and re-opens), and cancellation. The cancellation clear is an off-lock
bare assignment so it can't itself be cancelled while awaiting the lock and
leak the slot, wedging the breaker half-open forever.

Adds tests for: concurrent probes single-flighting, failure-path flag release
(re-probe allowed after next cooldown), and cancellation-path flag release.

Closes #65

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@JumpTechCode
JumpTechCode force-pushed the fix/breaker-half-open-single-flight branch from 4937788 to 27c5b1e Compare June 16, 2026 06:11
@JumpTechCode
JumpTechCode merged commit 501cfd0 into main Jun 16, 2026
9 checks passed
@JumpTechCode
JumpTechCode deleted the fix/breaker-half-open-single-flight branch June 16, 2026 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enrichment: half-open circuit breaker is not single-flighted — concurrent callers all probe the unhealthy dependency

1 participant