fix(enrichment): single-flight the half-open circuit-breaker probe (#65)#77
Merged
Merged
Conversation
In half_open the breaker released its lock before the trial call, so every caller arriving after cooldown saw `state == half_open`, skipped the cooldown branch, and ran its own trial — N concurrent callers all probing the unhealthy dependency instead of one trial deciding for the rest. Gate the trial behind an in-flight flag set under the lock: only the first post-cooldown caller probes; concurrent callers short-circuit with CircuitOpenError (consumed downstream as a degraded FetcherResult, never raised). The flag is cleared on every exit — success, failure (which still counts and re-opens), and cancellation. The cancellation clear is an off-lock bare assignment so it can't itself be cancelled while awaiting the lock and leak the slot, wedging the breaker half-open forever. Adds tests for: concurrent probes single-flighting, failure-path flag release (re-probe allowed after next cooldown), and cancellation-path flag release. Closes #65 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
JumpTechCode
force-pushed
the
fix/breaker-half-open-single-flight
branch
from
June 16, 2026 06:11
4937788 to
27c5b1e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #65.
In the
half_openstate the breaker released its lock before the trial call, so every caller arriving after cooldown sawstate == half_open, skipped the cooldown branch, and ran its own trial — N concurrent callers all probing the unhealthy dependency instead of one trial deciding for the rest. The breaker is an exported, reusable primitive advertised as correct, and nothing covered concurrentcall().Fix
_half_open_in_flightflag set under the lock: only the first post-cooldown caller probes; concurrent callers short-circuit withCircuitOpenError.CircuitOpenErrorfrom a rejected concurrent probe is consumed downstream as a degradedFetcherResult(invariant 1), never raised to the user — so rejecting (fail-fast) is correct and avoids coupling N callers' latency to one slow probe.Tests (all verified to fail without the fix via mutation)
test_half_open_single_flights_concurrent_probes— only one caller runsfn; the rest getCircuitOpenError.test_half_open_probe_reopens_then_allows_next_probe_after_cooldown— failure-path flag release.test_half_open_cancelled_probe_releases_slot— cancellation-path flag release (anti-wedge).Closed-state path is untouched (
probingstaysFalse); all 491 unit tests pass, ruff + mypy --strict clean.Note
The
wait_for5s-timeout path raisesTimeoutError(anException, notCancelledError) tocall, so it lands in the failure branch — flag cleared, failure counted, breaker re-opened. No leak on the timeout path.