fix(reaper): surface and quarantine poison rows instead of silent recirculation (#91, #107)#110
Merged
Merged
Conversation
…irculation (#91, #107) Per-item reap failure isolation (#74) returns a normal ReapReport(reaped, failed), but the worker escalation (#75) only fires when run_once *raises* — a normal return reset consecutive_failures and never inspected `failed`. So a reservation whose reap reliably fails recirculated at the queue head every tick (its rolled-back status flip left it held with the oldest deadline), stranding its reserved estimate indefinitely with no backoff, escalation, or alert; and >= batch_size such rows starved healthy reservations behind them while the worker reported healthy. Two schema-free changes close the gap: - runner: a tick that returns a structured result reporting failures while reaping nothing (reaped == 0, failed > 0) made no forward progress and now feeds consecutive_failures / backoff / escalation exactly like a raised tick, raising WorkerStalled past the threshold. A tick that still clears healthy rows (reaped > 0) resets the counter, so one poison row never takes the whole reaper down. The result is inspected by duck type, so the idempotency reaper's int return is never judged stalled. - reaper handler: per-row attempt counts accumulate across ticks; a row that fails max_reap_attempts times is quarantined — excluded from every future claim this process and logged as an error — so a permanent poison row surfaces instead of recirculating forever. State is per-process; a restart re-attempts. Also caps the backoff exponent (_backoff_seconds) so a large worker_max_consecutive_failures can't make 2 ** (n - 1) overflow before the clamp to the ceiling (#107). New setting reaper_max_reap_attempts (default 5), wired through app.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BjJ4QScUKbrMzF1jujL5Bg
This was referenced Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap (#91)
Per-item reap failure isolation (#74) rolls back a failed reap, excludes the id for the rest of the tick, and returns a normal
ReapReport(reaped, failed). The worker escalation (#75) only fires whenrun_onceraises — a normal return resetconsecutive_failures = 0and never inspectedfailed.So a reservation whose reap reliably fails (a
releasestatement-timeout on a hot balance row, aBalanceGuardViolation) recirculated at the queue head every tick — its rolled-back status flip left itheldwith the oldestttl_deadline— stranding itsreserved_microindefinitely with no backoff, escalation, or alert. If ≥batch_sizesuch rows exist, healthy reservations behind them are never reaped either, while the worker reports healthy. Strandedreserved_microproduces falseinsufficient_budgetdenials on that node.Fix (schema-free)
Runner liveness. A tick that returns a structured result reporting failures while reaping nothing (
reaped == 0 and failed > 0) made no forward progress and now feedsconsecutive_failures/ backoff / escalation exactly like a raised tick — raisingWorkerStalledpast the threshold so the orchestrator restarts and alerts. A tick that still clears healthy rows (reaped > 0) resets the counter, so one poison row never takes the whole reaper down. The result is inspected by duck type (getattr), so the runner stays generic/stdlib-only and the idempotency reaper'sintreturn is never judged stalled.Poison-row quarantine. The reservation reaper now tracks per-row failure counts across ticks; a row that fails
max_reap_attemptstimes is quarantined — excluded from every future claim this process and logged as an error — so a permanent poison row surfaces and stops recirculating/stranding unseen, while the reaper keeps serving healthy rows. State is per-process (a restart re-attempts — a fresh chance if contention cleared). A durable dead-letter would need a persisted attempt column; noted as the heavier follow-up.Backoff overflow (#107).
_backoff_secondscaps the exponent so a largeworker_max_consecutive_failures(settings enforce onlyge=1) can't make2 ** (n - 1)overflow before the clamp to the ceiling.New setting
reaper_max_reap_attempts(default 5), wired throughapp.py.Tests
failedon scalar results;_backoff_secondsclamps without overflow and grows-then-clamps.max_reap_attempts(and excludes it thereafter); never quarantines healthy rows.Local gate: ruff, mypy --strict, import-linter, full unit suite (483), reservation-reaper integration — all green.
Closes #91. Advances #107 (backoff-overflow item).