Skip to content

fix(reaper): surface and quarantine poison rows instead of silent recirculation (#91, #107)#110

Merged
JumpTechCode merged 1 commit into
mainfrom
fix/reaper-worker-liveness
Jul 20, 2026
Merged

fix(reaper): surface and quarantine poison rows instead of silent recirculation (#91, #107)#110
JumpTechCode merged 1 commit into
mainfrom
fix/reaper-worker-liveness

Conversation

@JumpTechCode

Copy link
Copy Markdown
Contributor

The gap (#91)

Per-item reap failure isolation (#74) rolls back a failed reap, excludes the id for the rest of the tick, and returns a normal ReapReport(reaped, failed). The worker escalation (#75) only fires when run_once raises — a normal return reset consecutive_failures = 0 and never inspected failed.

So a reservation whose reap reliably fails (a release statement-timeout on a hot balance row, a BalanceGuardViolation) recirculated at the queue head every tick — its rolled-back status flip left it held with the oldest ttl_deadline — stranding its reserved_micro indefinitely with no backoff, escalation, or alert. If ≥ batch_size such rows exist, healthy reservations behind them are never reaped either, while the worker reports healthy. Stranded reserved_micro produces false insufficient_budget denials on that node.

Fix (schema-free)

Runner liveness. A tick that returns a structured result reporting failures while reaping nothing (reaped == 0 and failed > 0) made no forward progress and now feeds consecutive_failures / backoff / escalation exactly like a raised tick — raising WorkerStalled past the threshold so the orchestrator restarts and alerts. A tick that still clears healthy rows (reaped > 0) resets the counter, so one poison row never takes the whole reaper down. The result is inspected by duck type (getattr), so the runner stays generic/stdlib-only and the idempotency reaper's int return is never judged stalled.

Poison-row quarantine. The reservation reaper now tracks per-row failure counts across ticks; a row that fails max_reap_attempts times is quarantined — excluded from every future claim this process and logged as an error — so a permanent poison row surfaces and stops recirculating/stranding unseen, while the reaper keeps serving healthy rows. State is per-process (a restart re-attempts — a fresh chance if contention cleared). A durable dead-letter would need a persisted attempt column; noted as the heavier follow-up.

Backoff overflow (#107). _backoff_seconds caps the exponent so a large worker_max_consecutive_failures (settings enforce only ge=1) can't make 2 ** (n - 1) overflow before the clamp to the ceiling.

New setting reaper_max_reap_attempts (default 5), wired through app.py.

Tests

  • runner: escalates on stalled (no-progress + failures) ticks; tolerates failures while making progress; ignores failed on scalar results; _backoff_seconds clamps without overflow and grows-then-clamps.
  • reaper: quarantines a persistent poison row after max_reap_attempts (and excludes it thereafter); never quarantines healthy rows.
  • Reservation-reaper integration suite still green.

Local gate: ruff, mypy --strict, import-linter, full unit suite (483), reservation-reaper integration — all green.

Closes #91. Advances #107 (backoff-overflow item).

…irculation (#91, #107)

Per-item reap failure isolation (#74) returns a normal ReapReport(reaped,
failed), but the worker escalation (#75) only fires when run_once *raises* — a
normal return reset consecutive_failures and never inspected `failed`. So a
reservation whose reap reliably fails recirculated at the queue head every tick
(its rolled-back status flip left it held with the oldest deadline), stranding
its reserved estimate indefinitely with no backoff, escalation, or alert; and
>= batch_size such rows starved healthy reservations behind them while the
worker reported healthy.

Two schema-free changes close the gap:

- runner: a tick that returns a structured result reporting failures while
  reaping nothing (reaped == 0, failed > 0) made no forward progress and now
  feeds consecutive_failures / backoff / escalation exactly like a raised tick,
  raising WorkerStalled past the threshold. A tick that still clears healthy
  rows (reaped > 0) resets the counter, so one poison row never takes the whole
  reaper down. The result is inspected by duck type, so the idempotency reaper's
  int return is never judged stalled.

- reaper handler: per-row attempt counts accumulate across ticks; a row that
  fails max_reap_attempts times is quarantined — excluded from every future
  claim this process and logged as an error — so a permanent poison row surfaces
  instead of recirculating forever. State is per-process; a restart re-attempts.

Also caps the backoff exponent (_backoff_seconds) so a large
worker_max_consecutive_failures can't make 2 ** (n - 1) overflow before the
clamp to the ceiling (#107).

New setting reaper_max_reap_attempts (default 5), wired through app.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjJ4QScUKbrMzF1jujL5Bg
@JumpTechCode
JumpTechCode merged commit 3b6fa2b into main Jul 20, 2026
8 checks passed
@JumpTechCode
JumpTechCode deleted the fix/reaper-worker-liveness branch July 20, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reaper: per-item reap failures bypass worker escalation and can strand reservations indefinitely

1 participant