Skip to content

fix(fusillade): fence reclaimed attempts and re-admit retries - #1356

Open
pjb157 wants to merge 14 commits into
peter/fix-cor-550-responses-db-burstsfrom
peter/fix-fusillade-attempt-fencing
Open

fix(fusillade): fence reclaimed attempts and re-admit retries#1356
pjb157 wants to merge 14 commits into
peter/fix-cor-550-responses-db-burstsfrom
peter/fix-fusillade-attempt-fencing

Conversation

@pjb157

@pjb157 pjb157 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • assign a fresh attempt token on every claim and fence all daemon-owned
    transitions by the exact request/daemon/attempt identity
  • reclaim abandoned work without age-reclaiming healthy processing requests
  • return automatic retries to pending so they reacquire ordinary model
    admission and respect the same in-flight limits as first attempts
  • retain terminal outcomes across transient persistence failures without
    invoking the provider again
  • make cancellation/terminal races parent-lock safe and keep their internal
    reroute signal out of the daemon's redacted fallback path

This PR is stacked on #1352, which supplies the bounded database admission used
by all response reads and writes. Merge #1352 first, then retarget this PR to
main.

Why

Pod identity alone does not distinguish two executions owned by the same
daemon. An abandoned request could be reclaimed while its previous execution
was still able to persist, allowing the old result to overwrite the new owner.
Automatic retry dispatch also bypassed normal model admission, so timeout
storms could exceed configured in-flight limits.

Ownership, reclaim, and retry semantics

  • every claim receives a non-nil attempt token; reclaim rotates it even when
    the daemon ID is unchanged
  • claimed work can be reclaimed after an age and compare-and-set recheck
  • processing work is reclaimed only when its owner is missing, dead, or stale,
    never solely because the request is old
  • reclaim runs in every daemon mode, including BatchOnly, before zero-capacity
    dispatch exits
  • stale attempts cannot process, complete, fail, cancel, retry, or overwrite a
    replacement attempt
  • automatic retries release their current admission, return to pending, and
    must be claimed and admitted again before provider execution

Persistence and concurrency guarantees

  • a per-dispatch admission UUID distinguishes ambiguous
    claimed -> processing commits from a concurrent clone of the same attempt
  • commit reconciliation uses bounded write admission and releases permits
    before pool retry backoff
  • transient terminal durability failures retry the prepared outcome; they do
    not call the provider again
  • terminal writes racing soft cancellation reacquire the batch parent lock and
    preserve the original completion/failure rather than synthesizing fallback
  • frozen batch counters and retry-version snapshots remain correct under late
    canceled-to-terminal writes
  • ZDR transformation is side-effect-free before the attempt CAS; request-key
    shredding happens only after the winning terminal commit
  • only an unreachable ZDR keystore is retriable; malformed/retired/config/crypto
    failures terminalize before dispatch
  • archive and live request schemas carry the same nullable ownership fields

Rollout and limitations

Both migrations are additive and nullable for rolling compatibility. Full
fencing applies after all daemon pods run the new binary. Code rollback may
leave the nullable columns in place; the processing_admission_id down
migration refuses to run while a non-null marker exists.

PostgreSQL and the provider cannot share an atomic transaction. A real
process/node death after dispatch therefore remains at-least-once provider
compute: recovery may execute again because the dead process cannot report its
result. This PR prevents live stale attempts from corrupting the replacement
and prevents retries from bypassing admission; it does not claim exactly-once
provider execution.

No Dynamo cancellation endpoint or transport-abort behavior is changed here.

Verification

  • Fusillade Arsenal: 269 tests
  • Fusillade daemon/library: 55 unit + 45 integration tests
  • Fusillade core: 23 tests + doctest
  • deterministic cancellation-snapshot concurrency regression
  • retry admission, provider-invocation-once, and terminal durability tests
  • archive schema parity: 5 tests
  • migration checksum, clean upgrade, revert, and re-apply
  • dwctl test compile and focused ZDR classifier test
  • SQLx workspace metadata regenerated and verified
  • formatting and diff checks
  • GitHub CI

Related: COR-550

Peter added 13 commits July 23, 2026 15:11
Keep terminal outcomes daemon-owned and persist them with the exact claim attempt token. Recover processor failures, panics, and retriable outcomes by returning requests to pending so every retry reacquires ordinary model and user admission.

Retry only positively classified transient persistence failures with shutdown-aware bounded backoff. Redact deterministic fallback failures, preserve ZDR transform safety before compare-and-set, and cover provider cancellation, terminal durability, attempt fencing, and retry admission.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 23, 2026

Copy link
Copy Markdown

Deploying control-layer with  Cloudflare Pages  Cloudflare Pages

Latest commit: b3b333a
Status: ✅  Deploy successful!
Preview URL: https://19a60869.control-layer.pages.dev
Branch Preview URL: https://peter-fix-fusillade-attempt.control-layer.pages.dev

View logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant