fix(fusillade): fence reclaimed attempts and re-admit retries - #1356
Open
pjb157 wants to merge 14 commits into
Open
fix(fusillade): fence reclaimed attempts and re-admit retries#1356pjb157 wants to merge 14 commits into
pjb157 wants to merge 14 commits into
Conversation
added 13 commits
July 23, 2026 15:11
Keep terminal outcomes daemon-owned and persist them with the exact claim attempt token. Recover processor failures, panics, and retriable outcomes by returning requests to pending so every retry reacquires ordinary model and user admission. Retry only positively classified transient persistence failures with shutdown-aware bounded backoff. Redact deterministic fallback failures, preserve ZDR transform safety before compare-and-set, and cover provider cancellation, terminal durability, attempt fencing, and retry admission.
Deploying control-layer with
|
| Latest commit: |
b3b333a
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://19a60869.control-layer.pages.dev |
| Branch Preview URL: | https://peter-fix-fusillade-attempt.control-layer.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
transitions by the exact request/daemon/attempt identity
pendingso they reacquire ordinary modeladmission and respect the same in-flight limits as first attempts
invoking the provider again
reroute signal out of the daemon's redacted fallback path
This PR is stacked on #1352, which supplies the bounded database admission used
by all response reads and writes. Merge #1352 first, then retarget this PR to
main.Why
Pod identity alone does not distinguish two executions owned by the same
daemon. An abandoned request could be reclaimed while its previous execution
was still able to persist, allowing the old result to overwrite the new owner.
Automatic retry dispatch also bypassed normal model admission, so timeout
storms could exceed configured in-flight limits.
Ownership, reclaim, and retry semantics
the daemon ID is unchanged
never solely because the request is old
dispatch exits
replacement attempt
pending, andmust be claimed and admitted again before provider execution
Persistence and concurrency guarantees
claimed -> processingcommits from a concurrent clone of the same attemptbefore pool retry backoff
not call the provider again
preserve the original completion/failure rather than synthesizing fallback
canceled-to-terminal writes
shredding happens only after the winning terminal commit
failures terminalize before dispatch
Rollout and limitations
Both migrations are additive and nullable for rolling compatibility. Full
fencing applies after all daemon pods run the new binary. Code rollback may
leave the nullable columns in place; the
processing_admission_iddownmigration refuses to run while a non-null marker exists.
PostgreSQL and the provider cannot share an atomic transaction. A real
process/node death after dispatch therefore remains at-least-once provider
compute: recovery may execute again because the dead process cannot report its
result. This PR prevents live stale attempts from corrupting the replacement
and prevents retries from bypassing admission; it does not claim exactly-once
provider execution.
No Dynamo cancellation endpoint or transport-abort behavior is changed here.
Verification
Related: COR-550