feat(dispatch): result landing complete/abort/JE convergence (slice 3/6) - #358
Open
as535364 wants to merge 3 commits into
Open
feat(dispatch): result landing complete/abort/JE convergence (slice 3/6)#358as535364 wants to merge 3 commits into
as535364 wants to merge 3 commits into
Conversation
…ce 3/6) complete: completing state + per-submission lock + delete-after-write (INV1/INV3/INV4). abort: requeue with per-reason counting; drain refunds the claim-time attempts bump so a claim+drain round trip nets zero (spec 7.5). Attempts-exhausted jobs converge to JE via the orphan scan; a failed JE landing leaves the job in Redis for the next sweep (ADR-0003). Mongo writers are injected callables; wiring lands in the keystone slice.
as535364
commented
Jul 24, 2026
as535364
left a comment
as535364
left a comment
Member
Author
There was a problem hiding this comment.
codex: review completed; actionable findings are tracked in the inline threads below.
…ites A Mongo landing has no completion bound, so a fixed lock TTL cannot hold INV3: once the lock expired mid-write, a rival complete could acquire it and land the same result twice (PR #358 review). A watchdog thread now extends the lock every TTL/3 for the whole landing (token lives on the Lock instance, not thread-local, so the watchdog thread may extend); the TTL only bounds crash recovery. Also pin the at-least-once landing contract at the seam: land_result must tolerate full re-invocation after a partial write (spec 17.2); mongo-side idempotency stays the keystone's pre-wiring task.
An extend failure means the lock key itself evaporated; this layer cannot abort a black-box landing already in flight, so the residual double-write window closes on the mongo side (fencing token / conditional write), as a pre-keystone task.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes Normal-OJ/Normal-OJ#63
Slice 3/6 of the pull-based dispatch rework (spec:
docs/specs/pull-based-job-dispatch.mdin the meta-repo, §6/§7.4/§7.5/§9; ADR-0003). Dark PR — nothing calls these functions yet.What
complete_begin(ownership → currency → markcompleting),complete_finish(delete-after-write cleanup with compare-and-delete on thecurrent_jobpointer),complete_restore(undocompletingafter a failed Mongo write, never stomps a reclaimed-away job),abort_requeue(requeue with per-reason counting).complete_job/abort_jobindispatch/job.py, returning outcome enums; HTTP status mapping is documented for slice 4.Lock, non-blocking) around every landing (INV3).claim_next_job; a failed JE write leaves the job in Redis for the next time-gated sweep (INV1, ADR-0003).land_result(sid)/land_je(sid)); the keystone slice wiresprocess_resultand a JE marker.Note for reviewers: drain refunds one attempt
Merged slice 2 counts attempts on every claim. If drain only skipped counting, each drain → requeue → re-claim cycle would still burn one attempt, so rolling restarts would erode a job's retry budget — against spec §7.5 ("drain must not shorten a job's life"). The drain branch therefore refunds the claim-time bump (floored at 0): a claim+drain round trip nets zero. Poison paths (
prep_failed/rejected/reclaim) are unchanged, so INV5 convergence still holds.Tests
tests/test_dispatch_landing.py(26 tests): INV1 (restore on failed landing, JE-retry after Mongo failure), INV3 (concurrent completes, reclaim-during-write, lock contention), INV4 (stale drop + log, rejudge-during-write pointer safety), abort matrix (3 reasons × attempt boundaries), JE sweep via scan, corrupted-hash reaping. Slice-2 suite untouched and green; full suite 585 passed.Review scope
This review is scoped to dark slice Normal-OJ/Normal-OJ#63 only. Known pre-keystone landing debts are tracked by #359 and #360; both natively block keystone Normal-OJ/Normal-OJ#66.