fix(ingestion): close lost-event race on persist failure (#62)#76
Merged
Conversation
The webhook handler marked the Redis idempotency key (SET NX EX 24h)
before persisting the incident. A transient failure in `ingest` after
the mark turned the sender's byte-identical retry into a silently
dropped event — the exact failure mode ADR 0001 ("never lose an event")
is meant to eliminate.
Wrap `ingest` in the handler: on failure, best-effort `unmark` the key
(new IdempotencyStore method) and return INFRA_UNAVAILABLE (503) so the
sender re-delivers and the now-cleared key lets the retry re-process.
If the unmark itself fails, still return 503 and let the key fall back
to its 24h TTL. The fix is safe because `ingest` is single-transaction:
a failure rolls back, so unmark never erases a persisted event.
Adds regression tests for ingest-fails-after-mark (with and without a
failing unmark) and a mark→unmark→processable-again round-trip.
Closes #62
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #62.
The webhook handler marked the Redis idempotency key (
SET NX EX 24h) before persisting the incident, with no rollback if persistence failed. A transientingestfailure after the mark turned the sender's byte-identical retry into a silently-dropped event — the exact failure mode ADR 0001 ("never lose an event") forbids.Fix
unmark(source, body)to theIdempotencyStoreprotocol +RedisIdempotencyStore(deletes the key; key derivation factored into_idempotency_key).ingestin the handler: on failure → log, best-effortunmark(compensating delete), returnINFRA_UNAVAILABLE(→ HTTP 503, retryable) so the sender re-delivers and the cleared key lets the retry re-process.unmarkitself fails, still return 503 and let the key fall back to its 24h TTL.Why this is safe:
ingestcommits in a single transaction — a failure rolls back atomically, sounmarkcan never erase a body that was actually persisted.Tests
test_ingest_failure_unmarks_idempotency_key— asserts 503 +unmarkawaited (fails if the compensating delete is removed).test_ingest_failure_still_503_when_unmark_also_fails— nested-failure path stays retryable.test_unmark_makes_body_processable_again/test_unmark_absent_key_is_noop— Redis round-trip + no-op.All 492 unit tests pass; ruff + mypy --strict clean.
Failure mode (the mantra)
On persist failure: logged (
ingest_failed_after_idempotency_mark), key cleared, breaker-agnostic 503 returned to the sender; if the unmark fails too it's logged (idempotency_unmark_failed) and the 24h TTL is the backstop.