Reset metadata between retry attempts per spec 3.4#130
Merged
Conversation
Spec observability 3.4 requires per-attempt scoping under retry middleware: each attempt sees only the metadata in scope at retry-loop entry plus that attempt's own writes; failed-attempt writes are discarded along with the attempt itself. The python RetryMiddleware was only managing the attempt_index ContextVar and leaving the invocation-metadata ContextVar untouched across attempts, so a set_invocation_metadata call inside a failed attempt remained visible on the next attempt. Capture the pre-attempt baseline once at retry-loop entry. Reset the metadata ContextVar to that baseline at the start of each iteration. Discard the failed attempt's writes on both retry-eligible and terminal failure paths. Leave the successful attempt's writes in place so downstream nodes see them. Two new unit tests pin the contract: one mirrors spec fixture 045's case shape (attempt 0 writes a marker and fails, attempt 1 reads and confirms the marker is absent, writes a new marker, and succeeds; downstream node reads the successful attempt's marker), the other exercises the middleware via compose_chain to verify that after a terminal failure the metadata is back at baseline with no leak from the final failed attempt. Closes the v0.12.0 cycle's partial claim on proposal 0048; conformance manifest flips back to implemented and the "Per-attempt retry scoping is partial in v0.12.0" docs callout disappears.
There was a problem hiding this comment.
Pull request overview
Closes an observability spec §3.4 gap by enforcing per-attempt scoping of invocation metadata under RetryMiddleware, ensuring metadata written during a failed attempt does not carry into subsequent attempts while successful-attempt writes remain visible downstream.
Changes:
- Reset invocation-metadata ContextVar to the retry-entry baseline at the start of each retry attempt; discard failed-attempt metadata on failure paths.
- Add unit tests pinning per-attempt discard behavior on both retry-success and terminal-failure scenarios.
- Update conformance manifest, docs, and changelog to reflect the now-implemented §3.4 behavior.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
src/openarmature/graph/middleware/retry.py |
Implements per-attempt metadata baseline/reset behavior inside the retry loop. |
tests/unit/test_observability_metadata.py |
Adds regression tests covering failed-attempt discard and terminal-failure discard semantics. |
docs/concepts/observability.md |
Updates §3.4 documentation to remove the “partial” caveat and describe per-attempt retry scoping as implemented. |
conformance.toml |
Marks proposal 0048 as implemented and updates the pinned test list to include the new per-attempt scoping tests. |
CHANGELOG.md |
Records the fix under [Unreleased] → Fixed. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The original PR-2c except-Exception branch caught transient retryable errors but missed cancellation. CancelledError extends BaseException, not Exception, so a node body that called set_invocation_metadata and then got cancelled would leak its writes upward into any code that caught the cancellation. The except-Exception was deliberate for retry semantics (cancellation MUST propagate, never retry), but it left a metadata cleanup hole. Add a parallel except-BaseException branch that resets the metadata token and re-raises. No retry, no on_retry callback, no sleep on this path — just clean up and propagate. New unit test mirrors the exact scenario: node writes a marker, raises CancelledError, retry middleware lets cancellation propagate (single attempt, no swallow per spec 6.1), and the post-failure metadata view is back at the pre-attempt baseline per spec 3.4. Conformance manifest 0048 pin enumeration extended to name the new test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 2c of the v0.12.0 cycle. Closes the per-attempt metadata scoping gap that PR #129's review pass surfaced and walked back.
Spec observability §3.4 mandates that under retry middleware, each attempt sees only the metadata in scope at retry-loop entry plus that attempt's own writes; writes from a prior attempt that subsequently failed are discarded along with the attempt. The python
RetryMiddlewarewas only managing theattempt_indexContextVar and leaving the invocation-metadata ContextVar untouched across attempts, so aset_invocation_metadata(...)call inside a failed attempt remained visible on the next attempt. This PR closes the gap.The fix captures the pre-attempt baseline once at retry-loop entry; resets the metadata ContextVar to that baseline at the start of each iteration; discards the failed attempt's writes on both retry-eligible and terminal failure paths; and leaves the successful attempt's writes in place so downstream nodes see them.
Coord context:
discuss-retry-metadata-scoping-gapthread. Spec cleared Q1 attribution (gap belongs to proposal 0034's contract, surfaced by 0048) and Q2 scope (Path A: small engine fix + unit test; defer fixture 045 activation to the broader 043-049 batch after the upcoming spec conformance-adapter capability ratifies the directive vocabulary). The implementation here follows that direction.Notable pieces
src/openarmature/graph/middleware/retry.pycarries the fix: pre-attempt baseline captured once before the while loop;_set_invocation_metadata(baseline)at each iteration;_reset_invocation_metadata(metadata_token)on failure paths; nothing on the success path so downstream sees the writes.tests/unit/test_observability_metadata.pyadds two tests:test_per_attempt_scoping_under_retry_discards_failed_attempt_writes(mirrors spec fixture 045 case shape end-to-end throughGraphBuilder.invoke) andtest_terminal_failure_discards_final_failed_attempt_writes(usescompose_chainto bypass the engine's outer reset so the post-retry metadata view is readable in the test scope).conformance.tomlflips proposal 0048 frompartialback toimplemented; drops thenotefield; rewrites the explanatory comment block to drop the "Open gap" paragraph and add the two new tests to the pin enumeration; integrates per-attempt scoping into the satisfied-contract list.docs/concepts/observability.mdremoves the!!! info "Per-attempt retry scoping is partial in v0.12.0"callout and restores the original "The read is per-attempt scoped under retry middleware" sentence to the metadata-read description.CHANGELOG.mdadds a### Fixedsubsection under[Unreleased]describing the bug and the fix.src/openarmature/AGENTS.mdregenerated to pick up the docs + conformance changes.Out of scope
RetryMiddleware.__call__itself.Test plan
uv run pytest tests/— 1100 passed (was 1098 pre-fix), 307 skipped, 0 failuresuv run pytest tests/unit/test_observability_metadata.py— 50 passed (2 new)uv run pytest tests/unit/test_middleware.py tests/unit/test_fan_out.py tests/unit/test_observability_otel.py— 74 passed, no retry / fan-out / OTel regressionsuv run python scripts/check_conformance_manifest.py— 51/51 entries consistentuv run ruff check .+uv run ruff format --check .— cleanuv run pyright src/ tests/— 0 errorsuv run mkdocs build --strict— clean