Skip to content

Reset metadata between retry attempts per spec 3.4#130

Merged
chris-colinsky merged 2 commits into
mainfrom
fix/retry-metadata-per-attempt-scoping
Jun 5, 2026
Merged

Reset metadata between retry attempts per spec 3.4#130
chris-colinsky merged 2 commits into
mainfrom
fix/retry-metadata-per-attempt-scoping

Conversation

@chris-colinsky

Copy link
Copy Markdown
Member

Summary

PR 2c of the v0.12.0 cycle. Closes the per-attempt metadata scoping gap that PR #129's review pass surfaced and walked back.

Spec observability §3.4 mandates that under retry middleware, each attempt sees only the metadata in scope at retry-loop entry plus that attempt's own writes; writes from a prior attempt that subsequently failed are discarded along with the attempt. The python RetryMiddleware was only managing the attempt_index ContextVar and leaving the invocation-metadata ContextVar untouched across attempts, so a set_invocation_metadata(...) call inside a failed attempt remained visible on the next attempt. This PR closes the gap.

The fix captures the pre-attempt baseline once at retry-loop entry; resets the metadata ContextVar to that baseline at the start of each iteration; discards the failed attempt's writes on both retry-eligible and terminal failure paths; and leaves the successful attempt's writes in place so downstream nodes see them.

Coord context: discuss-retry-metadata-scoping-gap thread. Spec cleared Q1 attribution (gap belongs to proposal 0034's contract, surfaced by 0048) and Q2 scope (Path A: small engine fix + unit test; defer fixture 045 activation to the broader 043-049 batch after the upcoming spec conformance-adapter capability ratifies the directive vocabulary). The implementation here follows that direction.

Notable pieces

  • src/openarmature/graph/middleware/retry.py carries the fix: pre-attempt baseline captured once before the while loop; _set_invocation_metadata(baseline) at each iteration; _reset_invocation_metadata(metadata_token) on failure paths; nothing on the success path so downstream sees the writes.
  • tests/unit/test_observability_metadata.py adds two tests: test_per_attempt_scoping_under_retry_discards_failed_attempt_writes (mirrors spec fixture 045 case shape end-to-end through GraphBuilder.invoke) and test_terminal_failure_discards_final_failed_attempt_writes (uses compose_chain to bypass the engine's outer reset so the post-retry metadata view is readable in the test scope).
  • conformance.toml flips proposal 0048 from partial back to implemented; drops the note field; rewrites the explanatory comment block to drop the "Open gap" paragraph and add the two new tests to the pin enumeration; integrates per-attempt scoping into the satisfied-contract list.
  • docs/concepts/observability.md removes the !!! info "Per-attempt retry scoping is partial in v0.12.0" callout and restores the original "The read is per-attempt scoped under retry middleware" sentence to the metadata-read description.
  • CHANGELOG.md adds a ### Fixed subsection under [Unreleased] describing the bug and the fix.
  • src/openarmature/AGENTS.md regenerated to pick up the docs + conformance changes.

Out of scope

  • Fixture 045 activation in the cross-capability parser (Path A defers it to the broader 043-049 batch).
  • Proposal 0050 call-level retry per-attempt scoping (forward-compat; happens in v0.14.0 when 0050 implementation opens).
  • Anything outside RetryMiddleware.__call__ itself.

Test plan

  • uv run pytest tests/ — 1100 passed (was 1098 pre-fix), 307 skipped, 0 failures
  • uv run pytest tests/unit/test_observability_metadata.py — 50 passed (2 new)
  • uv run pytest tests/unit/test_middleware.py tests/unit/test_fan_out.py tests/unit/test_observability_otel.py — 74 passed, no retry / fan-out / OTel regressions
  • uv run python scripts/check_conformance_manifest.py — 51/51 entries consistent
  • uv run ruff check . + uv run ruff format --check . — clean
  • uv run pyright src/ tests/ — 0 errors
  • uv run mkdocs build --strict — clean

Spec observability 3.4 requires per-attempt scoping under retry
middleware: each attempt sees only the metadata in scope at
retry-loop entry plus that attempt's own writes; failed-attempt
writes are discarded along with the attempt itself. The python
RetryMiddleware was only managing the attempt_index ContextVar
and leaving the invocation-metadata ContextVar untouched across
attempts, so a set_invocation_metadata call inside a failed
attempt remained visible on the next attempt.

Capture the pre-attempt baseline once at retry-loop entry. Reset
the metadata ContextVar to that baseline at the start of each
iteration. Discard the failed attempt's writes on both
retry-eligible and terminal failure paths. Leave the successful
attempt's writes in place so downstream nodes see them.

Two new unit tests pin the contract: one mirrors spec fixture
045's case shape (attempt 0 writes a marker and fails, attempt 1
reads and confirms the marker is absent, writes a new marker, and
succeeds; downstream node reads the successful attempt's marker),
the other exercises the middleware via compose_chain to verify
that after a terminal failure the metadata is back at baseline
with no leak from the final failed attempt.

Closes the v0.12.0 cycle's partial claim on proposal 0048;
conformance manifest flips back to implemented and the
"Per-attempt retry scoping is partial in v0.12.0" docs callout
disappears.
Copilot AI review requested due to automatic review settings June 5, 2026 04:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Closes an observability spec §3.4 gap by enforcing per-attempt scoping of invocation metadata under RetryMiddleware, ensuring metadata written during a failed attempt does not carry into subsequent attempts while successful-attempt writes remain visible downstream.

Changes:

  • Reset invocation-metadata ContextVar to the retry-entry baseline at the start of each retry attempt; discard failed-attempt metadata on failure paths.
  • Add unit tests pinning per-attempt discard behavior on both retry-success and terminal-failure scenarios.
  • Update conformance manifest, docs, and changelog to reflect the now-implemented §3.4 behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/openarmature/graph/middleware/retry.py Implements per-attempt metadata baseline/reset behavior inside the retry loop.
tests/unit/test_observability_metadata.py Adds regression tests covering failed-attempt discard and terminal-failure discard semantics.
docs/concepts/observability.md Updates §3.4 documentation to remove the “partial” caveat and describe per-attempt retry scoping as implemented.
conformance.toml Marks proposal 0048 as implemented and updates the pinned test list to include the new per-attempt scoping tests.
CHANGELOG.md Records the fix under [Unreleased]Fixed.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/openarmature/graph/middleware/retry.py
The original PR-2c except-Exception branch caught transient
retryable errors but missed cancellation. CancelledError extends
BaseException, not Exception, so a node body that called
set_invocation_metadata and then got cancelled would leak its
writes upward into any code that caught the cancellation. The
except-Exception was deliberate for retry semantics (cancellation
MUST propagate, never retry), but it left a metadata cleanup hole.

Add a parallel except-BaseException branch that resets the
metadata token and re-raises. No retry, no on_retry callback, no
sleep on this path — just clean up and propagate.

New unit test mirrors the exact scenario: node writes a marker,
raises CancelledError, retry middleware lets cancellation
propagate (single attempt, no swallow per spec 6.1), and the
post-failure metadata view is back at the pre-attempt baseline
per spec 3.4. Conformance manifest 0048 pin enumeration extended
to name the new test.
@chris-colinsky chris-colinsky merged commit 7a3e0b2 into main Jun 5, 2026
6 checks passed
@chris-colinsky chris-colinsky deleted the fix/retry-metadata-per-attempt-scoping branch June 5, 2026 04:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants