Extend production-observability example with accumulator pattern#133
Merged
chris-colinsky merged 4 commits intoJun 6, 2026
Merged
Conversation
Two pre-release polish items for the v0.12.0 cycle. CHANGELOG: the Unreleased Changed entry for the spec-pin advance originally said proposal 0052 "lands in a follow-on PR of this cycle" and 0054 "lands in a follow-on PR". Both have since landed (PRs 131 + 132). Rewrites the bullet to factually describe the final state: three proposals (0048, 0052, 0054) ship as fully implemented, two (0051, 0053) ship as textual-only. production-observability example: adds an LlmUsageAccumulator class plus a terminal persist node that demonstrates the queryable observer + drain_events_for pattern end-to-end. The accumulator subscribes to the LLM-namespace event stream, accumulates per- invocation token totals via current_invocation_id() bucket keys, and exposes convention-only get_bucket / drop methods. The persist node calls drain_events_for to synchronize on the deliver loop before reading the bucket so the rollup reflects every LLM call in the invocation, drops the bucket per the explicit-cleanup discipline, and prints a cost summary. The graph grows from respond -> END to respond -> persist -> END. Module-level singletons (_accumulator + _compiled_graph) keep the persist node closure-free and follow the existing _provider_instance precedent. Walkthrough doc updates the H1, overview, what-it-teaches list, captured-output sample, and reading-the-output walkthrough to cover the new pattern.
There was a problem hiding this comment.
Pull request overview
Extends the production-observability example to demonstrate the queryable observer accumulator + per-invocation drain_events_for pattern end-to-end, and updates docs/changelog to reflect the v0.12.0 cycle state.
Changes:
- Update the
[Unreleased]changelog entry to reflect proposals 0048/0052/0054 as implemented this cycle. - Extend
examples/production-observabilitywith anLlmUsageAccumulatorobserver and a terminalpersistnode that drains observer delivery before reading per-invocation usage. - Update the example walkthrough to document the new accumulator + drain pattern and output.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 8 comments.
| File | Description |
|---|---|
| examples/production-observability/main.py | Adds accumulator observer, new persist terminal node, and updates graph topology to demonstrate per-invocation drain + rollup. |
| docs/examples/production-observability.md | Updates walkthrough text and sample output to include the new persist/cost rollup behavior. |
| CHANGELOG.md | Rewrites the [Unreleased] spec-pin bullet from future tense to final implemented/text-only status. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The example's _format_otel_spans excluded the root openarmature.invocation span from its captured-output listing because two issues kept it from landing in the in-memory exporter and from showing usefully even when it did: 1. The OTel observer's shutdown() was never called, so the root invocation span stayed open and never moved into the exporter's finished-spans list. Adds otel_observer.shutdown() to the finally block after drain(), mirroring the pattern in the OTel unit tests. 2. The formatter's curated key set didn't include the invocation-level attributes the new span carries (openarmature.graph.entry_node, .spec_version, openarmature.implementation.name + .version). The formatter now picks the right key set based on span name: the invocation span surfaces its four invocation-level attrs only, inner-node spans surface the per-node + cross-cutting user.* + GenAI semconv attrs. Skipping cross-cutting attrs on the invocation line avoids repeating data that appears three more times below. Net visible change: the captured-OTel-spans block now opens with a [openarmature.invocation] line carrying implementation_name='openarmature-python' + implementation_version + spec_version + entry_node. Operators filtering traces by library version in Phoenix / Datadog / Honeycomb / Tempo / HyperDX read these directly from the root invocation span. Walkthrough doc's reading-the-output bullet now distinguishes the three OTel attribute families (invocation-level 5.1, cross-cutting 5.6, GenAI semconv) and explains why the invocation span only closes on observer shutdown().
Eight PR review threads, addressing four distinct issues. state.invocation_id -> current_invocation_id() in the example module docstring and walkthrough doc. The runnable persist() uses current_invocation_id() because State has no invocation_id field by default; the docstring snippets had drifted to the wrong shape. assert -> RuntimeError in persist(). The three runtime preconditions (_compiled_graph not None, _accumulator not None, current_invocation_id() not None) now raise explicit RuntimeError so the failure mode stays informative under python -O, which strips asserts and would otherwise produce silent None dereferences. InvocationCompletedEvent backstop cleanup in the accumulator. persist()'s drop is the fast path; if drain_events_for times out and the deliver loop later processes late-arriving LLM events, setdefault() would recreate a bucket that nothing ever cleans up. Adding InvocationCompletedEvent handling at the top of __call__ drops any leftover bucket on invocation completion. The drop is idempotent so it composes with persist()'s drop without harm. Defensive total_tokens derivation. LlmEventPayload makes all three usage fields optional; providers that emit prompt + completion but no total (anything non-OpenAI in practice) would leave bucket.total_tokens at zero while the sub-fields are correct. Now derives total from prompt + completion when total is None on the payload. build_graph() self-contained per the demo convention. Previously, persist() depended on _compiled_graph + _accumulator module globals that only main() populated, so a copy-pasting reader doing `graph = build_graph(); await graph.invoke(...)` would hit RuntimeError at persist time. build_graph() now owns the accumulator construction, the graph attach, and the global wiring. main() drops the duplicate construction and just attaches OTel + Langfuse on top.
Four stale-text findings from a second PR review pass — all caused by the previous review pass changing behavior without fully sweeping the surrounding documentation. Module docstring snippet now shows the full call shape: ``await graph.drain_events_for(current_invocation_id(), timeout=2.0)``, matching the runnable persist() pattern (the previous snippet stripped the await + timeout for brevity but under-described the API). The accumulator's drop() comment block was rewritten to describe the actual two-step lifecycle: fast-path explicit drop after read by the terminal node, plus the InvocationCompletedEvent backstop that the prior pass added. The old comment claimed "does NOT auto-drop on InvocationCompletedEvent" which directly contradicted the implementation. RuntimeError message in persist() now points readers at build_graph() instead of main() for the initialization pattern — the prior pass moved the singleton wiring into build_graph but left the error message pointing at the old call site. Walkthrough doc's "three observers attached at compile time" becomes "three observers attached before invoke", which is honest for both the build_graph-side accumulator attachment and the main-side OTel + Langfuse attachments. attach_observer happens after compile() in the OA API regardless of which function calls it.
chris-colinsky
deleted the
feature/v0.12.0-changelog-and-accumulator-example
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pre-release polish for the v0.12.0 cycle, ahead of PR 4 (the actual release prep). Two changes:
1. CHANGELOG tense fix
The
[Unreleased]### Changedentry for the spec-pin advance was originally written in PR 128 with future-tense language: proposal 0052 "lands in a follow-on PR of this cycle" and 0054 "lands in a follow-on PR". Both have since landed (PRs 131 + 132). One-bullet rewrite to factually describe the final state: three proposals (0048, 0052, 0054) ship as fully implemented this cycle, two (0051, 0053) ship as textual-only.2.
production-observabilityexample extensionThe §9 Queryable observer pattern (PR 129) +
drain_events_for(PR 131) were documented in the concepts page during the cycle, but no example demonstrated them end-to-end. This PR extends the existingproduction-observabilityexample so the headline operability story of v0.12.0 — per-invocation LLM cost attribution at request scope — is runnable.Notable pieces:
LlmUsageAccumulatorclass subscribes to the same event stream as the OTel + Langfuse observer pair but only records LLM-namespace events (event.namespace == LLM_NAMESPACEcarrying anLlmEventPayloadonpre_state). Accumulates per-invocation token totals viacurrent_invocation_id()bucket keys. Exposes convention-onlyget_bucket(invocation_id)anddrop(invocation_id)methods.persistterminal node callsawait graph.drain_events_for(state.invocation_id, timeout=2.0)to synchronize on the deliver loop, reads the accumulator's bucket, drops it per the §9.4 explicit-cleanup discipline, and prints a cost summary ([persist] LLM usage: prompt=X, completion=Y, total=Z across N call(s)). Handles the timeout-reached path with an inline[persist] drain incomplete: N events still pending after 2.0slog; production teams would also flip an SLO-breach metric.respond -> ENDtorespond -> persist -> END._accumulator,_compiled_graph) keep the persist node closure-free and match the existing_provider_instanceprecedent. Comment notes that an application server would use a request-scoped container instead.[persist]line, and a new walkthrough bullet explains the drain timeout fallback.Out of scope
production-observabilitykeeps the catalog at 13 and arcs naturally from "production observability wiring" to "and here's how the terminal node reads derived observer state").openarmature.implementation.name/.versionattributes from PR 132 surfacing in the demo output formatters (would feel like a third pass on the same example).## OpenTelemetry mapping/## Langfuse mapping(deferred per the user's scope choice earlier in this cycle).Test plan
uv run pytest tests/— 1123 passed, no regression from the import refactor or graph topology changeuv run pytest tests/test_examples_smoke.py -k production-observability— example still compiles viabuild_graph()uv run ruff check .+uv run ruff format --check .— cleanuv run pyright src/ tests/ examples/— 0 errorsuv run mkdocs build --strict— cleanuv run python scripts/check_conformance_manifest.py— 51/51 entries consistent