Extend production-observability example with accumulator pattern

CHANGELOG.md

@@ -8,7 +8,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The
 
 ### Changed
 
-- **Pinned spec advanced from v0.38.0 to v0.46.0.** Submodule + `[tool.openarmature].spec_version` + `conformance.toml` `spec_pin` advance together. Absorbs eight new proposals (0047-0054) into the conformance manifest. Two of them ship as part of the v0.12.0 cycle as textual-only acknowledgments with no code change required: **proposal 0051** (observability §8.4.1 Langfuse `trace.input` / `trace.output` implementation-surface caveat — documents that vendor SDK round-trip is required to project caller-side trace I/O updates onto the wire; the v0.11.0 (proposal 0043) caller-hook shape already matches the documented behavior) and **proposal 0053** (observability §3.4 shared-parent boundary clarification — tightens the structural-shared-parent classification to predicate the invocation span on whether at least one fan-out or parallel-branches dispatch is on the augmenter's call-stack path; behavior already matches via fixtures 034 + 039). Two more ship as implemented in this cycle: **proposal 0048** (read-symmetric metadata + queryable observer pattern docs — see *Added* below) and **proposal 0052** (implementation attribution attributes — landing in a follow-on PR of this cycle). The remaining proposals are marked `not-yet` in the conformance manifest with roadmap targets: 0047 + 0049 (v0.13.0 LLM provider hardening batch), 0050 (v0.14.0 retry & reliability batch), 0054 (per-invocation observer event drain — bundled into this v0.12.0 cycle alongside 0048 per the §9.4 accumulator-lifecycle pairing; lands in a follow-on PR).
+- **Pinned spec advanced from v0.38.0 to v0.46.0.** Submodule + `[tool.openarmature].spec_version` + `conformance.toml` `spec_pin` advance together. Absorbs eight new proposals (0047-0054) into the conformance manifest. Two ship as textual-only acknowledgments with no code change required: **proposal 0051** (observability §8.4.1 Langfuse `trace.input` / `trace.output` implementation-surface caveat — documents that vendor SDK round-trip is required to project caller-side trace I/O updates onto the wire; the v0.11.0 (proposal 0043) caller-hook shape already matches the documented behavior) and **proposal 0053** (observability §3.4 shared-parent boundary clarification — tightens the structural-shared-parent classification to predicate the invocation span on whether at least one fan-out or parallel-branches dispatch is on the augmenter's call-stack path; behavior already matches via fixtures 034 + 039). Three ship as fully implemented this cycle: **proposal 0048** (read-symmetric metadata + queryable observer pattern docs — see *Added* below), **proposal 0052** (implementation attribution attributes — see *Added* below), and **proposal 0054** (per-invocation observer event drain — see *Added* below; bundled with 0048 as the §9.4 accumulator-lifecycle pair). The remaining proposals are marked `not-yet` in the conformance manifest with roadmap targets: 0047 + 0049 (v0.13.0 LLM provider hardening batch) and 0050 (v0.14.0 retry & reliability batch).
 - **README and docs homepage refreshed around reasons-to-choose.** Replaced the 10-bullet "Why OpenArmature" feature inventory in `README.md` with 5 differentiating reasons (LLM-infused workflows to agents on one engine; crash-safe resume by contract; destination-pluggable observability with OTel + Langfuse, no SaaS lock-in; compile-time topology checks; spec + conformance). The docs homepage (`docs/index.md`) card grid carries the same five plus a sixth card retained from the previous grid for async-first / LLM-agnostic: workflows-to-agents, crash-safe, pluggable observability, bad-graphs-don't-compile, parallelism (fan-out + parallel-branches + nested correctness), async-first.
 - **Docs sweep: stale references and em-dash normalization.** Fixed three definite stale references (`spec_version='0.26.0'` in the Langfuse example output now reads `'0.38.0'`; the dangling `v0.16.1` qualifier dropped from the parallel-branches concept page; `compiled.attach_observer` corrected to `graph.attach_observer` in `non-obvious-shapes.md` for variable-name consistency with the rest of the docs). Swept em dashes out of the user-facing docs (130 instances across 17 files) per the convention set during the patterns expansion. mkdocs strict build clean; no broken intra-docs links.
 - **The checkpointing-and-migration example grows a crash-and-resume drama.** The first invoke of the v1 graph now hits a simulated transient failure inside `size_crew` (raises a `RuntimeError` on its first attempt only). The example catches `NodeException` at the `invoke()` boundary, prints what's saved on disk (`define_objective`'s position is already in `completed_positions`), then re-invokes with `resume_invocation=<id>`. The retried `size_crew` succeeds, `draft_timeline` runs, and the pipeline finishes - dramatizing the synchronous-checkpoint-by-contract reliability claim from the README pitch. The existing v1->v2 migration phase rides on top of the crash-survived checkpoint, so both reliability stories compose in one demo. Walk-through doc rewritten to cover both phases.

docs/examples/production-observability.md

@@ -1,4 +1,4 @@
-# Production observability with dual observers and timing middleware
+# Production observability with dual observers, timing middleware, and per-invocation cost rollup
 
 !!! info "Source"
     [https://github.com/LunarCommand/openarmature-python/blob/main/examples/production-observability/main.py](https://github.com/LunarCommand/openarmature-python/blob/main/examples/production-observability/main.py){target="_blank" rel="noopener"}
@@ -7,14 +7,19 @@ A single-turn lunar-mission Q&A endpoint instrumented the way you'd
 ship it: BOTH OTel and Langfuse observers attached to the same
 graph, caller hooks deriving domain-shaped `trace.input` /
 `trace.output` from State, the built-in `TimingMiddleware`
-recording per-node duration, and multi-tenant caller-supplied
-metadata propagating to both observers in one `invoke()` call.
+recording per-node duration, multi-tenant caller-supplied
+metadata propagating to both observers in one `invoke()` call, AND
+a third queryable-accumulator observer that a terminal `persist`
+node reads at request scope after synchronizing on the deliver
+loop with `drain_events_for`.
 
 ## Overview
 
-One node, one LLM call, two production-grade observability
-backends. The pipeline takes a question, calls the LLM, returns the
-answer. The interesting part is the observability wiring:
+Two nodes (`respond` then `persist`), one LLM call, three observers
+attached before invoke. The pipeline takes a question, calls the
+LLM, returns the answer, then synchronizes on the observer queue
+and rolls up token cost. The interesting part is the observability
+wiring:
 
 - `OTelObserver` attached with an `InMemorySpanExporter`
   (production swaps this for `BatchSpanProcessor` +
@@ -77,6 +82,23 @@ sees the same logical events represented two ways.
   `InMemorySpanExporter` records every Span. Production
   deployments swap each for a real exporter / SDK adapter; the
   observer call surface doesn't change.
+- **Queryable accumulator + `drain_events_for`**
+  ([queryable observer pattern](../concepts/observability.md)).
+  A third observer — `LlmUsageAccumulator` — subscribes to the
+  same event stream but only records the LLM-namespace events
+  carrying an `LlmEventPayload`. It accumulates per-invocation
+  token totals in memory, indexed by `current_invocation_id()`.
+  The terminal `persist` node calls
+  `await graph.drain_events_for(current_invocation_id(), timeout=2.0)`
+  to synchronize on the deliver loop, then reads the accumulator's
+  bucket and drops it. Without the drain, the bucket might be
+  missing the most-recent LLM event's tokens (the deliver loop
+  hasn't reached them yet). The `Observer` protocol itself stays
+  a single-callable shape; the accumulator just exposes its own
+  read methods (`get_bucket` / `drop`) that the persist node knows
+  about. This is the canonical shape for per-invocation cost
+  attribution at request scope, replacing the round-trip-through-
+  State workarounds that pre-v0.12.0 deployments used.
 
 ## How to run
 
@@ -105,13 +127,15 @@ request id:  <uuid>
 feature flag:v2-canary
 
 [timing] respond: 1234.5ms (success)
+[persist] LLM usage: prompt=42, completion=38, total=80 across 1 call(s)
 answer:      The primary objective of Apollo 11 was ...
 model:       gpt-4o-mini-2024-07-18
 
 --- captured OTel spans ---
-  [openarmature.invocation] 1240.0ms  openarmature.user.tenantId='demo-acme', ...
+  [openarmature.invocation] 1240.0ms  openarmature.graph.entry_node='respond', openarmature.graph.spec_version='0.46.0', openarmature.implementation.name='openarmature-python', openarmature.implementation.version='0.12.0'
   [respond] 1235.0ms  openarmature.node.name='respond', openarmature.user.tenantId='demo-acme', ...
-  [openarmature.llm.complete] 1200.0ms  gen_ai.system='openai', gen_ai.usage.input_tokens=42, ...
+  [openarmature.llm.complete] 1200.0ms  openarmature.user.tenantId='demo-acme', gen_ai.system='openai', gen_ai.usage.input_tokens=42, ...
+  [persist] 2.0ms  openarmature.node.name='persist', openarmature.user.tenantId='demo-acme', ...
 
 --- captured Langfuse trace ---
 Trace id=<uuid>
@@ -133,12 +157,33 @@ Trace id=<uuid>
   `TimingMiddleware` callback as soon as the respond chain returns.
   `outcome` is `"success"` here; a `ProviderRateLimit` would surface
   as `outcome="exception"` with `exception_category="provider_rate_limit"`.
+- **`[persist] LLM usage: ...`**: emitted by the `persist` node
+  after it drains the deliver loop and reads the
+  `LlmUsageAccumulator`'s bucket for this invocation. If the drain
+  times out (slow / hung observer), the persist line is prefixed by
+  a `[persist] drain incomplete: N events still pending after 2.0s`
+  surface — the production version of that log would also flip an
+  SLO-breach metric.
 - **OTel spans block**: one line per captured span, sorted by
   start time. The relevant attributes shown are a curated subset
   for readability; the full attribute set is on each `Span` object
-  for any reader inspecting them programmatically. Note the
-  `openarmature.user.*` attributes appearing on every span (the
-  cross-cutting attribute propagation from `invoke(metadata=...)`).
+  for any reader inspecting them programmatically. Note three
+  attribute families worth telling apart:
+    - The root `openarmature.invocation` span carries
+      `openarmature.graph.spec_version` plus the
+      `openarmature.implementation.name` / `.version` attribution
+      attributes. These are invocation-span-only (per spec §5.1) —
+      operators filtering by library version use these.
+    - The `openarmature.user.*` attributes appear on every span,
+      reflecting the cross-cutting propagation from
+      `invoke(metadata=...)`.
+    - `gen_ai.usage.*` lands on the LLM span only, sourced from the
+      provider's wire response.
+
+    The invocation span only lands in the exporter after the OTel
+    observer's `shutdown()` is called (closing the root span). The
+    demo calls it after `drain()` in the `finally` block; production
+    long-running processes call it at process exit.
 - **Langfuse trace block**: the same invocation as seen by the
   Langfuse data model. `trace.input` / `trace.output` come from the
   caller hooks (`{"question": ...}` / `{"answer": ..., "model": ...}`)