Render rerank observability (0060b)#207
Merged
Merged
Conversation
Render the typed rerank events in both bundled observers, replacing the safe-skip guards: the OTel openarmature.rerank.complete span (observability §5.5.13) and the Langfuse Retriever observation (§8.4.7, a new retriever() client/adapter surface + the searchUnits usageDetails convention on LangfuseUsage). Usage figures emit conditionally per proposal 0093; the span renders the scored results (RerankEvent.output_results per 0089) and is not gated by disable_llm_spans. Un-defer conformance fixtures 099-108 / 138 / 141 / 142 (109 metrics rides 0067). conformance.toml 0060 and 0089 move to implemented, completing the rerank capability and the 0089 output-rendering surface.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Completes proposal 0060 “rerank” observability by wiring typed rerank events into the bundled OTel and Langfuse observers, updating conformance to un-defer rerank observability fixtures, and marking proposals 0060/0089 as implemented.
Changes:
- Add OTel
openarmature.rerank.completespan emission fromRerankEvent/RerankFailedEvent(with payload + GenAI-semconv gating). - Add Langfuse “Retriever” observation emission (asType
"retriever") plussearchUnitsusageDetails support. - Update conformance harness + proposal status/notes to reflect implemented rerank observability fixtures.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/conformance/test_observability.py | Un-defer rerank observability fixtures; add rerank fixture harness and usageDetails assertions. |
| src/openarmature/observability/otel/observer.py | Render rerank spans from typed rerank events. |
| src/openarmature/observability/langfuse/observer.py | Render Langfuse Retriever observations from typed rerank events. |
| src/openarmature/observability/langfuse/client.py | Extend Langfuse client protocol/in-memory client with retriever observation + search_units usage field. |
| src/openarmature/observability/langfuse/adapter.py | Map retriever observation to Langfuse SDK (as_type="retriever") and searchUnits usage_details. |
| conformance.toml | Mark proposals 0060/0089 implemented and update proposal notes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The 0060 conformance note said Langfuse usageDetails is an empty map when usage is null, but the observer omits usageDetails entirely (usage=None, so the adapter adds no usage_details), which fixture 142 asserts. Correct the note to 'omitted when usage = null'.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the rerank capability (proposal 0060) with the observability half, the second of the two-PR split (0060a shipped the capability + events + reference reranker in #205). The bundled observers no longer safe-skip the typed rerank events.
What's here
_handle_rerank-> theopenarmature.rerank.completespan (observability section 5.5.13): the GenAI semconv subset +openarmature.rerank.{query_length, document_count, top_k, result_count}(top_komitted when None), conditionalgen_ai.usage.input_tokensandopenarmature.rerank.search_units(per proposal 0093), and payload-gatedquery/documents/results(the scored results fromRerankEvent.output_resultsper 0089). Parented lineage-aware, back-dated bylatency_ms, not gated bydisable_llm_spans._handle_rerank-> a dedicated Retriever observation (section 8.4.7,asType="retriever"):model+usageDetails.{input, searchUnits}+openarmature_*metadata + payload-gatedinput/output; ERROR-level onRerankFailedEventwitherror_categoryas the status message and no output.retriever()on theLangfuseClientProtocol +InMemoryLangfuseClient+ the SDK adapter (as_type="retriever"/ nativeLangfuseRetriever), and asearch_unitsfield onLangfuseUsagefor thesearchUnitsusageDetails convention.conformance.toml: 0060 and 0089 move toimplemented.Validation
tests/conformance/test_observability.py: 92 passed (the 13-fixture rerank family; 109 the only rerank skip)/review(six angles + live-SDK checks): no correctness defects. Follow-up cleanup (harness DRY, coverage tightening, src-side observer/adapter de-duplication) tracked separately.