fix: improve OpenClaw observability cost consistency#206
Conversation
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
github-actions
Bot
added
size:M
WalkthroughThis PR extends metric extraction and aggregation across ATIF and OpenInference observability modules to recognize, parse, and emit LLM cost fields ( ChangesLLM Cost Metrics Extraction and Aggregation
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/core/src/observability/atif.rs (1)
627-632:⚠️ Potential issue | 🟠 Major | ⚡ Quick winPreserve nested provider cost fields instead of dropping them.
Marking
"cost"as a fully known key strips the entire nested object frommetrics.extra, but this function only promotescost.totalintocost_usd. A payload like{"cost":{"total":0.0042,"input":0.003,"output":0.0012}}will now lose the provider-reported breakdown in ATIF.Suggested fix
const TOKEN_USAGE_KNOWN_KEYS: &[&str] = &[ "prompt_tokens", "completion_tokens", "cached_tokens", "cost_usd", - "cost", "prompt_token_ids", "completion_token_ids", "logprobs", ];Also applies to: 655-683
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/core/src/observability/atif.rs` around lines 627 - 632, Remove "cost" from TOKEN_USAGE_KNOWN_KEYS so the nested cost object is not treated as a fully-known key and dropped; instead, in the code path that promotes cost.total into the flat field (the logic that sets metrics.extra["cost_usd"] from a nested cost.total), explicitly read metrics.extra["cost"]["total"] (or the incoming payload's cost.total), set metrics.extra["cost_usd"] to that value if present, and leave the original metrics.extra["cost"] object intact. Update both TOKEN_USAGE_KNOWN_KEYS and the promotion logic (the block that extracts cost.total into cost_usd) so the provider breakdown (e.g., cost.input/cost.output) is preserved.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@crates/core/src/observability/atif.rs`:
- Around line 627-632: Remove "cost" from TOKEN_USAGE_KNOWN_KEYS so the nested
cost object is not treated as a fully-known key and dropped; instead, in the
code path that promotes cost.total into the flat field (the logic that sets
metrics.extra["cost_usd"] from a nested cost.total), explicitly read
metrics.extra["cost"]["total"] (or the incoming payload's cost.total), set
metrics.extra["cost_usd"] to that value if present, and leave the original
metrics.extra["cost"] object intact. Update both TOKEN_USAGE_KNOWN_KEYS and the
promotion logic (the block that extracts cost.total into cost_usd) so the
provider breakdown (e.g., cost.input/cost.output) is preserved.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 82c3639d-54b7-4ebe-b236-363143f52dcd
📒 Files selected for processing (7)
crates/core/src/observability/atif.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/atif_tests.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/observability/openinference_tests.rsdocs/supported-integrations/openclaw-plugin.mdxintegrations/openclaw/test/llm-replay.test.ts
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Check / Run
- GitHub Check: Preview docs
🧰 Additional context used
📓 Path-based instructions (28)
{docs/**,README.md,CONTRIBUTING.md}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
{docs/**,README.md,CONTRIBUTING.md}: For docs-only changes, run targeted checks only if commands, package names, or examples changed. Usejust docsfor docs-site builds andjust docs-linkcheckwhen links changed
Run docs site build withjust docs
Files:
docs/supported-integrations/openclaw-plugin.mdx
{docs/**,README.md,CONTRIBUTING.md,**/*.md}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
Run docs link validation with
just docs-linkcheckwhen links change
Files:
docs/supported-integrations/openclaw-plugin.mdx
{docs/**,README.md}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
Verify README and docs entry points still match current package names and paths for large or public-facing changes
Files:
docs/supported-integrations/openclaw-plugin.mdx
{docs/**,examples/**,README.md}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
Verify examples still run with documented commands for large or public-facing changes
Files:
docs/supported-integrations/openclaw-plugin.mdx
{docs/**,README.md,**/Cargo.toml,**/package.json,**/*.md}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
Ensure renamed public surfaces are reflected consistently in manifests and docs for large or public-facing changes
Files:
docs/supported-integrations/openclaw-plugin.mdx
**/*.{md,mdx,py,sh,yaml,yml,toml,json}
📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md)
Keep package names, repo references, and build commands current
Files:
docs/supported-integrations/openclaw-plugin.mdx
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md)
In MDX files, top-of-file comments must use JSX comment delimiters: {/* to open and */} to close. Do not use HTML comments for MDX SPDX headers.
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
Files:
docs/supported-integrations/openclaw-plugin.mdx
**/*.{html,md,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Include SPDX license header in HTML and Markdown files using HTML comment syntax
Files:
docs/supported-integrations/openclaw-plugin.mdx
docs/**/*.{md,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Update embedded documentation snippets, patch docs, and binding-support notes if examples or supported bindings changed
Files:
docs/supported-integrations/openclaw-plugin.mdx
docs/**
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run
just docsor./scripts/build-docs.sh htmlto regenerate ignored Fern API reference pages before validation for documentation site changes
Files:
docs/supported-integrations/openclaw-plugin.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/supported-integrations/openclaw-plugin.mdx
**/*{test,spec,smoke}.{js,ts,py}
📄 CodeRabbit inference engine (.agents/skills/contribute-integration/SKILL.md)
Relevant integration tests or smoke path must pass
Files:
integrations/openclaw/test/llm-replay.test.ts
{crates/adaptive/**/*.rs,**/*test*.{rs,py,go,ts,js},**/*adaptive*test*.{rs,py,go,ts,js},docs/plugins/adaptive/**}
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Maintain documented and tested validation and report behavior for adaptive surfaces
Files:
integrations/openclaw/test/llm-replay.test.tscrates/core/tests/unit/atif_tests.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/observability/atof_tests.rs
**/*.{wasm,js,ts}{,x}
📄 CodeRabbit inference engine (.agents/skills/maintain-packaging/SKILL.md)
Ensure WebAssembly package naming conventions are consistent with generated package expectations and downstream consumption
Files:
integrations/openclaw/test/llm-replay.test.ts
**/*.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
Run Node.js formatting with
npm run format --workspace=nemo-relay-nodeInclude SPDX license header in all JavaScript and TypeScript source files using double-slash comment syntax
Files:
integrations/openclaw/test/llm-replay.test.ts
**/*.{rs,py,js,ts,tsx,jsx,go,sh,toml,yaml,yml,md}
📄 CodeRabbit inference engine (AGENTS.md)
Keep SPDX headers on source, docs, scripts, and configuration files. The project is Apache-2.0.
Files:
integrations/openclaw/test/llm-replay.test.tscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/observability/atof_tests.rs
**/*.{rs,py,go,js,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions: Rust and Python use
snake_case, C FFI exports prefixednemo_relay_, Go usesPascalCasefor public APIs, Node.js usescamelCase.
Files:
integrations/openclaw/test/llm-replay.test.tscrates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/observability/atof_tests.rs
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Use
snake_casenaming convention for Rust identifiers (e.g.,nemo_relay_tool_call)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node work
**/*.rs: Runcargo fmt --allto format all Rust code
Runcargo clippy --workspace --all-targets -- -D warningsto enforce all clippy lints as errors
**/*.rs: Runcargo fmt --allwhen Rust files changed as part of WebAssembly work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files changed as part of WebAssembly work
**/*.rs: If any Rust code changed, always runjust test-rust
If any Rust code changed, also runcargo fmt --all
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings
Run Rust formatting withcargo fmt --all
Run Rust linting withcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Usecargo fmtfor Rust code formatting
Runcargo clippy -- -D warningsto lint Rust code and treat all warnings as errors
Use Rust snake_case naming convention for Rust identifiers
Include SPDX license header in all Rust source files using double-slash comment syntax
Validate Rust code withuv run pre-commit run --all-filesto enforce cargo fmt formatting check, cargo clippy lints, and cargo deny aud...
Files:
crates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/observability/atof_tests.rs
**/{Cargo.toml,**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/maintain-packaging/SKILL.md)
Maintain consistency between Rust package names in
Cargo.tomland their actual usage across the codebase
Files:
crates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/observability/atof_tests.rs
**/*.{h,hpp,c,cpp,rs}
📄 CodeRabbit inference engine (.agents/skills/maintain-packaging/SKILL.md)
Ensure FFI header and library naming follows consistent conventions across platform-specific builds
Files:
crates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/observability/atof_tests.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/observability/atof_tests.rs
**/*.{rs,toml}
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update Rust crate names and module prefixes during coordinated rename operations
Files:
crates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/observability/atof_tests.rs
crates/core/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/coreor shared runtime semantics, also usevalidate-changefor broader validation
crates/core/**/*.rs: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths. Keep errors explicit and binding-appropriate at the wrapper layer.
Files:
crates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/observability/atof_tests.rs
crates/{core,adaptive}/**
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full matrix across Rust, Python, Go, Node.js, and WebAssembly
Files:
crates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/observability/atof_tests.rs
crates/**/*.rs
📄 CodeRabbit inference engine (AGENTS.md)
crates/**/*.rs: Keep async behavior on the existing tokio-based model. Bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
UseJson = serde_json::Valuein Rust-facing runtime APIs for JSON payload handling.
Files:
crates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/observability/atof_tests.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/core/tests/unit/atif_tests.rscrates/core/src/observability/atif.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/src/observability/openinference.rscrates/core/tests/unit/observability/atof_tests.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/core/tests/unit/atif_tests.rscrates/core/tests/unit/observability/openinference_tests.rscrates/core/tests/unit/observability/atof_tests.rs
crates/core/src/observability/{atif,otel,openinference}.rs
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
When changing event fields in ATIF, OpenTelemetry, or OpenInference observability surfaces, keep the core event model in
crates/core/src/observability/atif.rs,crates/core/src/observability/otel.rs, andcrates/core/src/observability/openinference.rsin sync
Files:
crates/core/src/observability/atif.rscrates/core/src/observability/openinference.rs
🧠 Learnings (1)
📚 Learning: 2026-05-07T18:04:44.387Z
Learnt from: mnajafian-nv
Repo: NVIDIA/NeMo-Flow PR: 67
File: integrations/openclaw/src/modules.ts:1-2
Timestamp: 2026-05-07T18:04:44.387Z
Learning: In NVIDIA/NeMo-Flow, TypeScript source files should use `//` line comments for SPDX headers (e.g., `// SPDX-FileCopyrightText: ...` and `// SPDX-License-Identifier: ...`) rather than C-style block comments (`/* ... */`). The repo’s copyright checker enforces this mapping, so `//` SPDX headers in `.ts` files should not be flagged as a style violation.
Applied to files:
integrations/openclaw/test/llm-replay.test.ts
🔇 Additional comments (13)
crates/core/src/observability/openinference.rs (4)
690-692: LGTM!
703-707: LGTM!
712-734: LGTM!
735-755: LGTM!crates/core/tests/unit/observability/atof_tests.rs (3)
304-305: LGTM!Also applies to: 413-413
339-340: LGTM!Also applies to: 422-422
376-377: LGTM!Also applies to: 437-437
crates/core/tests/unit/observability/openinference_tests.rs (4)
695-696: LGTM!Also applies to: 722-722
772-773: LGTM!Also applies to: 807-807
1503-1503: LGTM!
1554-1555: LGTM!Also applies to: 1602-1605
docs/supported-integrations/openclaw-plugin.mdx (1)
284-287: LGTM!integrations/openclaw/test/llm-replay.test.ts (1)
128-128: LGTM!Also applies to: 186-186
mnajafian-nv
added
Improvement
| NeMo Relay does not infer provider pricing; cost attributes are emitted only | ||
| when OpenClaw supplies explicit cost fields. |
There was a problem hiding this comment.
I think we should preemptively omit this.
| NeMo Relay does not infer provider pricing; cost attributes are emitted only | |
| when OpenClaw supplies explicit cost fields. |
2 participants
Overview
This PR propagates provider-reported cost through ATIF, ATOF, OpenInference, and OpenClaw replay outputs. It does not infer pricing from token counts.
Details
llm.cost.totalattribute.cost_usd.Where should the reviewer start?
Start with
crates/core/src/observability/atif.rsandcrates/core/src/observability/openinference.rs, then review the focused coverage incrates/core/tests/unit/atif_tests.rs,crates/core/tests/unit/observability/openinference_tests.rs, andintegrations/openclaw/test/llm-replay.test.ts.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Follow-up Work
The remaining OpenClaw observability items stay as follow-ups rather than expanding this PR:
llm.input_messages.*,llm.output_messages.*, andllm.tools.*coverage beyond readable summaries.Summary by CodeRabbit