Skip to content

feat(governance,metrics): live cost attribution + pre-call budget projection (feat-026)#31

Merged
kjoshi07 merged 8 commits into
mainfrom
feat-026-cost-attribution-and-projection
Jun 20, 2026
Merged

feat(governance,metrics): live cost attribution + pre-call budget projection (feat-026)#31
kjoshi07 merged 8 commits into
mainfrom
feat-026-cost-attribution-and-projection

Conversation

@kjoshi07

@kjoshi07 kjoshi07 commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

Implements feat-026 (spec on main). The live + pre-call delta over what already ships — both pure config over existing surfaces, no agent-code change.

Live attributed-cost metrics (forgesight-core)

  • forgesight.cost.attributed_usd counter keyed by the stamped ownership dimensions (team/owner/…) — the live counterpart of feat-022's offline chargeback rollup. A record missing a dimension buckets to <unattributed>.
  • forgesight.cost.budget_utilization gauge (spend/cap). The instrument is a generic gauge in core; the budget semantics stay in governance, which records through get_runtime().metrics — core never depends on governance (P1).
  • AttributionMetricsConfig on MetricConfig; default off (P2).

Pre-call budget projection (forgesight-governance)

  • BudgetInterceptor gains a projection mode + ProjectionConfig. A new precall() estimates the call's cost from caller-declared projected_tokens via the same PricingProvider and denies before the spend if a cap would breach — closing feat-020's post-hoc gap.
  • It's a guard, not a commit: never writes the running totals (actual cost on the completed record stays the sole writer), so a conservative over-estimate can't inflate the accumulator.
  • output_token_estimate ∈ max_tokens|input_ratio|fixed; on_unpriced ∈ allow|deny.

The minimal core hook

Runtime.run_precall_interceptors runs only interceptors exposing precall() — so policy/kill-switch behaviour is unchanged — re-raising GovernanceSignal to veto. LLMScope._enter invokes it on a start record when the caller passes projected_tokens; a deny propagates out of __enter__ (the call body never runs) and the run record still flushes (feat-020 invariant).

Charter

Rides locked surfaces only (cost attr, metadata mechanism, PricingProvider SPI, Interceptor/GovernanceSignal); custom metrics namespaced forgesight.cost.* (never gen_ai.*, P4/ADR-0005); both capabilities default off.

Validation (local, full gate)

  • ruff + mypy --strict clean, pytest 97.7%; all changed source files 96–100% covered.
  • Tests: attributed metric + <unattributed> bucket + disabled; gauge setter; projection deny/allow + estimate modes + on_unpriced + guard-not-commit + from_config; integration deny-before-call (body never runs) + projection-off behaves as feat-020 + live budget_utilization.

🤖 Generated with Claude Code

kjoshi07 and others added 8 commits June 20, 2026 11:14
@kjoshi07 @claude
feat(governance,metrics): live cost attribution + pre-call budget pro…
2442a0c 
…jection (feat-026)

The live + pre-call delta over what already ships — both config over existing
surfaces, no agent-code change.

Live attributed-cost metrics (forgesight-core):
- forgesight.cost.attributed_usd counter keyed by the stamped ownership
  dimensions (team/owner/…), the live counterpart of feat-022's offline
  chargeback rollup. A record missing a dimension buckets to <unattributed>.
- forgesight.cost.budget_utilization gauge (spend/cap). The instrument is a
  generic gauge in core; the budget semantics stay in governance, which records
  through the runtime's metrics — core never depends on governance (P1).
- AttributionMetricsConfig on MetricConfig; default off (P2).

Pre-call budget projection (forgesight-governance):
- BudgetInterceptor gains a projection mode + ProjectionConfig. A new precall()
  estimates the call's cost from caller-declared projected_tokens via the same
  PricingProvider and denies BEFORE the spend if a cap would breach — closing
  feat-020's post-hoc gap. It is a GUARD: it never commits to the running totals
  (actual cost on the completed record stays the sole writer).
- output_token_estimate max_tokens|input_ratio|fixed; on_unpriced allow|deny.

Minimal core hook: Runtime.run_precall_interceptors runs only interceptors that
expose precall() (so policy/kill-switch behaviour is unchanged), re-raising
GovernanceSignal to veto. LLMScope._enter runs it on a start record when the
caller passes projected_tokens; a deny propagates out of __enter__ (the call body
never runs) and the run record still flushes.

Rides locked surfaces only (cost attr, metadata mechanism, PricingProvider SPI,
Interceptor/GovernanceSignal); both capabilities default off. Full gate green
(97.7%); changed files 96-100% covered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kjoshi07 @claude
fix(otel,docs): OTLP/HTTP endpoint path + compose tag; add Bedrock e2…
c9ce10a 
…e example

End-to-end validation against real backends surfaced two bugs, fixed here:

- forgesight-otel: the OTLP/HTTP exporter does not append the /v1/traces signal
  path when endpoint is set explicitly, so `endpoint: http://host:4318` 404s.
  Append it automatically for http when the caller gives a base URL (an explicit
  path is kept). Now `http://localhost:4318` "just works"; gRPC still uses :4317
  + protocol="grpc". Tested.
- docker-compose.yml: jaeger tag `1.62` does not exist on Docker Hub -> `1.62.0`.
- docs: README quick-start and playbooks 02/04 pointed OTLP/HTTP at the gRPC port
  :4317 with the default http protocol; corrected to :4318.

examples/bedrock-e2e/: a real AWS Bedrock (Claude Haiku 4.5) call instrumented
with ForgeSight, exporting traces to Jaeger (OTLP) + metrics to Prometheus, with
the feat-023 audit trail and feat-026 attributed-cost/budget-utilization metrics,
then verifying each landed. Validated live: 4-span GenAI-semconv trace in Jaeger
with cost, attributed cost by team/owner, a tamper-evident audit chain, and
Prometheus scrape.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kjoshi07 @claude
chore(examples,stack): Grafana + 3 Bedrock agent examples (e2e valida…
83a53a5 
…ted)

Stack: add a Grafana service to docker-compose with provisioned datasources
(Prometheus + Jaeger) and a starter "ForgeSight — agent telemetry" dashboard
(runs, cost, failures, p95 latency, cost-by-provider, token usage). UI at :3000,
anonymous admin.

examples/agents/: three self-contained, real AWS Bedrock agents instrumented with
ForgeSight, exporting end-to-end to the local stack (traces → Jaeger, metrics →
Prometheus/Grafana, audit trail, attributed-cost metrics):
- react_agent  — ReAct loop: run → step(iteration) → llm_call/tool_call
- rag_agent    — retrieve (vector-search tool) → generate (LLM)
- multi_agent  — supervisor delegating to nested researcher/writer sub-agent runs

All three validated live against the stack: correct answers, hash-chained audit
intact, and proper nested span trees in Jaeger (the multi-agent run exports as one
7-span tree across three agents).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kjoshi07 @claude
fix(langfuse): OTLP/HTTP endpoint needs the /v1/traces signal path
d7b1dd1 
The Langfuse exporter posted to {host}/api/public/otel (the base), which 404s —
Langfuse's OTLP/HTTP ingest is the /v1/traces signal under it. Point at
{host}/api/public/otel/v1/traces via a tested `otlp_traces_endpoint` helper.

Verified live against Langfuse Cloud: a Bedrock run now lands as a trace with
cost + 2 observations (confirmed through the Langfuse public API).

Also: gitignore .env / .env.* so local secrets are never committed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kjoshi07 @claude
chore(examples): expose run() per agent + a long-lived demo_all runner
53aaf09 
Why: each agent example is a short-lived process — it serves Prometheus metrics
on :9464 for only seconds and its counters reset per process, so Prometheus (pull,
5s scrape) rarely captures them and the Grafana dashboard under-counts (it showed
"1 run" for several runs). Traces in Jaeger are always complete; only the pull
metrics need a long-lived target.

- Each agent now exposes run(client) (the body), reused by main() and by demo_all.
- demo_all configures ForgeSight once, runs all agents under one runtime (metrics
  accumulate), and keeps :9464 up ~30s so Prometheus scrapes the real totals.

Validated: while demo_all is alive, sum(agentforge_agent_runs_total)=5 (react + rag
+ supervisor/researcher/writer); it goes stale the instant the process exits —
demonstrating the pull-vs-short-lived-job issue and the fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kjoshi07 @claude
ci(secret-scan) + docs: gitleaks in CI; document audit/cost/Grafana/e…
e50905e 
…xamples

CI: add a `secret-scan` job (gitleaks over full git history, via its Docker image)
so no credential can land before/after the repo goes public. Verified locally:
70 commits scanned, no leaks; .env was never committed (and is now gitignored).

Docs (drift from feat-023/026 + the stack/examples work):
- README: add forgesight-audit + live attributed-cost/pre-call-projection to the
  capability table, extras table, and packages list; point the local-stack callout
  at the full stack incl. Grafana; link the new examples/agents (Bedrock) examples.
- New runbook: docs/runbooks/audit-trail.md (feat-023) + index row.
- Playbook 03: add Grafana (datasources + dashboard), the OTLP/HTTP base-URL note,
  and the short-lived-process → Prometheus under-count gotcha with the demo_all fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kjoshi07 @claude
docs(branding): new punchline + richer PyPI description/keywords/clas…
e115cc1 
…sifiers

Lead line (README hero + forgesight PyPI summary):
"Instrument any AI agent in a few lines — then ship traces, cost, metrics, evals,
budgets, and a tamper-evident audit trail to any backend by changing one line of
config. OpenTelemetry-first. Vendor-neutral. Never an agent-code change."

- forgesight (facade): punchy one-line description; keywords expanded
  (agent-observability, llm-observability, otel, genai, tracing, cost, finops,
  audit, governance); add Topic/Intended-Audience classifiers.
- All 16 packages: add Intended Audience :: Information Technology + Topic ::
  System :: Monitoring + Topic :: Scientific/Engineering :: Artificial Intelligence
  for PyPI discoverability.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kjoshi07 @claude
docs(readme): add a 30s proof-it-works demo GIF
bac5303 
examples/demo.py — a fully offline, self-contained demo (no backend/AWS/network):
instruments a 'pr-reviewer' agent and prints the captured trace tree, derived cost,
and a verified tamper-evident audit chain in ~1s. Recorded to docs/assets/demo.gif
with VHS (docs/assets/demo.tape regenerates it). Embedded at the top of the README.

The same telemetry ships to any backend by changing one config line — the demo just
captures in-memory so it runs anywhere.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kjoshi07 kjoshi07 merged commit e8c03d4 into main Jun 20, 2026
5 checks passed
@kjoshi07 kjoshi07 deleted the feat-026-cost-attribution-and-projection branch June 20, 2026 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kjoshi07