TokenHelm Observation

An enterprise AI Observability Platform that turns TokenHelm / tokenhelm-prompt usage events into cost, prompt, agent, workflow, session, and model analytics — built on a single canonical event contract that every layer agrees on.

The project is organized as four layers, each depending only on the one above it:

Observation Protocol      ← the canonical ObservationEvent contract (versioned, conformance-tested)
        ↓
Observation SDKs          ← emit protocol-compliant events (Python, TypeScript) — protocol parity
        ↓
Observation Platform      ← ingest → normalize → replay → analytics → dashboard (Next.js)
        ↓
Connectors                ← storage/event sources behind one interface (JSONL, DuckDB, Postgres)

This hierarchy is the defining architecture: the protocol is the source of truth, the SDKs produce it, the platform consumes it, and connectors move it — and every analytic is derived, never stored, so every number is reproducible from the raw immutable events.

The four layers

Layer	Location	What it is	Key docs
Observation Protocol	protocol/	The canonical ObservationEvent schema, the conformance kit (valid/invalid fixtures + manifest), and explicit Protocol / Schema / SDK version separation. The executable spec everything else conforms to.	protocol/README.md, ADR 0002, 0004
Observation SDKs	sdk/python/, sdk/typescript/	Idiomatic emitters that produce byte-identical protocol events. The contract guarantees protocol parity, not feature parity — both SDKs pass the same conformance suite.	ADR 0003
Observation Platform	frontend/	Next.js 15 / React 19 app: normalize → dedupe → replay → analytics (overview, prompts, agents, sessions, workflows, models, recommendations, alerts) → REST API → dashboard. Analytics are derived and reconciliation-gated.	ADR 0001, docs/architecture.md, docs/api.md
Connectors	frontend/lib/observation/*-source.ts	Pluggable EventSource backends behind one interface — jsonl (default), duckdb (scale), postgres (SQL). Each proves byte-identical analytics vs the JSONL fixtures before it ships.	docs/event-source-plugin.md, ADR 0005

The observe CLI (in sdk/typescript/src/cli/) is the protocol's command-line tooling — validate · lint · normalize · stats · replay · diff over any event stream.

Reference instrumentation (where the events come from)

The repo root is a working example application — three minimal Google ADK agent demos that emit real protocol events through cost_tracking.py (a TokenHelm BasePlugin):

Folder	Pattern	What it shows
single_agent/	Single LLM + tools	One agent calling Python functions; writing to session state
multi_agent/	Coordinator + delegation	An LLM routing requests to specialist sub-agents
pipeline_agent/	Sequential workflow	A fixed 3-stage pipeline passing data via output_key → {state}

cost_tracking.py is the canonical Python emitter: its after_model_callback fires on every model response (including tool round-trips and sub-agent delegation), folds reasoning tokens into output, prices each call, and appends an immutable ObservationEvent to usage_log.jsonl — the contract between the emitters and the platform.

Quickstart

Run the demos / emit events (ADK is pre-installed in .venv):

cp .env.example .env          # paste GOOGLE_API_KEY — https://aistudio.google.com/apikey
python run_demo.py "What's the weather in Tokyo?"
adk web .                     # interactive UI: pick an agent, inspect state/traces/events
python verify_tracking.py     # verify the tracking pipeline end-to-end, no API key needed

Run the platform (reads usage_log.jsonl):

cd frontend
npm install
npm run dev                   # http://localhost:3000
npm test                      # Vitest — offline analytics + reconciliation suite

Use the protocol CLI / SDKs:

# Python SDK
cd sdk/python && pip install -e . && python -m pytest

# TypeScript SDK + observe CLI
cd sdk/typescript && npm install && npm test
npx observe validate usage_log.jsonl

Switch storage connector (zero analytics changes):

EVENT_SOURCE=postgres PG_DSN=postgres://… npm run start   # or EVENT_SOURCE=duckdb

Governance

Development is gated by the CI workflow (.github/workflows/ci.yml) with five jobs — python-sdk, typescript-sdk, frontend, platform-verification, and the aggregate gates job that all PRs into main must pass. Architectural decisions are recorded as ADRs under docs/adr/; the five reconciliation identities (decimal-exact, replay-deterministic) are enforced as tests, not conventions. New connectors must prove byte-identical analytics against the JSONL fixtures before merging.

Notes

• Model: the demo agents use gemini-3-flash-preview; pricing in pricing.yaml is placeholder — replace with official rates before trusting dollar figures.
• Cost output: adk web prints tokenhelm's cost line to the terminal, not the browser; the trace view shows token counts only. Restart adk web after editing tracking wiring (module cache).
• Immutability: the platform never writes to usage_log.jsonl — emitters append, the platform reads. Every metric is reproducible by replaying the raw events.