Debug: Anthropic cache token cost investigation

python/scripts/debug-anthropic-cache-cost/README.md

@@ -0,0 +1,24 @@
+# Debug: Anthropic Cache Token Cost
+
+Reproduction script for investigating Anthropic cache token cost discrepancies in the PostHog Python SDK.
+
+## The Issue
+
+When using `posthog.ai.anthropic.AsyncAnthropic` with Anthropic's prompt caching, the stored `$ai_input_tokens` can end up as the **inclusive** value (input + cache_read tokens) instead of the **exclusive** value that the Anthropic API returns. This causes the cost calculator to overcharge by treating all tokens (including cached ones) at the full prompt rate.
+
+## What This Script Tests
+
+1. **Raw Anthropic API** - Confirms the API returns `input_tokens` exclusive of cached tokens
+2. **PostHog SDK wrapper** - Confirms the SDK correctly passes through the exclusive value
+3. **`posthog_properties` override** - Demonstrates how passing custom properties can override the SDK's correct values
+
+## Usage
+
+```bash
+cd python
+python scripts/debug-anthropic-cache-cost/run.py
+```
+
+Requires `ANTHROPIC_API_KEY` and `POSTHOG_API_KEY` in the root `.env` file.
+
+Results are written to `output.md` (gitignored) in this directory.

python/scripts/debug-anthropic-cache-cost/output.md

@@ -0,0 +1,323 @@
+# Anthropic Cache Token Cost - Debug Results
+
+**Date:** 2026-02-25 11:30 UTC
+**Model:** `claude-haiku-4-5-20251001`
+**PostHog Python SDK:** `v7.9.3`
+**Anthropic SDK:** `v0.75.0`
+
+---
+
+## TL;DR
+
+The PostHog Python SDK (`posthog.ai.anthropic.AsyncAnthropic`) correctly passes through the **exclusive** `input_tokens` value from the Anthropic API. The SDK does **not** modify or inflate this value.
+
+However, if you pass `posthog_properties` to `messages.create()` containing `$ai_input_tokens` or `$ai_total_tokens`, those values **override** the SDK's correct values. This can cause inflated cost calculations if the overriding values include cached tokens in the input count.
+
+---
+
+## Test 1: Raw Anthropic API Behavior
+
+Verified that the Anthropic API returns `input_tokens` **exclusive** of cached tokens:
+
+| Field | Value |
+|-------|-------|
+| `input_tokens` | 13 |
+| `cache_read_input_tokens` | 14100 |
+| `cache_creation_input_tokens` | 14100 |
+| `output_tokens` | 42 |
+
+**Result:** `input_tokens` (13) is much smaller than `cache_read_input_tokens` (14100), confirming Anthropic returns **exclusive** counts.
+
+---
+
+## Test 2: PostHog SDK Wrapper (No Custom Properties)
+
+Spied on `ph_client.capture()` to see what the SDK sends to PostHog:
+
+| Field | Anthropic API | PostHog SDK |
+|-------|--------------|-------------|
+| `input_tokens` / `$ai_input_tokens` | 13 | 13 |
+| `cache_read_input_tokens` / `$ai_cache_read_input_tokens` | 14100 | 14100 |
+| `$ai_total_tokens` | N/A | NOT SET |
+
+**Result:** SDK correctly passes through the exclusive value.
+
+The Anthropic wrapper does NOT set `$ai_total_tokens` - this is expected.
+
+---
+
+## Test 3: `posthog_properties` Override (Root Cause)
+
+### 3a) Direct Override Test
+
+Passed `posthog_properties={"$ai_input_tokens": 99999}` to `messages.create()`.
+
+**Result:** The SDK's correct value was **overridden** to 99999. This confirms `posthog_properties` takes precedence.
+
+### 3b) Simulated Customer Pattern
+
+Simulated what happens when a customer computes inclusive token counts and passes them via `posthog_properties`:
+
+```python
+# Customer code pattern that causes the overcharge:
+inclusive_input = response.usage.input_tokens + response.usage.cache_read_input_tokens
+total = inclusive_input + response.usage.output_tokens
+
+client.messages.create(
+    ...,
+    posthog_properties={
+        "$ai_input_tokens": inclusive_input,   # WRONG - includes cached tokens!
+        "$ai_total_tokens": total,
+    },
+)
+```
+
+| Field | Correct (Exclusive) | Wrong (Inclusive) |
+|-------|-------------------|-----------------|
+| `$ai_input_tokens` | 12 | 14113 |
+| `$ai_cache_read_input_tokens` | 14100 | 14100 |
+| `$ai_total_tokens` | not set | 14160 |
+
+### Cost Impact
+
+Using `claude-haiku-4-5` pricing ($1/Mtok input, $0.10/Mtok cache read):
+
+| | Correct | Overcharged |
+|--|---------|-------------|
+| Uncached input cost | $0.001422 | $0.015523 |
+| **Overcharge factor** | | **10.9x** |
+
+---
+
+## How to Check if This Affects You
+
+Look for any code that passes `posthog_properties` to `messages.create()` with token-related fields:
+
+```python
+# Search your codebase for patterns like:
+client.messages.create(
+    ...,
+    posthog_properties={
+        "$ai_input_tokens": ...,    # This overrides the SDK!
+        "$ai_total_tokens": ...,    # This also overrides!
+    },
+)
+```
+
+The SDK already extracts the correct exclusive `input_tokens` from the Anthropic response. You do **not** need to pass these values yourself.
+
+### What to Do
+
+**Option A (Recommended):** Remove `$ai_input_tokens` and `$ai_total_tokens` from your `posthog_properties`. The SDK handles these correctly.
+
+**Option B:** If you need to pass custom properties, make sure `$ai_input_tokens` uses the **exclusive** value from `response.usage.input_tokens` (not `input_tokens + cache_read_input_tokens`).
+
+---
+
+## Test 4: SDK Source Code Inspection
+
+Programmatically inspected the PostHog Python SDK source to prove `$ai_total_tokens` cannot come from the Anthropic code path:
+
+| Component | File | Sets `$ai_total_tokens`? |
+|-----------|------|------------------------|
+| Anthropic converter | `posthog/ai/anthropic/anthropic_converter.py` | No |
+| General AI utils | `posthog/ai/utils.py` | No |
+| OpenAI Agents processor | `posthog/ai/openai_agents/processor.py` | **Yes** (only place) |
+
+**The Anthropic converter** (`extract_anthropic_usage_from_response`, lines 206-231) only extracts:
+- `input_tokens` (exclusive of cache)
+- `output_tokens`
+- `cache_read_input_tokens`
+- `cache_creation_input_tokens`
+- `web_search_count`
+
+**The general utils** (`posthog/ai/utils.py`) only tags `$ai_input_tokens` and `$ai_output_tokens`. There is no `tag("$ai_total_tokens", ...)` anywhere in this file.
+
+**The only code that sets `$ai_total_tokens`** is in `posthog/ai/openai_agents/processor.py` (lines 539 and 696), which is exclusively for OpenAI Agents — a completely separate code path from the Anthropic wrapper.
+
+**Result:** If `$ai_total_tokens` appears in a stored event captured via `posthog.ai.anthropic.AsyncAnthropic`, it **must** have come from outside the SDK — most likely via `posthog_properties`.
+
+---
+
+## Key Evidence
+
+1. `$ai_total_tokens` is present in the stored events, but the PostHog Anthropic wrapper **never sets** this property (confirmed by source inspection in Test 4). Only OpenAI Agents sets it. This strongly suggests the value comes from `posthog_properties`.
+
+2. The stored `$ai_input_tokens` (48268) minus `$ai_cache_read_input_tokens` (45417) equals exactly the expected exclusive value (2851), which is what the Anthropic API returns as `input_tokens`.
+
+---
+
+## Raw Script Output
+
+<details>
+<summary>Click to expand full script output</summary>
+
+```
+======================================================================
+  Anthropic Cache Token Cost Debug Script
+  Investigating: $ai_input_tokens inclusive vs exclusive
+======================================================================
+
+======================================================================
+  STEP 1: Raw Anthropic API (no PostHog wrapper)
+======================================================================
+  Making two calls to populate cache, then checking usage on second call...
+
+  Call 1 (cache miss - should create cache)...
+
+  Call 1 usage:
+    input_tokens:                13
+    output_tokens:               39
+    cache_read_input_tokens:     0
+    cache_creation_input_tokens: 14100
+
+  Call 2 (cache hit expected)...
+
+  Call 2 usage:
+    input_tokens:                13
+    output_tokens:               42
+    cache_read_input_tokens:     14100
+    cache_creation_input_tokens: 0
+
+  Analysis:
+    input_tokens (from API):       13
+    cache_read_input_tokens:       14100
+    input + cache_read:            14113
+    => input_tokens is EXCLUSIVE of cache (correct Anthropic behavior)
+
+======================================================================
+  STEP 2: PostHog AsyncAnthropic wrapper (spy on capture)
+======================================================================
+
+  Call 1 (cache miss - populating cache)...
+
+  [SPY] ph_client.capture() called!
+    event:                         $ai_generation
+    $ai_provider:                  anthropic
+    $ai_model:                     claude-haiku-4-5-20251001
+    $ai_input_tokens:              13
+    $ai_output_tokens:             35
+    $ai_cache_read_input_tokens:   14100
+    $ai_cache_creation_input_tokens: None
+    $ai_total_tokens:              NOT SET
+
+  Raw API usage (call 1):
+    input_tokens:                13
+    output_tokens:               35
+    cache_read_input_tokens:     14100
+    cache_creation_input_tokens: 0
+
+  Call 2 (cache hit expected)...
+
+  [SPY] ph_client.capture() called!
+    event:                         $ai_generation
+    $ai_provider:                  anthropic
+    $ai_model:                     claude-haiku-4-5-20251001
+    $ai_input_tokens:              13
+    $ai_output_tokens:             38
+    $ai_cache_read_input_tokens:   14100
+    $ai_cache_creation_input_tokens: None
+    $ai_total_tokens:              NOT SET
+
+  Raw API usage (call 2):
+    input_tokens:                13
+    output_tokens:               38
+    cache_read_input_tokens:     14100
+    cache_creation_input_tokens: 0
+
+======================================================================
+  COMPARISON (Call 2 - cache hit)
+======================================================================
+    Anthropic API input_tokens:   13
+    SDK $ai_input_tokens:         13
+    cache_read_input_tokens:      14100
+    API input + cache_read:       14113
+
+    RESULT: SDK correctly passes through EXCLUSIVE input_tokens
+
+    OK: $ai_total_tokens is not set (expected for Anthropic wrapper)
+
+======================================================================
+  STEP 3: Test posthog_properties override (leading theory)
+======================================================================
+  Simulating customer passing their own inclusive token counts via posthog_properties...
+
+  3a) Simple override with hardcoded values...
+    Anthropic API input_tokens:   13
+    SDK $ai_input_tokens:         99999
+    $ai_total_tokens:             100599
+    => posthog_properties OVERRIDES the SDK's correct value
+
+  3b) Simulating customer computing inclusive tokens from the response...
+  (Customer code might do: input_tokens = usage.input_tokens + usage.cache_read_input_tokens)
+    Customer computes: inclusive_input = 13 + 14100 = 14113
+    Customer computes: total = 14113 + 47 = 14160
+    Anthropic API input_tokens:         12 (exclusive)
+    Anthropic API cache_read:           14100
+    Anthropic API output_tokens:        100
+    ---
+    Stored $ai_input_tokens:            14113 (inclusive!)
+    Stored $ai_cache_read_input_tokens: 14100
+    Stored $ai_total_tokens:            14160
+    ---
+    COST COMPARISON (haiku pricing: $1/Mtok input, $0.10/Mtok cache):
+
+    Correct (exclusive input_tokens = 12):
+      uncached: 12 * $1/Mtok    = $0.000012
+      cached:   14100 * $0.10/Mtok = $0.001410
+      total input cost:                    $0.001422
+
+    Wrong (inclusive input_tokens = 14113, treated as exclusive):
+      uncached: 14113 * $1/Mtok    = $0.014113
+      cached:   14100 * $0.10/Mtok = $0.001410
+      total input cost:                    $0.015523
+
+    OVERCHARGE: 10.9x (992% more)
+
+======================================================================
+  STEP 4: Prove $ai_total_tokens is never set by Anthropic SDK
+======================================================================
+  Inspecting SDK source code to confirm no code path sets $ai_total_tokens for Anthropic...
+
+  1) Anthropic converter (extract_anthropic_usage_from_response):
+     File: posthog/ai/anthropic/anthropic_converter.py
+     Contains 'total_tokens': False
+     => CONFIRMED: Anthropic converter does NOT extract total_tokens
+
+     Streaming extractor (extract_anthropic_usage_from_event):
+     Contains 'total_tokens': False
+
+  2) General AI utils (posthog/ai/utils.py):
+     File: posthog/ai/utils.py
+     Contains 'ai_total_tokens': False
+     => CONFIRMED: utils.py never tags $ai_total_tokens
+
+  3) OpenAI Agents processor (for contrast):
+     File: posthog/ai/openai_agents/processor.py
+     Contains '$ai_total_tokens': True
+     => This is the ONLY code path that sets $ai_total_tokens
+
+  CONCLUSION:
+     The Anthropic SDK code path NEVER sets $ai_total_tokens.
+     If $ai_total_tokens appears in a stored event from posthog.ai.anthropic,
+     it MUST have come from outside the SDK — most likely via posthog_properties.
+
+     Code references (posthog-python SDK):
+       - posthog/ai/anthropic/anthropic_converter.py:206-231
+         extract_anthropic_usage_from_response() -> only sets input_tokens, output_tokens, cache_*
+       - posthog/ai/utils.py -> tag('$ai_input_tokens', ...) and tag('$ai_output_tokens', ...)
+         NO tag('$ai_total_tokens', ...) anywhere
+       - posthog/ai/openai_agents/processor.py:539,696
+         ONLY place $ai_total_tokens is set (OpenAI Agents only, not Anthropic)
+
+======================================================================
+  DONE
+======================================================================
+  Check the output above to see if the bug is reproducible.
+  If Step 2 shows correct values but the customer sees wrong values,
+  the issue is likely in posthog_properties overrides (Step 3).
+  Step 4 proves via source inspection that $ai_total_tokens cannot
+  come from the Anthropic SDK code path.
+```
+
+</details>