Skip to content

fix(litellm): stop the metering callback from silently dropping spend (#103)#109

Merged
JumpTechCode merged 1 commit into
mainfrom
fix/litellm-callback-spend-drops
Jul 20, 2026
Merged

fix(litellm): stop the metering callback from silently dropping spend (#103)#109
JumpTechCode merged 1 commit into
mainfrom
fix/litellm-callback-spend-drops

Conversation

@JumpTechCode

Copy link
Copy Markdown
Contributor

The LiteLLM metering callback swallows metering errors so it can never break the host application. That safety net hid two paths where a completed call's spend was recorded nowhere.

1. cached > input → 422, swallowed

_map_usage set cached_input_tokens = cache_read while input_tokens = max(prompt_tokens - cache_creation, 0). Under litellm/provider field skew (the two counts coming from different accounting), the reported cache-read can exceed the derived input. The server's cached <= input validator then 422s the meter, and the callback's broad except swallows it — the spend is lost.

Fix: clamp cached_input to input_tokens, so the callback never emits an invalid split.

2. success/failure key collision → 409, swallowed

The success hook (truncated=False) and failure hook (truncated=True) of one call derive the same idempotency key from the response id, but the server folds truncated into the meter fingerprint. Same key + differing fingerprint → 409 idempotency_key_reuse, swallowed. In failure-then-success ordering, the dropped meter is the authoritative success one.

Fix: fold the hook discriminator into the key (-truncated suffix on the failure hook only), so the two hooks are distinct. A retry within a hook still dedups. The success key format is unchanged (no suffix) for backward compatibility.

Tests

  • test_cache_read_is_clamped_to_input_tokens — cached (90) > derived input (20) clamps to 20; the cached <= input invariant holds.
  • test_success_and_failure_meters_of_one_call_use_distinct_keys — no cross-hook collision.
  • test_failure_idempotency_key_is_stable_across_retries — dedup within a hook is preserved.

Local gate green: ruff, mypy --strict, import-linter, full unit suite (476 passed).

Closes #103.

…#103)

Two paths where a completed call's spend was swallowed by the callback's
broad except:

1. cached > input 422. `_map_usage` derived `cached_input` from a cache-read
   count that, under litellm/provider field skew, can exceed the derived
   `input_tokens = prompt_tokens - cache_creation`. The server's
   `cached <= input` validator then 422'd the meter and the callback swallowed
   it, losing the spend. `cached_input` is now clamped to `input_tokens` so the
   split is always valid.

2. success/failure key collision 409. A failure meter (truncated=True) and a
   success meter (truncated=False) of the same call derived the same
   idempotency key, but the server folds `truncated` into the meter
   fingerprint — so the second meter was rejected 409 idempotency_key_reuse and
   swallowed, dropping the authoritative success meter in failure-then-success
   ordering. The hook discriminator is now folded into the key so the two hooks
   are distinct; a retry within a hook still dedups. The success key format is
   unchanged (no suffix) for backward compatibility.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjJ4QScUKbrMzF1jujL5Bg
@JumpTechCode
JumpTechCode merged commit a78f210 into main Jul 20, 2026
8 checks passed
@JumpTechCode
JumpTechCode deleted the fix/litellm-callback-spend-drops branch July 20, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LiteLLM callback can silently drop metered spend (cached>input 422, and truncated/success key collision 409)

1 participant