fix(litellm): stop the metering callback from silently dropping spend (#103)#109
Merged
Merged
Conversation
…#103) Two paths where a completed call's spend was swallowed by the callback's broad except: 1. cached > input 422. `_map_usage` derived `cached_input` from a cache-read count that, under litellm/provider field skew, can exceed the derived `input_tokens = prompt_tokens - cache_creation`. The server's `cached <= input` validator then 422'd the meter and the callback swallowed it, losing the spend. `cached_input` is now clamped to `input_tokens` so the split is always valid. 2. success/failure key collision 409. A failure meter (truncated=True) and a success meter (truncated=False) of the same call derived the same idempotency key, but the server folds `truncated` into the meter fingerprint — so the second meter was rejected 409 idempotency_key_reuse and swallowed, dropping the authoritative success meter in failure-then-success ordering. The hook discriminator is now folded into the key so the two hooks are distinct; a retry within a hook still dedups. The success key format is unchanged (no suffix) for backward compatibility. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BjJ4QScUKbrMzF1jujL5Bg
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The LiteLLM metering callback swallows metering errors so it can never break the host application. That safety net hid two paths where a completed call's spend was recorded nowhere.
1.
cached > input→ 422, swallowed_map_usagesetcached_input_tokens = cache_readwhileinput_tokens = max(prompt_tokens - cache_creation, 0). Under litellm/provider field skew (the two counts coming from different accounting), the reported cache-read can exceed the derived input. The server'scached <= inputvalidator then 422s the meter, and the callback's broadexceptswallows it — the spend is lost.Fix: clamp
cached_inputtoinput_tokens, so the callback never emits an invalid split.2. success/failure key collision → 409, swallowed
The success hook (
truncated=False) and failure hook (truncated=True) of one call derive the same idempotency key from the response id, but the server foldstruncatedinto the meter fingerprint. Same key + differing fingerprint →409 idempotency_key_reuse, swallowed. In failure-then-success ordering, the dropped meter is the authoritative success one.Fix: fold the hook discriminator into the key (
-truncatedsuffix on the failure hook only), so the two hooks are distinct. A retry within a hook still dedups. The success key format is unchanged (no suffix) for backward compatibility.Tests
test_cache_read_is_clamped_to_input_tokens— cached (90) > derived input (20) clamps to 20; thecached <= inputinvariant holds.test_success_and_failure_meters_of_one_call_use_distinct_keys— no cross-hook collision.test_failure_idempotency_key_is_stable_across_retries— dedup within a hook is preserved.Local gate green: ruff, mypy --strict, import-linter, full unit suite (476 passed).
Closes #103.