Estimate token usage for vscode.lm providers that report none#7
Merged
Conversation
When a vscode.lm stream (e.g. GitHub Copilot) ends without the Positron usage data part, VscodeLmClient now synthesizes a finish-step with locally estimated usage via LanguageModelChat.countTokens(), marked providerMetadata.positai.usage.isEstimated so consumers display it as approximate. Input is counted once over the final post-transform payload (lazy, never-rejecting, bounded LRU memo); output is counted from the accumulated stream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When Posit Assistant runs in Positron and a user selects a GitHub Copilot model, requests go through VS Code's
vscode.lmAPI, which reports no token usage at all. As a result the Assistant showed no token counts for those conversations. This adds client-side token estimation for anyvscode.lmprovider that doesn't report usage: it counts tokens locally with the model's owncountTokens()(a local tokenizer, no network call) and synthesizes the same end-of-stream usage event the rest of the pipeline already consumes. The synthesized usage is tagged as estimated viaproviderMetadataso consumers can display it as approximate.Estimation triggers whenever a stream ends without having emitted a real usage part, so it covers Copilot today, extends to any other no-usage
vscode.lmprovider, and steps aside automatically if a provider ever starts reporting real usage. Positron's own providers, which already emit exact usage, are unaffected.Changes
src/positron/token-estimation.ts(new): input estimation from the final post-transform payload (messages + tool definitions + system parameter where applicable) with a per-message framing constant and a bounded module-level LRU memo keyed on model id plus a hash of the serialized message; output estimation from accumulated text and tool-call inputs. Exposed as a lazy thunk so providers reporting real usage never pay thecountTokenscost.src/positron/VscodeLmClient.ts: wires the estimation context into the response-stream converter and, at stream end with no usage part, synthesizes afinish-stepcarrying the estimated usage and theproviderMetadata.positai.usage.isEstimated = truemarker. Estimation never fails the request — errors degrade to no-usage with a logged warning. The synthesized finish reason (and the existing real-usage one) now reflectstool-callswhen the stream emitted a tool call.LanguageModelChatwith a scripted stream and deterministiccountTokenscovers estimation emitted for no-usage streams, suppressed when real usage arrives, the Copilot system prompt counted exactly once, memoization, error fallback, and the tool-call finish reason.Not included
The core plumbing and UI that surface this flag (the
~prefix, unpriced cost policy, TUI/web displays) live in the companion posit-dev/assistant PR, which pins to this branch.Accuracy note
These counts come from the Copilot extension's local tiktoken tokenizer, so they are approximate for non-OpenAI models served through
vscode.lmand are always presented as such. They deliberately do not apply any per-model correction factor. Anthropic's newer models (Opus 4.7/4.8, Sonnet 5, Fable 5) ship a tokenizer that produces roughly 1.0–1.35× as many tokens as the prior generation, but that ratio is old-Anthropic to new-Anthropic — not tiktoken to Anthropic — so applying it here would layer one guess on top of an unrelated one. Genuinely accurate counts for those models would require counting with the provider's own tokenizer (e.g. Anthropic'scount_tokens), which is future work rather than a multiplier.