Skip to content

Estimate token usage for vscode.lm providers that report none#7

Merged
wch merged 2 commits into
mainfrom
copilot-token-estimation
Jul 11, 2026
Merged

Estimate token usage for vscode.lm providers that report none#7
wch merged 2 commits into
mainfrom
copilot-token-estimation

Conversation

@wch

@wch wch commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

When Posit Assistant runs in Positron and a user selects a GitHub Copilot model, requests go through VS Code's vscode.lm API, which reports no token usage at all. As a result the Assistant showed no token counts for those conversations. This adds client-side token estimation for any vscode.lm provider that doesn't report usage: it counts tokens locally with the model's own countTokens() (a local tokenizer, no network call) and synthesizes the same end-of-stream usage event the rest of the pipeline already consumes. The synthesized usage is tagged as estimated via providerMetadata so consumers can display it as approximate.

Estimation triggers whenever a stream ends without having emitted a real usage part, so it covers Copilot today, extends to any other no-usage vscode.lm provider, and steps aside automatically if a provider ever starts reporting real usage. Positron's own providers, which already emit exact usage, are unaffected.

Changes

  • src/positron/token-estimation.ts (new): input estimation from the final post-transform payload (messages + tool definitions + system parameter where applicable) with a per-message framing constant and a bounded module-level LRU memo keyed on model id plus a hash of the serialized message; output estimation from accumulated text and tool-call inputs. Exposed as a lazy thunk so providers reporting real usage never pay the countTokens cost.
  • src/positron/VscodeLmClient.ts: wires the estimation context into the response-stream converter and, at stream end with no usage part, synthesizes a finish-step carrying the estimated usage and the providerMetadata.positai.usage.isEstimated = true marker. Estimation never fails the request — errors degrade to no-usage with a logged warning. The synthesized finish reason (and the existing real-usage one) now reflects tool-calls when the stream emitted a tool call.
  • Tests: a fake LanguageModelChat with a scripted stream and deterministic countTokens covers estimation emitted for no-usage streams, suppressed when real usage arrives, the Copilot system prompt counted exactly once, memoization, error fallback, and the tool-call finish reason.

Not included

The core plumbing and UI that surface this flag (the ~ prefix, unpriced cost policy, TUI/web displays) live in the companion posit-dev/assistant PR, which pins to this branch.

Accuracy note

These counts come from the Copilot extension's local tiktoken tokenizer, so they are approximate for non-OpenAI models served through vscode.lm and are always presented as such. They deliberately do not apply any per-model correction factor. Anthropic's newer models (Opus 4.7/4.8, Sonnet 5, Fable 5) ship a tokenizer that produces roughly 1.0–1.35× as many tokens as the prior generation, but that ratio is old-Anthropic to new-Anthropic — not tiktoken to Anthropic — so applying it here would layer one guess on top of an unrelated one. Genuinely accurate counts for those models would require counting with the provider's own tokenizer (e.g. Anthropic's count_tokens), which is future work rather than a multiplier.

wch added 2 commits July 10, 2026 16:36
When a vscode.lm stream (e.g. GitHub Copilot) ends without the Positron
usage data part, VscodeLmClient now synthesizes a finish-step with
locally estimated usage via LanguageModelChat.countTokens(), marked
providerMetadata.positai.usage.isEstimated so consumers display it as
approximate. Input is counted once over the final post-transform
payload (lazy, never-rejecting, bounded LRU memo); output is counted
from the accumulated stream.
@wch
wch merged commit 548bb4d into main Jul 11, 2026
1 check passed
@wch
wch deleted the copilot-token-estimation branch July 11, 2026 01:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant