Send hydrated image attachments to the agent as embedded bytes#327482
Open
ulugbekna wants to merge 2 commits into
Open
Send hydrated image attachments to the agent as embedded bytes#327482ulugbekna wants to merge 2 commits into
ulugbekna wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Hydrates URI-only image attachments into embedded bytes so Copilot and Codex can process their pixels.
Changes:
- Reads URI-backed images for model-facing conversions with resource fallback.
- Serializes pending-message synchronization.
- Adds image hydration regression tests.
Show a summary per file
| File | Description |
|---|---|
agentHostSessionHandler.ts |
Adds image hydration and asynchronous pending synchronization. |
agentHostChatContribution.test.ts |
Tests hydration, fallback, and inline-byte behavior. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Medium
| for (const p of pending) { | ||
| const variables = p.request.variableData?.variables ?? []; | ||
| const messageAttachments = this._variableEntriesToAttachments(variables, sessionResource, p.request.message.text); | ||
| const messageAttachments = await this._variableEntriesToAttachments(variables, sessionResource, p.request.message.text, true); |
| try { | ||
| // Read the original (non-rebased) URI; the registered | ||
| // vscode-agent-host file system provider handles remote reads. | ||
| const content = await this._fileService.readFile(refUri); |
Uri-only image attachments (e.g. hydrated edit-and-resend images) reached the model as a Resource path reference instead of pixels, so the Agents window agent could not actually see them. In _toImageAttachment's no-inline-bytes branch, read the referenced file via IFileService and emit an EmbeddedResource (base64) for model-facing conversions (turn + pending), falling back to the existing Resource path reference on any read failure. Draft input-state sync stays a lightweight path reference. The attachment-conversion caller chain (_convertVariableToAttachment, _variableEntriesToAttachments, _appendActiveEditorAttachments, _convertVariablesToAttachments, _syncPendingMessages, _inputStateToDraft) is now async. Per-session pending-message syncs are serialized and coalesced through a ThrottlerByKey so the added hydration I/O cannot interleave concurrent runs and dispatch a stale same-id message. Helps Copilot and Codex; Claude still drops EmbeddedResource today (pre-existing, out of scope). Follow-up to #327381 for #327329. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c77566c8-d94f-4245-b965-75ea069af547
A uri-only image in a chat draft is not model-facing, so the draft path keeps it as a lightweight MessageAttachmentKind.Resource reference and must not read file bytes on every debounced draft sync (hydrateImageBytes defaults to false). Locks in that behavior, which was previously untested. Follow-up to #327381 for #327329. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c77566c8-d94f-4245-b965-75ea069af547
ulugbekna
force-pushed
the
ulugbekna/send-hydrated-images-to-agent
branch
from
July 25, 2026 20:03
04ec1c2 to
84197b9
Compare
roblourens
requested changes
Jul 25, 2026
| try { | ||
| // Read the original (non-rebased) URI; the registered | ||
| // vscode-agent-host file system provider handles remote reads. | ||
| const content = await this._fileService.readFile(refUri); |
Member
There was a problem hiding this comment.
I haven't looked closely at attachments in a bit but I'm not sure this is the right approach- reading and serializing the image and inlining all content. If this is as file on disk, shouldn't we pass it around as a URI, and the agent host can read it if or when it needs to?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In the Agents window, a uri-only image attachment (no inline bytes) — e.g. a hydrated edit-and-resend image — reached the model as a
MessageAttachmentKind.Resourcepath reference, not pixels. Downstream, aResourceimage is never resolved to bytes before hitting the model:copilotAgentSession.ts_toSdkAttachment):Resourceimage →{ type: 'file', path }(path only)codexPromptResolver.ts):Resource→@<path>text mentionEmbeddedResourcecarries actual bytes.So the agent genuinely could not see those images. This is a follow-up to #327381 (which only restored the UI thumbnail preview) and addresses the model-perception half of #327329.
Fix
In
_toImageAttachment, in the no-inline-bytes branch, when the conversion is model-facing, read the referenced file via a newly injectedIFileServiceand emit anEmbeddedResource(base64data, preservingrange/_meta) so the model receives the real image. On any read failure it falls back to the existingResourcepath reference (never throws). The original (non-rebased)refUriis read directly — the registeredvscode-agent-host:file-system provider handles remote reads; rebasing remains only for the path-reference fallback.Hydration is gated by a
hydrateImageBytesflag so only model-facing conversions pay the I/O:_convertVariablesToAttachments) → hydrate_syncPendingMessages) → hydrate_inputStateToDraft) → stays a lightweight path reference (avoids file I/O + base64 bloat on every debounced draft sync)The conversion caller chain (
_convertVariableToAttachment,_variableEntriesToAttachments,_appendActiveEditorAttachments,_convertVariablesToAttachments,_syncPendingMessages,_inputStateToDraft) is nowasync.Because
_syncPendingMessagesnowawaits file I/O between reading its diff baseline and dispatching, per-session pending syncs are serialized and coalesced through aThrottlerByKey, so concurrent fire-and-forget runs can't interleave and dispatch a stale same-id message (latest run wins). This was flagged by code review.Scope / caveat
Helps Copilot and Codex. Claude still drops
EmbeddedResourcetoday (claudePromptResolver.ts) — a pre-existing limitation affecting all embedded images (including today's inline-bytes ones), out of scope here.No max-images-per-request cap exists on the pre-existing inline-bytes path, so there is nothing to apply symmetrically.
Tests
Added regression tests in
agentHostChatContribution.test.tsusing an injected fakeIFileService(TestFileService, noany/global stubs):EmbeddedResourcewith base64dataand correctcontentTypeIFileServiceread fails → falls back toMessageAttachmentKind.ResourceEmbeddedResource, with an assertion that the file service is never readValidation
npm run typecheck-client— passagentHostChatContribution.test.ts— 261 passing, 0 failingFollow-up to #327381. Fixes #327329.