Support image input for Grok#58
Open
hassaans wants to merge 1 commit into
Open
Conversation
Grok's translator only modelled text content, so any Anthropic `image`
block was rejected with "unsupported content block: image" — even though
grok-4.5 is multimodal. This makes vision unusable when driving Grok from
Claude Code (a pasted screenshot bricks the whole conversation, since the
image stays in history and every subsequent request re-sends it).
Add an `input_image` content part and translate Anthropic image blocks in
user messages to it, matching the Responses-style shape the Grok provider
already uses:
- base64 sources become a `data:<media_type>;base64,<data>` URL.
- url sources pass the URL through.
- Unknown image sources / unexpected block keys still fail loudly.
Verified end to end against grok-4.5: a solid magenta test image is
described correctly ("magenta") through the translated request.
count_tokens gains a rough per-image estimate so it stays exhaustive.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grok's translator only modelled text content, so any Anthropic
imageblock was rejected withunsupported content block: image— even though grok-4.5 is multimodal. This makes vision unusable when driving Grok from Claude Code, and it is more than a missing feature: a single pasted screenshot bricks the whole conversation, because the image stays in history and every subsequent request re-sends it (so every following turn 400s until the image is removed).Change
Add an
input_imagecontent part and translate Anthropicimageblocks in user messages to it, matching the Responses-style shape the Grok provider already uses:data:<media_type>;base64,<data>URL.count_tokensgains a rough per-image estimate so its match stays exhaustive.Verification
"magenta") through the translated request — so theinput_imageshape is accepted by xAI and vision actually works, not just compiles.input_image, url →input_image, unknown source / stray key rejected), matching the existing test conventions.cargo test --locked: all passing.cargo fmt --checkclean. No new clippy warnings.