test(e2e): cover all vision-capable providers in media-input suite#302
Open
ling-senpeng13 wants to merge 10 commits into
Open
test(e2e): cover all vision-capable providers in media-input suite#302ling-senpeng13 wants to merge 10 commits into
ling-senpeng13 wants to merge 10 commits into
Conversation
ling-senpeng13
force-pushed
the
test/e2e-media-input-all-providers
branch
from
8db10ce to
c242150
Compare
Extend Suite 25 to parametrize the media-input tests across every
vision-capable provider the server supports (OpenAI, Anthropic, Gemini),
each gated on its API key. The suite now documents the full media-input
support matrix:
- OpenAI → attaches media, reads the image. Passes.
- Anthropic → server drops media (fixed in conductor-oss#1238, pending
release). Positive case skipped.
- Gemini → server drops media too: GeminiChatModel.convertMessage
uses only UserMessage.getText() (same bug as Anthropic).
Positive case skipped until fixed.
The counterfactual (no media -> token absent) runs for all three.
Verified: openai passes; anthropic/gemini positive cases skip with
documented reasons; counterfactuals pass where a key is present.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the coarse "other 8 providers don't wire image input" note with an accurate matrix derived by reading each provider's conductor-ai ChatModel. Media input depends on whether the message converter forwards UserMessage.getMedia() or only getText(): FORWARDS media: openai + azureopenai (OpenAIResponsesChatModel), and mistral/ollama/bedrock (Spring AI stock chat models). DROPS media: anthropic (fixed in conductor-oss#1238), gemini (same bug), cohere, huggingface, grok, perplexity (custom text-only converters). Behaviour unchanged; comments/docstring only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ling-senpeng13
force-pushed
the
test/e2e-media-input-all-providers
branch
from
c242150 to
bff49bb
Compare
Refine the provider matrix: Bedrock maps media via BedrockProxyChatModel.mapMediaToContentBlock -> ImageBlock, and note that spring-ai's MediaFetcher (SSRF-guarded URL fetch: blocks loopback/link- local, 40 MB cap) exists only in newer spring-ai — the versions resolved here (1.0.2/1.1.2) map bytes/URL without it, and conductor-ai usually pre-downloads media to bytes anyway. Comment-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Gemini parametrized case could never run — there is no GOOGLE_AI_API_KEY in any environment we control, so it only ever skipped. Remove the case (and its skip constant) to avoid a permanently-skipped test. Gemini stays documented in the provider matrix as a media-dropping converter (fix: conductor-oss#1241); it's simply not parametrized, like the other providers we can't exercise here (azure/mistral/ollama/bedrock). Suite is now OpenAI (runs) + Anthropic (skipped until conductor-oss#1238 ships). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Azure OpenAI reuses OpenAIResponsesChatModel and forwards media as input_image content parts (verified in AzureOpenAI.getChatModel -> OpenAIResponsesChatModel.convertMessage). Comment-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified in Spring AI sources: MistralAiChatModel maps getMedia() to image_url MediaContent; OllamaChatModel maps it to .images(base64). Restructure the FORWARDS block to name each provider's concrete mechanism. Comment-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…y ref Grok/Perplexity media fix is conductor-oss#1243 (update the matrix from the tracking issue). Add the Spring AI multimodality reference for the framework-mapped providers (mistral/ollama/bedrock). Comment-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
HuggingFace DOES support multimodal, but only via its OpenAI-compatible router endpoint (https://router.huggingface.co/v1/responses, input_text/ input_image). conductor-ai currently uses the legacy text-generation API ({inputs} -> generated_text) with no messages or media, so adding vision means migrating the provider to the router endpoint — not a converter tweak like #1238/#1241/#1243. Update the matrix accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add StabilityAI (and image-gen-only providers generally) to the provider matrix as explicitly out of scope for media INPUT — no chat model (getChatModel throws), so there's no vision path; they generate images (cf. Suite 7), not receive them. Also refresh the HuggingFace note to reference its router migration (conductor-oss#1245) instead of describing it as still-legacy. Comment-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cohere IS vision-capable (e.g. command-a-vision-07-2025) and its v2 chat API accepts image_url content parts; it was dropping media (converter + bare-String content DTO). Fixed in conductor-oss#1246. Update the matrix entry from an unqualified drop to reference the fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #301 (base branch
test/e2e-python-media-input).What
Extends Suite 25 to parametrize the media-input tests across every vision-capable provider the server supports — OpenAI, Anthropic, Gemini — each gated on its API key. The suite now doubles as the media-input support matrix.
Support matrix (does the server attach image media to the provider request?)
openai/gpt-4o-minianthropic/claude-sonnet-4-5google_gemini/gemini-2.5-flashGeminiChatModel.convertMessageuses onlyUserMessage.getText()(same bug #1238 fixes for Anthropic); positive case skippedThe other providers (azure/bedrock/cohere/grok/mistral/ollama/perplexity/huggingface) don't wire image input, so they're out of scope.
Each skip has a documented reason string, so the suite is self-explaining and each case flips to a real test the moment its server-side support lands.
New finding: Gemini has the same media-drop bug
While enumerating providers I found
GeminiChatModel.convertMessage()dropsUserMessage.getMedia()exactly like Anthropic did — so Gemini vision input is also non-functional server-side. It needs the same one-liner fix as #1238 (convert media into GeminiinlineDataparts). Tracked here via the skipped[gemini]case; happy to send that conductor-oss fix separately.Test results (local)
🤖 Generated with Claude Code