Open
Conversation
Contributor
📝 WalkthroughWalkthroughPreserves multimodal tool results across chat flows by introducing Changes
sequenceDiagram
participant Tool as Tool Execution
participant Norm as Normalization
participant Chat as Chat Logic
participant Adapter as Provider Adapter
participant Provider as OpenAI/Anthropic API
Tool->>Norm: result (string | object | Array<ContentPart>)
Norm->>Norm: normalizeToolResultContent(result)
Norm-->>Chat: ToolResultContent (string | ContentPart[])
Chat->>Chat: store/emit ToolResultContent in conversation & events
Chat->>Adapter: send ModelMessage with multimodal content
Adapter->>Adapter: map ContentPart[] via provider converters
Adapter-->>Provider: send function_call_output / tool_result with structured output
Provider->>Provider: process multimodal content (images/text)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/typescript/ai-client/src/types.ts`:
- Around line 154-160: The change to ToolResultPart (type ToolResultPart) made
content a union (ToolResultContent = string | Array<ContentPart>) which breaks
UI consumers; revert content back to string to preserve backwards compatibility
and instead add a new optional field (e.g., contentParts?: Array<ContentPart>)
to carry structured parts, updating the ToolResultContent typedef if necessary
and leaving ToolResultState untouched; ensure the interface remains: content:
string, contentParts?: Array<ContentPart> so existing React/Vue renderers (which
expect part.content as string) keep working while new consumers can use
contentParts.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 382c7b77-8ddf-4ff2-a42d-c1c2c4a90606
📒 Files selected for processing (14)
.changeset/multimodal-tool-results.mdpackages/typescript/ai-anthropic/src/adapters/text.tspackages/typescript/ai-anthropic/tests/anthropic-adapter.test.tspackages/typescript/ai-client/src/types.tspackages/typescript/ai-code-mode/models-eval/metrics.tspackages/typescript/ai-openai/src/adapters/text.tspackages/typescript/ai-openai/tests/openai-adapter.test.tspackages/typescript/ai/src/activities/chat/index.tspackages/typescript/ai/src/activities/chat/messages.tspackages/typescript/ai/src/activities/chat/stream/message-updaters.tspackages/typescript/ai/src/activities/chat/stream/processor.tspackages/typescript/ai/src/activities/chat/tools/tool-calls.tspackages/typescript/ai/src/types.tspackages/typescript/ai/tests/message-converters.test.ts
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
packages/typescript/ai-react-ui/src/chat-message.tsx (1)
69-89: Avoid silently dropping unsupported tool-result part types.Line [87] returns
nullfor unknown part types, so valid multimodal payloads outsidetext/imagerender as empty content. A minimal fallback improves debuggability and user feedback.Suggested fallback
default: - return null + return ( + <div key={index} data-tool-result-part-type="unsupported"> + Unsupported tool result part: {part.type} + </div> + )Based on learnings: Maintain type safety through multimodal content support (image, audio, video, document) with model capability awareness.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-react-ui/src/chat-message.tsx` around lines 69 - 89, The renderer currently returns null for unknown part.type values inside content.map, which silently drops supported multimodal parts; update the fallback in the map (the switch default) to render a visible placeholder instead of null — include part.type and basic info (e.g., a small div with a data-tool-result-part-type attribute and minimal message or serialized metadata) so developers/users see unsupported/unknown types; reference the content.map callback and getContentPartSourceUrl usage for locating the switch and ensure the new fallback preserves keys (index) and accessibility attributes.packages/typescript/ai-vue-ui/src/message-part.vue (1)
152-165: Add a default branch for unsupported content-part types.Current rendering only handles
textandimage; other multimodal parts disappear silently in default UI. A small fallback avoids invisible tool output.Suggested fallback branch
<template v-for="(contentPart, index) in part.content" :key="index"> <div v-if="contentPart.type === 'text'" data-tool-result-part-type="text" > {{ contentPart.content }} </div> <img v-else-if="contentPart.type === 'image'" data-tool-result-part-type="image" :src="getContentPartSourceUrl(contentPart.source)" alt="Tool result" /> + <div v-else data-tool-result-part-type="unsupported"> + Unsupported tool result part: {{ contentPart.type }} + </div> </template>Based on learnings: Maintain type safety through multimodal content support (image, audio, video, document) with model capability awareness.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-vue-ui/src/message-part.vue` around lines 152 - 165, The template currently only renders contentPart.type === 'text' and 'image' so any other multimodal parts vanish; update the template in message-part.vue (the v-for over part.content and the contentPart.type checks) to add a default v-else branch that renders a visible fallback (e.g., a small placeholder showing the contentPart.type and/or a safe stringify of contentPart) and optionally logs a console.warn with the contentPart to aid debugging; keep use of getContentPartSourceUrl for image branch unchanged and ensure the fallback is small and clearly labeled so unsupported audio/video/document parts don't disappear silently.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/typescript/ai-react-ui/src/chat-message.tsx`:
- Around line 9-13: The ToolResultContentSource type allows a missing mimeType
even for data sources, which can produce invalid URIs like
data:undefined;base64,...; fix this by turning ToolResultContentSource into a
discriminated union where the branch with type: 'data' requires mimeType (e.g.,
{ type: 'data'; value: string; mimeType: string }) while the branch(s) for type:
'url' keep mimeType optional, then update any code that constructs data URIs
(references to ToolResultContentSource usages) to rely on the required mimeType
for the 'data' branch.
In `@packages/typescript/ai-solid-ui/src/chat-message.tsx`:
- Around line 10-14: ToolResultContentSource currently allows mimeType to be
undefined but code that renders tool results (e.g., the consumer that reads
source.mimeType in chat-message.tsx) expects mimeType when source.type ===
'data'; change ToolResultContentSource into a discriminated union: one variant
for { type: 'url'; value: string; mimeType?: string } and one for { type:
'data'; value: string; mimeType: string } so that mimeType is required for data
blobs, then update any callsites or tests that create a {type: 'data'} source to
include mimeType and adjust rendering logic to rely on the strengthened type.
In `@packages/typescript/ai-vue-ui/src/message-part.vue`:
- Around line 6-10: ToolResultContentSource allows mimeType to be optional even
when type === 'data', which can produce malformed data: URLs where
source.mimeType is missing; change the type to a discriminated union so the
'data' variant requires mimeType (e.g. { type: 'data', value: string, mimeType:
string } vs { type: 'url', value: string, mimeType?: string }) and update any
usages in message-part.vue that interpolate source.mimeType to rely on the
stronger type (or add a runtime guard) so you never build a data: URL without a
defined mimeType.
---
Nitpick comments:
In `@packages/typescript/ai-react-ui/src/chat-message.tsx`:
- Around line 69-89: The renderer currently returns null for unknown part.type
values inside content.map, which silently drops supported multimodal parts;
update the fallback in the map (the switch default) to render a visible
placeholder instead of null — include part.type and basic info (e.g., a small
div with a data-tool-result-part-type attribute and minimal message or
serialized metadata) so developers/users see unsupported/unknown types;
reference the content.map callback and getContentPartSourceUrl usage for
locating the switch and ensure the new fallback preserves keys (index) and
accessibility attributes.
In `@packages/typescript/ai-vue-ui/src/message-part.vue`:
- Around line 152-165: The template currently only renders contentPart.type ===
'text' and 'image' so any other multimodal parts vanish; update the template in
message-part.vue (the v-for over part.content and the contentPart.type checks)
to add a default v-else branch that renders a visible fallback (e.g., a small
placeholder showing the contentPart.type and/or a safe stringify of contentPart)
and optionally logs a console.warn with the contentPart to aid debugging; keep
use of getContentPartSourceUrl for image branch unchanged and ensure the
fallback is small and clearly labeled so unsupported audio/video/document parts
don't disappear silently.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 8bf5ce1b-3477-42d5-aef2-685b572cfe5c
📒 Files selected for processing (6)
.changeset/multimodal-tool-results.mdpackages/typescript/ai-react-ui/src/chat-message.tsxpackages/typescript/ai-solid-ui/src/chat-message.tsxpackages/typescript/ai-vue-ui/src/chat-message.vuepackages/typescript/ai-vue-ui/src/message-part.vuepackages/typescript/ai-vue-ui/src/types.ts
✅ Files skipped from review due to trivial changes (1)
- packages/typescript/ai-vue-ui/src/types.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- .changeset/multimodal-tool-results.md
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/typescript/ai-react-ui/src/chat-message.tsx (1)
70-100: Consider more stable keys if content parts have unique identifiers.Using array index as a React key works here since tool result content is static after being received and won't be reordered. However, if
ContentPartobjects ever include a unique identifier (like anidfield), using that would provide more stable rendering behavior.The rendering logic correctly handles the union type with a type guard for strings and a switch statement for structured parts.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-react-ui/src/chat-message.tsx` around lines 70 - 100, The renderToolResultContent function uses array index as React keys; update it to prefer a stable unique identifier when available by checking each ContentPart for a unique field (e.g., part.id or part.key) and using that as the key in the returned elements, falling back to index only when no identifier exists; ensure the switch cases (text/image/default) and their JSX elements use this computed key so rendering remains stable if ContentPart gains an id field in the future.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@packages/typescript/ai-react-ui/src/chat-message.tsx`:
- Around line 70-100: The renderToolResultContent function uses array index as
React keys; update it to prefer a stable unique identifier when available by
checking each ContentPart for a unique field (e.g., part.id or part.key) and
using that as the key in the returned elements, falling back to index only when
no identifier exists; ensure the switch cases (text/image/default) and their JSX
elements use this computed key so rendering remains stable if ContentPart gains
an id field in the future.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f21ffe0f-9d75-47e4-b38e-23b9fe1e9269
📒 Files selected for processing (3)
packages/typescript/ai-react-ui/src/chat-message.tsxpackages/typescript/ai-solid-ui/src/chat-message.tsxpackages/typescript/ai-vue-ui/src/message-part.vue
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/typescript/ai-solid-ui/src/chat-message.tsx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preserve multimodal tool results
✅ Checklist
pnpm run test:pr.* There is no CONTRIBUTING.md?
🚀 Release Impact
Fixes #363
Summary by CodeRabbit
Bug Fixes
New Features
Tests