diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a886c3c..96509923 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,25 @@ # Changelog +## 0.6.0 + +**Note on versioning:** This release jumps from 0.3.2 to 0.6.0, skipping versions 0.4.0 and 0.5.0. This is intentional and aligns the `@convex-dev/agent` package version with the AI SDK v6 major version for clearer compatibility signaling. Going forward, the minor version of this package will track the AI SDK major version to make it easier for developers to identify which version of the AI SDK is supported. + +- Breaking: Requires AI SDK v6 and drops support for AI SDK v5. Projects pinned + to v5 must upgrade their AI SDK dependencies before updating to this + version. +- Aligns this package's message and tool invocation types with the AI SDK v6 + APIs to reduce casting/adapter code when integrating with the core SDK. +- Updates internal helpers to use the AI SDK v6 request/response shapes and + error handling semantics. +- Migration from 0.3.x: + - Update your AI SDK dependency to v6 in `package.json` and reinstall + dependencies. + - Rebuild and run your type checker to surface any call sites that depend on + the old AI SDK v5 types or message shapes. + - Review any custom integrations that relied on deprecated v5-only helpers + and update them to the new AI SDK v6-compatible APIs. + - See Vercel's [v6 migration guide](https://ai-sdk.dev/docs/migration-guides/migration-guide-6-0) for details on AI SDK changes. + ## 0.3.2 - Fix deleteByOrder spanning many messages @@ -306,8 +326,7 @@ data at rest is backwards compatible. - Adds a `rawRequestResponseHandler` argument to the Agent that is a good spot to log or save all raw request/responses if you're trying to debug model behavior, headers, etc. -- Centralizes the example model usage so you can swap openai for openrouter / - grok in one place. +- Centralizes the example model usage so you can swap models in one place. - StorageOptions now takes a better argument name `saveMessages?: "all" | "none" | "promptAndOutput";`, deprecating `save{All,Any}InputMessages` and `saveOutputMessages`. diff --git a/TYPE_FIX_SUMMARY.md b/TYPE_FIX_SUMMARY.md new file mode 100644 index 00000000..f11e3ad2 --- /dev/null +++ b/TYPE_FIX_SUMMARY.md @@ -0,0 +1,63 @@ +# AI SDK v6 Type Error Fix Summary + +## Problem +The build fails with TypeScript errors after upgrading to AI SDK v6. The main issues are: +1. `ToolCallPart` type now requires `input` field (not optional), but stored data may only have deprecated `args` field +2. Tool-result output types missing newer types like `execution-denied` and extended content types +3. Generated component types out of sync with updated validators + +## Changes Made + +### 1. Fixed `tool-call` Part Handling (src/mapping.ts) +- Updated `toModelMessageContent()` to ensure `input` is always present by falling back to `args` or `{}` +- Updated `serializeContent()` and `fromModelMessageContent()` to handle both `input` and legacy `args` fields +- This fixes the core issue where AI SDK v6's `ToolCallPart` expects non-nullable `input` + +### 2. Fixed Tool Approval Response Handling (src/client/search.ts) +- Updated `filterOutOrphanedToolMessages()` to handle tool-approval-response parts that don't have `toolCallId` +- Tool-approval-response only has `approvalId`, not `toolCallId` + +### 3. Updated Generated Component Types (src/component/_generated/component.ts) +Made manual updates to sync with validators (normally done via `convex codegen`): +- Added `input: any` field to all tool-call type definitions +- Made `args` optional (`args?: any`) in tool-call types +- Added `execution-denied` output type to tool-result +- Added extended content types: `file-data`, `file-url`, `file-id`, `image-data`, `image-url`, `image-file-id`, `custom` +- Added `providerOptions` to text types in content values + +## Remaining Issues (5 TypeScript errors) + +The remaining errors are due to a structural mismatch in the generated component types: +- Generated types have BOTH `experimental_content` (deprecated) and `output` (new) fields on tool-result +- Our validators only define `output`, not `experimental_content` +- TypeScript is comparing our new output types against the old experimental_content types +- This cannot be fixed manually - requires proper component regeneration + +### To Complete the Fix: +1. Run `convex codegen --component-dir ./src/component` with a valid Convex deployment +2. This will regenerate `src/component/_generated/component.ts` from the validators +3. The regenerated types will: + - Remove the deprecated `experimental_content` field + - Use only the `output` field with correct types + - Properly match the validator definitions + +### Error Locations: +- `src/client/index.ts:1052` - addMessages call +- `src/client/index.ts:1103` - addMessages call +- `src/client/index.ts:1169` - updateMessage call +- `src/client/messages.ts:141` - addMessages call +- `src/client/start.ts:265` - addMessages call + +All errors have the same root cause: content value types in tool-result output don't match experimental_content expectations. + +## Testing Plan +Once component types are regenerated: +1. Run `npm run build` - should complete without errors +2. Run `npm test` - ensure no regressions +3. Test with actual AI SDK v6 workflow - verify tool-call handling works with both new `input` and legacy `args` fields + +## Notes +- The mapping functions in `src/mapping.ts` correctly handle both old and new formats +- Data with only `args` will be converted to have `input` (with `args` as fallback) +- Data with `input` will work directly +- This provides backward compatibility while supporting AI SDK v6's requirements diff --git a/docs/agent-usage.mdx b/docs/agent-usage.mdx index 9da4e5e2..37b584e0 100644 --- a/docs/agent-usage.mdx +++ b/docs/agent-usage.mdx @@ -43,7 +43,7 @@ requiring the LLM to always pass through full context to each tool call. It also allows dynamically choosing a model or other options for the Agent. ```ts -import { Agent } from "@convex-dev/agent"; +import { Agent, stepCountIs } from "@convex-dev/agent"; import { type LanguageModel } from "ai"; import type { ActionCtx } from "./_generated/server"; import type { Id } from "./_generated/dataModel"; @@ -63,7 +63,7 @@ function createAuthorAgent( researchCharacter: researchCharacterTool(ctx, bookId), writeChapter: writeChapterTool(ctx, bookId), }, - maxSteps: 10, // Alternative to stopWhen: stepCountIs(10) + stopWhen: stepCountIs(10), }); } ``` @@ -249,7 +249,7 @@ const supportAgent = new Agent(components.agent, { }, }), // Standard AI SDK tool - myTool: tool({ description, parameters, execute: () => {}}), + myTool: tool({ description, inputSchema: parameters, execute: () => {}}), }, // Used for limiting the number of steps when tool calls are involved. // NOTE: if you want tool calls to happen automatically with a single call, diff --git a/docs/getting-started.mdx b/docs/getting-started.mdx index b174c8b0..647b5550 100644 --- a/docs/getting-started.mdx +++ b/docs/getting-started.mdx @@ -40,7 +40,7 @@ successfully run once before you start defining Agents. ```ts import { components } from "./_generated/api"; -import { Agent } from "@convex-dev/agent"; +import { Agent, stepCountIs } from "@convex-dev/agent"; import { openai } from "@ai-sdk/openai"; const agent = new Agent(components.agent, { @@ -48,7 +48,7 @@ const agent = new Agent(components.agent, { languageModel: openai.chat("gpt-4o-mini"), instructions: "You are a weather forecaster.", tools: { getWeather, getGeocoding }, - maxSteps: 3, + stopWhen: stepCountIs(3), }); ``` diff --git a/docs/tools.mdx b/docs/tools.mdx index 179176e7..4eb4f464 100644 --- a/docs/tools.mdx +++ b/docs/tools.mdx @@ -66,7 +66,7 @@ export const ideaSearch = createTool({ async function createTool(ctx: ActionCtx, teamId: Id<"teams">) { const myTool = tool({ description: "My tool", - parameters: z.object({...}).describe("The arguments for the tool"), + inputSchema: z.object({...}).describe("The arguments for the tool"), execute: async (args, options): Promise => { return await ctx.runQuery(internal.foo.bar, args); }, diff --git a/example/convex/modelsForDemo.ts b/example/convex/modelsForDemo.ts index 18c2bbeb..10295a8d 100644 --- a/example/convex/modelsForDemo.ts +++ b/example/convex/modelsForDemo.ts @@ -1,29 +1,26 @@ -import { openrouter } from "@openrouter/ai-sdk-provider"; import { type EmbeddingModel } from "ai"; -import type { LanguageModelV2 } from "@ai-sdk/provider"; +import type { LanguageModelV3 } from "@ai-sdk/provider"; import { anthropic } from "@ai-sdk/anthropic"; import { openai } from "@ai-sdk/openai"; import { groq } from "@ai-sdk/groq"; import { mockModel } from "@convex-dev/agent"; -let languageModel: LanguageModelV2; -let textEmbeddingModel: EmbeddingModel; +let languageModel: LanguageModelV3; +let textEmbeddingModel: EmbeddingModel; if (process.env.ANTHROPIC_API_KEY) { languageModel = anthropic.chat("claude-opus-4-20250514"); } else if (process.env.OPENAI_API_KEY) { languageModel = openai.chat("gpt-4o-mini"); - textEmbeddingModel = openai.textEmbeddingModel("text-embedding-3-small"); + textEmbeddingModel = openai.embedding("text-embedding-3-small"); } else if (process.env.GROQ_API_KEY) { languageModel = groq.languageModel( "meta-llama/llama-4-scout-17b-16e-instruct", ); -} else if (process.env.OPENROUTER_API_KEY) { - languageModel = openrouter.chat("openai/gpt-4o-mini") as LanguageModelV2; } else { languageModel = mockModel({}); console.warn( - "Run `npx convex env set GROQ_API_KEY=` or `npx convex env set OPENAI_API_KEY=` or `npx convex env set OPENROUTER_API_KEY=` from the example directory to set the API key.", + "Run `npx convex env set GROQ_API_KEY=` or `npx convex env set OPENAI_API_KEY=` from the example directory to set the API key.", ); } diff --git a/example/convex/threads.ts b/example/convex/threads.ts index 1b650d47..794cf258 100644 --- a/example/convex/threads.ts +++ b/example/convex/threads.ts @@ -73,7 +73,6 @@ export const updateThreadTitle = action({ object: { title, summary }, } = await thread.generateObject( { - mode: "json", schemaDescription: "Generate a title and summary for the thread. The title should be a single sentence that captures the main topic of the thread. The summary should be a short description of the thread that could be used to describe it to someone who hasn't read it.", schema: z.object({ diff --git a/example/tsconfig.json b/example/tsconfig.json index a71fcacd..f0af7adf 100644 --- a/example/tsconfig.json +++ b/example/tsconfig.json @@ -11,7 +11,7 @@ "strict": true, "forceConsistentCasingInFileNames": true, "module": "ESNext", - "moduleResolution": "Bundler", + "moduleResolution": "bundler", "resolveJsonModule": true, "isolatedModules": true, "jsx": "react-jsx", diff --git a/package-lock.json b/package-lock.json index 50bd9dbc..bea11838 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,27 +1,26 @@ { "name": "@convex-dev/agent", - "version": "0.3.2", + "version": "0.6.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@convex-dev/agent", - "version": "0.3.2", + "version": "0.6.0", "license": "Apache-2.0", "devDependencies": { - "@ai-sdk/anthropic": "2.0.39", - "@ai-sdk/groq": "2.0.26", - "@ai-sdk/openai": "2.0.57", - "@ai-sdk/provider": "2.0.0", - "@ai-sdk/provider-utils": "3.0.14", - "@convex-dev/rag": "0.6.0", + "@ai-sdk/anthropic": "^3.0.13", + "@ai-sdk/groq": "^3.0.8", + "@ai-sdk/openai": "^3.0.10", + "@ai-sdk/provider": "^3.0.3", + "@ai-sdk/provider-utils": "^4.0.6", + "@convex-dev/rag": "0.6.1", "@convex-dev/rate-limiter": "0.3.0", "@convex-dev/workflow": "0.3.2", "@edge-runtime/vm": "5.0.0", "@eslint/js": "9.38.0", "@hookform/resolvers": "5.2.2", "@langchain/textsplitters": "0.1.0", - "@openrouter/ai-sdk-provider": "1.2.0", "@radix-ui/react-accordion": "1.2.12", "@radix-ui/react-checkbox": "1.3.3", "@radix-ui/react-label": "2.1.7", @@ -33,7 +32,7 @@ "@types/react": "19.2.2", "@types/react-dom": "19.2.2", "@vitejs/plugin-react": "4.7.0", - "ai": "5.0.82", + "ai": "^6.0.35", "autoprefixer": "10.4.21", "chokidar-cli": "3.0.0", "class-variance-authority": "0.7.1", @@ -52,7 +51,7 @@ "globals": "16.4.0", "lucide-react": "0.548.0", "npm-run-all2": "8.0.4", - "ollama-ai-provider": "1.2.0", + "ollama-ai-provider": "^1.2.0", "openai": "5.23.2", "path-exists-cli": "^2.0.0", "pkg-pr-new": "0.0.60", @@ -77,8 +76,8 @@ "@ungap/structured-clone": "^1.3.0" }, "peerDependencies": { - "@ai-sdk/provider-utils": "^3.0.7", - "ai": "^5.0.29", + "@ai-sdk/provider-utils": "^4.0.6", + "ai": "^6.0.35", "convex": "^1.24.8", "convex-helpers": "^0.1.103", "react": "^18.3.1 || ^19.0.0" @@ -129,14 +128,14 @@ "license": "MIT" }, "node_modules/@ai-sdk/anthropic": { - "version": "2.0.39", - "resolved": "https://registry.npmjs.org/@ai-sdk/anthropic/-/anthropic-2.0.39.tgz", - "integrity": "sha512-8YckXsPN9e0NfU4zZvP23xCIKNESyYb1Y/xVllI1fZ+uVsd/shoz2zplbeGVQHPjXHWfY9aT5tF98Lp920HDIQ==", + "version": "3.0.13", + "resolved": "https://registry.npmjs.org/@ai-sdk/anthropic/-/anthropic-3.0.13.tgz", + "integrity": "sha512-62UqSpZWuR8pU2ZLc1IgPYiNdH01blAcaNEjrQtx4wCN7L2fUTXm/iG6Tq9qRCiRED+8eQ43olggbf0fbguqkA==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@ai-sdk/provider": "2.0.0", - "@ai-sdk/provider-utils": "3.0.14" + "@ai-sdk/provider": "3.0.3", + "@ai-sdk/provider-utils": "4.0.6" }, "engines": { "node": ">=18" @@ -146,15 +145,15 @@ } }, "node_modules/@ai-sdk/gateway": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/@ai-sdk/gateway/-/gateway-2.0.3.tgz", - "integrity": "sha512-/vCoMKtod+A74/BbkWsaAflWKz1ovhX5lmJpIaXQXtd6gyexZncjotBTbFM8rVJT9LKJ/Kx7iVVo3vh+KT+IJg==", + "version": "3.0.14", + "resolved": "https://registry.npmjs.org/@ai-sdk/gateway/-/gateway-3.0.14.tgz", + "integrity": "sha512-udVpkDaQ00jMcBvtGGvmkEBU31XidsHB4E8HIF9l7/H7lyjOS/EtXzN2adoupDg5j1/VjjSI3Ny5P1zHUvLyMA==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@ai-sdk/provider": "2.0.0", - "@ai-sdk/provider-utils": "3.0.14", - "@vercel/oidc": "3.0.3" + "@ai-sdk/provider": "3.0.3", + "@ai-sdk/provider-utils": "4.0.6", + "@vercel/oidc": "3.1.0" }, "engines": { "node": ">=18" @@ -163,15 +162,25 @@ "zod": "^3.25.76 || ^4.1.8" } }, + "node_modules/@ai-sdk/gateway/node_modules/@vercel/oidc": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/@vercel/oidc/-/oidc-3.1.0.tgz", + "integrity": "sha512-Fw28YZpRnA3cAHHDlkt7xQHiJ0fcL+NRcIqsocZQUSmbzeIKRpwttJjik5ZGanXP+vlA4SbTg+AbA3bP363l+w==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">= 20" + } + }, "node_modules/@ai-sdk/groq": { - "version": "2.0.26", - "resolved": "https://registry.npmjs.org/@ai-sdk/groq/-/groq-2.0.26.tgz", - "integrity": "sha512-X3B531H9WsKPCCwqPbeywEKc0HCPiwgnuVyvYnrX3VFSTcYbvpiylNrNbv0fvsBd/scNIU7AdpIyK0KscBdBIg==", + "version": "3.0.8", + "resolved": "https://registry.npmjs.org/@ai-sdk/groq/-/groq-3.0.8.tgz", + "integrity": "sha512-NUh5TWpX62Ar9zaJpMxkwO5V0+neKC5vNu6Pd28qmOV4YPwNR6YgOXRlYXnDXi5w3wULVJrRAg6OBSal9vz2iA==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@ai-sdk/provider": "2.0.0", - "@ai-sdk/provider-utils": "3.0.14" + "@ai-sdk/provider": "3.0.3", + "@ai-sdk/provider-utils": "4.0.6" }, "engines": { "node": ">=18" @@ -181,14 +190,14 @@ } }, "node_modules/@ai-sdk/openai": { - "version": "2.0.57", - "resolved": "https://registry.npmjs.org/@ai-sdk/openai/-/openai-2.0.57.tgz", - "integrity": "sha512-ad2e4Ah9KdLnchMcWFv2FfU1JCwm50b3+UZq2VhkO8qLYEh2kh/aVQomZyAsIbx5ft5nOv2KmDwZrefXkeKttQ==", + "version": "3.0.10", + "resolved": "https://registry.npmjs.org/@ai-sdk/openai/-/openai-3.0.10.tgz", + "integrity": "sha512-G6HJORN0rKuCFrqIUiYchjl2b4UjzKvv3VcNuW7xwQIdI8EcdB9Pr8ZaR9nEImK9E639nM8gCfvFEUM1xwGaCA==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@ai-sdk/provider": "2.0.0", - "@ai-sdk/provider-utils": "3.0.14" + "@ai-sdk/provider": "3.0.3", + "@ai-sdk/provider-utils": "4.0.6" }, "engines": { "node": ">=18" @@ -198,9 +207,9 @@ } }, "node_modules/@ai-sdk/provider": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/@ai-sdk/provider/-/provider-2.0.0.tgz", - "integrity": "sha512-6o7Y2SeO9vFKB8lArHXehNuusnpddKPk7xqL7T2/b+OvXMRIXUO1rR4wcv1hAFUAT9avGZshty3Wlua/XA7TvA==", + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider/-/provider-3.0.3.tgz", + "integrity": "sha512-qGPYdoAuECaUXPrrz0BPX1SacZQuJ6zky0aakxpW89QW1hrY0eF4gcFm/3L9Pk8C5Fwe+RvBf2z7ZjDhaPjnlg==", "dev": true, "license": "Apache-2.0", "dependencies": { @@ -211,15 +220,15 @@ } }, "node_modules/@ai-sdk/provider-utils": { - "version": "3.0.14", - "resolved": "https://registry.npmjs.org/@ai-sdk/provider-utils/-/provider-utils-3.0.14.tgz", - "integrity": "sha512-CYRU6L7IlR7KslSBVxvlqlybQvXJln/PI57O8swhOaDIURZbjRP2AY3igKgUsrmWqqnFFUHP+AwTN8xqJAknnA==", + "version": "4.0.6", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider-utils/-/provider-utils-4.0.6.tgz", + "integrity": "sha512-o/SP1GQOrpXAzHjMosPHI0Pu+YkwxIMndSjSLrEXtcVixdrjqrGaA9I7xJcWf+XpRFJ9byPHrKYnprwS+36gMg==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@ai-sdk/provider": "2.0.0", - "@standard-schema/spec": "^1.0.0", - "eventsource-parser": "^3.0.5" + "@ai-sdk/provider": "3.0.3", + "@standard-schema/spec": "^1.1.0", + "eventsource-parser": "^3.0.6" }, "engines": { "node": ">=18" @@ -562,9 +571,9 @@ "peer": true }, "node_modules/@convex-dev/rag": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/@convex-dev/rag/-/rag-0.6.0.tgz", - "integrity": "sha512-28ND1mbYawxJQiIWUAUxewrhOgXEmTlCp+mINSi1OM3UgtzpoyVlbSdE+XXJX2eiYUhzbbpoN08rl9CebsKm0Q==", + "version": "0.6.1", + "resolved": "https://registry.npmjs.org/@convex-dev/rag/-/rag-0.6.1.tgz", + "integrity": "sha512-0vgruuAIxLzKgVoslz8BGMzixQ6IOZy+1dI5QD2jcnRNItIe1X+xziYfM09RWqgvEkPfxp5Cox3N1qf6bmR/hQ==", "dev": true, "license": "Apache-2.0", "dependencies": { @@ -576,6 +585,74 @@ "convex-helpers": "^0.1.94" } }, + "node_modules/@convex-dev/rag/node_modules/@ai-sdk/gateway": { + "version": "2.0.23", + "resolved": "https://registry.npmjs.org/@ai-sdk/gateway/-/gateway-2.0.23.tgz", + "integrity": "sha512-qmX7afPRszUqG5hryHF3UN8ITPIRSGmDW6VYCmByzjoUkgm3MekzSx2hMV1wr0P+llDeuXb378SjqUfpvWJulg==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "2.0.0", + "@ai-sdk/provider-utils": "3.0.19", + "@vercel/oidc": "3.0.5" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@convex-dev/rag/node_modules/@ai-sdk/provider": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider/-/provider-2.0.0.tgz", + "integrity": "sha512-6o7Y2SeO9vFKB8lArHXehNuusnpddKPk7xqL7T2/b+OvXMRIXUO1rR4wcv1hAFUAT9avGZshty3Wlua/XA7TvA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "json-schema": "^0.4.0" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/@convex-dev/rag/node_modules/@ai-sdk/provider-utils": { + "version": "3.0.19", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider-utils/-/provider-utils-3.0.19.tgz", + "integrity": "sha512-W41Wc9/jbUVXVwCN/7bWa4IKe8MtxO3EyA0Hfhx6grnmiYlCvpI8neSYWFE0zScXJkgA/YK3BRybzgyiXuu6JA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "2.0.0", + "@standard-schema/spec": "^1.0.0", + "eventsource-parser": "^3.0.6" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@convex-dev/rag/node_modules/ai": { + "version": "5.0.116", + "resolved": "https://registry.npmjs.org/ai/-/ai-5.0.116.tgz", + "integrity": "sha512-+2hYJ80/NcDWuv9K2/MLP3cTCFgwWHmHlS1tOpFUKKcmLbErAAlE/S2knsKboc3PNAu8pQkDr2N3K/Vle7ENgQ==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/gateway": "2.0.23", + "@ai-sdk/provider": "2.0.0", + "@ai-sdk/provider-utils": "3.0.19", + "@opentelemetry/api": "1.9.0" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, "node_modules/@convex-dev/rate-limiter": { "version": "0.3.0", "resolved": "https://registry.npmjs.org/@convex-dev/rate-limiter/-/rate-limiter-0.3.0.tgz", @@ -608,9 +685,9 @@ } }, "node_modules/@convex-dev/workpool": { - "version": "0.3.0", - "resolved": "https://registry.npmjs.org/@convex-dev/workpool/-/workpool-0.3.0.tgz", - "integrity": "sha512-nY8Ub0pmfuxZ2rcnNwVeESYPyJqLU4h+afodEdg8Ifnr+vcFUuee/p69vMFmeqC2y4yo9IDPHrdiVZVyjbBE7A==", + "version": "0.3.1", + "resolved": "https://registry.npmjs.org/@convex-dev/workpool/-/workpool-0.3.1.tgz", + "integrity": "sha512-uw4Mi+irhhoYA/KwaMo5wXyYJ7BbxqeaLcCZbst3t1SxPN5488rpnR0OwBcPDCmwcdQjBVHOx+q8S4GUjq0Csg==", "dev": true, "license": "Apache-2.0", "peer": true, @@ -1460,9 +1537,9 @@ } }, "node_modules/@langchain/core": { - "version": "0.3.79", - "resolved": "https://registry.npmjs.org/@langchain/core/-/core-0.3.79.tgz", - "integrity": "sha512-ZLAs5YMM5N2UXN3kExMglltJrKKoW7hs3KMZFlXUnD7a5DFKBYxPFMeXA4rT+uvTxuJRZPCYX0JKI5BhyAWx4A==", + "version": "0.3.80", + "resolved": "https://registry.npmjs.org/@langchain/core/-/core-0.3.80.tgz", + "integrity": "sha512-vcJDV2vk1AlCwSh3aBm/urQ1ZrlXFFBocv11bz/NBUfLWD5/UDNMzwPdaAd2dKvNmTWa9FM2lirLU3+JCf4cRA==", "dev": true, "license": "MIT", "peer": true, @@ -1484,18 +1561,6 @@ "node": ">=18" } }, - "node_modules/@langchain/core/node_modules/ansi-styles": { - "version": "5.2.0", - "dev": true, - "license": "MIT", - "peer": true, - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/chalk/ansi-styles?sponsor=1" - } - }, "node_modules/@langchain/textsplitters": { "version": "0.1.0", "resolved": "https://registry.npmjs.org/@langchain/textsplitters/-/textsplitters-0.1.0.tgz", @@ -1832,20 +1897,6 @@ "@octokit/openapi-types": "^20.0.0" } }, - "node_modules/@openrouter/ai-sdk-provider": { - "version": "1.2.0", - "resolved": "https://registry.npmjs.org/@openrouter/ai-sdk-provider/-/ai-sdk-provider-1.2.0.tgz", - "integrity": "sha512-stuIwq7Yb7DNmk3GuCtz+oS3nZOY4TXEV3V5KsknDGQN7Fpu3KRMQVWRc1J073xKdf0FC9EHOctSyzsACmp5Ag==", - "dev": true, - "license": "Apache-2.0", - "engines": { - "node": ">=18" - }, - "peerDependencies": { - "ai": "^5.0.0", - "zod": "^3.24.1 || ^v4" - } - }, "node_modules/@opentelemetry/api": { "version": "1.9.0", "resolved": "https://registry.npmjs.org/@opentelemetry/api/-/api-1.9.0.tgz", @@ -2883,9 +2934,9 @@ ] }, "node_modules/@standard-schema/spec": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/@standard-schema/spec/-/spec-1.0.0.tgz", - "integrity": "sha512-m2bOd0f2RT9k8QJx1JN85cZYyH1RqFBdlwtkSlf4tBDYLCiiZnv1fIIwacK6cqwXavOydf0NPToMQgpKq+dVlA==", + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@standard-schema/spec/-/spec-1.1.0.tgz", + "integrity": "sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==", "dev": true, "license": "MIT" }, @@ -3366,9 +3417,9 @@ "license": "ISC" }, "node_modules/@vercel/oidc": { - "version": "3.0.3", - "resolved": "https://registry.npmjs.org/@vercel/oidc/-/oidc-3.0.3.tgz", - "integrity": "sha512-yNEQvPcVrK9sIe637+I0jD6leluPxzwJKx/Haw6F4H77CdDsszUn5V3o96LPziXkSNE2B83+Z3mjqGKBK/R6Gg==", + "version": "3.0.5", + "resolved": "https://registry.npmjs.org/@vercel/oidc/-/oidc-3.0.5.tgz", + "integrity": "sha512-fnYhv671l+eTTp48gB4zEsTW/YtRgRPnkI2nT7x6qw5rkI1Lq2hTmQIpHPgyThI0znLK+vX2n9XxKdXZ7BUbbw==", "dev": true, "license": "Apache-2.0", "engines": { @@ -3552,15 +3603,15 @@ } }, "node_modules/ai": { - "version": "5.0.82", - "resolved": "https://registry.npmjs.org/ai/-/ai-5.0.82.tgz", - "integrity": "sha512-wmZZfsU40qB77umrcj3YzMSk6cUP5gxLXZDPfiSQLBLegTVXPUdSJC603tR7JB5JkhBDzN5VLaliuRKQGKpUXg==", + "version": "6.0.35", + "resolved": "https://registry.npmjs.org/ai/-/ai-6.0.35.tgz", + "integrity": "sha512-MxgtU6CjnegH1rhRfomM0gptKxP6r+9sxbLvYq36C1l85+o0LacqbXLdNVYzqab+lHN4q7ZP3QS8wZq4YkZahA==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@ai-sdk/gateway": "2.0.3", - "@ai-sdk/provider": "2.0.0", - "@ai-sdk/provider-utils": "3.0.14", + "@ai-sdk/gateway": "3.0.14", + "@ai-sdk/provider": "3.0.3", + "@ai-sdk/provider-utils": "4.0.6", "@opentelemetry/api": "1.9.0" }, "engines": { @@ -3600,6 +3651,20 @@ "url": "https://github.com/chalk/ansi-regex?sponsor=1" } }, + "node_modules/ansi-styles": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-5.2.0.tgz", + "integrity": "sha512-Cxwpt2SfTzTtXcfOlzGEee8O+c+MmUgGrNiBcXnuWxuFJHe6a5Hz7qwhwe5OgaSYI0IJvkLqWX1ASG+cJOkEiA==", + "dev": true, + "license": "MIT", + "peer": true, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/chalk/ansi-styles?sponsor=1" + } + }, "node_modules/any-promise": { "version": "1.3.0", "resolved": "https://registry.npmjs.org/any-promise/-/any-promise-1.3.0.tgz", @@ -7080,9 +7145,9 @@ } }, "node_modules/langsmith": { - "version": "0.3.75", - "resolved": "https://registry.npmjs.org/langsmith/-/langsmith-0.3.75.tgz", - "integrity": "sha512-4cl/KOxq99/c0MtlzXd6rpmOvMUuRHrJTRFzEwz/G+zDygeFm6bbKaa5XRu/VDZs1FsFGKL2WJYNbjFfL2Cg3Q==", + "version": "0.3.87", + "resolved": "https://registry.npmjs.org/langsmith/-/langsmith-0.3.87.tgz", + "integrity": "sha512-XXR1+9INH8YX96FKWc5tie0QixWz6tOqAsAKfcJyPkE0xPep+NDz0IQLR32q4bn10QK3LqD2HN6T3n6z1YLW7Q==", "dev": true, "license": "MIT", "peer": true, @@ -7091,7 +7156,6 @@ "chalk": "^4.1.2", "console-table-printer": "^2.12.1", "p-queue": "^6.6.2", - "p-retry": "4", "semver": "^7.6.3", "uuid": "^10.0.0" }, @@ -12016,14 +12080,14 @@ } }, "node_modules/zod-to-json-schema": { - "version": "3.24.6", - "resolved": "https://registry.npmjs.org/zod-to-json-schema/-/zod-to-json-schema-3.24.6.tgz", - "integrity": "sha512-h/z3PKvcTcTetyjl1fkj79MHNEjm+HpD6NXheWjzOekY7kV+lwDYnHw+ivHkijnCSMz1yJaWBD9vu/Fcmk+vEg==", + "version": "3.25.1", + "resolved": "https://registry.npmjs.org/zod-to-json-schema/-/zod-to-json-schema-3.25.1.tgz", + "integrity": "sha512-pM/SU9d3YAggzi6MtR4h7ruuQlqKtad8e9S0fmxcMi+ueAK5Korys/aWcV9LIIHTVbj01NdzxcnXSN+O74ZIVA==", "dev": true, "license": "ISC", "peer": true, "peerDependencies": { - "zod": "^3.24.1" + "zod": "^3.25 || ^4" } }, "node_modules/zod-validation-error": { diff --git a/package.json b/package.json index 37e7d4cc..c899bf51 100644 --- a/package.json +++ b/package.json @@ -7,7 +7,7 @@ "email": "support@convex.dev", "url": "https://github.com/get-convex/agent/issues" }, - "version": "0.3.2", + "version": "0.6.0", "license": "Apache-2.0", "keywords": [ "convex", @@ -69,8 +69,8 @@ } }, "peerDependencies": { - "@ai-sdk/provider-utils": "^3.0.7", - "ai": "^5.0.29", + "@ai-sdk/provider-utils": "^4.0.6", + "ai": "^6.0.35", "convex": "^1.24.8", "convex-helpers": "^0.1.103", "react": "^18.3.1 || ^19.0.0" @@ -81,19 +81,18 @@ } }, "devDependencies": { - "@ai-sdk/anthropic": "2.0.39", - "@ai-sdk/groq": "2.0.26", - "@ai-sdk/openai": "2.0.57", - "@ai-sdk/provider": "2.0.0", - "@ai-sdk/provider-utils": "3.0.14", - "@convex-dev/rag": "0.6.0", + "@ai-sdk/anthropic": "^3.0.13", + "@ai-sdk/groq": "^3.0.8", + "@ai-sdk/openai": "^3.0.10", + "@ai-sdk/provider": "^3.0.3", + "@ai-sdk/provider-utils": "^4.0.6", + "@convex-dev/rag": "0.6.1", "@convex-dev/rate-limiter": "0.3.0", "@convex-dev/workflow": "0.3.2", "@edge-runtime/vm": "5.0.0", "@eslint/js": "9.38.0", "@hookform/resolvers": "5.2.2", "@langchain/textsplitters": "0.1.0", - "@openrouter/ai-sdk-provider": "1.2.0", "@radix-ui/react-accordion": "1.2.12", "@radix-ui/react-checkbox": "1.3.3", "@radix-ui/react-label": "2.1.7", @@ -105,7 +104,7 @@ "@types/react": "19.2.2", "@types/react-dom": "19.2.2", "@vitejs/plugin-react": "4.7.0", - "ai": "5.0.82", + "ai": "^6.0.35", "autoprefixer": "10.4.21", "chokidar-cli": "3.0.0", "class-variance-authority": "0.7.1", @@ -124,7 +123,7 @@ "globals": "16.4.0", "lucide-react": "0.548.0", "npm-run-all2": "8.0.4", - "ollama-ai-provider": "1.2.0", + "ollama-ai-provider": "^1.2.0", "openai": "5.23.2", "path-exists-cli": "^2.0.0", "pkg-pr-new": "0.0.60", diff --git a/src/UIMessages.combineUIMessages.test.ts b/src/UIMessages.combineUIMessages.test.ts new file mode 100644 index 00000000..f696de58 --- /dev/null +++ b/src/UIMessages.combineUIMessages.test.ts @@ -0,0 +1,239 @@ +import { describe, it, expect } from "vitest"; +import { combineUIMessages, type UIMessage } from "./UIMessages.js"; + +describe("combineUIMessages", () => { + it("should preserve all tool calls when combining messages", () => { + const messages: UIMessage[] = [ + { + id: "msg1", + key: "thread-1-0", + order: 1, + stepOrder: 0, + status: "streaming", + role: "assistant", + parts: [ + { + type: "tool-toolA", + toolCallId: "call_A", + state: "input-available", + input: {}, + }, + ], + text: "", + _creationTime: Date.now(), + }, + { + id: "msg1", + key: "thread-1-0", + order: 1, + stepOrder: 0, + status: "streaming", + role: "assistant", + parts: [ + { + type: "tool-toolB", + toolCallId: "call_B", + state: "input-available", + input: {}, + }, + ], + text: "", + _creationTime: Date.now(), + }, + ]; + + const result = combineUIMessages(messages); + + expect(result).toHaveLength(1); + expect(result[0].parts).toHaveLength(2); + + const toolCallIds = result[0].parts + .filter((p) => p.type.startsWith("tool-")) + .map((p: any) => p.toolCallId); + + expect(toolCallIds).toContain("call_A"); + expect(toolCallIds).toContain("call_B"); + }); + + it("should accumulate tool calls progressively (issue #182)", () => { + // Simulating: A(started) → B → C → A(result) + const messages: UIMessage[] = [ + { + id: "msg1", + key: "thread-1-0", + order: 1, + stepOrder: 0, + status: "streaming", + role: "assistant", + parts: [ + { + type: "tool-toolA", + toolCallId: "call_A", + state: "input-available", + input: {}, + }, + ], + text: "", + _creationTime: Date.now(), + }, + { + id: "msg1", + key: "thread-1-0", + order: 1, + stepOrder: 0, + status: "streaming", + role: "assistant", + parts: [ + { + type: "tool-toolA", + toolCallId: "call_A", + state: "input-available", + input: {}, + }, + { + type: "tool-toolB", + toolCallId: "call_B", + state: "input-available", + input: {}, + }, + ], + text: "", + _creationTime: Date.now(), + }, + { + id: "msg1", + key: "thread-1-0", + order: 1, + stepOrder: 0, + status: "streaming", + role: "assistant", + parts: [ + { + type: "tool-toolA", + toolCallId: "call_A", + state: "input-available", + input: {}, + }, + { + type: "tool-toolB", + toolCallId: "call_B", + state: "input-available", + input: {}, + }, + { + type: "tool-toolC", + toolCallId: "call_C", + state: "input-available", + input: {}, + }, + ], + text: "", + _creationTime: Date.now(), + }, + { + id: "msg1", + key: "thread-1-0", + order: 1, + stepOrder: 0, + status: "success", + role: "assistant", + parts: [ + { + type: "tool-toolA", + toolCallId: "call_A", + state: "output-available", + input: {}, + output: "success", + }, + { + type: "tool-toolB", + toolCallId: "call_B", + state: "input-available", + input: {}, + }, + { + type: "tool-toolC", + toolCallId: "call_C", + state: "input-available", + input: {}, + }, + ], + text: "", + _creationTime: Date.now(), + }, + ]; + + const result = combineUIMessages(messages); + + expect(result).toHaveLength(1); + expect(result[0].parts).toHaveLength(3); + + const toolCallIds = result[0].parts + .filter((p) => p.type.startsWith("tool-")) + .map((p: any) => p.toolCallId); + + // All tool calls should be present + expect(toolCallIds).toContain("call_A"); + expect(toolCallIds).toContain("call_B"); + expect(toolCallIds).toContain("call_C"); + + // Tool A should have the final state (output-available) + const toolA = result[0].parts.find( + (p: any) => p.type === "tool-toolA" && p.toolCallId === "call_A", + ) as any; + expect(toolA.state).toBe("output-available"); + expect(toolA.output).toBe("success"); + }); + + it("should merge tool calls with same toolCallId", () => { + const messages: UIMessage[] = [ + { + id: "msg1", + key: "thread-1-0", + order: 1, + stepOrder: 0, + status: "streaming", + role: "assistant", + parts: [ + { + type: "tool-toolA", + toolCallId: "call_A", + state: "input-available", + input: { test: "input" }, + }, + ], + text: "", + _creationTime: Date.now(), + }, + { + id: "msg1", + key: "thread-1-0", + order: 1, + stepOrder: 0, + status: "success", + role: "assistant", + parts: [ + { + type: "tool-toolA", + toolCallId: "call_A", + state: "output-available", + input: { test: "input" }, + output: "completed", + }, + ], + text: "", + _creationTime: Date.now(), + }, + ]; + + const result = combineUIMessages(messages); + + expect(result).toHaveLength(1); + expect(result[0].parts).toHaveLength(1); + + const toolCall = result[0].parts[0] as any; + expect(toolCall.toolCallId).toBe("call_A"); + expect(toolCall.state).toBe("output-available"); + expect(toolCall.output).toBe("completed"); + }); +}); diff --git a/src/UIMessages.ts b/src/UIMessages.ts index d5f4a43d..45011f77 100644 --- a/src/UIMessages.ts +++ b/src/UIMessages.ts @@ -8,6 +8,7 @@ import { type SourceUrlUIPart, type StepStartUIPart, type TextUIPart, + type ToolResultPart, type ToolUIPart, type UIDataTypes, type UITools, @@ -54,7 +55,7 @@ export type UIMessage< * @param meta - The metadata to add to the MessageDocs. * @returns */ -export function fromUIMessages( +export async function fromUIMessages( messages: UIMessage[], meta: { threadId: string; @@ -64,59 +65,71 @@ export function fromUIMessages( providerOptions?: ProviderOptions; metadata?: METADATA; }, -): (MessageDoc & { streaming: boolean; metadata?: METADATA })[] { - return messages.flatMap((uiMessage) => { - const stepOrder = uiMessage.stepOrder; - const commonFields = { - ...pick(meta, [ - "threadId", - "userId", - "model", - "provider", - "providerOptions", - "metadata", - ]), - ...omit(uiMessage, ["parts", "role", "key", "text", "userId"]), - userId: uiMessage.userId ?? meta.userId, - status: uiMessage.status === "streaming" ? "pending" : "success", - streaming: uiMessage.status === "streaming", - // to override - _id: uiMessage.id, - tool: false, - } satisfies MessageDoc & { streaming: boolean; metadata?: METADATA }; - const modelMessages = convertToModelMessages([uiMessage]); - return modelMessages - .map((modelMessage, i) => { - if (modelMessage.content.length === 0) { - return undefined; - } - const message = fromModelMessage(modelMessage); - const tool = isTool(message); - const doc: MessageDoc & { streaming: boolean; metadata?: METADATA } = { - ...commonFields, - _id: uiMessage.id + `-${i}`, - stepOrder: stepOrder + i, - message, - tool, - text: extractText(message), - reasoning: extractReasoning(message), - finishReason: tool ? "tool-calls" : "stop", - sources: fromSourceParts(uiMessage.parts), - }; - if (Array.isArray(modelMessage.content)) { - const providerOptions = modelMessage.content.find( - (c) => c.providerOptions, - )?.providerOptions; - if (providerOptions) { - // convertToModelMessages changes providerMetadata to providerOptions - doc.providerMetadata = providerOptions; - doc.providerOptions ??= providerOptions; +): Promise<(MessageDoc & { streaming: boolean; metadata?: METADATA })[]> { + const nested = await Promise.all( + messages.map(async (uiMessage) => { + const stepOrder = uiMessage.stepOrder; + const commonFields = { + ...pick(meta, [ + "threadId", + "userId", + "model", + "provider", + "providerOptions", + "metadata", + ]), + ...omit(uiMessage, ["parts", "role", "key", "text", "userId"]), + userId: uiMessage.userId ?? meta.userId, + status: uiMessage.status === "streaming" ? "pending" : "success", + streaming: uiMessage.status === "streaming", + // to override + _id: uiMessage.id, + tool: false, + } satisfies MessageDoc & { streaming: boolean; metadata?: METADATA }; + const modelMessages = await convertToModelMessages([uiMessage]); + return modelMessages + .map((modelMessage, i) => { + if (modelMessage.content.length === 0) { + return undefined; } - } - return doc; - }) - .filter((d) => d !== undefined); - }); + const message = fromModelMessage(modelMessage); + const tool = isTool(message); + const doc: MessageDoc & { streaming: boolean; metadata?: METADATA } = + { + ...commonFields, + _id: uiMessage.id + `-${i}`, + stepOrder: stepOrder + i, + message, + tool, + text: extractText(message), + reasoning: extractReasoning(message), + finishReason: tool ? "tool-calls" : "stop", + sources: fromSourceParts(uiMessage.parts), + }; + if (Array.isArray(modelMessage.content)) { + // Find a content part with providerOptions (type assertion needed for SDK compatibility) + const partWithProviderOptions = modelMessage.content.find( + (c): c is typeof c & { providerOptions: unknown } => + "providerOptions" in c && c.providerOptions !== undefined, + ); + if (partWithProviderOptions?.providerOptions) { + // convertToModelMessages changes providerMetadata to providerOptions + const providerOptions = + partWithProviderOptions.providerOptions as + | Record> + | undefined; + if (providerOptions) { + doc.providerMetadata = providerOptions; + doc.providerOptions ??= providerOptions; + } + } + } + return doc; + }) + .filter((d) => d !== undefined); + }), + ); + return nested.flat(); } function fromSourceParts(parts: UIMessage["parts"]): Infer[] { @@ -464,10 +477,50 @@ function createAssistantUIMessage< break; } case "tool-result": { + const typedPart = contentPart as unknown as ToolResultPart & { + output: { type: string; value?: unknown; reason?: string }; + }; + + // Check if this is an execution-denied result + if (typedPart.output?.type === "execution-denied") { + const call = allParts.find( + (part) => + part.type === `tool-${contentPart.toolName}` && + "toolCallId" in part && + part.toolCallId === contentPart.toolCallId, + ) as ToolUIPart | undefined; + + if (call) { + call.state = "output-denied"; + if (!("approval" in call) || !call.approval) { + (call as ToolUIPart & { approval?: object }).approval = { + id: "", + approved: false, + reason: typedPart.output.reason, + }; + } else { + const approval = ( + call as ToolUIPart & { + approval: { approved?: boolean; reason?: string }; + } + ).approval; + approval.approved = false; + approval.reason = typedPart.output.reason; + } + } + break; + } + const output = - typeof contentPart.output?.type === "string" - ? contentPart.output.value - : contentPart.output; + typeof typedPart.output?.type === "string" + ? typedPart.output.value + : typedPart.output; + // Check for error at both the content part level (isError) and message level + // isError may exist on stored tool results but isn't in ToolResultPart type + const hasError = + (contentPart as { isError?: boolean }).isError || message.error; + const errorText = + message.error || (hasError ? String(output) : undefined); const call = allParts.find( (part) => part.type === `tool-${contentPart.toolName}` && @@ -475,9 +528,9 @@ function createAssistantUIMessage< part.toolCallId === contentPart.toolCallId, ) as ToolUIPart | undefined; if (call) { - if (message.error) { + if (hasError) { call.state = "output-error"; - call.errorText = message.error; + call.errorText = errorText ?? "Unknown error"; call.output = output; } else { call.state = "output-available"; @@ -488,13 +541,13 @@ function createAssistantUIMessage< "Tool result without preceding tool call.. adding anyways", contentPart, ); - if (message.error) { + if (hasError) { allParts.push({ type: `tool-${contentPart.toolName}`, toolCallId: contentPart.toolCallId, state: "output-error", input: undefined, - errorText: message.error, + errorText: errorText ?? "Unknown error", callProviderMetadata: message.providerMetadata, } satisfies ToolUIPart); } else { @@ -510,8 +563,70 @@ function createAssistantUIMessage< } break; } + case "tool-approval-request": { + // Find the matching tool call + const typedPart = contentPart as { + toolCallId: string; + approvalId: string; + }; + const toolCallPart = allParts.find( + (part) => + "toolCallId" in part && part.toolCallId === typedPart.toolCallId, + ) as ToolUIPart | undefined; + + if (toolCallPart) { + toolCallPart.state = "approval-requested"; + (toolCallPart as ToolUIPart & { approval?: object }).approval = { + id: typedPart.approvalId, + }; + } else { + console.warn( + "Tool approval request without preceding tool call", + contentPart, + ); + } + break; + } + case "tool-approval-response": { + // Find the tool call that has this approval by matching approval.id + const typedPart = contentPart as { + approvalId: string; + approved: boolean; + reason?: string; + }; + const toolCallPart = allParts.find( + (part) => + "approval" in part && + (part as ToolUIPart & { approval?: { id: string } }).approval + ?.id === typedPart.approvalId, + ) as ToolUIPart | undefined; + + if (toolCallPart) { + if (typedPart.approved) { + toolCallPart.state = "approval-responded"; + (toolCallPart as ToolUIPart & { approval?: object }).approval = { + id: typedPart.approvalId, + approved: true, + reason: typedPart.reason, + }; + } else { + toolCallPart.state = "output-denied"; + (toolCallPart as ToolUIPart & { approval?: object }).approval = { + id: typedPart.approvalId, + approved: false, + reason: typedPart.reason, + }; + } + } else { + console.warn( + "Tool approval response without matching approval request", + contentPart, + ); + } + break; + } default: { - const maybeSource = contentPart as SourcePart; + const maybeSource = contentPart as unknown as SourcePart; if (maybeSource.type === "source") { allParts.push(toSourcePart(maybeSource)); } else { @@ -588,11 +703,12 @@ export function combineUIMessages(messages: UIMessage[]): UIMessage[] { const previousPartIndex = newParts.findIndex( (p) => getToolCallId(p) === toolCallId, ); - const previousPart = newParts.splice(previousPartIndex, 1)[0]; - if (!previousPart) { + if (previousPartIndex === -1) { + // Tool call not found in previous parts, add it as new newParts.push(part); continue; } + const previousPart = newParts.splice(previousPartIndex, 1)[0]; newParts.push(mergeParts(previousPart, part)); } acc.push({ diff --git a/src/client/createTool.ts b/src/client/createTool.ts index 556682f8..05767e83 100644 --- a/src/client/createTool.ts +++ b/src/client/createTool.ts @@ -1,9 +1,15 @@ -import type { FlexibleSchema } from "@ai-sdk/provider-utils"; -import type { Tool, ToolCallOptions, ToolSet } from "ai"; +import type { ToolResultOutput } from "@ai-sdk/provider-utils"; +import type { + FlexibleSchema, + ModelMessage, + Tool, + ToolExecutionOptions, + ToolSet, +} from "ai"; import { tool } from "ai"; -import type { Agent } from "./index.js"; import type { GenericActionCtx, GenericDataModel } from "convex/server"; import type { ProviderOptions } from "../validators.js"; +import type { Agent } from "./index.js"; export type ToolCtx = GenericActionCtx & { @@ -13,79 +19,241 @@ export type ToolCtx = messageId?: string; }; +/** + * Function that is called to determine if the tool needs approval before it can be executed. + */ +export type ToolNeedsApprovalFunctionCtx< + INPUT, + Ctx extends ToolCtx = ToolCtx, +> = ( + ctx: Ctx, + input: INPUT, + options: { + /** + * The ID of the tool call. You can use it e.g. when sending tool-call related information with stream data. + */ + toolCallId: string; + /** + * Messages that were sent to the language model to initiate the response that contained the tool call. + * The messages **do not** include the system prompt nor the assistant response that contained the tool call. + */ + messages: ModelMessage[]; + /** + * Additional context. + * + * Experimental (can break in patch releases). + */ + experimental_context?: unknown; + }, +) => boolean | PromiseLike; + +export type ToolExecuteFunctionCtx< + INPUT, + OUTPUT, + Ctx extends ToolCtx = ToolCtx, +> = ( + ctx: Ctx, + input: INPUT, + options: ToolExecutionOptions, +) => AsyncIterable | PromiseLike; + +type NeverOptional = 0 extends 1 & N + ? Partial + : [N] extends [never] + ? Partial> + : T; + +export type ToolOutputPropertiesCtx< + INPUT, + OUTPUT, + Ctx extends ToolCtx = ToolCtx, +> = NeverOptional< + OUTPUT, + | { + /** + * An async function that is called with the arguments from the tool call and produces a result. + * If `execute` (or `handler`) is not provided, the tool will not be executed automatically. + * + * @param input - The input of the tool call. + * @param options.abortSignal - A signal that can be used to abort the tool call. + */ + execute: ToolExecuteFunctionCtx; + outputSchema?: FlexibleSchema; + handler?: never; + } + | { + /** @deprecated Use execute instead. */ + handler: ToolExecuteFunctionCtx; + outputSchema?: FlexibleSchema; + execute?: never; + } + | { + outputSchema: FlexibleSchema; + execute?: never; + handler?: never; + } +>; + +export type ToolInputProperties = + | { + /** + * The schema of the input that the tool expects. + * The language model will use this to generate the input. + * It is also used to validate the output of the language model. + * + * You can use descriptions on the schema properties to make the input understandable for the language model. + */ + inputSchema: FlexibleSchema; + args?: never; + } + | { + /** + * The schema of the input that the tool expects. The language model will use this to generate the input. + * It is also used to validate the output of the language model. + * Use descriptions to make the input understandable for the language model. + * + * @deprecated Use inputSchema instead. + */ + args: FlexibleSchema; + inputSchema?: never; + }; + /** * This is a wrapper around the ai.tool function that adds extra context to the * tool call, including the action context, userId, threadId, and messageId. * @param tool The tool. See https://sdk.vercel.ai/docs/ai-sdk-core/tools-and-tool-calling - * but swap parameters for args and handler for execute. + * Currently contains deprecated parameters `args` and `handler` to maintain backwards compatibility + * but these will be removed in the future. Use `inputSchema` and `execute` instead, respectively. + * * @returns A tool to be used with the AI SDK. */ -export function createTool(def: { - /** - An optional description of what the tool does. - Will be used by the language model to decide whether to use the tool. - Not used for provider-defined tools. +export function createTool( + def: { + /** + * An optional description of what the tool does. + * Will be used by the language model to decide whether to use the tool. + * Not used for provider-defined tools. */ - description?: string; - /** - The schema of the input that the tool expects. The language model will use this to generate the input. - It is also used to validate the output of the language model. - Use descriptions to make the input understandable for the language model. + description?: string; + /** + * An optional title of the tool. */ - args: FlexibleSchema; - /** - An async function that is called with the arguments from the tool call and produces a result. - If not provided, the tool will not be executed automatically. - - @args is the input of the tool call. - @options.abortSignal is a signal that can be used to abort the tool call. + title?: string; + /** + * Additional provider-specific metadata. They are passed through + * to the provider from the AI SDK and enable provider-specific + * functionality that can be fully encapsulated in the provider. */ - handler: ( - ctx: Ctx, - args: INPUT, - options: ToolCallOptions, - ) => PromiseLike | AsyncIterable; - /** - * Provide the context to use, e.g. when defining the tool at runtime. - */ - ctx?: Ctx; - /** - * Optional function that is called when the argument streaming starts. - * Only called when the tool is used in a streaming context. - */ - onInputStart?: ( - ctx: Ctx, - options: ToolCallOptions, - ) => void | PromiseLike; - /** - * Optional function that is called when an argument streaming delta is available. - * Only called when the tool is used in a streaming context. - */ - onInputDelta?: ( - ctx: Ctx, - options: { inputTextDelta: string } & ToolCallOptions, - ) => void | PromiseLike; - /** - * Optional function that is called when a tool call can be started, - * even if the execute function is not provided. - */ - onInputAvailable?: ( - ctx: Ctx, - options: { - input: [INPUT] extends [never] ? undefined : INPUT; - } & ToolCallOptions, - ) => void | PromiseLike; + providerOptions?: ProviderOptions; + } & ToolInputProperties & { + /** + * An optional list of input examples that show the language + * model what the input should look like. + */ + inputExamples?: Array<{ + input: NoInfer; + }>; + /** + * Whether the tool needs approval before it can be executed. + */ + needsApproval?: + | boolean + | ToolNeedsApprovalFunctionCtx< + [INPUT] extends [never] ? unknown : INPUT, + Ctx + >; + /** + * Strict mode setting for the tool. + * + * Providers that support strict mode will use this setting to determine + * how the input should be generated. Strict mode will always produce + * valid inputs, but it might limit what input schemas are supported. + */ + strict?: boolean; + /** + * Provide the context to use, e.g. when defining the tool at runtime. + */ + ctx?: Ctx; + /** + * Optional function that is called when the argument streaming starts. + * Only called when the tool is used in a streaming context. + */ + onInputStart?: ( + ctx: Ctx, + options: ToolExecutionOptions, + ) => void | PromiseLike; + /** + * Optional function that is called when an argument streaming delta is available. + * Only called when the tool is used in a streaming context. + */ + onInputDelta?: ( + ctx: Ctx, + options: { inputTextDelta: string } & ToolExecutionOptions, + ) => void | PromiseLike; + /** + * Optional function that is called when a tool call can be started, + * even if the execute function is not provided. + */ + onInputAvailable?: ( + ctx: Ctx, + options: { + input: [INPUT] extends [never] ? unknown : INPUT; + } & ToolExecutionOptions, + ) => void | PromiseLike; + } & ToolOutputPropertiesCtx & { + /** + * Optional conversion function that maps the tool result to an output that can be used by the language model. + * + * If not provided, the tool result will be sent as a JSON object. + */ + toModelOutput?: ( + ctx: Ctx, + options: { + /** + * The ID of the tool call. You can use it e.g. when sending tool-call related information with stream data. + */ + toolCallId: string; + /** + * The input of the tool call. + */ + input: [INPUT] extends [never] ? unknown : INPUT; + /** + * The output of the tool call. + */ + output: 0 extends 1 & OUTPUT + ? any + : [OUTPUT] extends [never] + ? any + : NoInfer; + }, + ) => ToolResultOutput | PromiseLike; + }, +): Tool { + const inputSchema = def.inputSchema ?? def.args; + if (!inputSchema) + throw new Error("To use a Convex tool, you must provide an `inputSchema` (or `args`)"); - // Extra AI SDK pass-through options. - providerOptions?: ProviderOptions; -}): Tool { - const t = tool({ + const executeHandler = def.execute ?? def.handler; + if (!executeHandler && !def.outputSchema) + throw new Error( + "To use a Convex tool, you must either provide an execute" + + " handler function, define an outputSchema, or both", + ); + + const t = tool({ type: "function", __acceptsCtx: true, ctx: def.ctx, description: def.description, - inputSchema: def.args, - execute(args: INPUT, options: ToolCallOptions) { + title: def.title, + providerOptions: def.providerOptions, + inputSchema, + inputExamples: def.inputExamples, + needsApproval(this: Tool, input, options) { + const needsApproval = def.needsApproval; + if (!needsApproval || typeof needsApproval === "boolean") + return Boolean(needsApproval); + if (!getCtx(this)) { throw new Error( "To use a Convex tool, you must either provide the ctx" + @@ -93,9 +261,28 @@ export function createTool(def: { " call it (which injects the ctx, userId and threadId)", ); } - return def.handler(getCtx(this), args, options); + return needsApproval(getCtx(this), input, options); }, - providerOptions: def.providerOptions, + strict: def.strict, + ...(executeHandler + ? { + execute( + this: Tool, + input: INPUT, + options: ToolExecutionOptions, + ) { + if (!getCtx(this)) { + throw new Error( + "To use a Convex tool, you must either provide the ctx" + + " at definition time (dynamically in an action), or use the Agent to" + + " call it (which injects the ctx, userId and threadId)", + ); + } + return executeHandler(getCtx(this), input, options); + }, + } + : {}), + outputSchema: def.outputSchema, }); if (def.onInputStart) { t.onInputStart = def.onInputStart.bind(t, getCtx(t)); @@ -106,6 +293,9 @@ export function createTool(def: { if (def.onInputAvailable) { t.onInputAvailable = def.onInputAvailable.bind(t, getCtx(t)); } + if (def.toModelOutput) { + t.toModelOutput = def.toModelOutput.bind(t, getCtx(t)); + } return t; } diff --git a/src/client/files.ts b/src/client/files.ts index 6ee78506..ae56581e 100644 --- a/src/client/files.ts +++ b/src/client/files.ts @@ -92,7 +92,7 @@ export async function storeFile( storageId: newStorageId, hash, filename, - mimeType: blob.type, + mediaType: blob.type, }); const url = (await ctx.storage.getUrl(storageId as Id<"_storage">))!; if (storageId !== newStorageId) { @@ -142,8 +142,10 @@ export async function getFile( if (!url) { throw new Error(`File not found in storage: ${file.storageId}`); } + // Support both mediaType (preferred) and mimeType (deprecated) + const mediaType = file.mediaType ?? file.mimeType ?? ""; return { - ...getParts(url, file.mimeType, file.filename), + ...getParts(url, mediaType, file.filename), file: { fileId, url, diff --git a/src/client/index.test.ts b/src/client/index.test.ts index f6e97269..9a2c4e18 100644 --- a/src/client/index.test.ts +++ b/src/client/index.test.ts @@ -193,6 +193,7 @@ describe("filterOutOrphanedToolMessages", () => { type: "tool-call", toolCallId: "1", toolName: "tool1", + input: { test: "test" }, args: { test: "test" }, }, ], diff --git a/src/client/index.ts b/src/client/index.ts index 107a3970..bf6bd5b9 100644 --- a/src/client/index.ts +++ b/src/client/index.ts @@ -6,6 +6,7 @@ import type { } from "@ai-sdk/provider-utils"; import type { CallSettings, + EmbeddingModel, GenerateObjectResult, GenerateTextResult, LanguageModel, @@ -86,6 +87,7 @@ import type { UsageHandler, QueryCtx, AgentPrompt, + Output, } from "./types.js"; import { streamText } from "./streamText.js"; import { errorToString, willContinue } from "./utils.js"; @@ -243,6 +245,14 @@ export class Agent< }, ) {} + /** + * Get the embedding model, prioritizing embeddingModel over textEmbeddingModel. + * @private + */ + private getEmbeddingModel(): EmbeddingModel | undefined { + return this.options.embeddingModel ?? this.options.textEmbeddingModel; + } + /** * Start a new thread with the agent. This will have a fresh history, though if * you pass in a userId you can have it search across other threads for relevant @@ -416,9 +426,7 @@ export class Agent< ...args, tools: (args.tools ?? this.options.tools) as Tools, system: args.system ?? this.options.instructions, - stopWhen: (args.stopWhen ?? this.options.stopWhen) as - | StopCondition - | Array>, + stopWhen: (args.stopWhen ?? this.options.stopWhen) as any, }, { ...this.options, @@ -444,8 +452,7 @@ export class Agent< */ async generateText< TOOLS extends ToolSet | undefined = undefined, - OUTPUT = never, - OUTPUT_PARTIAL = never, + OUTPUT extends Output = never, >( ctx: ActionCtx & CustomCtx, threadOpts: { userId?: string | null; threadId?: string }, @@ -454,7 +461,7 @@ export class Agent< * {@link generateText} function, along with Agent prompt options. */ generateTextArgs: AgentPrompt & - TextArgs, + TextArgs, options?: Options, ): Promise< GenerateTextResult & @@ -469,7 +476,7 @@ export class Agent< type Tools = TOOLS extends undefined ? AgentTools : TOOLS; const steps: StepResult[] = []; try { - const result = (await generateText({ + const result = (await generateText({ ...args, prepareStep: async (options) => { const result = await generateTextArgs.prepareStep?.(options); @@ -504,8 +511,7 @@ export class Agent< */ async streamText< TOOLS extends ToolSet | undefined = undefined, - OUTPUT = never, - PARTIAL_OUTPUT = never, + OUTPUT extends Output = never, >( ctx: ActionCtx & CustomCtx, threadOpts: { userId?: string | null; threadId?: string }, @@ -514,7 +520,7 @@ export class Agent< * {@link streamText} function, along with Agent prompt options. */ streamTextArgs: AgentPrompt & - StreamingTextArgs, + StreamingTextArgs, /** * The {@link ContextOptions} and {@link StorageOptions} * options to use for fetching contextual messages and saving input/output messages. @@ -535,12 +541,12 @@ export class Agent< ): Promise< StreamTextResult< TOOLS extends undefined ? AgentTools : TOOLS, - PARTIAL_OUTPUT + OUTPUT > & GenerationOutputMetadata > { type Tools = TOOLS extends undefined ? AgentTools : TOOLS; - return streamText( + return streamText( ctx, this.component, { @@ -548,9 +554,7 @@ export class Agent< model: streamTextArgs.model ?? this.options.languageModel, tools: (streamTextArgs.tools ?? this.options.tools) as Tools, system: streamTextArgs.system ?? this.options.instructions, - stopWhen: (streamTextArgs.stopWhen ?? this.options.stopWhen) as - | StopCondition - | Array>, + stopWhen: (streamTextArgs.stopWhen ?? this.options.stopWhen) as any, }, { ...threadOpts, @@ -746,7 +750,7 @@ export class Agent< const { skipEmbeddings, ...rest } = args; if (args.embeddings) { embeddings = args.embeddings; - } else if (!skipEmbeddings && this.options.textEmbeddingModel) { + } else if (!skipEmbeddings && this.getEmbeddingModel()) { if (!("runAction" in ctx)) { console.warn( "You're trying to save messages and generate embeddings, but you're in a mutation. " + @@ -862,9 +866,10 @@ export class Agent< contextOptions, getEmbedding: async (text) => { assert("runAction" in ctx); + const embeddingModel = this.getEmbeddingModel(); assert( - this.options.textEmbeddingModel, - "A textEmbeddingModel is required to be set on the Agent that you're doing vector search with", + embeddingModel, + "An embeddingModel (or textEmbeddingModel) is required to be set on the Agent that you're doing vector search with", ); return { embedding: ( @@ -876,7 +881,7 @@ export class Agent< values: [text], }) ).embeddings[0], - textEmbeddingModel: this.options.textEmbeddingModel, + embeddingModel: embeddingModel, }; }, }); @@ -975,10 +980,10 @@ export class Agent< .join(", "), ); } - const { textEmbeddingModel } = this.options; - if (!textEmbeddingModel) { + const embeddingModel = this.getEmbeddingModel(); + if (!embeddingModel) { throw new Error( - "No embeddings were generated for the messages. You must pass a textEmbeddingModel to the agent constructor.", + "No embeddings were generated for the messages. You must pass an embeddingModel (or textEmbeddingModel) to the agent constructor.", ); } await generateAndSaveEmbeddings( @@ -989,7 +994,7 @@ export class Agent< agentName: this.options.name, threadId: messages[0].threadId, userId: messages[0].userId, - textEmbeddingModel, + embeddingModel, }, messages, ); @@ -1440,7 +1445,7 @@ export class Agent< } as GenerateObjectArgs>; const ctx = ( options?.customCtx - ? { ...ctx_, ...options.customCtx(ctx_, targetArgs, llmArgs) } + ? { ...ctx_, ...options.customCtx(ctx_, targetArgs, llmArgs as any) } : ctx_ ) as GenericActionCtx & CustomCtx; const value = await this.generateObject(ctx, targetArgs, llmArgs, { diff --git a/src/client/mockModel.ts b/src/client/mockModel.ts index b7868836..004a3299 100644 --- a/src/client/mockModel.ts +++ b/src/client/mockModel.ts @@ -1,7 +1,7 @@ import type { - LanguageModelV2, - LanguageModelV2Content, - LanguageModelV2StreamPart, + LanguageModelV3, + LanguageModelV3Content, + LanguageModelV3StreamPart, } from "@ai-sdk/provider"; import { simulateReadableStream, type ProviderMetadata } from "ai"; import { assert, pick } from "convex-helpers"; @@ -12,14 +12,20 @@ B B B B B B B B B B B B B B B C C C C C C C C C C C C C C C D D D D D D D D D D D D D D D `; -const DEFAULT_USAGE = { outputTokens: 10, inputTokens: 3, totalTokens: 13 }; +const DEFAULT_USAGE = { + outputTokens: 10, + inputTokens: 3, + totalTokens: 13, + inputTokenDetails: undefined, + outputTokenDetails: undefined, +}; export type MockModelArgs = { - provider?: LanguageModelV2["provider"]; - modelId?: LanguageModelV2["modelId"]; + provider?: LanguageModelV3["provider"]; + modelId?: LanguageModelV3["modelId"]; supportedUrls?: - | LanguageModelV2["supportedUrls"] - | (() => LanguageModelV2["supportedUrls"]); + | LanguageModelV3["supportedUrls"] + | (() => LanguageModelV3["supportedUrls"]); chunkDelayInMs?: number; initialDelayInMs?: number; /** A list of the responses for multiple steps. @@ -27,15 +33,15 @@ export type MockModelArgs = { * then the next list would be after the tool response or another tool call. * Tool responses come from actual tool calls! */ - contentSteps?: LanguageModelV2Content[][]; + contentSteps?: LanguageModelV3Content[][]; /** A single list of content responded from each step. * Provide contentSteps instead if you want to do multi-step responses with * tool calls. */ - content?: LanguageModelV2Content[]; + content?: LanguageModelV3Content[]; // provide either content, contentResponses or doGenerate & doStream - doGenerate?: LanguageModelV2["doGenerate"]; - doStream?: LanguageModelV2["doStream"]; + doGenerate?: LanguageModelV3["doGenerate"]; + doStream?: LanguageModelV3["doStream"]; providerMetadata?: ProviderMetadata; fail?: | boolean @@ -49,23 +55,23 @@ function atMostOneOf(...args: unknown[]) { return args.filter(Boolean).length <= 1; } -export function mockModel(args?: MockModelArgs): LanguageModelV2 { +export function mockModel(args?: MockModelArgs): LanguageModelV3 { return new MockLanguageModel(args ?? {}); } -export class MockLanguageModel implements LanguageModelV2 { - readonly specificationVersion = "v2"; +export class MockLanguageModel implements LanguageModelV3 { + readonly specificationVersion = "v3"; - private _supportedUrls: () => LanguageModelV2["supportedUrls"]; + private _supportedUrls: () => LanguageModelV3["supportedUrls"]; - readonly provider: LanguageModelV2["provider"]; - readonly modelId: LanguageModelV2["modelId"]; + readonly provider: LanguageModelV3["provider"]; + readonly modelId: LanguageModelV3["modelId"]; - doGenerate: LanguageModelV2["doGenerate"]; - doStream: LanguageModelV2["doStream"]; + doGenerate: LanguageModelV3["doGenerate"]; + doStream: LanguageModelV3["doStream"]; - doGenerateCalls: Parameters[0][] = []; - doStreamCalls: Parameters[0][] = []; + doGenerateCalls: Parameters[0][] = []; + doStreamCalls: Parameters[0][] = []; constructor(args: MockModelArgs) { assert( @@ -95,19 +101,19 @@ export class MockLanguageModel implements LanguageModelV2 { "Mock error message"; const metadata = pick(args, ["providerMetadata"]); - const chunkResponses: LanguageModelV2StreamPart[][] = contentSteps.map( + const chunkResponses: LanguageModelV3StreamPart[][] = contentSteps.map( (content) => { - const chunks: LanguageModelV2StreamPart[] = [ + const chunks: LanguageModelV3StreamPart[] = [ { type: "stream-start", warnings: [] }, ]; chunks.push( - ...content.flatMap((c, ci): LanguageModelV2StreamPart[] => { + ...content.flatMap((c, ci): LanguageModelV3StreamPart[] => { if (c.type !== "text" && c.type !== "reasoning") { return [c]; } const metadata = pick(c, ["providerMetadata"]); const deltas = c.text.split(" "); - const parts: LanguageModelV2StreamPart[] = []; + const parts: LanguageModelV3StreamPart[] = []; if (c.type === "reasoning") { parts.push({ type: "reasoning-start", @@ -122,7 +128,7 @@ export class MockLanguageModel implements LanguageModelV2 { delta: (di ? " " : "") + delta, id: `reasoning-${ci}`, ...metadata, - }) satisfies LanguageModelV2StreamPart, + }) satisfies LanguageModelV3StreamPart, ), ); parts.push({ @@ -144,7 +150,7 @@ export class MockLanguageModel implements LanguageModelV2 { delta: (di ? " " : "") + delta, id: `txt-${ci}`, ...metadata, - }) satisfies LanguageModelV2StreamPart, + }) satisfies LanguageModelV3StreamPart, ), ); parts.push({ @@ -166,7 +172,7 @@ export class MockLanguageModel implements LanguageModelV2 { type: "finish", finishReason: fail ? "error" : "stop", usage: DEFAULT_USAGE, - ...metadata, + ...(metadata as any), }); return chunks; }, @@ -187,7 +193,7 @@ export class MockLanguageModel implements LanguageModelV2 { content: contentSteps[callIndex % contentSteps.length], finishReason: "stop" as const, usage: DEFAULT_USAGE, - ...metadata, + ...(metadata as any), warnings: [], }; callIndex++; diff --git a/src/client/saveInputMessages.ts b/src/client/saveInputMessages.ts index 9e1cac64..8c659516 100644 --- a/src/client/saveInputMessages.ts +++ b/src/client/saveInputMessages.ts @@ -31,7 +31,7 @@ export async function saveInputMessages( storageOptions?: { saveMessages?: "all" | "promptAndOutput"; }; - } & Pick, + } & Pick, ): Promise<{ promptMessageId: string | undefined; pendingMessage: MessageDoc; @@ -64,7 +64,7 @@ export async function saveInputMessages( model: string; } | undefined; - if (args.textEmbeddingModel && toSave.length) { + if ((args.embeddingModel ?? args.textEmbeddingModel) && toSave.length) { assert( "runAction" in ctx, "You must be in an action context to generate embeddings", diff --git a/src/client/search.test.ts b/src/client/search.test.ts index c2749c92..a3c1c034 100644 --- a/src/client/search.test.ts +++ b/src/client/search.test.ts @@ -159,6 +159,7 @@ describe("search.ts", () => { type: "tool-call", toolCallId: "call_123", toolName: "test", + input: {}, args: {}, }, ], @@ -202,6 +203,7 @@ describe("search.ts", () => { type: "tool-call", toolCallId: "call_orphaned", toolName: "test", + input: {}, args: {}, }, ], @@ -234,6 +236,170 @@ describe("search.ts", () => { expect(result[0]._id).toBe("0"); expect(result[1]._id).toBe("3"); }); + + it("should keep tool calls with approval responses (but no tool-result yet)", () => { + const messages: MessageDoc[] = [ + { + _id: "1", + message: { + role: "assistant", + content: [ + { type: "text", text: "I'll run the dangerous tool" }, + { + type: "tool-call", + toolCallId: "call_123", + toolName: "dangerousTool", + input: { action: "delete" }, + args: { action: "delete" }, + }, + { + type: "tool-approval-request", + toolCallId: "call_123", + approvalId: "approval_456", + }, + ], + }, + order: 1, + } as MessageDoc, + { + _id: "2", + message: { + role: "tool", + content: [ + { + type: "tool-approval-response", + approvalId: "approval_456", + approved: true, + }, + ], + }, + order: 2, + } as MessageDoc, + ]; + + const result = filterOutOrphanedToolMessages(messages); + expect(result).toHaveLength(2); + // The assistant message should still contain the tool-call + expect(result[0]._id).toBe("1"); + const assistantContent = result[0].message?.content; + expect(Array.isArray(assistantContent)).toBe(true); + if (Array.isArray(assistantContent)) { + const toolCall = assistantContent.find((p) => p.type === "tool-call"); + expect(toolCall).toBeDefined(); + expect(toolCall?.toolCallId).toBe("call_123"); + } + // The tool message with approval response should be kept + expect(result[1]._id).toBe("2"); + }); + + it("should filter out tool calls with approval request but NO approval response", () => { + const messages: MessageDoc[] = [ + { + _id: "1", + message: { + role: "assistant", + content: [ + { type: "text", text: "I'll run the dangerous tool" }, + { + type: "tool-call", + toolCallId: "call_123", + toolName: "dangerousTool", + input: { action: "delete" }, + args: { action: "delete" }, + }, + { + type: "tool-approval-request", + toolCallId: "call_123", + approvalId: "approval_456", + }, + ], + }, + order: 1, + } as MessageDoc, + // No approval response provided + ]; + + const result = filterOutOrphanedToolMessages(messages); + expect(result).toHaveLength(1); + // The assistant message should have the tool-call filtered out + const assistantContent = result[0].message?.content; + expect(Array.isArray(assistantContent)).toBe(true); + if (Array.isArray(assistantContent)) { + // Text and approval-request should remain, but tool-call should be filtered + expect(assistantContent).toHaveLength(2); + expect(assistantContent.find((p) => p.type === "text")).toBeDefined(); + expect( + assistantContent.find((p) => p.type === "tool-approval-request"), + ).toBeDefined(); + expect( + assistantContent.find((p) => p.type === "tool-call"), + ).toBeUndefined(); + } + }); + + it("should handle mix of tool calls with results and with approvals", () => { + const messages: MessageDoc[] = [ + { + _id: "1", + message: { + role: "assistant", + content: [ + { + type: "tool-call", + toolCallId: "call_with_result", + toolName: "safeTool", + input: {}, + args: {}, + }, + { + type: "tool-call", + toolCallId: "call_with_approval", + toolName: "dangerousTool", + input: {}, + args: {}, + }, + { + type: "tool-approval-request", + toolCallId: "call_with_approval", + approvalId: "approval_789", + }, + ], + }, + order: 1, + } as MessageDoc, + { + _id: "2", + message: { + role: "tool", + content: [ + { + type: "tool-result", + toolCallId: "call_with_result", + result: "success", + }, + { + type: "tool-approval-response", + approvalId: "approval_789", + approved: true, + }, + ], + }, + order: 2, + } as MessageDoc, + ]; + + const result = filterOutOrphanedToolMessages(messages); + expect(result).toHaveLength(2); + // Both tool calls should be kept + const assistantContent = result[0].message?.content; + expect(Array.isArray(assistantContent)).toBe(true); + if (Array.isArray(assistantContent)) { + const toolCalls = assistantContent.filter( + (p) => p.type === "tool-call", + ); + expect(toolCalls).toHaveLength(2); + } + }); }); describe("fetchContextMessages", () => { diff --git a/src/client/search.ts b/src/client/search.ts index 6e42832e..e3a73dc4 100644 --- a/src/client/search.ts +++ b/src/client/search.ts @@ -36,10 +36,20 @@ const DEFAULT_VECTOR_SCORE_THRESHOLD = 0.0; // the 8k token limit for some models. const MAX_EMBEDDING_TEXT_LENGTH = 10_000; -export type GetEmbedding = (text: string) => Promise<{ - embedding: number[]; - textEmbeddingModel: string | EmbeddingModel; -}>; +export type GetEmbedding = (text: string) => Promise< + | { + embedding: number[]; + /** @deprecated Use embeddingModel instead. */ + textEmbeddingModel: string | EmbeddingModel; + embeddingModel?: string | EmbeddingModel; + } + | { + embedding: number[]; + /** @deprecated Use embeddingModel instead. */ + textEmbeddingModel?: string | EmbeddingModel; + embeddingModel: string | EmbeddingModel; + } +>; /** * Fetch the context messages for a thread. @@ -178,8 +188,10 @@ export async function fetchRecentAndSearchMessages( if (!embedding && args.getEmbedding) { const embeddingFields = await args.getEmbedding(text); embedding = embeddingFields.embedding; - embeddingModel = embeddingFields.textEmbeddingModel - ? getModelName(embeddingFields.textEmbeddingModel) + const effectiveModel = + embeddingFields.embeddingModel ?? embeddingFields.textEmbeddingModel; + embeddingModel = effectiveModel + ? getModelName(effectiveModel) : undefined; // TODO: if the text matches the target message, save the embedding // for the target message and return the embeddingId on the message. @@ -225,12 +237,19 @@ export async function fetchRecentAndSearchMessages( /** * Filter out tool messages that don't have both a tool call and response. + * For the approval workflow, tool calls with approval responses (but no tool-results yet) + * should also be kept. * @param docs The messages to filter. * @returns The filtered messages. */ export function filterOutOrphanedToolMessages(docs: MessageDoc[]) { const toolCallIds = new Set(); const toolResultIds = new Set(); + // Track approval workflow: toolCallId → approvalId + const approvalRequestsByToolCallId = new Map(); + // Track which approvalIds have responses + const approvalResponseIds = new Set(); + const result: MessageDoc[] = []; for (const doc of docs) { if (doc.message && Array.isArray(doc.message.content)) { @@ -239,17 +258,43 @@ export function filterOutOrphanedToolMessages(docs: MessageDoc[]) { toolCallIds.add(content.toolCallId); } else if (content.type === "tool-result") { toolResultIds.add(content.toolCallId); + } else if (content.type === "tool-approval-request") { + const approvalRequest = content as { + type: "tool-approval-request"; + toolCallId: string; + approvalId: string; + }; + approvalRequestsByToolCallId.set( + approvalRequest.toolCallId, + approvalRequest.approvalId, + ); + } else if (content.type === "tool-approval-response") { + const approvalResponse = content as { + type: "tool-approval-response"; + approvalId: string; + }; + approvalResponseIds.add(approvalResponse.approvalId); } } } } + + // Helper: check if tool call has a corresponding approval response + const hasApprovalResponse = (toolCallId: string) => { + const approvalId = approvalRequestsByToolCallId.get(toolCallId); + return approvalId !== undefined && approvalResponseIds.has(approvalId); + }; + for (const doc of docs) { if ( doc.message?.role === "assistant" && Array.isArray(doc.message.content) ) { const content = doc.message.content.filter( - (p) => p.type !== "tool-call" || toolResultIds.has(p.toolCallId), + (p) => + p.type !== "tool-call" || + toolResultIds.has(p.toolCallId) || + hasApprovalResponse(p.toolCallId), ); if (content.length) { result.push({ @@ -261,9 +306,14 @@ export function filterOutOrphanedToolMessages(docs: MessageDoc[]) { }); } } else if (doc.message?.role === "tool") { - const content = doc.message.content.filter((c) => - toolCallIds.has(c.toolCallId), - ); + const content = doc.message.content.filter((c) => { + // tool-result parts have toolCallId + if (c.type === "tool-result") { + return toolCallIds.has(c.toolCallId); + } + // tool-approval-response parts don't have toolCallId, so include them + return true; + }); if (content.length) { result.push({ ...doc, @@ -294,7 +344,10 @@ export async function embedMessages( userId: string | undefined; threadId: string | undefined; agentName?: string; - } & Pick, + } & Pick< + Config, + "usageHandler" | "textEmbeddingModel" | "embeddingModel" | "callSettings" + >, messages: (ModelMessage | Message)[], ): Promise< | { @@ -304,7 +357,9 @@ export async function embedMessages( } | undefined > { - if (!options.textEmbeddingModel) { + const textEmbeddingModel = + options.embeddingModel ?? options.textEmbeddingModel; + if (!textEmbeddingModel) { return undefined; } let embeddings: @@ -340,7 +395,7 @@ export async function embedMessages( if (textEmbeddings.embeddings.length > 0) { const dimension = textEmbeddings.embeddings[0].length; validateVectorDimension(dimension); - const model = getModelName(options.textEmbeddingModel); + const model = getModelName(textEmbeddingModel); embeddings = { vectors: embeddingsOrNull, dimension, model }; } return embeddings; @@ -355,7 +410,19 @@ export async function embedMessages( */ export async function embedMany( ctx: ActionCtx, - { + args: { + userId: string | undefined; + threadId: string | undefined; + values: string[]; + abortSignal?: AbortSignal; + headers?: Record; + agentName?: string; + } & Pick< + Config, + "usageHandler" | "textEmbeddingModel" | "embeddingModel" | "callSettings" + >, +): Promise<{ embeddings: number[][] }> { + const { userId, threadId, values, @@ -364,24 +431,17 @@ export async function embedMany( agentName, usageHandler, textEmbeddingModel, + embeddingModel, callSettings, - }: { - userId: string | undefined; - threadId: string | undefined; - values: string[]; - abortSignal?: AbortSignal; - headers?: Record; - agentName?: string; - } & Pick, -): Promise<{ embeddings: number[][] }> { - const embeddingModel = textEmbeddingModel; + } = args; + const effectiveEmbeddingModel = embeddingModel ?? textEmbeddingModel; assert( - embeddingModel, - "a textEmbeddingModel is required to be set for vector search", + effectiveEmbeddingModel, + "an embeddingModel (or textEmbeddingModel) is required to be set for vector search", ); const result = await embedMany_({ ...callSettings, - model: embeddingModel, + model: effectiveEmbeddingModel, values, abortSignal, headers, @@ -391,13 +451,15 @@ export async function embedMany( userId, threadId, agentName, - model: getModelName(embeddingModel), - provider: getProviderName(embeddingModel), + model: getModelName(effectiveEmbeddingModel), + provider: getProviderName(effectiveEmbeddingModel), providerMetadata: undefined, usage: { inputTokens: result.usage.tokens, outputTokens: 0, totalTokens: result.usage.tokens, + inputTokenDetails: {} as any, + outputTokenDetails: {} as any, }, }); } @@ -418,17 +480,28 @@ export async function generateAndSaveEmbeddings( threadId: string | undefined; userId: string | undefined; agentName?: string; - textEmbeddingModel: EmbeddingModel; + /** + * @deprecated Use embeddingModel instead. + */ + textEmbeddingModel?: EmbeddingModel; + embeddingModel?: EmbeddingModel; } & Pick, messages: MessageDoc[], ) { + const effectiveEmbeddingModel = + args.embeddingModel ?? args.textEmbeddingModel; + if (!effectiveEmbeddingModel) { + throw new Error( + "an embeddingModel (or textEmbeddingModel) is required to generate and save embeddings", + ); + } const toEmbed = messages.filter((m) => !m.embeddingId && m.message); if (toEmbed.length === 0) { return; } const embeddings = await embedMessages( ctx, - args, + { ...args, embeddingModel: effectiveEmbeddingModel }, toEmbed.map((m) => m.message!), ); if (embeddings && embeddings.vectors.some((v) => v !== null)) { @@ -473,7 +546,8 @@ export async function fetchContextWithPrompt( order: number | undefined; stepOrder: number | undefined; }> { - const { threadId, userId, textEmbeddingModel } = args; + const { threadId, userId, textEmbeddingModel, embeddingModel } = args; + const effectiveEmbeddingModel = embeddingModel ?? textEmbeddingModel; const promptArray = getPromptArray(args.prompt); @@ -496,8 +570,8 @@ export async function fetchContextWithPrompt( contextOptions: args.contextOptions ?? {}, getEmbedding: async (text) => { assert( - textEmbeddingModel, - "A textEmbeddingModel is required to be set on the Agent that you're doing vector search with", + effectiveEmbeddingModel, + "An embeddingModel (or textEmbeddingModel) is required to be set on the Agent that you're doing vector search with", ); return { embedding: ( @@ -505,10 +579,10 @@ export async function fetchContextWithPrompt( ...args, userId, values: [text], - textEmbeddingModel, + embeddingModel: effectiveEmbeddingModel, }) ).embeddings[0], - textEmbeddingModel, + embeddingModel: effectiveEmbeddingModel, }; }, }, @@ -531,7 +605,7 @@ export async function fetchContextWithPrompt( promptArray.push(promptMessage.message); } } - if (!promptMessage.embeddingId && textEmbeddingModel) { + if (!promptMessage.embeddingId && effectiveEmbeddingModel) { // Lazily generate embeddings for the prompt message, if it doesn't have // embeddings yet. This can happen if the message was saved in a mutation // where the LLM is not available. @@ -541,7 +615,7 @@ export async function fetchContextWithPrompt( { ...args, userId, - textEmbeddingModel, + embeddingModel: effectiveEmbeddingModel, }, [promptMessage], ); diff --git a/src/client/streamText.ts b/src/client/streamText.ts index 2666e3f2..ea500dac 100644 --- a/src/client/streamText.ts +++ b/src/client/streamText.ts @@ -17,6 +17,7 @@ import type { AgentPrompt, GenerationOutputMetadata, Options, + Output, } from "./types.js"; import { startGeneration } from "./start.js"; import type { Agent } from "./index.js"; @@ -32,8 +33,7 @@ import { errorToString, willContinue } from "./utils.js"; */ export async function streamText< TOOLS extends ToolSet, - OUTPUT = never, - PARTIAL_OUTPUT = never, + OUTPUT extends Output = never, >( ctx: ActionCtx, component: AgentComponent, @@ -43,7 +43,7 @@ export async function streamText< */ streamTextArgs: AgentPrompt & Omit< - Parameters>[0], + Parameters>[0], "model" | "prompt" | "messages" > & { /** @@ -73,7 +73,7 @@ export async function streamText< saveStreamDeltas?: boolean | StreamingOptions; agentForToolCtx?: Agent; }, -): Promise & GenerationOutputMetadata> { +): Promise & GenerationOutputMetadata> { const { threadId } = options ?? {}; const { args, userId, order, stepOrder, promptMessageId, ...call } = await startGeneration(ctx, component, streamTextArgs, options); @@ -141,7 +141,7 @@ export async function streamText< await call.save({ step }, createPendingMessage); return args.onStepFinish?.(step); }, - }) as StreamTextResult; + }) as StreamTextResult; const stream = streamer?.consumeStream( result.toUIMessageStream>(), ); diff --git a/src/client/types.ts b/src/client/types.ts index 0d4d3372..fa23bc10 100644 --- a/src/client/types.ts +++ b/src/client/types.ts @@ -21,6 +21,14 @@ import type { CallSettings, generateObject, } from "ai"; + +export interface Output { + name: string; + responseFormat: any; + parseCompleteOutput: any; + parsePartialOutput: any; + createElementStreamTransform: any; +} import type { GenericActionCtx, GenericDataModel, @@ -99,8 +107,21 @@ export type Config = { * const myAgent = new Agent(components.agent, { * ... * textEmbeddingModel: openai.embedding("text-embedding-3-small") + * @deprecated — Use embeddingModel instead. + */ + textEmbeddingModel?: EmbeddingModel; + /** + * The model to use for text embeddings. Optional. + * If specified, it will use this for generating vector embeddings + * of chats, and can opt-in to doing vector search for automatic context + * on generateText, etc. + * e.g. + * import { openai } from "@ai-sdk/openai" + * const myAgent = new Agent(components.agent, { + * ... + * embeddingModel: openai.embedding("text-embedding-3-small") */ - textEmbeddingModel?: EmbeddingModel; + embeddingModel?: EmbeddingModel; /** * Options to determine what messages are included as context in message * generation. To disable any messages automatically being added, pass: @@ -330,14 +351,12 @@ export type AgentComponent = ComponentApi; export type TextArgs< AgentTools extends ToolSet, TOOLS extends ToolSet | undefined = undefined, - OUTPUT = never, - OUTPUT_PARTIAL = never, + OUTPUT extends Output = never, > = Omit< Parameters< typeof generateText< TOOLS extends undefined ? AgentTools : TOOLS, - OUTPUT, - OUTPUT_PARTIAL + OUTPUT > >[0], "model" | "prompt" | "messages" @@ -352,14 +371,12 @@ export type TextArgs< export type StreamingTextArgs< AgentTools extends ToolSet, TOOLS extends ToolSet | undefined = undefined, - OUTPUT = never, - OUTPUT_PARTIAL = never, + OUTPUT extends Output = never, > = Omit< Parameters< typeof streamText< TOOLS extends undefined ? AgentTools : TOOLS, - OUTPUT, - OUTPUT_PARTIAL + OUTPUT > >[0], "model" | "prompt" | "messages" @@ -474,15 +491,13 @@ export interface Thread { */ generateText< TOOLS extends ToolSet | undefined = undefined, - OUTPUT = never, - OUTPUT_PARTIAL = never, + OUTPUT extends Output = never, >( generateTextArgs: AgentPrompt & TextArgs< TOOLS extends undefined ? DefaultTools : TOOLS, TOOLS, - OUTPUT, - OUTPUT_PARTIAL + OUTPUT >, options?: Options, ): Promise< @@ -502,15 +517,13 @@ export interface Thread { */ streamText< TOOLS extends ToolSet | undefined = undefined, - OUTPUT = never, - PARTIAL_OUTPUT = never, + OUTPUT extends Output = never, >( streamTextArgs: AgentPrompt & StreamingTextArgs< TOOLS extends undefined ? DefaultTools : TOOLS, TOOLS, - OUTPUT, - PARTIAL_OUTPUT + OUTPUT >, options?: Options & { /** @@ -528,7 +541,7 @@ export interface Thread { ): Promise< StreamTextResult< TOOLS extends undefined ? DefaultTools : TOOLS, - PARTIAL_OUTPUT + OUTPUT > & ThreadOutputMetadata >; diff --git a/src/component/_generated/component.ts b/src/component/_generated/component.ts index 228108ff..f6708edc 100644 --- a/src/component/_generated/component.ts +++ b/src/component/_generated/component.ts @@ -56,7 +56,8 @@ export type ComponentApi = { filename?: string; hash: string; - mimeType: string; + mediaType?: string; + mimeType?: string; storageId: string; }, { fileId: string; storageId: string }, @@ -86,7 +87,8 @@ export type ComponentApi = filename?: string; hash: string; lastTouchedAt: number; - mimeType: string; + mediaType?: string; + mimeType?: string; refcount: number; storageId: string; }, @@ -114,7 +116,8 @@ export type ComponentApi = filename?: string; hash: string; lastTouchedAt: number; - mimeType: string; + mediaType?: string; + mimeType?: string; refcount: number; storageId: string; }>; @@ -182,6 +185,7 @@ export type ComponentApi = } | { image: string | ArrayBuffer; + mediaType?: string; mimeType?: string; providerOptions?: Record< string, @@ -192,7 +196,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -226,7 +231,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -262,8 +268,25 @@ export type ComponentApi = >; type: "redacted-reasoning"; } + | { + args?: any; + input: any; + providerExecuted?: boolean; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + toolName: string; + type: "tool-call"; + } | { args: any; + input?: any; providerExecuted?: boolean; providerMetadata?: Record< string, @@ -289,19 +312,120 @@ export type ComponentApi = >; isError?: boolean; output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } | { type: "content"; value: Array< - | { text: string; type: "text" } + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } | { data: string; mediaType: string; type: "media"; } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } >; }; providerExecuted?: boolean; @@ -349,38 +473,167 @@ export type ComponentApi = title: string; type: "source"; } + | { + approvalId: string; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + type: "tool-approval-request"; + } >; providerOptions?: Record>; role: "assistant"; } | { - content: Array<{ - args?: any; - experimental_content?: Array< - | { text: string; type: "text" } - | { data: string; mimeType?: string; type: "image" } - >; - isError?: boolean; - output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } - | { - type: "content"; - value: Array< - | { text: string; type: "text" } - | { data: string; mediaType: string; type: "media" } - >; - }; - providerExecuted?: boolean; - providerMetadata?: Record>; - providerOptions?: Record>; - result?: any; - toolCallId: string; - toolName: string; - type: "tool-result"; - }>; + content: Array< + | { + args?: any; + experimental_content?: Array< + | { text: string; type: "text" } + | { data: string; mimeType?: string; type: "image" } + >; + isError?: boolean; + output?: + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } + | { + type: "content"; + value: Array< + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } + | { + data: string; + mediaType: string; + type: "media"; + } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } + >; + }; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + result?: any; + toolCallId: string; + toolName: string; + type: "tool-result"; + } + | { + approvalId: string; + approved: boolean; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + reason?: string; + type: "tool-approval-response"; + } + >; providerOptions?: Record>; role: "tool"; } @@ -485,6 +738,7 @@ export type ComponentApi = } | { image: string | ArrayBuffer; + mediaType?: string; mimeType?: string; providerOptions?: Record< string, @@ -495,7 +749,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -529,7 +784,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -565,8 +821,25 @@ export type ComponentApi = >; type: "redacted-reasoning"; } + | { + args?: any; + input: any; + providerExecuted?: boolean; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + toolName: string; + type: "tool-call"; + } | { args: any; + input?: any; providerExecuted?: boolean; providerMetadata?: Record< string, @@ -592,19 +865,120 @@ export type ComponentApi = >; isError?: boolean; output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } | { type: "content"; value: Array< - | { text: string; type: "text" } + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } | { data: string; mediaType: string; type: "media"; } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } >; }; providerExecuted?: boolean; @@ -652,38 +1026,167 @@ export type ComponentApi = title: string; type: "source"; } + | { + approvalId: string; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + type: "tool-approval-request"; + } >; providerOptions?: Record>; role: "assistant"; } | { - content: Array<{ - args?: any; - experimental_content?: Array< - | { text: string; type: "text" } - | { data: string; mimeType?: string; type: "image" } - >; - isError?: boolean; - output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } - | { - type: "content"; - value: Array< - | { text: string; type: "text" } - | { data: string; mediaType: string; type: "media" } - >; - }; - providerExecuted?: boolean; - providerMetadata?: Record>; - providerOptions?: Record>; - result?: any; - toolCallId: string; - toolName: string; - type: "tool-result"; - }>; + content: Array< + | { + args?: any; + experimental_content?: Array< + | { text: string; type: "text" } + | { data: string; mimeType?: string; type: "image" } + >; + isError?: boolean; + output?: + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } + | { + type: "content"; + value: Array< + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } + | { + data: string; + mediaType: string; + type: "media"; + } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } + >; + }; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + result?: any; + toolCallId: string; + toolName: string; + type: "tool-result"; + } + | { + approvalId: string; + approved: boolean; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + reason?: string; + type: "tool-approval-response"; + } + >; providerOptions?: Record>; role: "tool"; } @@ -839,6 +1342,7 @@ export type ComponentApi = } | { image: string | ArrayBuffer; + mediaType?: string; mimeType?: string; providerOptions?: Record>; type: "image"; @@ -846,7 +1350,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -874,7 +1379,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -902,7 +1408,8 @@ export type ComponentApi = type: "redacted-reasoning"; } | { - args: any; + args?: any; + input: any; providerExecuted?: boolean; providerMetadata?: Record< string, @@ -913,6 +1420,22 @@ export type ComponentApi = toolName: string; type: "tool-call"; } + | { + args: any; + input?: any; + providerExecuted?: boolean; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + toolName: string; + type: "tool-call"; + } | { args?: any; experimental_content?: Array< @@ -921,19 +1444,120 @@ export type ComponentApi = >; isError?: boolean; output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } | { type: "content"; value: Array< - | { text: string; type: "text" } + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } | { data: string; mediaType: string; type: "media"; } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } >; }; providerExecuted?: boolean; @@ -972,88 +1596,214 @@ export type ComponentApi = title: string; type: "source"; } + | { + approvalId: string; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record>; + toolCallId: string; + type: "tool-approval-request"; + } >; providerOptions?: Record>; role: "assistant"; } | { - content: Array<{ - args?: any; - experimental_content?: Array< - | { text: string; type: "text" } - | { data: string; mimeType?: string; type: "image" } - >; - isError?: boolean; - output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } - | { - type: "content"; - value: Array< - | { text: string; type: "text" } - | { data: string; mediaType: string; type: "media" } - >; - }; - providerExecuted?: boolean; - providerMetadata?: Record>; - providerOptions?: Record>; - result?: any; - toolCallId: string; - toolName: string; - type: "tool-result"; - }>; - providerOptions?: Record>; - role: "tool"; - } - | { - content: string; - providerOptions?: Record>; - role: "system"; - }; - model?: string; - order: number; - provider?: string; - providerMetadata?: Record>; - providerOptions?: Record>; - reasoning?: string; - reasoningDetails?: Array< - | { - providerMetadata?: Record>; - providerOptions?: Record>; - signature?: string; - text: string; - type: "reasoning"; - } - | { signature?: string; text: string; type: "text" } - | { data: string; type: "redacted" } - >; - sources?: Array< - | { - id: string; - providerMetadata?: Record>; - providerOptions?: Record>; - sourceType: "url"; - title?: string; - type?: "source"; - url: string; - } - | { - filename?: string; - id: string; - mediaType: string; - providerMetadata?: Record>; - providerOptions?: Record>; - sourceType: "document"; - title: string; - type: "source"; - } - >; - status: "pending" | "success" | "failed"; - stepOrder: number; - text?: string; - threadId: string; + content: Array< + | { + args?: any; + experimental_content?: Array< + | { text: string; type: "text" } + | { data: string; mimeType?: string; type: "image" } + >; + isError?: boolean; + output?: + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } + | { + type: "content"; + value: Array< + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } + | { + data: string; + mediaType: string; + type: "media"; + } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } + >; + }; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + result?: any; + toolCallId: string; + toolName: string; + type: "tool-result"; + } + | { + approvalId: string; + approved: boolean; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + reason?: string; + type: "tool-approval-response"; + } + >; + providerOptions?: Record>; + role: "tool"; + } + | { + content: string; + providerOptions?: Record>; + role: "system"; + }; + model?: string; + order: number; + provider?: string; + providerMetadata?: Record>; + providerOptions?: Record>; + reasoning?: string; + reasoningDetails?: Array< + | { + providerMetadata?: Record>; + providerOptions?: Record>; + signature?: string; + text: string; + type: "reasoning"; + } + | { signature?: string; text: string; type: "text" } + | { data: string; type: "redacted" } + >; + sources?: Array< + | { + id: string; + providerMetadata?: Record>; + providerOptions?: Record>; + sourceType: "url"; + title?: string; + type?: "source"; + url: string; + } + | { + filename?: string; + id: string; + mediaType: string; + providerMetadata?: Record>; + providerOptions?: Record>; + sourceType: "document"; + title: string; + type: "source"; + } + >; + status: "pending" | "success" | "failed"; + stepOrder: number; + text?: string; + threadId: string; tool: boolean; usage?: { cachedInputTokens?: number; @@ -1134,6 +1884,7 @@ export type ComponentApi = } | { image: string | ArrayBuffer; + mediaType?: string; mimeType?: string; providerOptions?: Record< string, @@ -1144,7 +1895,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -1178,7 +1930,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -1214,8 +1967,25 @@ export type ComponentApi = >; type: "redacted-reasoning"; } + | { + args?: any; + input: any; + providerExecuted?: boolean; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + toolName: string; + type: "tool-call"; + } | { args: any; + input?: any; providerExecuted?: boolean; providerMetadata?: Record< string, @@ -1241,19 +2011,120 @@ export type ComponentApi = >; isError?: boolean; output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } | { type: "content"; value: Array< - | { text: string; type: "text" } + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } | { data: string; mediaType: string; type: "media"; } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } >; }; providerExecuted?: boolean; @@ -1301,38 +2172,167 @@ export type ComponentApi = title: string; type: "source"; } + | { + approvalId: string; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + type: "tool-approval-request"; + } >; providerOptions?: Record>; role: "assistant"; } | { - content: Array<{ - args?: any; - experimental_content?: Array< - | { text: string; type: "text" } - | { data: string; mimeType?: string; type: "image" } - >; - isError?: boolean; - output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } - | { - type: "content"; - value: Array< - | { text: string; type: "text" } - | { data: string; mediaType: string; type: "media" } - >; - }; - providerExecuted?: boolean; - providerMetadata?: Record>; - providerOptions?: Record>; - result?: any; - toolCallId: string; - toolName: string; - type: "tool-result"; - }>; + content: Array< + | { + args?: any; + experimental_content?: Array< + | { text: string; type: "text" } + | { data: string; mimeType?: string; type: "image" } + >; + isError?: boolean; + output?: + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } + | { + type: "content"; + value: Array< + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } + | { + data: string; + mediaType: string; + type: "media"; + } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } + >; + }; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + result?: any; + toolCallId: string; + toolName: string; + type: "tool-result"; + } + | { + approvalId: string; + approved: boolean; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + reason?: string; + type: "tool-approval-response"; + } + >; providerOptions?: Record>; role: "tool"; } @@ -1455,6 +2455,7 @@ export type ComponentApi = } | { image: string | ArrayBuffer; + mediaType?: string; mimeType?: string; providerOptions?: Record>; type: "image"; @@ -1462,7 +2463,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -1490,7 +2492,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -1518,7 +2521,8 @@ export type ComponentApi = type: "redacted-reasoning"; } | { - args: any; + args?: any; + input: any; providerExecuted?: boolean; providerMetadata?: Record< string, @@ -1529,6 +2533,22 @@ export type ComponentApi = toolName: string; type: "tool-call"; } + | { + args: any; + input?: any; + providerExecuted?: boolean; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + toolName: string; + type: "tool-call"; + } | { args?: any; experimental_content?: Array< @@ -1537,19 +2557,120 @@ export type ComponentApi = >; isError?: boolean; output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } | { type: "content"; value: Array< - | { text: string; type: "text" } + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } | { data: string; mediaType: string; type: "media"; } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } >; }; providerExecuted?: boolean; @@ -1588,47 +2709,173 @@ export type ComponentApi = title: string; type: "source"; } + | { + approvalId: string; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record>; + toolCallId: string; + type: "tool-approval-request"; + } >; providerOptions?: Record>; role: "assistant"; } | { - content: Array<{ - args?: any; - experimental_content?: Array< - | { text: string; type: "text" } - | { data: string; mimeType?: string; type: "image" } - >; - isError?: boolean; - output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } - | { - type: "content"; - value: Array< - | { text: string; type: "text" } - | { data: string; mediaType: string; type: "media" } - >; - }; - providerExecuted?: boolean; - providerMetadata?: Record>; - providerOptions?: Record>; - result?: any; - toolCallId: string; - toolName: string; - type: "tool-result"; - }>; - providerOptions?: Record>; - role: "tool"; - } - | { - content: string; - providerOptions?: Record>; - role: "system"; - }; - model?: string; + content: Array< + | { + args?: any; + experimental_content?: Array< + | { text: string; type: "text" } + | { data: string; mimeType?: string; type: "image" } + >; + isError?: boolean; + output?: + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } + | { + type: "content"; + value: Array< + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } + | { + data: string; + mediaType: string; + type: "media"; + } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } + >; + }; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + result?: any; + toolCallId: string; + toolName: string; + type: "tool-result"; + } + | { + approvalId: string; + approved: boolean; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + reason?: string; + type: "tool-approval-response"; + } + >; + providerOptions?: Record>; + role: "tool"; + } + | { + content: string; + providerOptions?: Record>; + role: "system"; + }; + model?: string; order: number; provider?: string; providerMetadata?: Record>; @@ -1729,6 +2976,7 @@ export type ComponentApi = } | { image: string | ArrayBuffer; + mediaType?: string; mimeType?: string; providerOptions?: Record>; type: "image"; @@ -1736,7 +2984,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -1764,7 +3013,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -1792,7 +3042,8 @@ export type ComponentApi = type: "redacted-reasoning"; } | { - args: any; + args?: any; + input: any; providerExecuted?: boolean; providerMetadata?: Record< string, @@ -1803,6 +3054,22 @@ export type ComponentApi = toolName: string; type: "tool-call"; } + | { + args: any; + input?: any; + providerExecuted?: boolean; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + toolName: string; + type: "tool-call"; + } | { args?: any; experimental_content?: Array< @@ -1811,19 +3078,120 @@ export type ComponentApi = >; isError?: boolean; output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } | { type: "content"; value: Array< - | { text: string; type: "text" } + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } | { data: string; mediaType: string; type: "media"; } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } >; }; providerExecuted?: boolean; @@ -1862,38 +3230,164 @@ export type ComponentApi = title: string; type: "source"; } + | { + approvalId: string; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record>; + toolCallId: string; + type: "tool-approval-request"; + } >; providerOptions?: Record>; role: "assistant"; } | { - content: Array<{ - args?: any; - experimental_content?: Array< - | { text: string; type: "text" } - | { data: string; mimeType?: string; type: "image" } - >; - isError?: boolean; - output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } - | { - type: "content"; - value: Array< - | { text: string; type: "text" } - | { data: string; mediaType: string; type: "media" } - >; - }; - providerExecuted?: boolean; - providerMetadata?: Record>; - providerOptions?: Record>; - result?: any; - toolCallId: string; - toolName: string; - type: "tool-result"; - }>; + content: Array< + | { + args?: any; + experimental_content?: Array< + | { text: string; type: "text" } + | { data: string; mimeType?: string; type: "image" } + >; + isError?: boolean; + output?: + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } + | { + type: "content"; + value: Array< + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } + | { + data: string; + mediaType: string; + type: "media"; + } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } + >; + }; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + result?: any; + toolCallId: string; + toolName: string; + type: "tool-result"; + } + | { + approvalId: string; + approved: boolean; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + reason?: string; + type: "tool-approval-response"; + } + >; providerOptions?: Record>; role: "tool"; } @@ -1996,6 +3490,7 @@ export type ComponentApi = } | { image: string | ArrayBuffer; + mediaType?: string; mimeType?: string; providerOptions?: Record< string, @@ -2006,7 +3501,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -2040,7 +3536,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -2076,8 +3573,25 @@ export type ComponentApi = >; type: "redacted-reasoning"; } + | { + args?: any; + input: any; + providerExecuted?: boolean; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + toolName: string; + type: "tool-call"; + } | { args: any; + input?: any; providerExecuted?: boolean; providerMetadata?: Record< string, @@ -2103,19 +3617,120 @@ export type ComponentApi = >; isError?: boolean; output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } | { type: "content"; value: Array< - | { text: string; type: "text" } + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } | { data: string; mediaType: string; type: "media"; } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } >; }; providerExecuted?: boolean; @@ -2163,38 +3778,167 @@ export type ComponentApi = title: string; type: "source"; } + | { + approvalId: string; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + type: "tool-approval-request"; + } >; providerOptions?: Record>; role: "assistant"; } | { - content: Array<{ - args?: any; - experimental_content?: Array< - | { text: string; type: "text" } - | { data: string; mimeType?: string; type: "image" } - >; - isError?: boolean; - output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } - | { - type: "content"; - value: Array< - | { text: string; type: "text" } - | { data: string; mediaType: string; type: "media" } - >; - }; - providerExecuted?: boolean; - providerMetadata?: Record>; - providerOptions?: Record>; - result?: any; - toolCallId: string; - toolName: string; - type: "tool-result"; - }>; + content: Array< + | { + args?: any; + experimental_content?: Array< + | { text: string; type: "text" } + | { data: string; mimeType?: string; type: "image" } + >; + isError?: boolean; + output?: + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } + | { + type: "content"; + value: Array< + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } + | { + data: string; + mediaType: string; + type: "media"; + } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } + >; + }; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + result?: any; + toolCallId: string; + toolName: string; + type: "tool-result"; + } + | { + approvalId: string; + approved: boolean; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + reason?: string; + type: "tool-approval-response"; + } + >; providerOptions?: Record>; role: "tool"; } @@ -2241,6 +3985,7 @@ export type ComponentApi = } | { image: string | ArrayBuffer; + mediaType?: string; mimeType?: string; providerOptions?: Record>; type: "image"; @@ -2248,7 +3993,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -2276,7 +4022,8 @@ export type ComponentApi = | { data: string | ArrayBuffer; filename?: string; - mimeType: string; + mediaType?: string; + mimeType?: string; providerMetadata?: Record< string, Record @@ -2304,7 +4051,8 @@ export type ComponentApi = type: "redacted-reasoning"; } | { - args: any; + args?: any; + input: any; providerExecuted?: boolean; providerMetadata?: Record< string, @@ -2315,6 +4063,22 @@ export type ComponentApi = toolName: string; type: "tool-call"; } + | { + args: any; + input?: any; + providerExecuted?: boolean; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record< + string, + Record + >; + toolCallId: string; + toolName: string; + type: "tool-call"; + } | { args?: any; experimental_content?: Array< @@ -2323,19 +4087,120 @@ export type ComponentApi = >; isError?: boolean; output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } | { type: "content"; value: Array< - | { text: string; type: "text" } + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } | { data: string; mediaType: string; type: "media"; } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } >; }; providerExecuted?: boolean; @@ -2374,38 +4239,164 @@ export type ComponentApi = title: string; type: "source"; } + | { + approvalId: string; + providerMetadata?: Record< + string, + Record + >; + providerOptions?: Record>; + toolCallId: string; + type: "tool-approval-request"; + } >; providerOptions?: Record>; role: "assistant"; } | { - content: Array<{ - args?: any; - experimental_content?: Array< - | { text: string; type: "text" } - | { data: string; mimeType?: string; type: "image" } - >; - isError?: boolean; - output?: - | { type: "text"; value: string } - | { type: "json"; value: any } - | { type: "error-text"; value: string } - | { type: "error-json"; value: any } - | { - type: "content"; - value: Array< - | { text: string; type: "text" } - | { data: string; mediaType: string; type: "media" } - >; - }; - providerExecuted?: boolean; - providerMetadata?: Record>; - providerOptions?: Record>; - result?: any; - toolCallId: string; - toolName: string; - type: "tool-result"; - }>; + content: Array< + | { + args?: any; + experimental_content?: Array< + | { text: string; type: "text" } + | { data: string; mimeType?: string; type: "image" } + >; + isError?: boolean; + output?: + | { + providerOptions?: Record< + string, + Record + >; + type: "text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-text"; + value: string; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "error-json"; + value: any; + } + | { + providerOptions?: Record< + string, + Record + >; + reason?: string; + type: "execution-denied"; + } + | { + type: "content"; + value: Array< + | { + providerOptions?: Record< + string, + Record + >; + text: string; + type: "text"; + } + | { + data: string; + mediaType: string; + type: "media"; + } + | { + data: string; + filename?: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "file-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "file-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "file-id"; + } + | { + data: string; + mediaType: string; + providerOptions?: Record< + string, + Record + >; + type: "image-data"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "image-url"; + url: string; + } + | { + fileId: string | Record; + providerOptions?: Record< + string, + Record + >; + type: "image-file-id"; + } + | { + providerOptions?: Record< + string, + Record + >; + type: "custom"; + } + >; + }; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + result?: any; + toolCallId: string; + toolName: string; + type: "tool-result"; + } + | { + approvalId: string; + approved: boolean; + providerExecuted?: boolean; + providerMetadata?: Record>; + providerOptions?: Record>; + reason?: string; + type: "tool-approval-response"; + } + >; providerOptions?: Record>; role: "tool"; } diff --git a/src/component/files.ts b/src/component/files.ts index c6ce165b..89506f4c 100644 --- a/src/component/files.ts +++ b/src/component/files.ts @@ -9,7 +9,9 @@ const addFileArgs = v.object({ storageId: v.string(), hash: v.string(), filename: v.optional(v.string()), - mimeType: v.string(), + mediaType: v.optional(v.string()), + /** @deprecated Use `mediaType` instead. */ + mimeType: v.optional(v.string()), }); export const addFile = mutation({ @@ -25,6 +27,9 @@ export async function addFileHandler( ctx: MutationCtx, args: Infer, ) { + // Support both mediaType (preferred) and mimeType (deprecated) + const mediaType = args.mediaType ?? args.mimeType; + const existingFile = await ctx.db .query("files") .withIndex("hash", (q) => q.eq("hash", args.hash)) @@ -42,7 +47,11 @@ export async function addFileHandler( }; } const fileId = await ctx.db.insert("files", { - ...args, + storageId: args.storageId, + hash: args.hash, + filename: args.filename, + mediaType, + mimeType: args.mimeType, // Keep for backwards compatibility // We start out with it unused - when it's saved in a message we increment. refcount: 0, lastTouchedAt: Date.now(), diff --git a/src/component/messages.test.ts b/src/component/messages.test.ts index 9581b8f6..0c273307 100644 --- a/src/component/messages.test.ts +++ b/src/component/messages.test.ts @@ -68,7 +68,7 @@ describe("agent", () => { content: [ { type: "tool-call", - args: { a: 1 }, + input: { a: 1 }, toolCallId: "1", toolName: "tool", }, @@ -258,7 +258,7 @@ describe("agent", () => { content: [ { type: "tool-call", - args: { a: 1 }, + input: { a: 1 }, toolCallId: "1", toolName: "tool", }, @@ -389,7 +389,7 @@ describe("agent", () => { content: [ { type: "tool-call", - args: { a: 1 }, + input: { a: 1 }, toolCallId: "1", toolName: "tool", }, @@ -408,7 +408,7 @@ describe("agent", () => { content: [ { type: "tool-call", - args: { a: 2, b: 3 }, + input: { a: 2, b: 3 }, toolCallId: "1", toolName: "tool", }, @@ -422,7 +422,7 @@ describe("agent", () => { content: [ { type: "tool-call", - args: { a: 2, b: 3 }, + input: { a: 2, b: 3 }, toolCallId: "1", toolName: "tool", }, diff --git a/src/component/schema.ts b/src/component/schema.ts index da319828..644cc62e 100644 --- a/src/component/schema.ts +++ b/src/component/schema.ts @@ -149,7 +149,9 @@ export const schema = defineSchema({ files: defineTable({ storageId: v.string(), - mimeType: v.string(), + mediaType: v.optional(v.string()), + /** @deprecated Use `mediaType` instead. */ + mimeType: v.optional(v.string()), filename: v.optional(v.string()), hash: v.string(), refcount: v.number(), diff --git a/src/component/streams.ts b/src/component/streams.ts index ef9dc863..4c3499fe 100644 --- a/src/component/streams.ts +++ b/src/component/streams.ts @@ -545,7 +545,7 @@ export async function getStreamingMessagesWithMetadata( // We don't save messages that have already been saved const numToSkip = stepOrder - streamingMessage.stepOrder; const messages = await Promise.all( - fromUIMessages(uiMessages, streamingMessage) + (await fromUIMessages(uiMessages, streamingMessage)) .slice(numToSkip) .filter((m) => m.message !== undefined) .map(async (msg) => { diff --git a/src/deltas.test.ts b/src/deltas.test.ts index fab52758..6be75868 100644 --- a/src/deltas.test.ts +++ b/src/deltas.test.ts @@ -156,6 +156,49 @@ describe("UIMessageChunks", () => { }); }); +describe("UIMessageChunks - continuation stream", () => { + it("gracefully handles tool-result without tool-call in continuation stream after approval", async () => { + // This simulates what happens after tool approval: + // Stream A: tool-call, tool-approval-request -> finishes + // User approves + // Stream B: tool-result (referencing tool-call from Stream A) -> this test + // + // The AI SDK's readUIMessageStream expects tool-call before tool-result, + // but they're in different streams. The onError handler should gracefully + // ignore this error since stored messages provide the fallback. + const uiMessage = blankUIMessage( + { + streamId: "continuation-stream", + status: "streaming", + order: 1, + stepOrder: 0, + format: "UIMessageChunk", + agentName: "agent1", + }, + "thread1", + ); + + // Send a tool-result without the corresponding tool-call in this stream + // This would normally throw "No tool invocation found" error + const updatedMessage = await updateFromUIMessageChunks(uiMessage, [ + { type: "start" }, + { type: "start-step" }, + { + type: "tool-output-available", + toolCallId: "call_from_previous_stream", + output: "Tool execution result", + }, + { type: "finish-step" }, + { type: "finish" }, + ]); + + // The message should NOT be marked as failed - the error should be suppressed + expect(updatedMessage.status).not.toBe("failed"); + // The stream still processes (even if tool-output isn't reflected without tool-input) + expect(updatedMessage).toBeDefined(); + }); +}); + describe("mergeDeltas", () => { it("merges a single text-delta into a message", () => { const streamId = "s1"; @@ -533,4 +576,51 @@ describe("mergeDeltas", () => { }, ]); }); + + it("handles streaming tool-approval-request and updates tool state", () => { + const streamId = "s10"; + const deltas = [ + { + streamId, + start: 0, + end: 1, + parts: [ + { + type: "tool-call", + toolCallId: "call1", + toolName: "dangerousTool", + input: { action: "delete" }, + }, + ], + } satisfies StreamDelta, + { + streamId, + start: 1, + end: 2, + parts: [ + { + type: "tool-approval-request", + toolCallId: "call1", + approvalId: "approval1", + }, + ], + } satisfies StreamDelta, + ]; + const [[message], _, changed] = deriveUIMessagesFromTextStreamParts( + "thread1", + [{ streamId, order: 10, stepOrder: 0, status: "streaming" }], + [], + deltas, + ); + expect(message).toBeDefined(); + expect(message.role).toBe("assistant"); + expect(changed).toBe(true); + + const toolPart = message.parts.find( + (p) => p.type === "tool-dangerousTool", + ) as any; + expect(toolPart).toBeDefined(); + expect(toolPart.state).toBe("approval-requested"); + expect(toolPart.approval).toEqual({ id: "approval1" }); + }); }); diff --git a/src/deltas.ts b/src/deltas.ts index 739c66bb..b62f2338 100644 --- a/src/deltas.ts +++ b/src/deltas.ts @@ -66,22 +66,40 @@ export async function updateFromUIMessageChunks( }, }); let failed = false; + let suppressError = false; const messageStream = readUIMessageStream({ message: uiMessage, stream: partsStream, onError: (e) => { + const errorMessage = e instanceof Error ? e.message : String(e); + // Tool invocation errors can be safely ignored when streaming continuation + // after tool approval - the stored messages have the complete tool context + if (errorMessage.toLowerCase().includes("no tool invocation found")) { + // Silently suppress - this is expected after tool approval when the + // continuation stream has tool-result without the original tool-call + suppressError = true; + return; + } failed = true; console.error("Error in stream", e); }, terminateOnError: true, }); let message = uiMessage; - for await (const messagePart of messageStream) { - assert( - messagePart.id === message.id, - `Expecting to only make one UIMessage in a stream`, - ); - message = messagePart; + try { + for await (const messagePart of messageStream) { + assert( + messagePart.id === message.id, + `Expecting to only make one UIMessage in a stream`, + ); + message = messagePart; + } + } catch (e) { + // If we've already handled this error in onError and marked it as suppressed, + // don't rethrow - the stored messages provide the fallback + if (!suppressError) { + throw e; + } } if (failed) { message.status = "failed"; @@ -472,6 +490,25 @@ export function updateFromTextStreamParts( } break; } + case "tool-approval-request": { + const typedPart = part as unknown as { + type: "tool-approval-request"; + toolCallId: string; + approvalId: string; + }; + const toolPart = toolPartsById.get(typedPart.toolCallId); + if (toolPart) { + toolPart.state = "approval-requested"; + (toolPart as ToolUIPart & { approval?: object }).approval = { + id: typedPart.approvalId, + }; + } else { + console.warn( + `Expected tool call part ${typedPart.toolCallId} for approval request`, + ); + } + break; + } case "file": case "text-end": case "finish-step": @@ -483,7 +520,7 @@ export function updateFromTextStreamParts( break; default: { // Should never happen - const _: never = part; + // const _: never = part; console.warn(`Received unexpected part: ${JSON.stringify(part)}`); break; } diff --git a/src/fromUIMessages.test.ts b/src/fromUIMessages.test.ts index bf6ed895..4413c3c1 100644 --- a/src/fromUIMessages.test.ts +++ b/src/fromUIMessages.test.ts @@ -20,7 +20,7 @@ function baseMessageDoc( } describe("fromUIMessages round-trip tests", () => { - it("preserves essential data for simple user message", () => { + it("preserves essential data for simple user message", async () => { const originalMessages = [ baseMessageDoc({ message: { @@ -32,7 +32,7 @@ describe("fromUIMessages round-trip tests", () => { ]; const uiMessages = toUIMessages(originalMessages); - const backToMessageDocs = fromUIMessages(uiMessages, { + const backToMessageDocs = await fromUIMessages(uiMessages, { threadId: "thread1", }); @@ -54,7 +54,7 @@ describe("fromUIMessages round-trip tests", () => { } }); - it("preserves essential data for assistant message", () => { + it("preserves essential data for assistant message", async () => { const originalMessages = [ baseMessageDoc({ message: { @@ -66,7 +66,7 @@ describe("fromUIMessages round-trip tests", () => { ]; const uiMessages = toUIMessages(originalMessages); - const backToMessageDocs = fromUIMessages(uiMessages, { + const backToMessageDocs = await fromUIMessages(uiMessages, { threadId: "thread1", }); @@ -78,7 +78,7 @@ describe("fromUIMessages round-trip tests", () => { expect(backToMessageDocs[0].text).toBe("Hi there! How can I help?"); }); - it("preserves system messages correctly", () => { + it("preserves system messages correctly", async () => { const originalMessages = [ baseMessageDoc({ message: { @@ -90,7 +90,7 @@ describe("fromUIMessages round-trip tests", () => { ]; const uiMessages = toUIMessages(originalMessages); - const backToMessageDocs = fromUIMessages(uiMessages, { + const backToMessageDocs = await fromUIMessages(uiMessages, { threadId: "thread1", }); @@ -108,7 +108,7 @@ describe("fromUIMessages round-trip tests", () => { ); }); - it("preserves reasoning in assistant messages", () => { + it("preserves reasoning in assistant messages", async () => { const originalMessages = [ baseMessageDoc({ message: { @@ -130,7 +130,7 @@ describe("fromUIMessages round-trip tests", () => { ]; const uiMessages = toUIMessages(originalMessages); - const backToMessageDocs = fromUIMessages(uiMessages, { + const backToMessageDocs = await fromUIMessages(uiMessages, { threadId: "thread1", }); @@ -152,7 +152,7 @@ describe("fromUIMessages round-trip tests", () => { expect(backToMessageDocs[0].reasoning).toBe("Let me think about this..."); }); - it("handles tool calls and groups them correctly", () => { + it("handles tool calls and groups them correctly", async () => { // Tool calls get grouped into single UI message but expanded back to multiple message docs const originalMessages = [ baseMessageDoc({ @@ -166,6 +166,7 @@ describe("fromUIMessages round-trip tests", () => { type: "tool-call", toolName: "calculator", toolCallId: "call1", + input: { operation: "add", a: 2, b: 3 }, args: { operation: "add", a: 2, b: 3 }, }, ], @@ -196,7 +197,7 @@ describe("fromUIMessages round-trip tests", () => { const toTest = [originalMessages, [...originalMessages].reverse()]; for (const messages of toTest) { const uiMessages = toUIMessages(messages); - const backToMessageDocs = fromUIMessages(uiMessages, { + const backToMessageDocs = await fromUIMessages(uiMessages, { threadId: "thread1", }); @@ -230,7 +231,7 @@ describe("fromUIMessages round-trip tests", () => { } }); - it("preserves file attachments in user messages", () => { + it("preserves file attachments in user messages", async () => { const originalMessages = [ baseMessageDoc({ message: { @@ -252,7 +253,7 @@ describe("fromUIMessages round-trip tests", () => { ]; const uiMessages = toUIMessages(originalMessages); - const backToMessageDocs = fromUIMessages(uiMessages, { + const backToMessageDocs = await fromUIMessages(uiMessages, { threadId: "thread1", }); @@ -277,12 +278,12 @@ describe("fromUIMessages round-trip tests", () => { expect(fileContent).toBeDefined(); expect(fileContent).toMatchObject({ type: "file", - mimeType: "image/png", + mediaType: "image/png", }); } }); - it("preserves sources correctly", () => { + it("preserves sources correctly", async () => { const originalMessages = [ baseMessageDoc({ message: { @@ -315,7 +316,7 @@ describe("fromUIMessages round-trip tests", () => { ]; const uiMessages = toUIMessages(originalMessages); - const backToMessageDocs = fromUIMessages(uiMessages, { + const backToMessageDocs = await fromUIMessages(uiMessages, { threadId: "thread1", }); @@ -338,7 +339,7 @@ describe("fromUIMessages round-trip tests", () => { }); }); - it("preserves metadata when provided", () => { + it("preserves metadata when provided", async () => { const testMetadata = { customField: "customValue", timestamp: Date.now(), @@ -356,7 +357,7 @@ describe("fromUIMessages round-trip tests", () => { ]; const uiMessages = toUIMessages(originalMessages); - const backToMessageDocs = fromUIMessages(uiMessages, { + const backToMessageDocs = await fromUIMessages(uiMessages, { threadId: "thread1", }); @@ -367,7 +368,7 @@ describe("fromUIMessages round-trip tests", () => { expect(backToMessageDocs[0].metadata).toEqual(testMetadata); }); - it("handles streaming status correctly", () => { + it("handles streaming status correctly", async () => { const originalMessages = [ baseMessageDoc({ message: { @@ -381,7 +382,7 @@ describe("fromUIMessages round-trip tests", () => { ]; const uiMessages = toUIMessages(originalMessages); - const backToMessageDocs = fromUIMessages(uiMessages, { + const backToMessageDocs = await fromUIMessages(uiMessages, { threadId: "thread1", }); @@ -395,13 +396,13 @@ describe("fromUIMessages round-trip tests", () => { }); describe("fromUIMessages functionality tests", () => { - it("handles empty messages array", () => { + it("handles empty messages array", async () => { const uiMessages: UIMessage[] = []; - const result = fromUIMessages(uiMessages, { threadId: "thread1" }); + const result = await fromUIMessages(uiMessages, { threadId: "thread1" }); expect(result).toHaveLength(0); }); - it("correctly assigns thread ID", () => { + it("correctly assigns thread ID", async () => { const uiMessage: UIMessage = { id: "test-id", _creationTime: Date.now(), @@ -414,14 +415,14 @@ describe("fromUIMessages functionality tests", () => { parts: [{ type: "text", text: "Hello" }], }; - const result = fromUIMessages([uiMessage], { + const result = await fromUIMessages([uiMessage], { threadId: "custom-thread-id", }); expect(result).toHaveLength(1); expect(result[0].threadId).toBe("custom-thread-id"); }); - it("correctly determines tool status", () => { + it("correctly determines tool status", async () => { const toolUIMessage: UIMessage = { id: "tool-id", _creationTime: Date.now(), @@ -442,7 +443,9 @@ describe("fromUIMessages functionality tests", () => { ], }; - const result = fromUIMessages([toolUIMessage], { threadId: "thread1" }); + const result = await fromUIMessages([toolUIMessage], { + threadId: "thread1", + }); expect(result.length).toBeGreaterThan(0); // Should have tool messages @@ -450,7 +453,7 @@ describe("fromUIMessages functionality tests", () => { expect(toolMessages.length).toBeGreaterThan(0); }); - it("handles tool calls without responses", () => { + it("handles tool calls without responses", async () => { const toolUIMessage: UIMessage = { id: "tool-id", _creationTime: Date.now(), @@ -471,7 +474,9 @@ describe("fromUIMessages functionality tests", () => { ], }; - const result = fromUIMessages([toolUIMessage], { threadId: "thread1" }); + const result = await fromUIMessages([toolUIMessage], { + threadId: "thread1", + }); expect(result.length).toBeGreaterThan(0); // Should have tool messages diff --git a/src/mapping.test.ts b/src/mapping.test.ts index f730a775..c802b468 100644 --- a/src/mapping.test.ts +++ b/src/mapping.test.ts @@ -209,4 +209,52 @@ describe("mapping", () => { const { fileIds } = await serializeContent(ctx, component, content); expect(fileIds).toBeUndefined(); }); + + test("tool-approval-request is preserved after serialization", async () => { + const approvalRequest = { + type: "tool-approval-request" as const, + approvalId: "approval-123", + toolCallId: "tool-call-456", + }; + const { content } = await serializeContent( + {} as ActionCtx, + {} as AgentComponent, + [approvalRequest], + ); + expect(content).toHaveLength(1); + expect((content as unknown[])[0]).toMatchObject(approvalRequest); + }); + + test("tool-approval-response with approved: true is preserved", async () => { + const approvalResponse = { + type: "tool-approval-response" as const, + approvalId: "approval-123", + approved: true, + reason: "User approved", + }; + const { content } = await serializeContent( + {} as ActionCtx, + {} as AgentComponent, + [approvalResponse], + ); + expect(content).toHaveLength(1); + expect((content as unknown[])[0]).toMatchObject(approvalResponse); + }); + + test("tool-approval-response with approved: false is preserved", async () => { + const approvalResponse = { + type: "tool-approval-response" as const, + approvalId: "approval-123", + approved: false, + reason: "User denied", + providerExecuted: false, + }; + const { content } = await serializeContent( + {} as ActionCtx, + {} as AgentComponent, + [approvalResponse], + ); + expect(content).toHaveLength(1); + expect((content as unknown[])[0]).toMatchObject(approvalResponse); + }); }); diff --git a/src/mapping.ts b/src/mapping.ts index 86253c31..cdfd6806 100644 --- a/src/mapping.ts +++ b/src/mapping.ts @@ -36,6 +36,8 @@ import { type SourcePart, vToolResultOutput, type MessageDoc, + vToolApprovalRequest, + vToolApprovalResponse, } from "./validators.js"; import type { ActionCtx, AgentComponent } from "./client/types.js"; import type { MutationCtx } from "./client/types.js"; @@ -45,6 +47,8 @@ import { convertUint8ArrayToBase64, type ProviderOptions, type ReasoningPart, + type ToolApprovalRequest, + type ToolApprovalResponse, } from "@ai-sdk/provider-utils"; import { parse, validate } from "convex-helpers/validators"; import { @@ -155,6 +159,8 @@ export function toModelMessageUsage(usage: Usage): LanguageModelUsage { totalTokens: usage.totalTokens, reasoningTokens: usage.reasoningTokens, cachedInputTokens: usage.cachedInputTokens, + inputTokenDetails: {} as any, + outputTokenDetails: {} as any, }; } @@ -165,18 +171,33 @@ export function serializeWarnings( return undefined; } return warnings.map((warning) => { - if (warning.type !== "unsupported-setting") { - return warning; + if (warning.type === "compatibility") { + return { + type: "unsupported-setting", + setting: warning.feature, + details: warning.details, + }; } - return { ...warning, setting: warning.setting.toString() }; - }); + return warning; + }) as any; } export function toModelMessageWarnings( warnings: MessageWithMetadata["warnings"], ): CallWarning[] | undefined { - // We don't need to do anythign here for now - return warnings; + if (!warnings) { + return undefined; + } + return warnings.map((warning) => { + if (warning.type === "unsupported-setting") { + return { + type: "compatibility", + feature: warning.setting, + details: warning.details, + }; + } + return warning; + }) as any; } export async function serializeNewMessagesInStep( @@ -307,7 +328,7 @@ export async function serializeContent( } return { type: part.type, - mimeType: getMimeOrMediaType(part), + mediaType: getMimeOrMediaType(part), ...metadata, image, } satisfies Infer; @@ -327,15 +348,18 @@ export async function serializeContent( type: part.type, data, filename: part.filename, - mimeType: getMimeOrMediaType(part)!, + mediaType: getMimeOrMediaType(part)!, ...metadata, } satisfies Infer; } case "tool-call": { - const args = "input" in part ? part.input : part.args; + // Handle legacy data where only args field exists + const input = part.input ?? (part as any)?.args ?? {}; return { type: part.type, - args: args ?? null, + input, + /** @deprecated Use `input` instead. */ + args: input, toolCallId: part.toolCallId, toolName: part.toolName, providerExecuted: part.providerExecuted, @@ -363,13 +387,31 @@ export async function serializeContent( case "source": { return part satisfies Infer; } + case "tool-approval-request": { + return { + type: part.type, + approvalId: part.approvalId, + toolCallId: part.toolCallId, + ...metadata, + } satisfies Infer; + } + case "tool-approval-response": { + return { + type: part.type, + approvalId: part.approvalId, + approved: part.approved, + reason: part.reason, + providerExecuted: part.providerExecuted, + ...metadata, + } satisfies Infer; + } default: - return part satisfies Infer; + return null; } }), ); return { - content: serialized as SerializedContent, + content: serialized.filter((p) => p !== null) as SerializedContent, fileIds: fileIds.length > 0 ? fileIds : undefined, }; } @@ -378,57 +420,78 @@ export function fromModelMessageContent(content: Content): Message["content"] { if (typeof content === "string") { return content; } - return content.map((part) => { - const metadata: { - providerOptions?: ProviderOptions; - providerMetadata?: ProviderMetadata; - } = {}; - if ("providerOptions" in part) { - metadata.providerOptions = part.providerOptions as ProviderOptions; - } - if ("providerMetadata" in part) { - metadata.providerMetadata = part.providerMetadata as ProviderMetadata; - } - switch (part.type) { - case "text": - return part satisfies Infer; - case "image": - return { - type: part.type, - mimeType: getMimeOrMediaType(part), - ...metadata, - image: serializeDataOrUrl(part.image), - } satisfies Infer; - case "file": - return { - type: part.type, - data: serializeDataOrUrl(part.data), - filename: part.filename, - mimeType: getMimeOrMediaType(part)!, - ...metadata, - } satisfies Infer; - case "tool-call": - return { - type: part.type, - args: part.input ?? null, - toolCallId: part.toolCallId, - toolName: part.toolName, - providerExecuted: part.providerExecuted, - ...metadata, - } satisfies Infer; - case "tool-result": - return normalizeToolResult(part, metadata); - case "reasoning": - return { - type: part.type, - text: part.text, - ...metadata, - } satisfies Infer; - // Not in current generation output, but could be in historical messages - default: - return part satisfies Infer; - } - }) as Message["content"]; + return content + .map((part) => { + const metadata: { + providerOptions?: ProviderOptions; + providerMetadata?: ProviderMetadata; + } = {}; + if ("providerOptions" in part) { + metadata.providerOptions = part.providerOptions as ProviderOptions; + } + if ("providerMetadata" in part) { + metadata.providerMetadata = part.providerMetadata as ProviderMetadata; + } + switch (part.type) { + case "text": + return part satisfies Infer; + case "image": + return { + type: part.type, + mediaType: getMimeOrMediaType(part), + ...metadata, + image: serializeDataOrUrl(part.image), + } satisfies Infer; + case "file": + return { + type: part.type, + data: serializeDataOrUrl(part.data), + filename: part.filename, + mediaType: getMimeOrMediaType(part)!, + ...metadata, + } satisfies Infer; + case "tool-call": + // Handle legacy data where only args field exists + return { + type: part.type, + input: part.input ?? (part as any)?.args ?? {}, + /** @deprecated Use `input` instead. */ + args: part.input ?? (part as any)?.args ?? {}, + toolCallId: part.toolCallId, + toolName: part.toolName, + providerExecuted: part.providerExecuted, + ...metadata, + } satisfies Infer; + case "tool-result": + return normalizeToolResult(part, metadata); + case "reasoning": + return { + type: part.type, + text: part.text, + ...metadata, + } satisfies Infer; + case "tool-approval-request": + return { + type: part.type, + approvalId: part.approvalId, + toolCallId: part.toolCallId, + ...metadata, + } satisfies Infer; + case "tool-approval-response": + return { + type: part.type, + approvalId: part.approvalId, + approved: part.approved, + reason: part.reason, + providerExecuted: part.providerExecuted, + ...metadata, + } satisfies Infer; + // Not in current generation output, but could be in historical messages + default: + return null; + } + }) + .filter((p) => p !== null) as Message["content"]; } export function toModelMessageContent( @@ -437,84 +500,103 @@ export function toModelMessageContent( if (typeof content === "string") { return content; } - return content.map((part) => { - const metadata: { - providerOptions?: ProviderOptions; - providerMetadata?: ProviderMetadata; - } = {}; - if ("providerOptions" in part) { - metadata.providerOptions = part.providerOptions; - } - if ("providerMetadata" in part) { - metadata.providerMetadata = part.providerMetadata; - } - switch (part.type) { - case "text": - return { - type: part.type, - text: part.text, - ...metadata, - } satisfies TextPart; - case "image": - return { - type: part.type, - image: toModelMessageDataOrUrl(part.image), - mediaType: getMimeOrMediaType(part), - ...metadata, - } satisfies ImagePart; - case "file": - return { - type: part.type, - data: toModelMessageDataOrUrl(part.data), - filename: part.filename, - mediaType: getMimeOrMediaType(part)!, - ...metadata, - } satisfies FilePart; - case "tool-call": { - const input = "input" in part ? part.input : part.args; - return { - type: part.type, - input: input ?? null, - toolCallId: part.toolCallId, - toolName: part.toolName, - providerExecuted: part.providerExecuted, - ...metadata, - } satisfies ToolCallPart; + return content + .map((part) => { + const metadata: { + providerOptions?: ProviderOptions; + providerMetadata?: ProviderMetadata; + } = {}; + if ("providerOptions" in part) { + metadata.providerOptions = part.providerOptions; } - case "tool-result": { - return normalizeToolResult(part, metadata); + if ("providerMetadata" in part) { + metadata.providerMetadata = part.providerMetadata; } - case "reasoning": - return { - type: part.type, - text: part.text, - ...metadata, - } satisfies ReasoningPart; - case "redacted-reasoning": - // TODO: should we just drop this? - return { - type: "reasoning", - text: "", - ...metadata, - providerOptions: metadata.providerOptions - ? { - ...Object.fromEntries( - Object.entries(metadata.providerOptions ?? {}).map( - ([key, value]) => [ - key, - { ...value, redactedData: part.data }, - ], + switch (part.type) { + case "text": + return { + type: part.type, + text: part.text, + ...metadata, + } satisfies TextPart; + case "image": + return { + type: part.type, + image: toModelMessageDataOrUrl(part.image), + mediaType: getMimeOrMediaType(part), + ...metadata, + } satisfies ImagePart; + case "file": + return { + type: part.type, + data: toModelMessageDataOrUrl(part.data), + filename: part.filename, + mediaType: getMimeOrMediaType(part)!, + ...metadata, + } satisfies FilePart; + case "tool-call": { + // Handle legacy data where only args field exists + const input = part.input ?? (part as any)?.args ?? {}; + return { + type: part.type, + input, + toolCallId: part.toolCallId, + toolName: part.toolName, + providerExecuted: part.providerExecuted, + ...metadata, + } satisfies ToolCallPart; + } + case "tool-result": { + return normalizeToolResult(part, metadata); + } + case "reasoning": + return { + type: part.type, + text: part.text, + ...metadata, + } satisfies ReasoningPart; + case "redacted-reasoning": + // TODO: should we just drop this? + return { + type: "reasoning", + text: "", + ...metadata, + providerOptions: metadata.providerOptions + ? { + ...Object.fromEntries( + Object.entries(metadata.providerOptions ?? {}).map( + ([key, value]) => [ + key, + { ...value, redactedData: part.data }, + ], + ), ), - ), - } - : undefined, - } satisfies ReasoningPart; - case "source": - return part satisfies SourcePart; - default: - return part satisfies Content; - } - }) as Content; + } + : undefined, + } satisfies ReasoningPart; + case "source": + return part satisfies SourcePart; + case "tool-approval-request": + return { + type: part.type, + approvalId: part.approvalId, + toolCallId: part.toolCallId, + ...metadata, + } satisfies ToolApprovalRequest; + case "tool-approval-response": + return { + type: part.type, + approvalId: part.approvalId, + approved: part.approved, + reason: part.reason, + providerExecuted: part.providerExecuted, + ...metadata, + } satisfies ToolApprovalResponse; + default: + return null; + } + }) + .filter((p) => p !== null) as Content; } export function normalizeToolOutput( @@ -544,11 +626,15 @@ function normalizeToolResult( ): ToolResultPart & Infer { return { type: part.type, - output: - part.output ?? - normalizeToolOutput("result" in part ? part.result : undefined), + output: part.output + ? validate(vToolResultOutput, part.output) + ? (part.output as any) + : normalizeToolOutput(JSON.stringify(part.output)) + : normalizeToolOutput("result" in part ? part.result : undefined), toolCallId: part.toolCallId, toolName: part.toolName, + // Preserve isError flag for error reporting + ...("isError" in part && part.isError ? { isError: true } : {}), ...metadata, } satisfies ToolResultPart; } diff --git a/src/react/useThreadMessages.ts b/src/react/useThreadMessages.ts index ecd6e8f3..eae6f833 100644 --- a/src/react/useThreadMessages.ts +++ b/src/react/useThreadMessages.ts @@ -16,7 +16,7 @@ import type { PaginationOptions, PaginationResult, } from "convex/server"; -import { useMemo } from "react"; +import { useMemo, useState, useEffect } from "react"; import type { SyncStreamsReturnValue } from "../client/types.js"; import { sorted } from "../shared.js"; import { fromUIMessages } from "../UIMessages.js"; @@ -239,12 +239,26 @@ export function useStreamingThreadMessages>( args === "skip" ? undefined : (args.startOrder ?? undefined); const queryOptions = { startOrder, ...options }; const uiMessages = useStreamingUIMessages(query, queryArgs, queryOptions); - if (args === "skip") { - return undefined; - } - // TODO: we aren't passing through as much metadata as we could here. - // We could share the stream metadata logic with useStreamingUIMessages. - return uiMessages - ?.map((m) => fromUIMessages([m], { threadId: args.threadId })) - .flat(); + const [messages, setMessages] = useState | undefined>(); + + useEffect(() => { + if (args === "skip" || !uiMessages) { + setMessages(undefined); + return; + } + let isMounted = true; + (async () => { + const nested = await Promise.all( + uiMessages.map((m) => fromUIMessages([m], { threadId: args.threadId })), + ); + if (isMounted) { + setMessages(nested.flat()); + } + })(); + return () => { + isMounted = false; + }; + }, [uiMessages, args === "skip" ? undefined : args.threadId]); + + return messages; } diff --git a/src/shared.ts b/src/shared.ts index ffedbeba..79452e53 100644 --- a/src/shared.ts +++ b/src/shared.ts @@ -4,6 +4,7 @@ import type { ReasoningPart, ToolCallPart, ToolResultPart, + ToolApprovalRequest, } from "@ai-sdk/provider-utils"; import type { ModelMessage, @@ -55,6 +56,7 @@ export function joinText( | ToolCallPart | ToolResultPart | MessageContentParts + | ToolApprovalRequest )[], ) { return parts diff --git a/src/toUIMessages.test.ts b/src/toUIMessages.test.ts index 58bcaacb..73707cbe 100644 --- a/src/toUIMessages.test.ts +++ b/src/toUIMessages.test.ts @@ -90,6 +90,7 @@ describe("toUIMessages", () => { type: "tool-call", toolName: "myTool", toolCallId: "call1", + input: "an arg", args: "an arg", }, ], @@ -219,6 +220,7 @@ describe("toUIMessages", () => { }, { type: "tool-call", + input: "What's the meaning of life?", args: "What's the meaning of life?", toolCallId: "call1", toolName: "myTool", @@ -310,6 +312,7 @@ describe("toUIMessages", () => { type: "tool-call", toolName: "myTool", toolCallId: "call1", + input: { query: "test" }, args: { query: "test" }, }, ], @@ -356,6 +359,7 @@ describe("toUIMessages", () => { type: "tool-call", toolName: "myTool", toolCallId: "call1", + input: "hi", args: "hi", }, ], @@ -387,6 +391,7 @@ describe("toUIMessages", () => { type: "tool-call", toolName: "myTool", toolCallId: "call1", + input: "", args: "", }, ], @@ -448,6 +453,7 @@ describe("toUIMessages", () => { type: "tool-call", toolName: "calculator", toolCallId: "call1", + input: { operation: "add", a: 2, b: 3 }, args: { operation: "add", a: 2, b: 3 }, }, { @@ -490,6 +496,7 @@ describe("toUIMessages", () => { type: "tool-call", toolName: "calculator", toolCallId: "call1", + input: { operation: "add", a: 1, b: 2 }, args: { operation: "add", a: 1, b: 2 }, }, ], @@ -561,6 +568,9 @@ describe("toUIMessages", () => { text: "**Finding the Time**\n\nI've pinpointed the core task: obtaining the current time in Paris. It involves using the `dateTime` tool. I've identified \"Europe/Paris\" as the necessary timezone identifier to provide to the tool. My next step is to test the tool.\n\n\n", }, { + input: { + timezone: "Europe/Paris", + }, args: { timezone: "Europe/Paris", }, @@ -690,6 +700,7 @@ describe("toUIMessages", () => { type: "tool-call", toolName: "calculator", toolCallId: "call1", + input: { operation: "add", a: 40, b: 2 }, args: { operation: "add", a: 40, b: 2 }, }, ], @@ -729,6 +740,109 @@ describe("toUIMessages", () => { expect(textParts[0].text).toBe("The result is 42."); }); + it("shows output-error state when tool result has isError: true (issue #162)", () => { + const messages = [ + // Tool call + baseMessageDoc({ + _id: "msg1", + order: 1, + stepOrder: 1, + tool: true, + message: { + role: "assistant", + content: [ + { + type: "tool-call", + toolName: "generateImage", + toolCallId: "call1", + input: { id: "invalid-id" }, + args: { id: "invalid-id" }, + }, + ], + }, + }), + // Tool result with error + baseMessageDoc({ + _id: "msg2", + order: 1, + stepOrder: 2, + tool: true, + message: { + role: "tool", + content: [ + { + type: "tool-result", + toolCallId: "call1", + toolName: "generateImage", + output: { + type: "text", + value: + 'ArgumentValidationError: Value does not match validator.\nPath: .id\nValue: "invalid-id"\nValidator: v.id("images")', + }, + isError: true, + }, + ], + }, + }), + ]; + + const uiMessages = toUIMessages(messages); + + expect(uiMessages).toHaveLength(1); + expect(uiMessages[0].role).toBe("assistant"); + + const toolParts = uiMessages[0].parts.filter( + (p) => p.type === "tool-generateImage", + ); + expect(toolParts).toHaveLength(1); + + const toolPart = toolParts[0] as any; + expect(toolPart.toolCallId).toBe("call1"); + // Should show output-error, not output-available + expect(toolPart.state).toBe("output-error"); + expect(toolPart.output).toContain("ArgumentValidationError"); + }); + + it("shows output-error when tool result has isError: true without tool call present (issue #162)", () => { + // This simulates the case where the tool-call message wasn't saved + const messages = [ + baseMessageDoc({ + _id: "msg1", + order: 1, + stepOrder: 1, + tool: true, + message: { + role: "tool", + content: [ + { + type: "tool-result", + toolCallId: "call1", + toolName: "generateImage", + output: { + type: "text", + value: "Error: Something went wrong", + }, + isError: true, + }, + ], + }, + }), + ]; + + const uiMessages = toUIMessages(messages); + + expect(uiMessages).toHaveLength(1); + expect(uiMessages[0].role).toBe("assistant"); + + const toolParts = uiMessages[0].parts.filter( + (p) => p.type === "tool-generateImage", + ); + expect(toolParts).toHaveLength(1); + + const toolPart = toolParts[0] as any; + expect(toolPart.state).toBe("output-error"); + }); + describe("userId preservation", () => { it("preserves userId in user messages", () => { const messages = [ @@ -796,6 +910,7 @@ describe("toUIMessages", () => { type: "tool-call", toolName: "myTool", toolCallId: "call1", + input: {}, args: {}, }, ], @@ -855,4 +970,300 @@ describe("toUIMessages", () => { expect(uiMessages[0].userId).toBeUndefined(); }); }); + + describe("tool approval workflow", () => { + it("sets state to approval-requested when tool-approval-request is present", () => { + const messages = [ + baseMessageDoc({ + _id: "msg1", + order: 1, + stepOrder: 1, + tool: true, + message: { + role: "assistant", + content: [ + { + type: "tool-call", + toolName: "dangerousTool", + toolCallId: "call1", + input: { action: "delete" }, + args: { action: "delete" }, + }, + { + type: "tool-approval-request", + approvalId: "approval1", + toolCallId: "call1", + }, + ], + }, + }), + ]; + + const uiMessages = toUIMessages(messages); + + expect(uiMessages).toHaveLength(1); + const toolPart = uiMessages[0].parts.find( + (p) => p.type === "tool-dangerousTool", + ) as any; + expect(toolPart).toBeDefined(); + expect(toolPart.state).toBe("approval-requested"); + expect(toolPart.approval).toEqual({ id: "approval1" }); + }); + + it("sets state to approval-responded when tool-approval-response with approved: true", () => { + const messages = [ + baseMessageDoc({ + _id: "msg1", + order: 1, + stepOrder: 1, + tool: true, + message: { + role: "assistant", + content: [ + { + type: "tool-call", + toolName: "dangerousTool", + toolCallId: "call1", + input: { action: "delete" }, + args: { action: "delete" }, + }, + { + type: "tool-approval-request", + approvalId: "approval1", + toolCallId: "call1", + }, + ], + }, + }), + baseMessageDoc({ + _id: "msg2", + order: 1, + stepOrder: 2, + tool: true, + message: { + role: "tool", + content: [ + { + type: "tool-approval-response", + approvalId: "approval1", + approved: true, + reason: "User confirmed", + }, + ], + }, + }), + ]; + + const uiMessages = toUIMessages(messages); + + expect(uiMessages).toHaveLength(1); + const toolPart = uiMessages[0].parts.find( + (p) => p.type === "tool-dangerousTool", + ) as any; + expect(toolPart).toBeDefined(); + expect(toolPart.state).toBe("approval-responded"); + expect(toolPart.approval).toEqual({ + id: "approval1", + approved: true, + reason: "User confirmed", + }); + }); + + it("sets state to output-denied when tool-approval-response with approved: false", () => { + const messages = [ + baseMessageDoc({ + _id: "msg1", + order: 1, + stepOrder: 1, + tool: true, + message: { + role: "assistant", + content: [ + { + type: "tool-call", + toolName: "dangerousTool", + toolCallId: "call1", + input: { action: "delete" }, + args: { action: "delete" }, + }, + { + type: "tool-approval-request", + approvalId: "approval1", + toolCallId: "call1", + }, + ], + }, + }), + baseMessageDoc({ + _id: "msg2", + order: 1, + stepOrder: 2, + tool: true, + message: { + role: "tool", + content: [ + { + type: "tool-approval-response", + approvalId: "approval1", + approved: false, + reason: "User declined the operation", + }, + ], + }, + }), + ]; + + const uiMessages = toUIMessages(messages); + + expect(uiMessages).toHaveLength(1); + const toolPart = uiMessages[0].parts.find( + (p) => p.type === "tool-dangerousTool", + ) as any; + expect(toolPart).toBeDefined(); + expect(toolPart.state).toBe("output-denied"); + expect(toolPart.approval).toEqual({ + id: "approval1", + approved: false, + reason: "User declined the operation", + }); + }); + + it("sets state to output-denied when tool-result has execution-denied output", () => { + const messages = [ + baseMessageDoc({ + _id: "msg1", + order: 1, + stepOrder: 1, + tool: true, + message: { + role: "assistant", + content: [ + { + type: "tool-call", + toolName: "dangerousTool", + toolCallId: "call1", + input: { action: "delete" }, + args: { action: "delete" }, + }, + ], + }, + }), + baseMessageDoc({ + _id: "msg2", + order: 1, + stepOrder: 2, + tool: true, + message: { + role: "tool", + content: [ + { + type: "tool-result", + toolCallId: "call1", + toolName: "dangerousTool", + output: { + type: "execution-denied", + reason: "Tool execution was denied by the user", + }, + }, + ], + }, + }), + ]; + + const uiMessages = toUIMessages(messages); + + expect(uiMessages).toHaveLength(1); + const toolPart = uiMessages[0].parts.find( + (p) => p.type === "tool-dangerousTool", + ) as any; + expect(toolPart).toBeDefined(); + expect(toolPart.state).toBe("output-denied"); + expect(toolPart.approval).toEqual({ + id: "", + approved: false, + reason: "Tool execution was denied by the user", + }); + }); + + it("handles full approval flow: request → approved → executed → output-available", () => { + const messages = [ + baseMessageDoc({ + _id: "msg1", + order: 1, + stepOrder: 1, + tool: true, + message: { + role: "assistant", + content: [ + { + type: "tool-call", + toolName: "dangerousTool", + toolCallId: "call1", + input: { action: "delete" }, + args: { action: "delete" }, + }, + { + type: "tool-approval-request", + approvalId: "approval1", + toolCallId: "call1", + }, + ], + }, + }), + baseMessageDoc({ + _id: "msg2", + order: 1, + stepOrder: 2, + tool: true, + message: { + role: "tool", + content: [ + { + type: "tool-approval-response", + approvalId: "approval1", + approved: true, + }, + ], + }, + }), + baseMessageDoc({ + _id: "msg3", + order: 1, + stepOrder: 3, + tool: true, + message: { + role: "tool", + content: [ + { + type: "tool-result", + toolCallId: "call1", + toolName: "dangerousTool", + output: { + type: "json", + value: { deleted: true }, + }, + }, + ], + }, + }), + ]; + + const uiMessages = toUIMessages(messages); + + expect(uiMessages).toHaveLength(1); + const toolPart = uiMessages[0].parts.find( + (p) => p.type === "tool-dangerousTool", + ) as any; + expect(toolPart).toBeDefined(); + // After tool-result, state should be output-available + expect(toolPart.state).toBe("output-available"); + expect(toolPart.output).toEqual({ deleted: true }); + // approval should still be preserved from earlier + expect(toolPart.approval).toEqual({ + id: "approval1", + approved: true, + reason: undefined, + }); + }); + }); }); diff --git a/src/validators.ts b/src/validators.ts index 8d664ea4..f8e03132 100644 --- a/src/validators.ts +++ b/src/validators.ts @@ -1,4 +1,5 @@ -import { v, type Infer, type Validator, type Value } from "convex/values"; +import type { Infer, Validator, Value } from "convex/values"; +import { v } from "convex/values"; import { vVectorDimension } from "./component/vector/tables.js"; // const deprecated = v.optional(v.any()) as unknown as VNull; @@ -42,6 +43,8 @@ export const vTextPart = v.object({ export const vImagePart = v.object({ type: v.literal("image"), image: v.union(v.string(), v.bytes()), + mediaType: v.optional(v.string()), + /** @deprecated Use `mediaType` instead. */ mimeType: v.optional(v.string()), providerOptions, }); @@ -50,7 +53,9 @@ export const vFilePart = v.object({ type: v.literal("file"), data: v.union(v.string(), v.bytes()), filename: v.optional(v.string()), - mimeType: v.string(), + mediaType: v.optional(v.string()), + /** @deprecated Use `mediaType` instead. */ + mimeType: v.optional(v.string()), providerOptions, providerMetadata, }); @@ -110,15 +115,34 @@ export const vSourcePart = v.union( ); export type SourcePart = Infer; -export const vToolCallPart = v.object({ - type: v.literal("tool-call"), - toolCallId: v.string(), - toolName: v.string(), - args: v.any(), - providerExecuted: v.optional(v.boolean()), - providerOptions, - providerMetadata, -}); +// Union type to support both old (args) and new (input) formats +// Both include input for type hint support +export const vToolCallPart = v.union( + // New format: input is primary, args is optional for backwards compat + v.object({ + type: v.literal("tool-call"), + toolCallId: v.string(), + toolName: v.string(), + input: v.any(), + /** @deprecated Use `input` instead. */ + args: v.optional(v.any()), + providerExecuted: v.optional(v.boolean()), + providerOptions, + providerMetadata, + }), + // Legacy format: args is present, input is optional + v.object({ + type: v.literal("tool-call"), + toolCallId: v.string(), + toolName: v.string(), + /** @deprecated Use `input` instead. */ + args: v.any(), + input: v.optional(v.any()), + providerExecuted: v.optional(v.boolean()), + providerOptions, + providerMetadata, + }), +); const vToolResultContent = v.array( v.union( @@ -132,25 +156,155 @@ const vToolResultContent = v.array( ); export const vToolResultOutput = v.union( - v.object({ type: v.literal("text"), value: v.string() }), - v.object({ type: v.literal("json"), value: v.any() }), - v.object({ type: v.literal("error-text"), value: v.string() }), - v.object({ type: v.literal("error-json"), value: v.any() }), + v.object({ type: v.literal("text"), value: v.string(), providerOptions }), + v.object({ type: v.literal("json"), value: v.any(), providerOptions }), + v.object({ + type: v.literal("error-text"), + value: v.string(), + providerOptions, + }), + v.object({ type: v.literal("error-json"), value: v.any(), providerOptions }), + v.object({ + type: v.literal("execution-denied"), + reason: v.optional(v.string()), + providerOptions, + }), v.object({ type: v.literal("content"), value: v.array( v.union( - v.object({ type: v.literal("text"), text: v.string() }), + v.object({ + type: v.literal("text"), + text: v.string(), + providerOptions, + }), + /** @deprecated Use `image-data` or `file-data` instead. */ v.object({ type: v.literal("media"), data: v.string(), mediaType: v.string(), }), + v.object({ + type: v.literal("file-data"), + /** Base-64 encoded */ + data: v.string(), + /** + * IANA media type. + * @see https://www.iana.org/assignments/media-types/media-types.xhtml + */ + mediaType: v.string(), + filename: v.optional(v.string()), + providerOptions, + }), + v.object({ + type: v.literal("file-url"), + url: v.string(), + providerOptions, + }), + v.object({ + type: v.literal("file-id"), + /** + * ID of the file. + * + * If you use multiple providers, you need to + * specify the provider specific ids using + * the Record option. The key is the provider + * name, e.g. 'openai' or 'anthropic'. + */ + fileId: v.union(v.string(), v.record(v.string(), v.string())), + providerOptions, + }), + v.object({ + type: v.literal("image-data"), + data: v.string(), + /** + * IANA media type. + * @see https://www.iana.org/assignments/media-types/media-types.xhtml + */ + mediaType: v.string(), + providerOptions, + }), + v.object({ + type: v.literal("image-url"), + url: v.string(), + providerOptions, + }), + v.object({ + /** + * Images that are referenced using a provider file id. + */ + type: v.literal("image-file-id"), + /** + * Image that is referenced using a provider file id. + * + * If you use multiple providers, you need to + * specify the provider specific ids using + * the Record option. The key is the provider + * name, e.g. 'openai' or 'anthropic'. + */ + fileId: v.union(v.string(), v.record(v.string(), v.string())), + providerOptions, + }), + v.object({ + /** + * Custom content part. This can be used to implement + * provider-specific content parts. + */ + type: v.literal("custom"), + providerOptions, + }), ), ), }), ); +/** + * Tool approval request prompt part. + */ +export const vToolApprovalRequest = v.object({ + type: v.literal("tool-approval-request"), + /** + * ID of the tool approval. + */ + approvalId: v.string(), + /** + * ID of the tool call that the approval request is for. + */ + toolCallId: v.string(), + /** @todo Should we continue to include? */ + providerMetadata, + /** @todo Should we continue to include? */ + providerOptions, +}); + +/** + * Tool approval response prompt part. + */ +export const vToolApprovalResponse = v.object({ + type: v.literal("tool-approval-response"), + /** + * ID of the tool approval. + */ + approvalId: v.string(), + /** + * Flag indicating whether the approval was granted or denied. + */ + approved: v.boolean(), + /** + * Optional reason for the approval or denial. + */ + reason: v.optional(v.string()), + /** + * Flag indicating whether the tool call is provider-executed. + * Only provider-executed tool approval responses should be sent to the model. + */ + providerExecuted: v.optional(v.boolean()), + /** @todo Should we continue to include? */ + providerMetadata, + /** @todo Should we continue to include? */ + providerOptions, +}); + export const vToolResultPart = v.object({ type: v.literal("tool-result"), toolCallId: v.string(), @@ -169,7 +323,9 @@ export const vToolResultPart = v.object({ args: v.optional(v.any()), experimental_content: v.optional(vToolResultContent), }); -export const vToolContent = v.array(vToolResultPart); +export const vToolContent = v.array( + v.union(vToolResultPart, vToolApprovalResponse), +); export const vAssistantContent = v.union( v.string(), @@ -182,6 +338,7 @@ export const vAssistantContent = v.union( vToolCallPart, vToolResultPart, vSourcePart, + vToolApprovalRequest, ), ), ); @@ -462,7 +619,7 @@ export const vStreamMessage = v.object({ agentName: v.optional(v.string()), model: v.optional(v.string()), provider: v.optional(v.string()), - providerOptions: v.optional(vProviderOptions), // Sent to model + providerOptions, // Sent to model }); export type StreamMessage = Infer; @@ -490,7 +647,7 @@ export const vMessageDoc = v.object({ agentName: v.optional(v.string()), model: v.optional(v.string()), provider: v.optional(v.string()), - providerOptions: v.optional(vProviderOptions), // Sent to model + providerOptions, // Sent to model // The result message: v.optional(vMessage),