Skip to content

onToolStart/onToolEnd callbacks never fire for built-in web_search tools #113

Description

@jlevy

Bug: onToolStart/onToolEnd callbacks never fire for built-in web_search

Observed

When using fillForm() with enableWebSearch: true and FillCallbacks with
onToolStart/onToolEnd, only custom additionalTools produce per-call callbacks.
Built-in provider web search tools (e.g. web_search) appear only in the aggregated
turnStats.toolCalls summary (e.g. web_search(17)) — no individual
onToolStart/onToolEnd callbacks fire.

Expected

onToolStart and onToolEnd should fire for every tool call, including built-in
provider web search tools, so consumers get per-call visibility (name, input, output,
duration).

Root Cause

In liveAgent.ts, wrapToolsWithCallbacks() only wraps tools that have an execute
function:

const execute = (tool as any).execute;
if (typeof execute === 'function') {
  wrapped[name] = wrapTool(name, tool, execute, callbacks);
} else {
  // Pass through declarative tools unchanged
  wrapped[name] = tool;
}

Provider web search tools (loaded via loadWebSearchTools()) are declarative AI SDK
tools
— they have a schema but no execute function. The AI SDK handles their
execution internally during generateText(), bypassing the callback wrapper entirely.

The aggregated turnStats.toolCalls count works because it comes from the AI SDK
response metadata, not from the callback wrapper.

Suggested Fix

Option A (recommended): Hook into the AI SDK's onStepFinish or tool result
metadata to emit synthetic onToolStart/onToolEnd callbacks for declarative tools
after each generateText() call.

Option B: Convert provider web search tools from declarative to executable wrappers
that call the provider API directly, so wrapToolsWithCallbacks can intercept them.

Option A is less invasive and preserves the AI SDK's native handling.

Reproduction

import { fillForm } from 'markform';

const result = await fillForm({
  form: myFormMarkdown,
  model: 'openai/gpt-5-mini',
  enableWebSearch: true,
  additionalTools: { my_tool: myTool },
  callbacks: {
    onToolStart: ({ name, input }) => console.log(`tool start: ${name}`),
    onToolEnd: ({ name }) => console.log(`tool end: ${name}`),
  },
});
// Logs: "tool start: my_tool", "tool end: my_tool"
// Does NOT log: "tool start: web_search" (despite web_search appearing in turnStats)

Affected Version

markform 0.1.17

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions