Skip to content

refactor(agent): simplify tool disclosure, index tool schemas#94

Merged
shallyan merged 6 commits into
mainfrom
refactor/simplify_tool_search
Jul 8, 2026
Merged

refactor(agent): simplify tool disclosure, index tool schemas#94
shallyan merged 6 commits into
mainfrom
refactor/simplify_tool_search

Conversation

@shallyan

@shallyan shallyan commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Progressive tool disclosure previously exposed three meta-tools:
tool_search, tool_describe, and tool_call. tool_describe was a separate
round-trip to fetch a hidden tool's parameter schema after tool_search. This
PR collapses the flow to two meta-tools:

  • Drop tool_describe. tool_search now returns each hit's parameter
    schema alongside its name and description, so the model can go straight to
    tool_call. When arguments don't fit the schema the registry already returns
    a correctable validation error, so the extra lookup step added no correctness.
  • Index tool parameter schemas. Parameter names, their descriptions, and
    enum values are folded into the BM25 index body (name x3 + description +
    schema keywords). Discriminating keywords often live in the schema rather than
    the one-line description (a repo owner, a channel id, an image size), so
    indexing them lifts retrieval recall without changing the schema the model
    sees. The index signature now keys on (name, description, parameters), so a
    parameter change under hot-reload correctly triggers a rebuild.

Offline validation on the ToolRet benchmark (7,961 queries, 44k-tool corpus,
Raven's own BM25 kernel) shows folding the schema into the index consistently
lifts nDCG@10 by ~4-5 points over indexing name + description alone.

Type

  • Fix
  • Feature
  • Docs
  • CI / tooling
  • Refactor
  • Other

Verification

uv run pytest tests/test_tool_search.py tests/test_agent_loop_tool_search.py -q
# 35 passed

uv run ruff check raven/agent/tools/tool_index.py raven/agent/tools/tool_search.py \
    raven/config/schema.py tests/test_tool_search.py tests/test_agent_loop_tool_search.py
# All checks passed!

uv run ruff format --check raven/agent/tools/tool_index.py raven/agent/tools/tool_search.py \
    tests/test_tool_search.py
# already formatted

New unit coverage: schema-keyword extraction (names, descriptions, enums,
nesting, depth cap, non-dict input), rebuild-on-parameters-change, search hits
carrying parameters, meta-tool set no longer including tool_describe.

  • Relevant tests pass locally
  • Relevant lint / type checks pass locally
  • User-facing docs or screenshots are updated when needed

Risk

  • Security impact considered
  • Backward compatibility considered
  • Rollback path is clear for risky changes

Progressive tool disclosure is opt-in (tools.tool_search.enabled defaults to
false), so the default agent path is unaffected. tool_describe is an internal
meta-tool with no public API; removing it changes only the enabled-feature flow.
tool_search hits are larger now (they carry parameter schemas), but only when
the model actively searches - the per-turn tool list and prompt-cache prefix are
untouched. Rollback: revert the branch.

Related Issues

N/A

shallyan and others added 3 commits July 6, 2026 22:36
Progressive tool disclosure exposed three meta-tools; tool_describe was a
separate round-trip to fetch a tool's parameter schema after tool_search.
Collapse to two: tool_search now returns each hit's parameter schema
alongside name and description, so the model can go straight to tool_call.

Also fold parameter names, descriptions and enum values into the BM25 index
body so discriminating keywords that live in the schema (a repo owner, a
channel id, an image size) lift recall. Lower the default search_result_limit
from 10 to 5 to offset the larger per-hit payload.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
The prior commit lowered tool_search's default result count to 5 to offset
the larger per-hit payload from folding parameter schemas into hits. Restore
it to 10: recall from more candidates outweighs the marginal payload, and a
caller can still cap results per query via the limit argument.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
The OpenRouter-backed disclosure e2e needs a live LLM and network, so it
cannot run as a deterministic unit test. The search -> call flow (hits carry
parameter schemas, tool_describe removed) stays covered by the deterministic
cases in tests/test_tool_search.py.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
@shallyan shallyan force-pushed the refactor/simplify_tool_search branch from f4e56fe to c8ace52 Compare July 6, 2026 15:02
@shallyan shallyan closed this Jul 6, 2026
@shallyan shallyan reopened this Jul 6, 2026
Align the JSON schema maximum for the tool_search limit param with the
default search_result_limit of 10, so the model cannot request more
results than the configured default.

Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
@shallyan shallyan requested review from 0xKT and cyfyifanchen July 7, 2026 08:37
@shallyan shallyan requested a review from JasonJarvan July 8, 2026 03:38
@shallyan shallyan merged commit ae3024d into main Jul 8, 2026
7 checks passed
@shallyan shallyan deleted the refactor/simplify_tool_search branch July 8, 2026 06:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants