investigation: measure tool schema token weight in system prompt

[Aaronontheweb]

Summary

Every LLM call sends the JSON schema for every available tool in the request. Netclaw currently has ~30-40 tools registered, each with parameter schemas, descriptions, and examples. Measure how many tokens of each turn's input are actually tool definitions vs conversation content. The measurement itself is the deliverable — the decision on what to do with it depends on the numbers.

Why

Tool schemas are stable per-session — they don't change between turns unless the tool set itself changes (progressive disclosure, skill auto-load, etc.). That means they should live inside the cacheable prefix and ride along the #608 fix for free. But there are two ways they could secretly be costing us:

1. If the schemas are large enough (say, 5k+ tokens of pure schema), the uncached cost on the FIRST turn of every session is non-trivial. Even with caching, you pay the full cost once per session + any turn where the tool set changes.

2. If the schemas sit in a position where they break prefix cache stability (e.g., if the tool list re-serializes with different ordering or whitespace each turn), they could be poisoning the cache we just worked so hard to make stable.

Method

Use the eval suite + Multi-Turn Cache Evolution table from the post-#608 baseline as a starting point. multi_turn_text_growth has 5 short chit-chat turns — minimal conversation payload, so the input size is almost entirely {persisted prompt + tool schemas + session block}.

From the current post-fix baseline (memory 6b42a0e4-8210-4e55-b9ca-8ff65c527cac):

multi_turn_text_growth  1  input=5380  cached=4707  uncached=673
multi_turn_text_growth  2  input=5038  cached=4864  uncached=174

That's roughly 5000 tokens of "static baseline" on each turn. SOUL.md + AGENTS.md + TOOLING.md account for some of it, tool schemas account for some of it. The question is what fraction.

Concrete steps

1. Tap the Netclaw-side serialization point where tool schemas get added to the outgoing request. Log or capture the serialized JSON length of the tools field on one representative call.
2. Compare to the persisted system prompt length from ISystemPromptProvider.GetSystemPrompt().
3. Compute the ratio: how much of a 5000-token static prefix is tool schemas? If it's >40%, trimming has measurable impact. If it's <10%, it's not worth optimizing.

Potential follow-up actions (depend on the measurement)

Not filing as separate issues until we have numbers, but candidates are:

• Trim tool descriptions — if descriptions are verbose and repetitive, condensing them saves the same tokens on every single turn.
• More aggressive progressive disclosure — Netclaw already has search_tools for dynamic discovery. If the token savings are large, we could move more tools behind progressive discovery and let the agent pull them on demand.
• Tool schema compression — some providers let you use a shorter format (OpenAI has tool_choice: "auto" with a parallel-friendly format); not available on all providers but worth checking.

Out of scope

• Any code changes based on the measurement — this issue is just the measurement. Filing decisions on subsequent actions will happen after we see real numbers.
• Non-measurement changes to the tool serialization pipeline.

Related

• System prompt cache busting: reorder dynamic context layers + move memory recall out of system role #608 — cache-prefix stability (the work that makes this question interesting)
• Memory reference: 6b42a0e4-8210-4e55-b9ca-8ff65c527cac (post-fix baseline eval numbers, when the memorizer box comes back online)

[Aaronontheweb]

Concrete breakdown from a fresh single-turn chat -p against a self-hosted llama.cpp openai-compatible endpoint, validated on a current dev build (post #946 / #947) where the LLM prompt summary diagnostic now lands at Debug. For a one-line user message (17 chars), the wire payload was input=14932 tokens with this structure: messages=4 system=3 user=1 assistant=0 tool=0 totalChars=40140 toolOptions=25 browserToolOptions=0.

#	Role	Built by	Char weight
0	System	SystemPromptAssembler.Assemble (SOUL + AGENTS + TOOLING + project instructions)	16,506
1	System	SessionMessageAssembler.BuildStaticContextBlock: OnceAtStart IContextLayerProviders plus [session] plus [attachments] (when file_read granted)	~23,000
2	User	the prompt	17
3	System	BuildVolatileContextBlock: [memory-recall] (empty here, itemCount=0), [current-time] (the one EveryTurn layer), [working-context] (empty), [active-background-jobs] (empty)	small

Slot [1] is dominated by ToolIndexContextLayer, i.e., the per-call cost of carrying descriptions for 25 registered tools. The other layers there are minor: SkillIndexContextLayer is the compressed name+description index (~22 skills, roughly 3.3k chars), MemoryIndexContextLayer and SubAgentDiscoveryContextLayer are small. The actual JSON tools= array on the wire is additional to this and not counted in totalChars. That's the measurement this issue is specifically asking for, and we still don't have it.

Worth flagging: zero skill bodies are inlined into the prompt at session start. Skill content is loaded lazily via file_read (or /name slash command, or keyword match through SystemSkillSyncService enrichment) into the volatile block on a later turn. So debate about "auto-loading skills" is a non-factor for the static prefix budget — only the index entries are present.

Comparison with the prior day

158 prompt-summary lines from the prior day vs. 6 from today, filtered to autonomous-channel calls (toolOptions=25):

	count	min	p25	median	p75	max
Prior day	120	17,408	31,579	46,994	85,207	257,699
Today	6	27,912	37,060	43,090	43,712	57,161

Today's median sits below the prior day's. Today's sample is biased toward early-turn calls without chained tool-result accumulation, which is why the upper tail is missing. 14-15k-token input on a fresh single-line prompt isn't a regression — it's the static cost of identity + tool index + tool schemas combined.

Profiling extensions to close out the investigation

Three steps, smallest-change-first, that would let us answer the issue without more code spelunking:

1. Add toolSchemaChars to LLM prompt summary. One field on the existing log line that counts the serialized tools= JSON length as it goes on the wire. Today we have toolOptions=25 (count) but not byte weight. With this single field, the ratio the issue asks for can be computed from one log line.

2. Add per-system-message char breakdown to the prompt summary. Something like persistedChars=… staticBlockChars=… volatileBlockChars=…. Shows the [persisted | static | volatile] split at a glance and would have collapsed today's investigation into one grep.

3. Emit individual IContextLayerProvider sizes at Debug the first time the static block is built (toolIndex=… skillIndex=… memoryIndex=… subagentDiscovery=… sessionBlock=… attachmentsHint=…). Targeted instrumentation that shows whether the [tools] index dominates slot [1], versus identity files dominating slot [0].

Step 1 alone closes the literal ask in the issue. Steps 2 and 3 produce a durable view of the static prefix that future audits can reuse without re-instrumenting.

[Aaronontheweb]

Surfaced a dogfood observation today that maps onto this. A fresh chat -p "hi are you there?" session reported in=14932 tokens — Aaron's mental baseline was ~9k for a comparable empty turn. The daemon log shows System prompt set (16506 chars) for that session, which fully accounts for ~4k of the prompt budget. The other ~7k is somewhere in the (tool defs + skill index + working context + memory recall) layers — exactly the space this issue is about characterizing.

Concrete sizes captured today:

• Resources/AGENTS.md (embedded resource): 11,771 bytes (≈ 3k tokens). Grew during the approval-policy-v2 PR by ~40 lines / ~500 tokens for the "Declaring Project Scope" guidance.
• ~/.netclaw/identity/SOUL.md: 1,931 bytes
• ~/.netclaw/identity/TOOLING.md: 2,637 bytes
• Total identity content: ~16.3 KB / ~4k tokens.

That leaves a ~7k-token gap unaccounted for between identity content and observed in=14932. Could be tool schemas (the original concern of this issue), but also plausibly skill index growth, the new working-context block, or memory recall when items hit the floor.

Suggested addition to scope

Netclaw.Daemon.Configuration.LoggingChatClient.LogPromptSummary already emits totalChars=N system=N user=N assistant=N tool=N. Extending that emission to attribute totalChars across the actual production-side contributors would turn this into a continuously-running measurement instead of a one-time investigation:

• system_prompt_chars (the assembled SOUL+AGENTS+TOOLING+project_instructions output)
• tool_schema_chars (the JSON-serialized tools field — the original ask of this issue)
• skill_index_chars (the skill-discovery menu layer)
• working_context_chars (recent files / project_dir block)
• memory_recall_chars (injected durable_fact items for the turn)

Each is independently measurable at the chat-client layer (we already have access to the request before serialization). Logging these per-turn would let any future regression in any one bucket be detected from the daemon log without requiring a one-off investigation.

If that scope creep is unwelcome, the original tool-schema-only measurement still answers the headline question — just noting that the same instrumentation hook would cover the broader observability gap with marginal extra effort.

Cross-ref: relevant in the context of the approval-policy-v2 + path-extraction PRs (#940, branch openspec/approval-policy-v2) where AGENTS.md grew specifically to support the new approval guidance.