Selecting the wrong tool

[PenguinDOOM]

Type: Bug

This is a random occurrence.

The custom agent is instructed to use askQuestions last. However, it may mistakenly call renameSymbol.
While the selection of the wrong tool is random, the selected tool will always call renameSymbol.

VS Code version: Code - Insiders 1.116.0-insider (47add6d, 2026-04-10T20:25:41+09:00)
OS version: Windows_NT x64 10.0.26200
Modes:

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-14700F (28 x 2112)
GPU Status	2d_canvas: enabled GPU0: VENDOR= 0x10de, DEVICE=0x2705 [NVIDIA GeForce RTX 4070 Ti SUPER], DRIVER_VENDOR=NVIDIA, DRIVER_VERSION=32.0.15.9597 ACTIVE GPU1: VENDOR= 0x1414, DEVICE=0x008c [Microsoft Basic Render Driver], DRIVER_VERSION=10.0.26100.7309 Machine model name: Machine model version: direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off trees_in_viz: disabled_off video_decode: enabled video_encode: enabled webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	undefined
Memory (System)	63.84GB (47.55GB free)
Process Argv	--crash-reporter-id 2ec5f96c-c0f3-4a7f-a64c-d65afeffe46b
Screen Reader	no
VM	0%

Extensions (41)

Extension	Author (truncated)	Version
UnityShader	ash	1.0.13
markdown-preview-github-styles	bie	2.2.0
path-intellisense	chr	2.10.0
csharpier-vscode	csh	10.0.1
prettier-vscode	esb	12.4.0
remotehub	Git	0.64.0
vscode-github-actions	git	0.31.3
discord-vscode	icr	5.9.2
markdown-live-editor	jis	0.5.3
zenkaku-hankaku	mas	1.0.0
vscode-copilot-auto-retry	Max	0.3.1
vscode-github-actions	me-	3.0.1
zenkaku	mos	0.0.3
vscode-language-pack-ja	MS-	1.110.2026041113
csdevkit	ms-	3.10.14
csharp	ms-	2.130.5
vscode-dotnet-runtime	ms-	3.0.0
debugpy	ms-	2026.3.2026032701
python	ms-	2026.5.2026032701
vscode-pylance	ms-	2026.2.100
vscode-python-envs	ms-	1.27.2026040901
remote-ssh	ms-	0.123.2026033115
remote-ssh-edit	ms-	0.87.0
remote-wsl	ms-	0.104.3
azure-repos	ms-	0.40.0
cmake-tools	ms-	1.22.28
cpp-devtools	ms-	0.4.6
cpptools	ms-	1.31.4
cpptools-extension-pack	ms-	1.5.1
powershell	ms-	2025.4.0
remote-explorer	ms-	0.6.2026031809
remote-repositories	ms-	0.42.0
indent-rainbow	ode	8.3.1
fix-json	oli	0.2.0
material-icon-theme	PKi	5.33.1
vscode-yaml	red	1.22.2026041108
trailing-spaces	sha	0.4.1
shader	sle	1.1.5
pdf	tom	1.2.2
vstuc	vis	1.2.1
gpg-indicator	wdh	0.7.5

(1 theme extensions excluded)

A/B Experiments

vsliv368cf:30146710
pythonvspyt551:31249597
nativeloc1:31118317
dwcopilot:31158714
dwoutputs:31242946
copilot_t_ci:31333650
g012b348:31231168
pythonrdcb7:31268811
pythonpcpt1cf:31399617
6518g693:31302842
82j33506:31327384
6abeh943:31336334
envsactivate1:31349248
editstats-enabled:31346256
cloudbuttont:31366566
3efgi100_wstrepl:31403338
use-responses-api:31390341
je187915:31401257
cp_cls_c_966_ss:31454199
inlinechat_v2_hd992725:31445440
4je02754:31455664
8hhj4413:31478653
ge8j1254_inline_auto_hint_haiku:31490507
cp_cls_c_1081:31454833
conptydll_true:31485575
ia-use-proxy-models-svc:31446143
e9c30283:31453065
test_treatment2:31471001
g_xe_63ac8346:31491249
idci7584:31454084
nes_chat_context_disabled:31451402
e3e4d672:31454087
ei9d7968:31462942
534a6447_copy:31478741
7ef87755:31455235
nes-extended-on:31455475
quick_suggest_off_75197330:31462668
89g7j272:31480197
7e884298:31462391
4f032803_compaction:31493295
t-some:31466503
7e187181:31482583
cpptoolsoff-v2:31475362
i2gc6536:31472020
1g3cj959:31491811
h08i8180:31475367
ec1eg703:31491645
client_tst_t:31486874
po_split_t:31488387
ddid_c:31478205
getcmakediagnosticsoff:31489825
ja75b849:31492254
61f49681:31488289
7df3h592:31491241
fixed-ctrl-prompt-cache:31487637
pro_ctrl:31484701
41e7a136_t:31485218
cp_cls_c_1082:31491634
719di409_sum_t:31493350
0he74425_c:31489327
nes-cj-befline:31490295
nes-async-ew-check-on:31490308
logging_enabled_new:31490725

[PenguinDOOM]

log
0df6bd4f.md

[PenguinDOOM]

I give them instructions to avoid making mistakes as much as possible, but they still make mistakes.
And it seems the agents don't understand why they made the mistakes.

[raye-deng]

Random tool selection by AI agents is a known and under-appreciated problem. The agent has access to multiple tools, and in some contexts it picks the wrong one — not because of a misunderstanding, but because the tool-selection heuristics have no deterministic guardrail.

What makes this particularly hard to debug:

1. Non-deterministic by nature: The same prompt can produce different tool selections across runs, making it impossible to reliably reproduce
2. Instruction-following gap: The agent "knows" it should use askQuestions last, but the tool-selection layer doesn't always respect that ordering constraint
3. Cascade effects: Once the wrong tool is selected, the subsequent execution path diverges from the intended workflow, and the user sees unexpected behavior

This connects to a broader pattern we see in AI-generated code: the agent selects the right general area but the wrong specific implementation. For example, calling a deprecated overload instead of the current API, or importing from a subpath that doesn't exist. The intent is correct, but the execution is off by one layer of specificity.

The most effective mitigation we've found is deterministic validation of agent outputs: regardless of which tool the agent selected, verify the result against the expected contract. This catches both wrong-tool and right-tool-wrong-usage scenarios.

For CI pipelines with AI-generated code: npx @opencodereview/cli scan . --sla L1 validates agent outputs against objective criteria, independent of tool selection.

[roblourens]

It looks like the askQuestions tool was not enabled in this run

[PenguinDOOM]

It looks like the askQuestions tool was not enabled in this run

If the AI ​​saw this log, it would say the same thing as you.
VS Code features, including askQuestions, are randomly unloaded. This log has indeed been unloaded.
It is mandatory to call activate_vs_code_interaction before calling askQuestions.
However, if not called, the agent will either say "not found" or will always call vscode_renameSymbol.

[PenguinDOOM]

The call is successful on lines 3262 and 3689, but otherwise it calls vscode_renameSymbol.

[PenguinDOOM]

For now, I'll create a hook and see how it goes with ask and additional context.

[PenguinDOOM]

I've discovered something new.
If renameSymbol is disabled, vscode_listCodeUsages is called.