Skip to content

LLM chat missing 10+ MCP tool definitions — cannot create device identities or use newer tools #724

@amiable-dev

Description

@amiable-dev

Problem

The LLM chat cannot create device identities, send MIDI, switch modes, validate config, or use any of the tools added after v4.23.0. When a device shows is_configured: false, the LLM correctly identifies it needs conductor_create_device_identity (per system prompt instructions) but cannot call it because the tool schema was never sent to the LLM provider.

User-visible behavior

The LLM says:

"Since I don't have access to a conductor_create_device_identity function in my available tools..."

Then suggests manual workarounds instead of performing the operation.

Root Cause

Tool definition mismatch between the GUI and daemon layers.

The GUI's get_mcp_tool_definitions() (conductor-gui/src-tauri/src/llm_commands.rs:190-465) only registers 16 tools — the original set from v4.11.0-v4.15.0 plus the ADR-016 signal tools. But the daemon's mcp_tools.rs defines 25+ tools.

Tools registered in GUI (sent to LLM) — 16 total:

Tool Risk Tier
conductor_get_status ReadOnly
conductor_list_devices ReadOnly
conductor_get_config ReadOnly
conductor_list_mappings ReadOnly
conductor_get_mapping ReadOnly
conductor_get_topology_summary ReadOnly
conductor_create_mapping ConfigChange
conductor_update_mapping ConfigChange
conductor_delete_mapping ConfigChange
conductor_batch_changes ConfigChange
conductor_start_midi_learn Stateful
conductor_stop_midi_learn Stateful
conductor_get_signal_pulse Frontend-intercepted
conductor_get_recent_events Frontend-intercepted
conductor_get_loop_analysis Frontend-intercepted
conductor_get_mapping_stats Frontend-intercepted

Tools MISSING from GUI (defined in daemon but not sent to LLM) — 11+ tools:

Tool Risk Tier Added In
conductor_create_device_identity ConfigChange v4.25.0
conductor_set_device_enabled Stateful v4.23.0
conductor_scan_ports HardwareIO v4.23.0
conductor_list_device_bindings ReadOnly v4.23.0
conductor_send_midi HardwareIO v4.26.67
conductor_switch_mode Stateful v4.26.69
conductor_validate_config ReadOnly v4.26.66
conductor_switch_profile Stateful daemon
conductor_get_active_profile ReadOnly daemon
conductor_send_sysex HardwareIO daemon
conductor_device_reset HardwareIO daemon
conductor_list_plugins ReadOnly daemon
conductor_plugin_info ReadOnly daemon
conductor_enable_plugin ConfigChange daemon
conductor_disable_plugin ConfigChange daemon

Second issue: system prompt references tools the LLM can't call

The system prompt (chat.js:1350-1435) mentions conductor_create_device_identity, conductor_switch_mode, conductor_batch_changes (this one IS registered), etc. — creating confusion where the LLM knows what it should do but can't actually do it.

Recommended Fix

1. Add missing tool definitions to get_mcp_tool_definitions() in llm_commands.rs

Add ToolDefinition entries for at minimum these high-value tools:

  • conductor_create_device_identitycritical for multi-device workflows
  • conductor_set_device_enabled — mute/unmute devices
  • conductor_scan_ports — trigger port rescan
  • conductor_list_device_bindings — view device binding state
  • conductor_send_midi — send MIDI messages
  • conductor_switch_mode — switch active mode
  • conductor_validate_config — validate config

Copy parameter schemas from conductor-daemon/src/daemon/mcp_tools.rs:322-382 (and other tool definitions) to ensure consistency.

2. Consider generating tool definitions from a single source

The current pattern of maintaining parallel tool definitions in the daemon (mcp_tools.rs) and GUI (llm_commands.rs) is error-prone. Options:

  • Shared crate: Move ToolDefinition list to conductor-core and import in both
  • Code generation: Generate GUI definitions from daemon definitions at build time
  • Runtime import: Have the GUI fetch tool definitions from the daemon via IPC

3. Add a test to catch future drift

#[test]
fn test_gui_tools_subset_of_daemon_tools() {
    let gui_tools = get_mcp_tool_definitions();
    let daemon_tools = conductor_daemon::get_tool_definitions();
    for tool in &gui_tools {
        assert!(daemon_tools.iter().any(|d| d.name == tool.name),
            "GUI tool '{}' not found in daemon", tool.name);
    }
}

Scope

  • Primary: conductor-gui/src-tauri/src/llm_commands.rs — add missing ToolDefinition entries
  • Secondary: conductor-gui/ui/src/lib/stores/chat.js — update system prompt to match available tools exactly
  • Optional: Refactor to single source of truth for tool definitions

Acceptance Criteria

  • conductor_create_device_identity works from chat (LLM can register devices)
  • All multi-device tools (list_device_bindings, set_device_enabled, scan_ports) callable from chat
  • conductor_send_midi, conductor_switch_mode, conductor_validate_config callable from chat
  • System prompt tool list matches actual tool definitions exactly
  • Test exists to prevent future tool definition drift
  • Existing tool calls continue to work (no regression)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions