Scan Request: helix-pilot (GUI Automation MCP Server)

[tsunamayo7]

Tool Information

• Name: helix-pilot
• Repository: https://github.com/tsunamayo7/helix-pilot
• Version: v1.0.0
• Type: MCP Server (stdio)
• Language: Python

Description

helix-pilot is a local Vision LLM-powered MCP server that enables AI agents to observe and interact with Windows desktop GUI applications. It uses local Ollama models (no cloud API required) to analyze screenshots and execute GUI operations via PyAutoGUI and Win32 API.

Key Security Characteristics

• Local-only processing: All LLM inference runs on local Ollama, no data sent to external servers
• No network access required: Operates entirely on localhost
• Configurable safety: Region-based click restrictions, coordinate validation
• Read + Write operations: Screen capture (read) and mouse/keyboard control (write)
• Python/FastMCP based: Standard MCP stdio transport

Why Scan This Tool

helix-pilot handles desktop GUI automation which inherently requires exec-level permissions. A ToolTrust security grade would help users understand the risk profile and make informed decisions about deployment.

Requested By

@tsunamayo7 (tool author)

[m13v]

gui automation MCP servers have a unique risk profile that's different from file or network MCPs. the core issue is that helix-pilot (and tools like it) can interact with ANY visible UI element on the desktop, not just the target application. a screenshot-based approach has the advantage that the agent only sees what's on screen, but pyautogui can move the mouse anywhere and type into anything. the coordinate validation and region-based click restrictions mentioned in helix-pilot's security characteristics are the right instinct but they're fragile. if a window moves or resizes, the restricted region no longer covers the intended app. a more robust approach is to use accessibility APIs to constrain interactions to a specific application's UI tree by process ID. that way even if the agent tries to interact with another app, the automation layer refuses because the target element doesn't belong to the allowed process.

[m13v]

the desktop automation framework we built constrains interactions to specific process accessibility trees. the element interaction layer with process-scoped targeting: https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/element.rs and the MCP server that exposes these as tools with built-in scoping: https://github.com/mediar-ai/terminator/blob/main/crates/terminator-mcp-agent/src/server.rs