-
Notifications
You must be signed in to change notification settings - Fork 0
feat(qa): adb interaction tools for QA agents (screenshot/input/logcat) #423
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,183 @@ | ||
| """ | ||
| Android QA Harness Tools | ||
| ======================== | ||
|
|
||
| MCP tools that let QA agents drive an Android app on an emulator or | ||
| attached device via adb: screenshots, input injection, and logcat reads. | ||
| The Android analog of the Electron MCP toolset. | ||
|
|
||
| Only registered when the project is an Android Gradle project, keeping | ||
| tool context clean for everything else. | ||
| """ | ||
|
|
||
| import base64 | ||
| import logging | ||
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| try: | ||
| from claude_agent_sdk import tool | ||
|
|
||
| SDK_AVAILABLE = True | ||
| except ImportError: | ||
| SDK_AVAILABLE = False | ||
|
|
||
| LOGCAT_TAIL_CHARS = 8_000 | ||
|
|
||
|
|
||
| def create_android_tools(spec_dir: Path, project_dir: Path) -> list: | ||
|
Check failure on line 28 in apps/backend/agents/tools_pkg/tools/android_harness.py
|
||
| """ | ||
| Create Android device tools for QA agents. | ||
|
|
||
| Args: | ||
| spec_dir: Path to the spec directory (unused, kept for registry symmetry) | ||
| project_dir: Path to the project root; gates registration | ||
|
|
||
| Returns: | ||
| List of tool functions (empty for non-Android projects) | ||
| """ | ||
| if not SDK_AVAILABLE: | ||
| return [] | ||
|
|
||
| try: | ||
| from core.android_device import is_android_project | ||
| except ImportError: | ||
| return [] | ||
|
|
||
| if not is_android_project(project_dir): | ||
| return [] | ||
|
|
||
| tools = [] | ||
|
|
||
| @tool( | ||
| "android_screenshot", | ||
| "Take a screenshot of the Android emulator or attached device via " | ||
| "adb. Use it to visually verify UI state after launching the app or " | ||
| "interacting with it. Optionally pass a device serial when several " | ||
| "devices are connected.", | ||
| {"serial": str}, | ||
| ) | ||
| async def android_screenshot(args: dict[str, Any]) -> dict[str, Any]: | ||
| """Capture and return a device screenshot.""" | ||
| from core.android_device import take_screenshot | ||
|
|
||
| try: | ||
| image, mime_or_error = take_screenshot( | ||
| serial=(args.get("serial") or "").strip() or None | ||
| ) | ||
|
Comment on lines
+64
to
+67
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win Offload blocking ADB calls from the MCP event loop. Each async handler synchronously waits in As per path instructions, focus on proper “async patterns.” Also applies to: 118-133, 157-162 🤖 Prompt for AI AgentsSource: Path instructions |
||
| except Exception as e: | ||
| logging.exception("Error during android_screenshot") | ||
| return _text_result(f"Error taking screenshot: {e}") | ||
|
|
||
| if image is None: | ||
| return _text_result(f"Screenshot failed: {mime_or_error}") | ||
|
|
||
| return { | ||
| "content": [ | ||
| { | ||
| "type": "image", | ||
| "data": base64.b64encode(image).decode("ascii"), | ||
| "mimeType": mime_or_error, | ||
| } | ||
| ] | ||
| } | ||
|
|
||
| tools.append(android_screenshot) | ||
|
|
||
| @tool( | ||
| "android_input", | ||
| "Send input to the Android emulator or attached device via adb. " | ||
| "Actions: 'tap' (x, y), 'swipe' (x, y, x2, y2, optional " | ||
| "duration_ms), 'text' (text typed into the focused field), " | ||
| "'keyevent' (key: KEYCODE_* name or numeric code, e.g. KEYCODE_BACK). " | ||
| "Use android_screenshot before and after to verify the effect.", | ||
| { | ||
| "action": str, | ||
| "x": int, | ||
| "y": int, | ||
| "x2": int, | ||
| "y2": int, | ||
| "duration_ms": int, | ||
| "text": str, | ||
| "key": str, | ||
| "serial": str, | ||
| }, | ||
| ) | ||
| async def android_input(args: dict[str, Any]) -> dict[str, Any]: | ||
| """Inject a tap, swipe, text, or key event.""" | ||
| from core.android_device import ( | ||
| send_keyevent, | ||
| send_swipe, | ||
| send_tap, | ||
| send_text, | ||
| ) | ||
|
|
||
| action = (args.get("action") or "").strip().lower() | ||
| serial = (args.get("serial") or "").strip() or None | ||
|
|
||
| try: | ||
| if action == "tap": | ||
| result = send_tap(args.get("x") or 0, args.get("y") or 0, serial) | ||
| elif action == "swipe": | ||
| result = send_swipe( | ||
| args.get("x") or 0, | ||
| args.get("y") or 0, | ||
| args.get("x2") or 0, | ||
| args.get("y2") or 0, | ||
| args.get("duration_ms") or 300, | ||
| serial, | ||
|
Comment on lines
+120
to
+128
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win Preserve explicit zero-valued inputs.
Proposed fix- args.get("duration_ms") or 300,
+ args.get("duration_ms", 300),
...
- lines=args.get("lines") or 200,
+ lines=args.get("lines", 200),Also applies to: 159-159 🤖 Prompt for AI Agents |
||
| ) | ||
| elif action == "text": | ||
| result = send_text(args.get("text") or "", serial) | ||
| elif action == "keyevent": | ||
| result = send_keyevent(args.get("key") or "", serial) | ||
| else: | ||
| return _text_result("Unknown action: use tap, swipe, text, or keyevent") | ||
| except Exception as e: | ||
| logging.exception("Error during android_input") | ||
| return _text_result(f"Error sending input: {e}") | ||
|
|
||
| status = "OK" if result.ok else "FAILED" | ||
| detail = result.output.strip() | ||
| return _text_result(f"{status}: {action}" + (f"\n{detail}" if detail else "")) | ||
|
|
||
| tools.append(android_input) | ||
|
|
||
| @tool( | ||
| "android_logcat", | ||
| "Read recent Android log output (adb logcat) for debugging. " | ||
| "Optionally filter lines by a substring (e.g. your app's tag or " | ||
| "package name) and limit the number of recent lines.", | ||
| {"lines": int, "filter_text": str, "serial": str}, | ||
| ) | ||
| async def android_logcat(args: dict[str, Any]) -> dict[str, Any]: | ||
| """Fetch recent logcat lines.""" | ||
| from core.android_device import read_logcat | ||
|
|
||
| try: | ||
| result = read_logcat( | ||
| lines=args.get("lines") or 200, | ||
| filter_text=(args.get("filter_text") or "").strip() or None, | ||
| serial=(args.get("serial") or "").strip() or None, | ||
| ) | ||
| except Exception as e: | ||
| logging.exception("Error during android_logcat") | ||
| return _text_result(f"Error reading logcat: {e}") | ||
|
|
||
| if not result.ok: | ||
| return _text_result(f"logcat failed: {result.output}") | ||
|
|
||
| output = result.output[-LOGCAT_TAIL_CHARS:] | ||
| return _text_result(output if output.strip() else "(no matching log lines)") | ||
|
Comment on lines
+157
to
+171
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift Do not return unscoped device logs to the agent. With no filter, this forwards raw device-wide logcat output into the agent context. Attached-device logs may contain identifiers, tokens, or other user data. Require an app/package scope by default and redact sensitive patterns before returning results. As per path instructions, check for proper “security considerations.” 🤖 Prompt for AI AgentsSource: Path instructions |
||
|
|
||
| tools.append(android_logcat) | ||
|
|
||
| return tools | ||
|
|
||
|
|
||
| def _text_result(text: str) -> dict[str, Any]: | ||
| """Wrap text in the MCP tool result envelope.""" | ||
| return {"content": [{"type": "text", "text": text}]} | ||
|
|
||
|
|
||
| __all__ = ["create_android_tools"] | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: OBenner/Auto-Coding
Length of output: 149
🏁 Script executed:
Repository: OBenner/Auto-Coding
Length of output: 23954
🏁 Script executed:
Repository: OBenner/Auto-Coding
Length of output: 26494
🏁 Script executed:
Repository: OBenner/Auto-Coding
Length of output: 7356
🏁 Script executed:
Repository: OBenner/Auto-Coding
Length of output: 27099
🌐 Web query:
claude-agent-sdk@tooldict schema required properties optional JSON schema tool decorator💡 Result:
In the Claude Agent SDK for Python, the
@tooldecorator uses the input_schema parameter to define the tool's arguments [1][2]. You have three ways to define this schema [2]: 1. Dictionary mapping (Simple): You provide a dictionary where keys are parameter names and values are Python types (e.g., {"name": str, "age": int}) [2][3]. In this mode, all parameters are treated as required [4]. 2. TypedDict (Complex): You can provide a TypedDict class for more structured, complex schemas [2]. 3. JSON Schema (Full Control): You can provide a full JSON Schema dictionary [1][2]. This is necessary if you need features like enums, specific ranges, nested objects, or explicit control over required/optional fields [5][4]. For JSON Schema definitions, the standard JSON schema structure applies, where you define your properties under the "properties" key and specify which are required using the "required" array [1][6]. To handle optional parameters in Python when using the simple dictionary mapping: Because the dictionary schema treats every key as required, the recommended pattern for optional parameters is to omit them from the schema entirely, mention them in the tool's description string, and access them using args.get within your handler function [4]. When using the full JSON Schema approach, you simply define your properties and omit the parameter name from the "required" list in the schema definition [1][6]. Top Results: [1][5][2][4]Citations:
Make the optional tool args optional.
android_screenshot,android_input, andandroid_logcatcurrently use the dict-based@toolschema, which makes every listed property required. That forcesserial,x2/y2/duration_ms,text,key, andfilter_textto be passed even though the handlers already treat them as optional. Switch to full JSON Schema (or aTypedDict) and mark only the action-specific fields as required.🤖 Prompt for AI Agents