diff --git a/README.md b/README.md index 3f1cd14..7b79fe0 100644 --- a/README.md +++ b/README.md @@ -73,6 +73,21 @@ Or add it to your own client manually: } ``` +### macOS image capture + +`get_app_state` and action tools return a PNG screenshot together with the accessibility tree. On macOS, screenshot size can be tuned with optional environment variables. Set them before starting the `open-computer-use` runtime process; changing shell environment values afterward does not affect an already running process. + +| Variable | Default | Meaning | +| --- | --- | --- | +| `OPEN_COMPUTER_USE_IMAGE_CAPTURE_TIMEOUT` | `5` | Seconds to wait for ScreenCaptureKit before omitting the screenshot image content from the result. | +| `OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION` | `1280` | Integer long-edge pixel cap for the returned PNG. | +| `OPEN_COMPUTER_USE_IMAGE_MAX_BYTES` | `900000` | Best-effort byte budget for the encoded PNG. | +| `OPEN_COMPUTER_USE_IMAGE_MIN_SCALE` | `0.25` | Byte-budget retry multiplier applied after the dimension cap. For example, a 500 px capped long edge with `0.25` may retry down to 125 px; it never enlarges the PNG chosen by the dimension cap. | + +Invalid, non-finite, or non-positive values fall back to the defaults. `OPEN_COMPUTER_USE_IMAGE_MIN_SCALE` values above `1` also fall back. + +Coordinate tools keep using the actual returned PNG dimensions, so downsampling does not change click or drag mapping. + ### Skill Install the skill directly: diff --git a/README.zh-CN.md b/README.zh-CN.md index 2cd60f8..739b8d5 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -72,6 +72,21 @@ ocu install-codex-mcp } ``` +### macOS 截图尺寸 + +`get_app_state` 和 action 类工具会和 accessibility tree 一起返回 PNG 截图。macOS 上可以通过可选环境变量调整截图尺寸。请在启动 `open-computer-use` runtime 进程前设置这些变量;之后再修改 shell 环境不会影响已经运行的进程。 + +| 变量 | 默认值 | 作用 | +| --- | --- | --- | +| `OPEN_COMPUTER_USE_IMAGE_CAPTURE_TIMEOUT` | `5` | ScreenCaptureKit 等待秒数;超时会从结果里省略截图 image 内容。 | +| `OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION` | `1280` | 返回 PNG 的整数长边像素上限。 | +| `OPEN_COMPUTER_USE_IMAGE_MAX_BYTES` | `900000` | 编码后 PNG 的 best-effort 字节预算。 | +| `OPEN_COMPUTER_USE_IMAGE_MIN_SCALE` | `0.25` | 作为长边上限之后的字节预算重试倍率。例如长边已限制到 500 px,`0.25` 允许继续重试缩小到 125 px;它不会放大长边上限已经选出的 PNG。 | + +非法值、非有限数或非正数会回退到默认值。`OPEN_COMPUTER_USE_IMAGE_MIN_SCALE` 大于 `1` 时也会回退。 + +坐标类工具会读取实际返回 PNG 的尺寸做映射,因此降采样不会改变 click 或 drag 的坐标换算。 + ### Skill 一键安装skill: diff --git a/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift b/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift index 1f02d00..3bcc7e6 100644 --- a/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift +++ b/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift @@ -88,6 +88,7 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDele private let dragPadView = DragPadView(frame: NSRect(x: 0, y: 0, width: 320, height: 120)) private var scrollView: NSScrollView! private var counter = 0 + private var selectedText: String? private weak var observedScrollView: NSScrollView? private var commandObserver: NSObjectProtocol? private let headless: Bool @@ -286,15 +287,18 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDele switch (command.kind, command.identifier) { case ("set_value", "fixture-input"): inputField.stringValue = command.value ?? "" + selectedText = nil updateExportedState() case ("click", "fixture-increment"): handleIncrement() case ("click", "fixture-input"): window.makeFirstResponder(inputField) + selectedText = nil updateExportedState() case ("click", "fixture-key-capture"): window.makeFirstResponder(keyCaptureView) keyCaptureView.needsDisplay = true + selectedText = nil updateExportedState() case ("scroll", "fixture-scroll-view"): let delta = CGFloat(120 * (command.pages ?? 1)) @@ -321,17 +325,83 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDele case ("type_text", "fixture-input"): inputField.stringValue += command.value ?? "" window.makeFirstResponder(inputField) + selectedText = nil + updateExportedState() + case ("select_text", "fixture-input"): + selectTextInFixtureInput(command) updateExportedState() case ("press_key", "fixture-key-capture"): keyLabel.stringValue = "Last key: \(command.value ?? "unknown")" window.makeFirstResponder(keyCaptureView) keyCaptureView.needsDisplay = true + selectedText = nil updateExportedState() default: break } } + private func selectTextInFixtureInput(_ command: FixtureCommand) { + let value = inputField.stringValue + guard let target = command.value, + let range = fixtureTextSelectionMatch(in: value, text: target, prefix: command.prefix, suffix: command.suffix) + else { + selectedText = nil + return + } + + let mode = command.selection ?? "text" + let selectedRange = switch mode { + case "cursor_before": + NSRange(location: range.location, length: 0) + case "cursor_after": + NSRange(location: range.location + range.length, length: 0) + default: + range + } + + window.makeFirstResponder(inputField) + if let editor = inputField.currentEditor() { + editor.selectedRange = selectedRange + } + selectedText = mode == "text" ? (value as NSString).substring(with: range) : nil + } + + private func fixtureTextSelectionMatch(in value: String, text: String, prefix: String?, suffix: String?) -> NSRange? { + let haystack = value as NSString + let target = text as NSString + guard target.length > 0 else { + return nil + } + + let prefix = prefix ?? "" + let suffix = suffix ?? "" + var searchRange = NSRange(location: 0, length: haystack.length) + var matchedRange: NSRange? + + while searchRange.length >= target.length { + let found = haystack.range(of: text, options: [], range: searchRange) + if found.location == NSNotFound { + break + } + + let afterStart = found.location + found.length + let before = haystack.substring(with: NSRange(location: 0, length: found.location)) + let after = haystack.substring(with: NSRange(location: afterStart, length: haystack.length - afterStart)) + if (prefix.isEmpty || before.hasSuffix(prefix)) && (suffix.isEmpty || after.hasPrefix(suffix)) { + if matchedRange != nil { + return nil + } + matchedRange = found + } + + let nextLocation = found.location + max(found.length, 1) + searchRange = NSRange(location: nextLocation, length: haystack.length - nextLocation) + } + + return matchedRange + } + private func updateExportedState() { guard let contentView = window.contentView else { return @@ -341,6 +411,7 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDele windowTitle: window.title, windowBounds: FixtureRect(rect: windowBoundsInQuartzCoordinates()), focusedIdentifier: focusedIdentifier(), + selectedText: selectedText, elements: [ element(identifier: "fixture-window", index: 0, role: "standard window", title: window.title, value: nil, actions: ["Raise"], rect: CGRect(x: 0, y: 0, width: window.frame.width, height: window.frame.height)), element(identifier: "fixture-increment", index: 1, role: "button", title: incrementButton.title, value: nil, actions: [], rect: localRect(for: incrementButton, in: contentView)), diff --git a/apps/OpenComputerUseLinux/main.go b/apps/OpenComputerUseLinux/main.go index ec75d85..84f22a4 100644 --- a/apps/OpenComputerUseLinux/main.go +++ b/apps/OpenComputerUseLinux/main.go @@ -25,7 +25,9 @@ var version = "0.1.54" //go:embed runtime.py var linuxRuntimeScript string -const serverInstructions = "Computer Use tools let you interact with Linux desktop apps by performing UI actions.\n\nBegin by calling `get_app_state` every turn you want to use Computer Use to get the latest state before acting. The available tools are list_apps, get_app_state, click, perform_secondary_action, scroll, drag, type_text, press_key, and set_value.\n\nPrefer element-targeted interactions over coordinate clicks when an index for the targeted element is available. Linux actions use AT-SPI2 semantic actions and editable text APIs first. Coordinate mouse and key synthesis are best-effort fallbacks and are not a universal Wayland background input model." +const serverInstructions = "Computer Use tools let you interact with Linux desktop apps by performing UI actions.\n\nBegin by calling `get_app_state` every turn you want to use Computer Use to get the latest state before acting. The available tools are list_apps, get_app_state, click, perform_secondary_action, scroll, select_text, drag, type_text, press_key, and set_value.\n\nPrefer element-targeted interactions over coordinate clicks when an index for the targeted element is available. Linux actions use AT-SPI2 semantic actions and editable text APIs first. Coordinate mouse and key synthesis are best-effort fallbacks and are not a universal Wayland background input model." + +const appStateNoChangeMessage = "There has been no change" type toolDefinition struct { Name string `json:"name"` @@ -50,6 +52,27 @@ func textResult(text string, isError bool) toolCallResult { return toolCallResult{Content: []contentItem{{Type: "text", Text: text}}, IsError: isError} } +type appStateOutputOptions struct { + IncludeImage bool + ForceImage bool + MaxTextChars *int + OnlyChanges bool +} + +type appStateOutputCache struct { + RenderedText string + ScreenshotPNGBase64 string + ImageIncluded bool +} + +func (c appStateOutputCache) sameSnapshot(other appStateOutputCache) bool { + return c.RenderedText == other.RenderedText && c.ScreenshotPNGBase64 == other.ScreenshotPNGBase64 +} + +func defaultAppStateOutputOptions() appStateOutputOptions { + return appStateOutputOptions{IncludeImage: true} +} + type appDescriptor struct { Name string `json:"name"` BundleIdentifier string `json:"bundleIdentifier,omitempty"` @@ -119,17 +142,49 @@ func (s *appSnapshot) renderedText() string { } func (s *appSnapshot) result() toolCallResult { + result, _ := s.appStateResult(defaultAppStateOutputOptions(), nil) + return result +} + +func (s *appSnapshot) appStateResult(options appStateOutputOptions, previous *appStateOutputCache) (toolCallResult, appStateOutputCache) { + renderedText := s.renderedText() + screenshot := "" + if s != nil { + screenshot = s.ScreenshotPNGBase64 + } + current := appStateOutputCache{RenderedText: renderedText, ScreenshotPNGBase64: screenshot} + if options.OnlyChanges && previous != nil && previous.sameSnapshot(current) { + return textResult(appStateNoChangeMessage, false), current + } + result := toolCallResult{ - Content: []contentItem{{Type: "text", Text: s.renderedText()}}, + Content: []contentItem{{Type: "text", Text: limitedAppStateText(renderedText, options.MaxTextChars)}}, } - if s != nil && s.ScreenshotPNGBase64 != "" { + if options.IncludeImage && screenshot != "" && (options.ForceImage || previous == nil || !previous.ImageIncluded || previous.ScreenshotPNGBase64 != screenshot) { result.Content = append(result.Content, contentItem{ Type: "image", - Data: s.ScreenshotPNGBase64, + Data: screenshot, MimeType: "image/png", }) + current.ImageIncluded = true } - return result + return result, current +} + +func limitedAppStateText(text string, maxChars *int) string { + if maxChars == nil || *maxChars == 0 { + return text + } + runes := []rune(text) + if len(runes) <= *maxChars { + return text + } + suffix := fmt.Sprintf("\n\n[truncated after %d characters]", *maxChars) + suffixRunes := []rune(suffix) + if *maxChars <= len(suffixRunes) { + return string(runes[:*maxChars]) + } + return string(runes[:*maxChars-len(suffixRunes)]) + suffix } type linuxRequest struct { @@ -148,6 +203,9 @@ type linuxRequest struct { Direction string `json:"direction,omitempty"` Pages float64 `json:"pages,omitempty"` Text string `json:"text,omitempty"` + Prefix string `json:"prefix,omitempty"` + Suffix string `json:"suffix,omitempty"` + Selection string `json:"selection,omitempty"` Key string `json:"key,omitempty"` Value string `json:"value,omitempty"` WindowBounds *frame `json:"windowBounds,omitempty"` @@ -162,11 +220,12 @@ type linuxResponse struct { } type service struct { - snapshots map[string]*appSnapshot + snapshots map[string]*appSnapshot + appStateOutputs map[string]appStateOutputCache } func newService() *service { - return &service{snapshots: map[string]*appSnapshot{}} + return &service{snapshots: map[string]*appSnapshot{}, appStateOutputs: map[string]appStateOutputCache{}} } func (s *service) callTool(name string, args map[string]any) toolCallResult { @@ -174,7 +233,11 @@ func (s *service) callTool(name string, args map[string]any) toolCallResult { case "list_apps": return s.listApps() case "get_app_state": - return s.getAppState(requiredString(args, "app"), optionalBool(args, "show_full_text")) + options, errText := appStateOutputOptionsFromArgs(args) + if errText != "" { + return textResult(errText, true) + } + return s.getAppState(requiredString(args, "app"), optionalBool(args, "show_full_text"), options) case "click": return s.click( requiredString(args, "app"), @@ -197,6 +260,15 @@ func (s *service) callTool(name string, args map[string]any) toolCallResult { requiredElementIndex(args), floatValue(optionalFloat(args, "pages"), 1), ) + case "select_text": + return s.selectText( + requiredString(args, "app"), + requiredElementIndex(args), + requiredString(args, "text"), + optionalString(args, "prefix"), + optionalString(args, "suffix"), + defaultString(optionalString(args, "selection"), "text"), + ) case "drag": return s.drag( requiredString(args, "app"), @@ -230,7 +302,7 @@ func (s *service) listApps() toolCallResult { return textResult(response.Text, false) } -func (s *service) getAppState(app string, showFullText bool) toolCallResult { +func (s *service) getAppState(app string, showFullText bool, outputOptions appStateOutputOptions) toolCallResult { if app == "" { return textResult("Missing required argument: app", true) } @@ -238,7 +310,13 @@ func (s *service) getAppState(app string, showFullText bool) toolCallResult { if result.IsError { return result } - return snapshot.result() + keys := appStateCacheKeys(app, snapshot) + previous := s.previousAppStateOutput(keys) + result, current := snapshot.appStateResult(outputOptions, previous) + for _, key := range keys { + s.appStateOutputs[key] = current + } + return result } func (s *service) click(app, elementIndex string, x, y *float64, clickCount int, mouseButton string) toolCallResult { @@ -317,6 +395,30 @@ func (s *service) scroll(app, direction, elementIndex string, pages float64) too return s.actionResult(app, linuxRequest{Tool: "scroll", App: app, Element: record, Direction: normalized, Pages: pages}) } +func (s *service) selectText(app, elementIndex, text, prefix, suffix, selection string) toolCallResult { + if app == "" { + return textResult("Missing required argument: app", true) + } + if elementIndex == "" { + return textResult("Missing required argument: element_index", true) + } + if text == "" { + return textResult("Missing required argument: text", true) + } + if selection != "text" && selection != "cursor_before" && selection != "cursor_after" { + return textResult("Invalid selection: "+selection, true) + } + snapshot := s.currentSnapshot(app) + if snapshot == nil { + return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true) + } + record, err := lookupElement(snapshot, elementIndex) + if err != nil { + return textResult(err.Error(), true) + } + return s.actionResult(app, linuxRequest{Tool: "select_text", App: app, Element: record, Text: text, Prefix: prefix, Suffix: suffix, Selection: selection}) +} + func (s *service) drag(app string, fromX, fromY, toX, toY *float64) toolCallResult { if app == "" { return textResult("Missing required argument: app", true) @@ -967,6 +1069,92 @@ func optionalBool(args map[string]any, key string) bool { return value } +func appStateOutputOptionsFromArgs(args map[string]any) (appStateOutputOptions, string) { + options := defaultAppStateOutputOptions() + if value, ok := args["include_image"].(bool); ok { + options.IncludeImage = value + } + if value, ok := args["force_image"].(bool); ok { + options.ForceImage = value + } + if value, ok := args["only_changes"].(bool); ok { + options.OnlyChanges = value + } + maxTextChars, ok := optionalNonNegativeInt(args, "max_text_chars") + if !ok { + return options, "max_text_chars must be a non-negative integer" + } + options.MaxTextChars = maxTextChars + return options, "" +} + +func optionalNonNegativeInt(args map[string]any, key string) (*int, bool) { + value, exists := args[key] + if !exists || value == nil { + return nil, true + } + + switch value := value.(type) { + case json.Number: + integer, err := value.Int64() + if err != nil || integer < 0 || int64(int(integer)) != integer { + return nil, false + } + result := int(integer) + return &result, true + case float64: + return nonNegativeIntFloat(value) + case int: + if value < 0 { + return nil, false + } + result := value + return &result, true + case int64: + if value < 0 || int64(int(value)) != value { + return nil, false + } + result := int(value) + return &result, true + } + + return nil, false +} + +func nonNegativeIntFloat(value float64) (*int, bool) { + if math.IsNaN(value) || math.IsInf(value, 0) || math.Trunc(value) != value || value < 0 || value > float64(int(^uint(0)>>1)) { + return nil, false + } + result := int(value) + return &result, true +} + +func appStateCacheKeys(query string, snapshot *appSnapshot) []string { + seen := map[string]bool{} + keys := []string{} + rawKeys := []string{query} + if snapshot != nil { + rawKeys = append(rawKeys, snapshot.App.Name, snapshot.App.BundleIdentifier) + } + for _, key := range rawKeys { + normalized := strings.ToLower(strings.TrimSpace(key)) + if normalized != "" && !seen[normalized] { + seen[normalized] = true + keys = append(keys, normalized) + } + } + return keys +} + +func (s *service) previousAppStateOutput(keys []string) *appStateOutputCache { + for _, key := range keys { + if previous, ok := s.appStateOutputs[key]; ok { + return &previous + } + } + return nil +} + func intValue(value *float64, fallback int) int { if value == nil { return fallback @@ -1022,6 +1210,10 @@ func toolDefinitions() []toolDefinition { InputSchema: objectSchema(map[string]any{ "app": stringProperty("App name or bundle identifier"), "show_full_text": booleanProperty("Return full accessibility text without the default 500 character truncation. Defaults to false."), + "include_image": booleanProperty("Return a screenshot image block when one is available. Defaults to true."), + "force_image": booleanProperty("Return the screenshot even when it matches the previous app state for this app. Defaults to false."), + "max_text_chars": integerProperty("Maximum characters to return from the rendered accessibility text. Set 0 for no post-render cap.", 0), + "only_changes": booleanProperty("Return only a no-change message when the rendered state and screenshot match the previous get_app_state result for this app. Defaults to false."), }, []string{"app"}), }, { @@ -1060,6 +1252,19 @@ func toolDefinitions() []toolDefinition { "pages": numberProperty("Number of pages to scroll. Fractional values are supported. Defaults to 1"), }, []string{"app", "element_index", "direction"}), }, + { + Name: "select_text", + Description: "Select text inside a text element, or place the text cursor before or after it. Provide text exactly as it appears in the accessibility tree, including any Markdown formatting. If the text is not unique, provide surrounding prefix or suffix text to disambiguate it.", + Annotations: defaultAnnotations(), + InputSchema: objectSchema(map[string]any{ + "app": stringProperty("App name or bundle identifier"), + "element_index": stringProperty("Text element identifier"), + "text": stringProperty("Target text as shown in the accessibility tree"), + "prefix": stringProperty("Optional text immediately before the target, used to disambiguate repeated matches"), + "suffix": stringProperty("Optional text immediately after the target, used to disambiguate repeated matches"), + "selection": enumStringProperty("Whether to select the text or place the cursor before or after it. Defaults to text.", []string{"text", "cursor_before", "cursor_after"}), + }, []string{"app", "element_index", "text"}), + }, { Name: "set_value", Description: "Set the value of a settable accessibility element. This tool is part of plugin `Computer Use`.", @@ -1120,8 +1325,12 @@ func numberProperty(description string) map[string]any { return map[string]any{"type": "number", "description": description} } -func integerProperty(description string) map[string]any { - return map[string]any{"type": "integer", "description": description} +func integerProperty(description string, minimum ...int) map[string]any { + property := map[string]any{"type": "integer", "description": description} + if len(minimum) > 0 { + property["minimum"] = minimum[0] + } + return property } func main() { diff --git a/apps/OpenComputerUseLinux/main_test.go b/apps/OpenComputerUseLinux/main_test.go index 6e28938..dee3467 100644 --- a/apps/OpenComputerUseLinux/main_test.go +++ b/apps/OpenComputerUseLinux/main_test.go @@ -11,24 +11,105 @@ import ( ) func TestToolDefinitionCount(t *testing.T) { - if got := len(toolDefinitions()); got != 9 { - t.Fatalf("toolDefinitions() count = %d, want 9", got) + if got := len(toolDefinitions()); got != 10 { + t.Fatalf("toolDefinitions() count = %d, want 10", got) } } -func TestGetAppStateSchemaIncludesShowFullText(t *testing.T) { +func TestSelectTextSchemaMatchesCodexSurface(t *testing.T) { + tool := findToolDefinition(t, "select_text") + properties := tool.InputSchema["properties"].(map[string]any) + selection := properties["selection"].(map[string]any) + required := tool.InputSchema["required"].([]string) + + if tool.Description != "Select text inside a text element, or place the text cursor before or after it. Provide text exactly as it appears in the accessibility tree, including any Markdown formatting. If the text is not unique, provide surrounding prefix or suffix text to disambiguate it." { + t.Fatalf("unexpected select_text description: %q", tool.Description) + } + if strings.Join(required, ",") != "app,element_index,text" { + t.Fatalf("required = %#v, want [app element_index text]", required) + } + if strings.Join(selection["enum"].([]string), ",") != "text,cursor_before,cursor_after" { + t.Fatalf("selection enum = %#v", selection["enum"]) + } +} + +func TestGetAppStateSchemaIncludesOutputControls(t *testing.T) { tool := findToolDefinition(t, "get_app_state") properties := tool.InputSchema["properties"].(map[string]any) showFullText := properties["show_full_text"].(map[string]any) if got := showFullText["type"]; got != "boolean" { t.Fatalf("show_full_text type = %v, want boolean", got) } + for _, key := range []string{"include_image", "force_image", "only_changes"} { + property := properties[key].(map[string]any) + if got := property["type"]; got != "boolean" { + t.Fatalf("%s type = %v, want boolean", key, got) + } + } + maxTextChars := properties["max_text_chars"].(map[string]any) + if got := maxTextChars["type"]; got != "integer" { + t.Fatalf("max_text_chars type = %v, want integer", got) + } + if got := maxTextChars["minimum"]; got != 0 { + t.Fatalf("max_text_chars minimum = %v, want 0", got) + } required := tool.InputSchema["required"].([]string) if len(required) != 1 || required[0] != "app" { t.Fatalf("required = %#v, want [app]", required) } } +func TestAppStateResultOutputControls(t *testing.T) { + maxChars := 40 + snapshot := &appSnapshot{ + App: appDescriptor{Name: "Text Editor", BundleIdentifier: "org.gnome.TextEditor", PID: 123}, + WindowTitle: "Draft", + ScreenshotPNGBase64: "image-a", + TreeLines: []string{"0123456789abcdef"}, + } + + result, current := snapshot.appStateResult(appStateOutputOptions{IncludeImage: false, MaxTextChars: &maxChars}, nil) + if len(result.Content) != 1 || result.Content[0].Type != "text" { + t.Fatalf("content = %#v, want text only", result.Content) + } + if !strings.Contains(result.Content[0].Text, "[truncated after 40 characters]") { + t.Fatalf("text was not capped: %q", result.Content[0].Text) + } + if got := len([]rune(result.Content[0].Text)); got > maxChars { + t.Fatalf("text length = %d, want at most %d", got, maxChars) + } + + imageAfterTextOnly, currentWithImage := snapshot.appStateResult(defaultAppStateOutputOptions(), ¤t) + if len(imageAfterTextOnly.Content) != 2 || imageAfterTextOnly.Content[1].Type != "image" { + t.Fatalf("image after text-only content = %#v, want text and image", imageAfterTextOnly.Content) + } + deduped, _ := snapshot.appStateResult(defaultAppStateOutputOptions(), ¤tWithImage) + if len(deduped.Content) != 1 { + t.Fatalf("deduped content = %#v, want text only", deduped.Content) + } + forced, _ := snapshot.appStateResult(appStateOutputOptions{IncludeImage: true, ForceImage: true}, ¤tWithImage) + if len(forced.Content) != 2 || forced.Content[1].Type != "image" { + t.Fatalf("forced content = %#v, want text and image", forced.Content) + } + unchanged, _ := snapshot.appStateResult(appStateOutputOptions{IncludeImage: true, OnlyChanges: true}, ¤tWithImage) + if unchanged.Content[0].Text != appStateNoChangeMessage { + t.Fatalf("unchanged text = %q, want no-change message", unchanged.Content[0].Text) + } +} + +func TestMaxTextCharsRejectsInvalidValues(t *testing.T) { + for _, value := range []any{json.Number("0"), json.Number("10"), float64(2), 3, int64(4)} { + if _, ok := optionalNonNegativeInt(map[string]any{"max_text_chars": value}, "max_text_chars"); !ok { + t.Fatalf("max_text_chars %v should be valid", value) + } + } + for _, value := range []any{json.Number("-1"), json.Number("1.5"), json.Number("9223372036854775808"), float64(2.2), true, "10"} { + if _, ok := optionalNonNegativeInt(map[string]any{"max_text_chars": value}, "max_text_chars"); ok { + t.Fatalf("max_text_chars %v should be invalid", value) + } + } +} + func TestParseSnapshotArgsSupportsShowFullText(t *testing.T) { app, showFullText, err := parseSnapshotArgs([]string{"--show-full-text", "Text Editor"}) if err != nil { diff --git a/apps/OpenComputerUseLinux/runtime.py b/apps/OpenComputerUseLinux/runtime.py index a147234..39772af 100644 --- a/apps/OpenComputerUseLinux/runtime.py +++ b/apps/OpenComputerUseLinux/runtime.py @@ -691,6 +691,61 @@ def insert_text(root, text): ) +def text_selection_match(value, target, prefix=None, suffix=None): + if not target: + raise RuntimeError("Missing required argument: text") + prefix = prefix or "" + suffix = suffix or "" + matches = [] + start = 0 + while start <= len(value): + index = value.find(target, start) + if index < 0: + break + end = index + len(target) + before = value[:index] + after = value[end:] + if (not prefix or before.endswith(prefix)) and (not suffix or after.startswith(suffix)): + matches.append((index, end)) + start = index + max(len(target), 1) + if not matches: + raise RuntimeError("Target text not found in element") + if len(matches) > 1: + raise RuntimeError("Target text is ambiguous; provide prefix or suffix") + return matches[0] + + +def select_text_node(node, target, prefix=None, suffix=None, selection="text"): + if node is None: + raise RuntimeError("unknown element_index") + mode = (selection or "text").strip() or "text" + if mode not in {"text", "cursor_before", "cursor_after"}: + raise RuntimeError("Invalid selection: " + str(selection)) + if not bool(safe(node.is_text, False)): + raise RuntimeError("Cannot select text for an element that does not expose text") + text_iface = safe(node.get_text_iface) + if text_iface is None: + raise RuntimeError("Cannot select text for an element that does not expose text") + count = int(safe(lambda: Atspi.Text.get_character_count(text_iface), 0) or 0) + value = str(safe(lambda: Atspi.Text.get_text(text_iface, 0, count), "") or "") + start, end = text_selection_match(value, str(target), prefix, suffix) + if mode == "cursor_before": + end = start + elif mode == "cursor_after": + start = end + if mode == "text": + selections = safe(lambda: Atspi.Text.get_text_selections(text_iface), []) or [] + if selections: + if bool(safe(lambda: Atspi.Text.set_selection(text_iface, 0, start, end), False)): + return + if bool(safe(lambda: Atspi.Text.add_selection(text_iface, start, end), False)): + return + raise RuntimeError("Cannot select text for an element that does not expose a settable text selection range") + if bool(safe(lambda: Atspi.Text.set_caret_offset(text_iface, start), False)): + return + raise RuntimeError("Cannot place the text cursor for this element") + + def set_element_value(node, value): if node is not None and bool(safe(node.is_editable_text, False)): editable = safe(node.get_editable_text_iface) @@ -776,6 +831,14 @@ def perform_operation(operation): invoke_secondary_action(element, operation.get("action", "")) elif tool == "scroll": scroll_element(operation.get("direction", "down"), operation.get("pages", 1)) + elif tool == "select_text": + select_text_node( + element, + operation.get("text", ""), + operation.get("prefix"), + operation.get("suffix"), + operation.get("selection", "text"), + ) elif tool == "drag": from_x, from_y = screen_point( bounds, None, operation.get("from_x"), operation.get("from_y") diff --git a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift index c221879..0ab1c3e 100644 --- a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift +++ b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift @@ -196,8 +196,8 @@ enum OpenComputerUseSmokeSuite { try client.initialize() let tools = try client.listTools() - guard tools.count == 9 else { - throw SmokeError.message("Expected 9 tools, got \(tools.count)") + guard tools.count == 10 else { + throw SmokeError.message("Expected 10 tools, got \(tools.count)") } print("1. list_apps") @@ -258,7 +258,17 @@ enum OpenComputerUseSmokeSuite { ]) try expect(state.contains("set-value-ok-typed"), "type_text should append literal text to the focused text field") - print("8. press_key") + print("8. select_text") + state = try client.callTool("select_text", arguments: [ + "app": appName, + "element_index": index["fixture-input"]!.index, + "text": "value-ok", + "prefix": "set-", + "suffix": "-typed", + ]) + try expect(state.contains("Selected text: [value-ok]"), "select_text should select the requested substring") + + print("9. press_key") index = parseElementIndex(state) let keyCaptureFrame = index["fixture-key-capture"]!.frame _ = try client.callTool("click", arguments: [ @@ -272,7 +282,7 @@ enum OpenComputerUseSmokeSuite { ]) try expect(state.contains("Last key: Return"), "press_key should update the key capture view") - print("9. scroll") + print("10. scroll") index = parseElementIndex(state) let scrollIndex = index["fixture-scroll-view"]!.index state = try client.callTool("scroll", arguments: [ @@ -283,7 +293,7 @@ enum OpenComputerUseSmokeSuite { ]) try expect(!state.contains("Scroll offset: 0"), "scroll should move the scroll view") - print("10. drag") + print("11. drag") index = parseElementIndex(state) let dragFrame = index["fixture-drag-pad"]!.frame state = try client.callTool("drag", arguments: [ diff --git a/apps/OpenComputerUseWindows/main.go b/apps/OpenComputerUseWindows/main.go index fbff762..2e3c9b3 100644 --- a/apps/OpenComputerUseWindows/main.go +++ b/apps/OpenComputerUseWindows/main.go @@ -22,7 +22,9 @@ var version = "0.1.54" //go:embed runtime.ps1 var windowsRuntimeScript string -const serverInstructions = "Computer Use tools let you interact with Windows apps by performing UI actions.\n\nBegin by calling `get_app_state` every turn you want to use Computer Use to get the latest state before acting. The available tools are list_apps, get_app_state, click, perform_secondary_action, scroll, drag, type_text, press_key, and set_value.\n\nPrefer element-targeted interactions over coordinate clicks when an index for the targeted element is available. Windows actions use UI Automation patterns first and fall back to window messages when an app does not expose the needed pattern. The Windows runtime does not auto-launch apps, perform SetFocus, or use UIA text fallback by default, so background-capable actions do not intentionally steal the user's foreground focus." +const serverInstructions = "Computer Use tools let you interact with Windows apps by performing UI actions.\n\nBegin by calling `get_app_state` every turn you want to use Computer Use to get the latest state before acting. The available tools are list_apps, get_app_state, click, perform_secondary_action, scroll, select_text, drag, type_text, press_key, and set_value.\n\nPrefer element-targeted interactions over coordinate clicks when an index for the targeted element is available. Windows actions use UI Automation patterns first and fall back to window messages when an app does not expose the needed pattern. The Windows runtime does not auto-launch apps, perform SetFocus, use UIA text fallback, or perform UIA text selection by default, so background-capable actions do not intentionally steal the user's foreground focus." + +const appStateNoChangeMessage = "There has been no change" type toolDefinition struct { Name string `json:"name"` @@ -47,6 +49,27 @@ func textResult(text string, isError bool) toolCallResult { return toolCallResult{Content: []contentItem{{Type: "text", Text: text}}, IsError: isError} } +type appStateOutputOptions struct { + IncludeImage bool + ForceImage bool + MaxTextChars *int + OnlyChanges bool +} + +type appStateOutputCache struct { + RenderedText string + ScreenshotPNGBase64 string + ImageIncluded bool +} + +func (c appStateOutputCache) sameSnapshot(other appStateOutputCache) bool { + return c.RenderedText == other.RenderedText && c.ScreenshotPNGBase64 == other.ScreenshotPNGBase64 +} + +func defaultAppStateOutputOptions() appStateOutputOptions { + return appStateOutputOptions{IncludeImage: true} +} + type appDescriptor struct { Name string `json:"name"` BundleIdentifier string `json:"bundleIdentifier,omitempty"` @@ -116,17 +139,49 @@ func (s *appSnapshot) renderedText() string { } func (s *appSnapshot) result() toolCallResult { + result, _ := s.appStateResult(defaultAppStateOutputOptions(), nil) + return result +} + +func (s *appSnapshot) appStateResult(options appStateOutputOptions, previous *appStateOutputCache) (toolCallResult, appStateOutputCache) { + renderedText := s.renderedText() + screenshot := "" + if s != nil { + screenshot = s.ScreenshotPNGBase64 + } + current := appStateOutputCache{RenderedText: renderedText, ScreenshotPNGBase64: screenshot} + if options.OnlyChanges && previous != nil && previous.sameSnapshot(current) { + return textResult(appStateNoChangeMessage, false), current + } + result := toolCallResult{ - Content: []contentItem{{Type: "text", Text: s.renderedText()}}, + Content: []contentItem{{Type: "text", Text: limitedAppStateText(renderedText, options.MaxTextChars)}}, } - if s != nil && s.ScreenshotPNGBase64 != "" { + if options.IncludeImage && screenshot != "" && (options.ForceImage || previous == nil || !previous.ImageIncluded || previous.ScreenshotPNGBase64 != screenshot) { result.Content = append(result.Content, contentItem{ Type: "image", - Data: s.ScreenshotPNGBase64, + Data: screenshot, MimeType: "image/png", }) + current.ImageIncluded = true } - return result + return result, current +} + +func limitedAppStateText(text string, maxChars *int) string { + if maxChars == nil || *maxChars == 0 { + return text + } + runes := []rune(text) + if len(runes) <= *maxChars { + return text + } + suffix := fmt.Sprintf("\n\n[truncated after %d characters]", *maxChars) + suffixRunes := []rune(suffix) + if *maxChars <= len(suffixRunes) { + return string(runes[:*maxChars]) + } + return string(runes[:*maxChars-len(suffixRunes)]) + suffix } type psRequest struct { @@ -145,6 +200,9 @@ type psRequest struct { Direction string `json:"direction,omitempty"` Pages float64 `json:"pages,omitempty"` Text string `json:"text,omitempty"` + Prefix string `json:"prefix,omitempty"` + Suffix string `json:"suffix,omitempty"` + Selection string `json:"selection,omitempty"` Key string `json:"key,omitempty"` Value string `json:"value,omitempty"` WindowBounds *frame `json:"windowBounds,omitempty"` @@ -159,11 +217,12 @@ type psResponse struct { } type service struct { - snapshots map[string]*appSnapshot + snapshots map[string]*appSnapshot + appStateOutputs map[string]appStateOutputCache } func newService() *service { - return &service{snapshots: map[string]*appSnapshot{}} + return &service{snapshots: map[string]*appSnapshot{}, appStateOutputs: map[string]appStateOutputCache{}} } func (s *service) callTool(name string, args map[string]any) toolCallResult { @@ -171,7 +230,11 @@ func (s *service) callTool(name string, args map[string]any) toolCallResult { case "list_apps": return s.listApps() case "get_app_state": - return s.getAppState(requiredString(args, "app"), optionalBool(args, "show_full_text")) + options, errText := appStateOutputOptionsFromArgs(args) + if errText != "" { + return textResult(errText, true) + } + return s.getAppState(requiredString(args, "app"), optionalBool(args, "show_full_text"), options) case "click": return s.click( requiredString(args, "app"), @@ -194,6 +257,15 @@ func (s *service) callTool(name string, args map[string]any) toolCallResult { requiredElementIndex(args), floatValue(optionalFloat(args, "pages"), 1), ) + case "select_text": + return s.selectText( + requiredString(args, "app"), + requiredElementIndex(args), + requiredString(args, "text"), + optionalString(args, "prefix"), + optionalString(args, "suffix"), + defaultString(optionalString(args, "selection"), "text"), + ) case "drag": return s.drag( requiredString(args, "app"), @@ -227,7 +299,7 @@ func (s *service) listApps() toolCallResult { return textResult(response.Text, false) } -func (s *service) getAppState(app string, showFullText bool) toolCallResult { +func (s *service) getAppState(app string, showFullText bool, outputOptions appStateOutputOptions) toolCallResult { if app == "" { return textResult("Missing required argument: app", true) } @@ -235,7 +307,13 @@ func (s *service) getAppState(app string, showFullText bool) toolCallResult { if result.IsError { return result } - return snapshot.result() + keys := appStateCacheKeys(app, snapshot) + previous := s.previousAppStateOutput(keys) + result, current := snapshot.appStateResult(outputOptions, previous) + for _, key := range keys { + s.appStateOutputs[key] = current + } + return result } func (s *service) click(app, elementIndex string, x, y *float64, clickCount int, mouseButton string) toolCallResult { @@ -314,6 +392,33 @@ func (s *service) scroll(app, direction, elementIndex string, pages float64) too return s.actionResult(app, psRequest{Tool: "scroll", App: app, Element: record, Direction: normalized, Pages: pages}) } +func (s *service) selectText(app, elementIndex, text, prefix, suffix, selection string) toolCallResult { + if app == "" { + return textResult("Missing required argument: app", true) + } + if elementIndex == "" { + return textResult("Missing required argument: element_index", true) + } + if text == "" { + return textResult("Missing required argument: text", true) + } + if selection != "text" && selection != "cursor_before" && selection != "cursor_after" { + return textResult("Invalid selection: "+selection, true) + } + if !envFlagEnabled("OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_SELECTION") { + return textResult("UIA TextPattern selection is disabled by default because it may bring the target app to the foreground; set OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_SELECTION=1 to enable it.", true) + } + snapshot := s.currentSnapshot(app) + if snapshot == nil { + return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true) + } + record, err := lookupElement(snapshot, elementIndex) + if err != nil { + return textResult(err.Error(), true) + } + return s.actionResult(app, psRequest{Tool: "select_text", App: app, Element: record, Text: text, Prefix: prefix, Suffix: suffix, Selection: selection}) +} + func (s *service) drag(app string, fromX, fromY, toX, toY *float64) toolCallResult { if app == "" { return textResult("Missing required argument: app", true) @@ -550,6 +655,97 @@ func optionalBool(args map[string]any, key string) bool { return value } +func appStateOutputOptionsFromArgs(args map[string]any) (appStateOutputOptions, string) { + options := defaultAppStateOutputOptions() + if value, ok := args["include_image"].(bool); ok { + options.IncludeImage = value + } + if value, ok := args["force_image"].(bool); ok { + options.ForceImage = value + } + if value, ok := args["only_changes"].(bool); ok { + options.OnlyChanges = value + } + maxTextChars, ok := optionalNonNegativeInt(args, "max_text_chars") + if !ok { + return options, "max_text_chars must be a non-negative integer" + } + options.MaxTextChars = maxTextChars + return options, "" +} + +func optionalNonNegativeInt(args map[string]any, key string) (*int, bool) { + value, exists := args[key] + if !exists || value == nil { + return nil, true + } + + switch value := value.(type) { + case json.Number: + integer, err := value.Int64() + if err != nil || integer < 0 || int64(int(integer)) != integer { + return nil, false + } + result := int(integer) + return &result, true + case float64: + return nonNegativeIntFloat(value) + case int: + if value < 0 { + return nil, false + } + result := value + return &result, true + case int64: + if value < 0 || int64(int(value)) != value { + return nil, false + } + result := int(value) + return &result, true + } + + return nil, false +} + +func nonNegativeIntFloat(value float64) (*int, bool) { + if math.IsNaN(value) || math.IsInf(value, 0) || math.Trunc(value) != value || value < 0 || value > float64(int(^uint(0)>>1)) { + return nil, false + } + result := int(value) + return &result, true +} + +func appStateCacheKeys(query string, snapshot *appSnapshot) []string { + seen := map[string]bool{} + keys := []string{} + rawKeys := []string{query} + if snapshot != nil { + rawKeys = append(rawKeys, snapshot.App.Name, snapshot.App.BundleIdentifier) + } + for _, key := range rawKeys { + normalized := strings.ToLower(strings.TrimSpace(key)) + if normalized != "" && !seen[normalized] { + seen[normalized] = true + keys = append(keys, normalized) + } + } + return keys +} + +func (s *service) previousAppStateOutput(keys []string) *appStateOutputCache { + for _, key := range keys { + if previous, ok := s.appStateOutputs[key]; ok { + return &previous + } + } + return nil +} + +func envFlagEnabled(name string) bool { + value := strings.TrimSpace(os.Getenv(name)) + return value == "1" || strings.EqualFold(value, "true") || strings.EqualFold(value, "yes") || strings.EqualFold(value, "on") +} + func intValue(value *float64, fallback int) int { if value == nil { return fallback @@ -605,6 +801,10 @@ func toolDefinitions() []toolDefinition { InputSchema: objectSchema(map[string]any{ "app": stringProperty("App name or bundle identifier"), "show_full_text": booleanProperty("Return full accessibility text without the default 500 character truncation. Defaults to false."), + "include_image": booleanProperty("Return a screenshot image block when one is available. Defaults to true."), + "force_image": booleanProperty("Return the screenshot even when it matches the previous app state for this app. Defaults to false."), + "max_text_chars": integerProperty("Maximum characters to return from the rendered accessibility text. Set 0 for no post-render cap.", 0), + "only_changes": booleanProperty("Return only a no-change message when the rendered state and screenshot match the previous get_app_state result for this app. Defaults to false."), }, []string{"app"}), }, { @@ -643,6 +843,19 @@ func toolDefinitions() []toolDefinition { "pages": numberProperty("Number of pages to scroll. Fractional values are supported. Defaults to 1"), }, []string{"app", "element_index", "direction"}), }, + { + Name: "select_text", + Description: "Select text inside a text element, or place the text cursor before or after it. Provide text exactly as it appears in the accessibility tree, including any Markdown formatting. If the text is not unique, provide surrounding prefix or suffix text to disambiguate it.", + Annotations: defaultAnnotations(), + InputSchema: objectSchema(map[string]any{ + "app": stringProperty("App name or bundle identifier"), + "element_index": stringProperty("Text element identifier"), + "text": stringProperty("Target text as shown in the accessibility tree"), + "prefix": stringProperty("Optional text immediately before the target, used to disambiguate repeated matches"), + "suffix": stringProperty("Optional text immediately after the target, used to disambiguate repeated matches"), + "selection": enumStringProperty("Whether to select the text or place the cursor before or after it. Defaults to text.", []string{"text", "cursor_before", "cursor_after"}), + }, []string{"app", "element_index", "text"}), + }, { Name: "set_value", Description: "Set the value of a settable accessibility element. This tool is part of plugin `Computer Use`.", @@ -703,8 +916,12 @@ func numberProperty(description string) map[string]any { return map[string]any{"type": "number", "description": description} } -func integerProperty(description string) map[string]any { - return map[string]any{"type": "integer", "description": description} +func integerProperty(description string, minimum ...int) map[string]any { + property := map[string]any{"type": "integer", "description": description} + if len(minimum) > 0 { + property["minimum"] = minimum[0] + } + return property } func main() { diff --git a/apps/OpenComputerUseWindows/main_test.go b/apps/OpenComputerUseWindows/main_test.go index a2adcf3..61fe1b3 100644 --- a/apps/OpenComputerUseWindows/main_test.go +++ b/apps/OpenComputerUseWindows/main_test.go @@ -8,24 +8,105 @@ import ( ) func TestToolDefinitionCount(t *testing.T) { - if got := len(toolDefinitions()); got != 9 { - t.Fatalf("toolDefinitions() count = %d, want 9", got) + if got := len(toolDefinitions()); got != 10 { + t.Fatalf("toolDefinitions() count = %d, want 10", got) } } -func TestGetAppStateSchemaIncludesShowFullText(t *testing.T) { +func TestSelectTextSchemaMatchesCodexSurface(t *testing.T) { + tool := findToolDefinition(t, "select_text") + properties := tool.InputSchema["properties"].(map[string]any) + selection := properties["selection"].(map[string]any) + required := tool.InputSchema["required"].([]string) + + if tool.Description != "Select text inside a text element, or place the text cursor before or after it. Provide text exactly as it appears in the accessibility tree, including any Markdown formatting. If the text is not unique, provide surrounding prefix or suffix text to disambiguate it." { + t.Fatalf("unexpected select_text description: %q", tool.Description) + } + if strings.Join(required, ",") != "app,element_index,text" { + t.Fatalf("required = %#v, want [app element_index text]", required) + } + if strings.Join(selection["enum"].([]string), ",") != "text,cursor_before,cursor_after" { + t.Fatalf("selection enum = %#v", selection["enum"]) + } +} + +func TestGetAppStateSchemaIncludesOutputControls(t *testing.T) { tool := findToolDefinition(t, "get_app_state") properties := tool.InputSchema["properties"].(map[string]any) showFullText := properties["show_full_text"].(map[string]any) if got := showFullText["type"]; got != "boolean" { t.Fatalf("show_full_text type = %v, want boolean", got) } + for _, key := range []string{"include_image", "force_image", "only_changes"} { + property := properties[key].(map[string]any) + if got := property["type"]; got != "boolean" { + t.Fatalf("%s type = %v, want boolean", key, got) + } + } + maxTextChars := properties["max_text_chars"].(map[string]any) + if got := maxTextChars["type"]; got != "integer" { + t.Fatalf("max_text_chars type = %v, want integer", got) + } + if got := maxTextChars["minimum"]; got != 0 { + t.Fatalf("max_text_chars minimum = %v, want 0", got) + } required := tool.InputSchema["required"].([]string) if len(required) != 1 || required[0] != "app" { t.Fatalf("required = %#v, want [app]", required) } } +func TestAppStateResultOutputControls(t *testing.T) { + maxChars := 40 + snapshot := &appSnapshot{ + App: appDescriptor{Name: "Notepad", BundleIdentifier: "notepad", PID: 123}, + WindowTitle: "Draft", + ScreenshotPNGBase64: "image-a", + TreeLines: []string{"0123456789abcdef"}, + } + + result, current := snapshot.appStateResult(appStateOutputOptions{IncludeImage: false, MaxTextChars: &maxChars}, nil) + if len(result.Content) != 1 || result.Content[0].Type != "text" { + t.Fatalf("content = %#v, want text only", result.Content) + } + if !strings.Contains(result.Content[0].Text, "[truncated after 40 characters]") { + t.Fatalf("text was not capped: %q", result.Content[0].Text) + } + if got := len([]rune(result.Content[0].Text)); got > maxChars { + t.Fatalf("text length = %d, want at most %d", got, maxChars) + } + + imageAfterTextOnly, currentWithImage := snapshot.appStateResult(defaultAppStateOutputOptions(), ¤t) + if len(imageAfterTextOnly.Content) != 2 || imageAfterTextOnly.Content[1].Type != "image" { + t.Fatalf("image after text-only content = %#v, want text and image", imageAfterTextOnly.Content) + } + deduped, _ := snapshot.appStateResult(defaultAppStateOutputOptions(), ¤tWithImage) + if len(deduped.Content) != 1 { + t.Fatalf("deduped content = %#v, want text only", deduped.Content) + } + forced, _ := snapshot.appStateResult(appStateOutputOptions{IncludeImage: true, ForceImage: true}, ¤tWithImage) + if len(forced.Content) != 2 || forced.Content[1].Type != "image" { + t.Fatalf("forced content = %#v, want text and image", forced.Content) + } + unchanged, _ := snapshot.appStateResult(appStateOutputOptions{IncludeImage: true, OnlyChanges: true}, ¤tWithImage) + if unchanged.Content[0].Text != appStateNoChangeMessage { + t.Fatalf("unchanged text = %q, want no-change message", unchanged.Content[0].Text) + } +} + +func TestMaxTextCharsRejectsInvalidValues(t *testing.T) { + for _, value := range []any{json.Number("0"), json.Number("10"), float64(2), 3, int64(4)} { + if _, ok := optionalNonNegativeInt(map[string]any{"max_text_chars": value}, "max_text_chars"); !ok { + t.Fatalf("max_text_chars %v should be valid", value) + } + } + for _, value := range []any{json.Number("-1"), json.Number("1.5"), json.Number("9223372036854775808"), float64(2.2), true, "10"} { + if _, ok := optionalNonNegativeInt(map[string]any{"max_text_chars": value}, "max_text_chars"); ok { + t.Fatalf("max_text_chars %v should be invalid", value) + } + } +} + func TestParseSnapshotArgsSupportsShowFullText(t *testing.T) { app, showFullText, err := parseSnapshotArgs([]string{"--show-full-text", "Notepad"}) if err != nil { @@ -131,7 +212,10 @@ func TestWindowsRuntimeForegroundActionsRequireOptIn(t *testing.T) { if !strings.Contains(windowsRuntimeScript, "OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_FALLBACK") { t.Fatal("Windows UIA text fallback must remain opt-in") } - if !strings.Contains(serverInstructions, "does not auto-launch apps, perform SetFocus, or use UIA text fallback by default") { + if !strings.Contains(windowsRuntimeScript, "OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_SELECTION") { + t.Fatal("Windows UIA text selection must remain opt-in") + } + if !strings.Contains(serverInstructions, "does not auto-launch apps, perform SetFocus, use UIA text fallback, or perform UIA text selection by default") { t.Fatal("MCP instructions must document the Windows background-focus policy") } } diff --git a/apps/OpenComputerUseWindows/runtime.ps1 b/apps/OpenComputerUseWindows/runtime.ps1 index 2a38ad3..912bc49 100644 --- a/apps/OpenComputerUseWindows/runtime.ps1 +++ b/apps/OpenComputerUseWindows/runtime.ps1 @@ -681,6 +681,75 @@ function Get-CurrentPatternOrNull($element, $pattern) { } } +function Find-TextSelectionMatch([string]$value, [string]$target, [string]$prefix, [string]$suffix) { + if ([string]::IsNullOrEmpty($target)) { + throw "Missing required argument: text" + } + + $foundRanges = @() + $start = 0 + while ($start -le $value.Length) { + $index = $value.IndexOf($target, $start, [StringComparison]::Ordinal) + if ($index -lt 0) { + break + } + + $afterStart = $index + $target.Length + $before = $value.Substring(0, $index) + $after = $value.Substring($afterStart) + $prefixMatches = [string]::IsNullOrEmpty($prefix) -or $before.EndsWith($prefix, [StringComparison]::Ordinal) + $suffixMatches = [string]::IsNullOrEmpty($suffix) -or $after.StartsWith($suffix, [StringComparison]::Ordinal) + if ($prefixMatches -and $suffixMatches) { + $foundRanges += [pscustomobject]@{ Start = $index; End = $afterStart } + } + + $start = $index + [math]::Max($target.Length, 1) + } + + if ($foundRanges.Count -eq 0) { + throw "Target text not found in element" + } + if ($foundRanges.Count -gt 1) { + throw "Target text is ambiguous; provide prefix or suffix" + } + return $foundRanges[0] +} + +function Invoke-SelectText($element, [string]$target, [string]$prefix, [string]$suffix, [string]$selection) { + if ($null -eq $element) { + throw "unknown element_index" + } + + $mode = if ([string]::IsNullOrWhiteSpace($selection)) { "text" } else { $selection.Trim() } + if (@("text", "cursor_before", "cursor_after") -notcontains $mode) { + throw "Invalid selection: $selection" + } + + $textPattern = Get-CurrentPatternOrNull $element ([Windows.Automation.TextPattern]::Pattern) + if ($null -eq $textPattern) { + throw "Cannot select text for an element that does not expose text" + } + + $document = $textPattern.DocumentRange + $value = $document.GetText(-1) + $match = Find-TextSelectionMatch $value $target $prefix $suffix + $start = [int]$match.Start + $end = [int]$match.End + if ($mode -eq "cursor_before") { + $end = $start + } elseif ($mode -eq "cursor_after") { + $start = $end + } + + $range = $document.Clone() + [void]$range.MoveEndpointByUnit([Windows.Automation.TextPatternRangeEndpoint]::Start, [Windows.Automation.TextUnit]::Character, $start) + [void]$range.MoveEndpointByUnit([Windows.Automation.TextPatternRangeEndpoint]::End, [Windows.Automation.TextUnit]::Character, -1 * ($value.Length - $end)) + if (-not (Test-EnvFlagEnabled "OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_SELECTION")) { + throw "UIA TextPattern selection is disabled by default because it may bring the target app to the foreground; set OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_SELECTION=1 to enable it." + } + $range.Select() +} + function Invoke-PreferredClick($element) { $invoke = Get-CurrentPatternOrNull $element ([Windows.Automation.InvokePattern]::Pattern) if ($null -ne $invoke) { @@ -914,6 +983,9 @@ try { Send-Scroll $hwnd $point.x $point.y $operation.direction ([double]$operation.pages) } } + "select_text" { + Invoke-SelectText $element $operation.text $operation.prefix $operation.suffix $operation.selection + } "drag" { Send-Drag $hwnd ([int][math]::Round($windowBounds.x + [double]$operation.from_x)) ([int][math]::Round($windowBounds.y + [double]$operation.from_y)) ([int][math]::Round($windowBounds.x + [double]$operation.to_x)) ([int][math]::Round($windowBounds.y + [double]$operation.to_y)) } diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 234452b..fdd296d 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,6 +1,6 @@ # 架构总览 -这个仓库当前已经从模板收敛成一个本地 `computer-use` 项目。主线仍是 Swift 实现的 macOS automation MCP server,同时新增了实验性的 Windows 和 Linux runtime,用独立 Go 二进制暴露同一组 9 个 Computer Use tools。 +这个仓库当前已经从模板收敛成一个本地 `computer-use` 项目。主线仍是 Swift 实现的 macOS automation MCP server,同时新增了实验性的 Windows 和 Linux runtime,用独立 Go 二进制暴露同一组 10 个 Computer Use tools。 ## 当前目录结构 @@ -9,7 +9,7 @@ - `apps/OpenComputerUseFixture` 本地 GUI fixture app,用来承载低风险、可预测的点击/输入/滚动/拖拽验证路径。 - `apps/OpenComputerUseSmokeSuite` - 端到端 smoke runner,会拉起 fixture 和 MCP server,并通过 JSON-RPC 真实调用 9 个 tools;同时也支持单独的 visual cursor idle smoke,用跨进程 observation file 断言等待下一次 move 时是 anchored tip + tiny rotate wobble,而不是横向漂移。 + 端到端 smoke runner,会拉起 fixture 和 MCP server,并通过 JSON-RPC 真实调用 10 个 tools;同时也支持单独的 visual cursor idle smoke,用跨进程 observation file 断言等待下一次 move 时是 anchored tip + tiny rotate wobble,而不是横向漂移。 - `apps/OpenComputerUseWindows` 实验性 Windows runtime。它不依赖 Swift 或 `.app` bundle,Go CLI/MCP 入口会嵌入 PowerShell UI Automation bridge,构建产物是 `open-computer-use.exe`,并随已有 npm 包的 `dist/windows//` bundled artifacts 分发。 - `apps/OpenComputerUseLinux` @@ -60,10 +60,10 @@ ### 3. Tool Service 层 -- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。 +- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 10 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。 - `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。 -- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。 -- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。 +- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。MCP schema 还暴露 `include_image`、`force_image`、`max_text_chars` 和 `only_changes`,让 host 可以把截图省略、重复截图去重、渲染文本后置截断和官方风格 `There has been no change` 响应下沉到 runtime,而不是只能在 host wrapper 里处理已经展开的 MCP result。 +- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` `1.0.857` 的 10 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。 - `open-computer-use call --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file ` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep ` 覆盖;遇到 `isError=true` 的 tool result 后停止。 - 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。 - 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。 @@ -77,6 +77,7 @@ - 动作型 tools 对普通 app 采用“非侵入优先,物理指针路径显式 opt-in”策略: - `perform_secondary_action` 只执行目标元素已经暴露出来的 AX action;无效 action 返回官方风格的 `... is not a valid secondary action for ...`,fixture 的 `Raise` 路径也不再为了测试去准备全局物理指针输入 - `set_value` 会先用 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 判断目标是否真的是可设置值元素,只有 settable 时才调用 `AXUIElementSetAttributeValue`;不可设置时返回官方风格的 non-settable 错误,不退到键盘输入、剪贴板或未公开的文本替换接口 + - `select_text` 使用目标文本元素暴露的原生选区能力:macOS 走 `kAXSelectedTextRangeAttribute`,Windows 走 UI Automation `TextPattern`,Linux 走 AT-SPI `Text` selection / caret API;按 `text` 加可选 `prefix` / `suffix` 做唯一匹配,支持 `cursor_before` / `cursor_after`,不使用剪贴板、全局按键或物理指针兜底。Windows UIA `TextPatternRange.Select()` 可能把目标窗口带到前台,因此默认禁用,只有设置 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_SELECTION=1` 时才执行。 - element-targeted `click` 的左键路径会先试原生列表的 `AXSelectedChildren` 选择,再试 `AXPress` / `AXConfirm` / `AXOpen` 这类真正语义化的激活动作;如果目标本身不可点,还会继续尝试其子孙 AX 元素(例如 Finder sidebar row 下面暴露 `AXOpen` 的 cell)和命中点附近的 AX hit-test 结果,最后才会退到 `postToPid` 定向鼠标事件。`AXRaise` / `kAXMainAttribute` / `kAXFocusedAttribute` 这类 activation-only fallback 只允许窗口级元素使用,避免普通静态文本或容器把“获得焦点”误报成“点击已处理”;`click_count > 1` 也会优先重复可用的 AX action - 对 renderer 合成的 summary `text` 行,`click` 不再默认点击父容器中心;这类元素使用左侧安全锚点。对普通 row/container/text 的后代候选,也会过滤右侧紧凑 hover action(例如 Lark / Electron 会话列表里的“完成”勾),避免把主行点击误操作成 side action;Electron/Lark 这类 app 的 WebArea 合成文本会优先寻找紧邻的行级 `AXPress` 祖先并静默执行,避免用物理鼠标点行。浏览器 WebArea(例如 Chrome/GitHub)不走这条 Electron-scoped 行级祖先点击优化,仍保留通用 link/container 点击路径。命中点如果只反查到覆盖整页的 Electron/WebArea 级 AX 元素,不再继续扫描整个大容器的子孙候选,避免一次行点击被远处的可点击元素截走。 - `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素 @@ -88,7 +89,7 @@ ### 4. Fixture Bridge - `OpenComputerUseFixture` 会把自己的窗口与元素状态写到临时 JSON 文件。 -- 对 fixture 的 `get_app_state` 和少量测试专用动作,会通过 `FixtureBridge` 走显式 command 通道。 +- 对 fixture 的 `get_app_state` 和少量测试专用动作,会通过 `FixtureBridge` 走显式 command 通道;`select_text` 也在这条 deterministic path 里记录选区文本,保证 smoke suite 能覆盖新 tool。 - 这个 bridge 只服务于仓库内 deterministic smoke path,不是面向真实第三方 app 的能力边界。 - 因为 SwiftPM 裸 executable 形式启动的 fixture 没有稳定的 bundle identifier,`list_apps` 会仅对 `OpenComputerUseFixture` 注入一个内部 synthetic identifier,保证 smoke suite 仍能覆盖 `list_apps`,普通第三方 app 仍按真实 bundle id 输出。 @@ -107,8 +108,8 @@ - Windows runtime 位于 `apps/OpenComputerUseWindows`,以 Go 维护 CLI、`call --calls` sequence、MCP JSON-RPC、tool schema 和进程内 snapshot cache。 - 构建入口是 `scripts/build-open-computer-use-windows.sh --arch arm64|amd64`,默认输出到 `dist/windows//open-computer-use.exe`;npm release package 会把两个 Windows artifact 内置到已有 root/alias packages,Node launcher 按 `process.platform/process.arch` 自动选择。 - Go runtime 通过 `go:embed` 带上 `runtime.ps1`,执行 tool call 时临时落盘并调用 Windows PowerShell。PowerShell bridge 使用 `System.Windows.Automation` 做 app/window/element discovery、tree rendering、UIA pattern action、ValuePattern set value 和 ScrollPattern scroll;当目标 app 不暴露对应 pattern 时,fallback 到 `PostMessage` / `SendMessage` 形式的 Win32 window message。 -- Windows runtime 默认只连接已经运行的 app,不会在 `get_app_state` 找不到进程时自动 `Start-Process`,也不会默认允许 `SetFocus` secondary action;这两条前台抢占路径分别需要 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_APP_LAUNCH=1` 和 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_FOCUS_ACTIONS=1` 显式打开。`type_text` 默认优先对可写文本控件的 child HWND 发送 `EM_SETSEL` / `EM_REPLACESEL`,不再默认走可能触发前台激活的 UIA `ValuePattern.SetValue` fallback;需要旧行为时必须设置 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_FALLBACK=1`。UIA pattern 和 Win32 message fallback 本身仍是 best-effort:很多控件可以在后台响应,但 Windows 没有一套对所有 GUI toolkit 都等价于 macOS AX 的后台键鼠输入模型。 -- 这 9 个 tool 的协议面与 macOS 主线保持一致:`list_apps`、`get_app_state`、`click`、`perform_secondary_action`、`scroll`、`drag`、`type_text`、`press_key`、`set_value`。其中 element-targeted action 会优先复用上一轮 `get_app_state` 的 runtime id / automation metadata,coordinate action 使用 screenshot/window-relative 坐标。 +- Windows runtime 默认只连接已经运行的 app,不会在 `get_app_state` 找不到进程时自动 `Start-Process`,也不会默认允许 `SetFocus` secondary action;这两条前台抢占路径分别需要 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_APP_LAUNCH=1` 和 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_FOCUS_ACTIONS=1` 显式打开。`type_text` 默认优先对可写文本控件的 child HWND 发送 `EM_SETSEL` / `EM_REPLACESEL`,不再默认走可能触发前台激活的 UIA `ValuePattern.SetValue` fallback;需要旧行为时必须设置 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_FALLBACK=1`。`select_text` 的 UIA `TextPatternRange.Select()` 同样默认禁用,需要 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_SELECTION=1`。UIA pattern 和 Win32 message fallback 本身仍是 best-effort:很多控件可以在后台响应,但 Windows 没有一套对所有 GUI toolkit 都等价于 macOS AX 的后台键鼠输入模型。 +- 这 10 个 tool 的协议面与 macOS 主线保持一致:`list_apps`、`get_app_state`、`click`、`perform_secondary_action`、`scroll`、`select_text`、`drag`、`type_text`、`press_key`、`set_value`。其中 element-targeted action 会优先复用上一轮 `get_app_state` 的 runtime id / automation metadata,coordinate action 使用 screenshot/window-relative 坐标。 - Windows UI Automation 需要运行在已登录用户的桌面 session 里。通过 SSH 作为脱离桌面的后台进程运行时,PowerShell 可以启动并返回 JSON,但系统可能不给它暴露顶层窗口;这种情况下 `list_apps` 会是空,`get_app_state` 可能返回 `appNotFound(...)`。 - 当前 Windows 侧仍是功能性第一版:没有 visual cursor overlay、没有 installer/onboarding、没有 code signing,也没有独立的 Windows smoke fixture。后续 TODO 记录在 `docs/exec-plans/active/20260422-windows-computer-use-runtime.md`。 @@ -120,7 +121,7 @@ - Linux 上最接近 macOS AX 的是 AT-SPI2/D-Bus accessibility,而不是一套统一的后台键鼠输入模型。第一版优先使用元素暴露的 AT-SPI action、EditableText 和 Value 接口;coordinate `click` / `drag` 与 `press_key` 使用 AT-SPI event synthesis fallback,在 Wayland 下只能按 best-effort 处理。 - Linux runtime 需要运行在已登录桌面用户 session 里。缺少 `XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS` 或 display 环境时,Go runtime 会在启动 Python AT-SPI bridge 前尝试从 `/run/user/` 和常见桌面进程自动发现当前用户的 session bus、display / Wayland 值;纯 SSH tty 如果找不到已登录桌面 session,可以启动二进制,但不能直接 inspect 或操作 GUI session。 - `get_app_state` 的 accessibility tree 在 GTK/GNOME app 上可能很深,Linux bridge 当前把 traversal depth 放宽到 64。截图通过 GDK root window best-effort capture;GNOME Wayland 可能返回黑图,bridge 会检测全黑采样并省略 image block。 -- 这 9 个 tool 的协议面与 macOS / Windows 保持一致:`list_apps`、`get_app_state`、`click`、`perform_secondary_action`、`scroll`、`drag`、`type_text`、`press_key`、`set_value`。其中 element-targeted action 会优先复用上一轮 `get_app_state` 的 runtime path metadata,coordinate action 使用 screenshot/window-relative 坐标。 +- 这 10 个 tool 的协议面与 macOS / Windows 保持一致:`list_apps`、`get_app_state`、`click`、`perform_secondary_action`、`scroll`、`select_text`、`drag`、`type_text`、`press_key`、`set_value`。其中 element-targeted action 会优先复用上一轮 `get_app_state` 的 runtime path metadata,coordinate action 使用 screenshot/window-relative 坐标。 - 当前 Linux 侧仍是功能性第一版:没有 visual cursor overlay、没有 installer/desktop entry,也没有独立 Linux fixture。后续 TODO 记录在 `docs/exec-plans/active/20260422-linux-computer-use-runtime.md`。 ## 关键边界 @@ -128,7 +129,7 @@ - 开源版当前不复刻官方闭源实现里的 caller signing、私有 IPC、完整 overlay choreography 和 plugin 自安装逻辑。 - 因为官方 `SkyComputerUseClient` 带有宿主侧 launch constraints,普通 stdio MCP client 在本机上可能被系统直接杀掉;如果要探测官方 bundled `computer-use`,`scripts/computer-use-cli` 的 app-server 模式现在只适合做工具清单和协议面观察。官方 `1.0.755` 的真实 tool call 还会经过 service-side sender authorization / active IPC client 追踪,外部 raw helper 即使走已签名 Codex binary,也可能返回 `Sender process is not authenticated`;需要真实使用官方工具时应走正常 Codex agent/tool 调用链,开源版则继续提供可直连的 `open-computer-use` MCP server。 - 当前权限引导已经具备可运行 app、深链、拖拽辅助,以及一版更接近官方的 accessory panel 入场动画和返回 affordance;点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑,并且在 overlay 可见期间会持续重申“排在目标 window 之上”,避免用户手动激活目标 app 后 cursor 被目标窗口重新盖住;但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。 -- screenshot 当前通过 `ScreenCaptureKit` 捕获目标窗口,并以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录;编码前会按最大尺寸和目标字节数自适应缩小,避免复杂页面的大 PNG 触发 host 侧 MCP result 降级,同时 coordinate tools 继续按实际返回的 screenshot pixel 尺寸映射坐标;单次 ScreenCaptureKit capture 会设置超时,超时后省略 image block 而不是卡住整个 `get_app_state`。 +- screenshot 当前通过 `ScreenCaptureKit` 捕获目标窗口,并以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录;编码前会按最大尺寸和目标字节数自适应缩小,避免复杂页面的大 PNG 触发 host 侧 MCP result 降级,同时 coordinate tools 继续按实际返回的 screenshot pixel 尺寸映射坐标。macOS 截图的 capture timeout、长边像素上限、PNG 字节预算和字节预算继续缩小时的最小缩放比例可分别通过 `OPEN_COMPUTER_USE_IMAGE_CAPTURE_TIMEOUT`、`OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION`、`OPEN_COMPUTER_USE_IMAGE_MAX_BYTES`、`OPEN_COMPUTER_USE_IMAGE_MIN_SCALE` 调整;单次 ScreenCaptureKit capture 超时后省略 image block,而不是卡住整个 `get_app_state`。 - 会话状态现在是进程内内存态,保存每个 app 最近一次 snapshot 和 element index 映射。 ## 主要验证路径 @@ -136,7 +137,7 @@ - 单元测试:`swift test` - standalone cursor 构建:`swift build --product StandaloneCursor` - cursor lab 构建:`swift build --product CursorMotion` -- 端到端 smoke:`./scripts/run-tool-smoke-tests.sh`(标准 9-tool smoke + visual cursor idle smoke;脚本默认以 headless 模式启动内部 fixture,避免在用户桌面弹出测试窗口) +- 端到端 smoke:`./scripts/run-tool-smoke-tests.sh`(标准 10-tool smoke + visual cursor idle smoke;脚本默认以 headless 模式启动内部 fixture,避免在用户桌面弹出测试窗口) - app 打包:`./scripts/build-open-computer-use-app.sh debug` - 权限 onboarding 端到端回归:`./scripts/run-permission-onboarding-e2e.sh`(需要当前 macOS 对被测 `open-computer-use` 已授予 Accessibility 与 Screen Recording;默认禁用 app-agent proxy 来测试当前 CLI 运行态,可用 `OPEN_COMPUTER_USE_E2E_CLI=/path/to/open-computer-use` 指定被测 CLI,或用 `OPEN_COMPUTER_USE_E2E_DISABLE_APP_AGENT_PROXY=0` 显式覆盖默认代理行为) - npm staging:`node ./scripts/npm/build-packages.mjs` diff --git a/docs/QUALITY_SCORE.md b/docs/QUALITY_SCORE.md index f5b3a47..4e566d0 100644 --- a/docs/QUALITY_SCORE.md +++ b/docs/QUALITY_SCORE.md @@ -11,10 +11,10 @@ | 区域 | 评分 | 原因 | 下一步 | | --- | --- | --- | --- | -| 产品面 | B | 已经有 Swift 本地 `computer-use` MCP server、默认 app 模式权限引导,以及一轮按官方 surface / result 行为收敛过的 9 个 tools。 | 继续收敛复杂 AX 场景下的 state rendering 细节、权限 UI 和更清晰的用户错误提示。 | -| Windows runtime | C | 已新增独立 Go `.exe`,通过 Windows UI Automation + Win32 window message 暴露同样 9 个 tools、MCP server 和 `call --calls`;默认不再自动启动 app、执行 `SetFocus`,或让 `type_text` 走可能抢前台的 UIA text fallback,并已接入 npm bundled artifact 分发,但仍是功能性第一版。 | 补交互式桌面 smoke、Windows fixture、installer/signing,以及更原生的 Go UIA 实现或更稳定的 bridge。 | -| Linux runtime | C | 已新增独立 Go binary,通过 Python GI / AT-SPI2 暴露同样 9 个 tools、MCP server 和 `call --calls`;Ubuntu GNOME VM 已跑通 `list_apps`、MCP tools list 和 Text Editor 8-tool sequence,并已接入 npm bundled artifact 分发,但截图在 GNOME Wayland 下仍只能 best-effort,coordinate input 也不是通用后台模型。 | 补 Linux fixture、可重复 smoke runner、portal/compositor screenshot 路径,以及更原生的 Go D-Bus/libatspi bridge。 | +| 产品面 | B | 已经有 Swift 本地 `computer-use` MCP server、默认 app 模式权限引导,以及一轮按官方 surface / result 行为收敛过的 10 个 tools。 | 继续收敛复杂 AX 场景下的 state rendering 细节、权限 UI 和更清晰的用户错误提示。 | +| Windows runtime | C | 已新增独立 Go `.exe`,通过 Windows UI Automation + Win32 window message 暴露同样 10 个 tools、MCP server 和 `call --calls`;默认不再自动启动 app、执行 `SetFocus`,或让 `type_text` 走可能抢前台的 UIA text fallback,并已接入 npm bundled artifact 分发,但仍是功能性第一版。 | 补交互式桌面 smoke、Windows fixture、installer/signing,以及更原生的 Go UIA 实现或更稳定的 bridge。 | +| Linux runtime | C | 已新增独立 Go binary,通过 Python GI / AT-SPI2 暴露同样 10 个 tools、MCP server 和 `call --calls`;Ubuntu GNOME VM 已跑通 `list_apps`、MCP tools list 和 Text Editor 8-tool sequence,并已接入 npm bundled artifact 分发,但截图在 GNOME Wayland 下仍只能 best-effort,coordinate input 也不是通用后台模型。 | 补 Linux fixture、可重复 smoke runner、portal/compositor screenshot 路径,以及更原生的 Go D-Bus/libatspi bridge。 | | 架构文档 | B | 顶层结构、fixture bridge、app 模式和验证路径已经落文档。 | 后续补 release artifact、code signing / notarization 和 host 集成方式。 | -| 测试 | B | `swift test` + smoke suite 已覆盖 9 个 tools 的回归,并新增了针对“前台焦点是否被抢占”的手工对比样本沉淀。 | 增加更多普通 app 的录制回归,减少只依赖 fixture 和一次性手工检查。 | +| 测试 | B | smoke suite 已覆盖 10 个 tools 的回归,并新增了针对“前台焦点是否被抢占”的手工对比样本沉淀。 | 增加更多普通 app 的录制回归,减少只依赖 fixture 和一次性手工检查。 | | 可观测性 | C | 已有 `doctor`、`snapshot`、smoke 输出,以及一组仓库内留档的官方 `computer-use` / 本仓库实现对比样本。 | 补统一日志级别、失败上下文和 release artifact 里的诊断信息,把一次性样本收敛成可重复采集流程。 | | 安全 | B | 已明确本地-only、权限边界和 fixture test bridge 的作用域,并将内置 denylist 收缩到密码管理器。 | 增加 session approval 和更清楚的敏感 app policy,避免策略长期硬编码在仓库里。 | diff --git a/docs/exec-plans/completed/20260702-select-text-codex-parity.md b/docs/exec-plans/completed/20260702-select-text-codex-parity.md new file mode 100644 index 0000000..f95b065 --- /dev/null +++ b/docs/exec-plans/completed/20260702-select-text-codex-parity.md @@ -0,0 +1,56 @@ +# `select_text` Codex parity + +## 目标 + +把官方 Codex `computer-use` `1.0.857` 新增的 `select_text` 工具补到 Open Computer Use 的 macOS、Windows、Linux runtime 中,并让 schema、dispatcher、fixture smoke、文档和历史记录同步进入 10-tool surface。 + +## 范围 + +- 包含:macOS Swift runtime、fixture bridge、smoke suite、Swift schema tests、Windows Go/PowerShell runtime、Linux Go/Python runtime、架构/skill/release note/history 文档。 +- 不包含:移除本仓库已有的 `show_full_text` 扩展、重写旧版 9-tool 历史逆向文档、发布版本号或替换用户已安装的 app bundle。 + +## 背景 + +- 相关文档:`docs/ARCHITECTURE.md`、`docs/HISTORY_GUIDE.md`、`docs/QUALITY_SCORE.md`、`skills/open-computer-use/references/usage.md`。 +- 相关代码路径:`packages/OpenComputerUseKit`、`apps/OpenComputerUseFixture`、`apps/OpenComputerUseSmokeSuite`、`apps/OpenComputerUseWindows`、`apps/OpenComputerUseLinux`。 +- 已知约束:保持非侵入优先,不用剪贴板、全局物理指针或前台抢占兜底实现文本选区。 + +## 风险 + +- 风险:不同平台文本 API 的索引单位不同,可能导致 emoji / 组合字符附近选区偏移。 +- 缓解方式:macOS 使用 `NSString` / `NSRange` 与 AX 的 UTF-16 range 语义对齐;Windows 使用 UIA `TextPatternRange` 字符端点;Linux 使用 AT-SPI `Text` offset。 +- 风险:目标文本重复时错误选中错误位置。 +- 缓解方式:要求唯一匹配,重复时提示补 `prefix` / `suffix` 消歧。 + +## 里程碑 + +1. 官方 surface 探测与 schema 收敛。 +2. macOS / fixture / smoke 实现。 +3. Windows / Linux runtime parity。 +4. 验证、文档、history 收尾。 + +## 验证方式 + +- 命令:`swift build --target OpenComputerUseKit` +- 命令:`swift build --product OpenComputerUseSmokeSuite` +- 命令:`swift build --product OpenComputerUse` +- 命令:`swift build --product OpenComputerUseFixture` +- 命令:`OPEN_COMPUTER_USE_VISUAL_CURSOR=0 .build/out/Products/Debug/OpenComputerUseSmokeSuite` +- 命令:`.build/out/Products/Debug/OpenComputerUseSmokeSuite --cursor-idle-only` +- 命令:`go test ./...` in `apps/OpenComputerUseWindows` +- 命令:`go test ./...` in `apps/OpenComputerUseLinux` +- 命令:`python3 -m py_compile apps/OpenComputerUseLinux/runtime.py` + +## 进度记录 + +- [x] 确认官方 `computer-use` `1.0.857` 暴露 10 个 tools,并记录 `select_text` schema。 +- [x] macOS tool definition、dispatcher、service 和 AX selected range 实现完成。 +- [x] Fixture bridge 和 smoke suite 增加 `select_text` 覆盖。 +- [x] Windows UI Automation `TextPattern` 和 Linux AT-SPI text selection parity 完成。 +- [x] 文档、quality score、skill usage、history 同步完成。 + +## 决策记录 + +- 2026-07-02:保留本仓库已有 `show_full_text` 扩展;官方 `1.0.857` 未继续暴露该字段,但它已是本仓库面向长文本读取的既有能力。 +- 2026-07-02:`select_text` 不使用剪贴板、全局按键或物理指针兜底;不支持原生文本 selection API 的元素直接返回能力边界错误。 +- 2026-07-02:旧 reverse-engineering 样本和早期 9-tool execution plan 保持历史语境,不批量重写;当前架构、skill、release note 和新 plan/history 记录 10-tool 状态。 diff --git a/docs/histories/2026-07/20260702-0126-configurable-image-capture.md b/docs/histories/2026-07/20260702-0126-configurable-image-capture.md new file mode 100644 index 0000000..5f023b8 --- /dev/null +++ b/docs/histories/2026-07/20260702-0126-configurable-image-capture.md @@ -0,0 +1,31 @@ +## [2026-07-02 01:26] | Task: configurable image capture + +### Execution Context +* **Agent ID**: `Amp` +* **Base Model**: `Amp deep mode, model not exposed` +* **Runtime**: `macOS SwiftPM` + +### User Query +> `get_app_state` 和动作工具返回的截图占用大量上下文,希望能按 MCP host 的预算调小图片,同时保持真实桌面操作和坐标映射安全。 + +### Changes Overview +**Scope:** `OpenComputerUseKit` macOS screenshot capture, README, architecture docs, release notes + +**Key Actions:** +- **[Config]**: macOS 截图捕获新增 `OPEN_COMPUTER_USE_IMAGE_CAPTURE_TIMEOUT`、`OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION`、`OPEN_COMPUTER_USE_IMAGE_MAX_BYTES`、`OPEN_COMPUTER_USE_IMAGE_MIN_SCALE` 四个环境变量。 +- **[Runtime]**: runtime 进程读取环境配置后,`WindowCapture` 使用同一份配置控制 ScreenCaptureKit 超时、PNG 长边上限、编码后字节预算和降采样比例下限。 +- **[Scaling]**: 修复 `maxDimension / nativeSize` 小于 `minScale` 时 resize 循环不执行、从而返回原图的问题;现在显式的长边上限会优先生效,`minScale` 只限制在该长边上限基础上按字节预算继续缩小时的下限。 +- **[Validation]**: 非法环境变量值会回退到默认值;返回 PNG 会遵守长边上限,按字节预算继续缩小时也会按实际返回 PNG 尺寸换算坐标。 +- **[Tests]**: 补充单元测试覆盖环境变量解析、非法配置回退、`minScale` 夹取行为和缩小后 PNG 尺寸到窗口坐标的换算。 +- **[Docs]**: 同步 README、中文 README、架构文档和功能发布记录。 + +### Design Intent (Why) +默认截图边界已经能避免一部分过大的 PNG,但不同 MCP host 对 image block 的上下文成本差异很大。把已有边界开放成环境变量可以保持默认兼容,同时让 Codex、Claude、Gemini 或其他 host 根据自己的预算调小图片。坐标类工具继续从返回 PNG 读取实际尺寸再映射回窗口坐标,因此降采样不会破坏 click / drag 的坐标语义。 + +### Files Modified +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift` +- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift` +- `README.md` +- `README.zh-CN.md` +- `docs/ARCHITECTURE.md` +- `docs/releases/feature-release-notes.md` diff --git a/docs/histories/2026-07/20260702-1202-select-text-codex-parity.md b/docs/histories/2026-07/20260702-1202-select-text-codex-parity.md new file mode 100644 index 0000000..a0f6181 --- /dev/null +++ b/docs/histories/2026-07/20260702-1202-select-text-codex-parity.md @@ -0,0 +1,45 @@ +## [2026-07-02 12:02] | Task: 对齐官方 `select_text` 工具 + +### Execution Context +* **Agent ID**: `Amp` +* **Base Model**: `Amp deep mode, model not exposed` +* **Runtime**: `macOS SwiftPM + Go tests` + +### User Query +> 继续上一轮 Open Computer Use 改进,特别是从 Codex 学到的 `select_text`,并补完无效 image env 诊断、测试、文档和 Codex parity。 + +### Changes Overview +**Scope:** macOS Swift runtime, fixture/smoke, Windows runtime, Linux runtime, docs/skill + +**Key Actions:** +- **[Official parity]**: 按官方 `computer-use` `1.0.857` 把 MCP tool surface 扩到 10 个 tools,新增 `select_text` schema、dispatcher 和跨平台实现。 +- **[Native selection]**: macOS 使用 `kAXSelectedTextRangeAttribute`,Linux 使用 AT-SPI `Text` selection / caret API,Windows 使用显式 opt-in 的 UI Automation `TextPattern`,不走剪贴板或全局输入兜底。 +- **[Fixture coverage]**: Fixture bridge 支持 `select_text` 并在 smoke suite 中验证 `prefix` / `suffix` 消歧后的选区结果。 +- **[Diagnostics]**: macOS image capture env 的无效非空值现在会打印忽略诊断,空值和缺省仍静默使用默认值。 +- **[Docs]**: 更新架构、质量水位、skill usage、release note,并新增 completed execution plan。 + +### Design Intent (Why) +官方 Codex `computer-use` 已从 9-tool surface 演进到 10-tool surface,`select_text` 是编辑类任务里比剪贴板、全局键盘或物理鼠标更安全的语义动作。把它落到三套 runtime 的原生 accessibility text API 上,可以保持本仓库“非侵入优先”的能力边界,同时减少 host 与官方工具面的提示词差异。 + +### Files Modified +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift` +- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift` +- `apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift` +- `apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift` +- `apps/OpenComputerUseWindows/main.go` +- `apps/OpenComputerUseWindows/runtime.ps1` +- `apps/OpenComputerUseWindows/main_test.go` +- `apps/OpenComputerUseLinux/main.go` +- `apps/OpenComputerUseLinux/runtime.py` +- `apps/OpenComputerUseLinux/main_test.go` +- `docs/ARCHITECTURE.md` +- `docs/QUALITY_SCORE.md` +- `docs/releases/feature-release-notes.md` +- `skills/open-computer-use/SKILL.md` +- `skills/open-computer-use/references/usage.md` +- `docs/exec-plans/completed/20260702-select-text-codex-parity.md` diff --git a/docs/histories/2026-07/20260702-1634-app-resolution-ranking.md b/docs/histories/2026-07/20260702-1634-app-resolution-ranking.md new file mode 100644 index 0000000..2703597 --- /dev/null +++ b/docs/histories/2026-07/20260702-1634-app-resolution-ranking.md @@ -0,0 +1,24 @@ +## [2026-07-02 16:34] | Task: Fix app-name resolution picking windowless helper processes + +### 🤖 Execution Context +* **Agent ID**: Claude Code (Amp) +* **Base Model**: Claude +* **Runtime**: macOS, swift test via Xcode-beta toolchain + +### 📥 User Query +> get_app_state with `"app":"Safari"` fails with `Apple event error -10005: cgWindowNotFound` while `"app":"com.apple.Safari"` works; fix the underlying cause instead of documenting workarounds. + +### 🛠 Changes Overview +**Scope:** packages/OpenComputerUseKit + +**Key Actions:** +- **[Root cause]**: `AppDiscovery.resolvedRunningApp` matched apps by display name OR executable name with `first(where:)` over an active-first sorted list. Bitwarden's Safari-extension helper (`exec=safari`, activationPolicy=accessory, reports `isActive=true`) matched the query "Safari" via the executable fallback and won over the real Safari, then failed window capture with `cgWindowNotFound`. +- **[Fix]**: Extracted a pure `bestResolutionIndex(of:matching:)` ranking that prefers regular-activation-policy apps and display-name matches: regular name > regular executable > accessory name > accessory executable. +- **[Tests]**: Added three `bestResolutionIndex` unit tests including the Bitwarden/Safari regression case. + +### 🧠 Design Intent (Why) +Name queries should never resolve to background helper processes when a regular app with a matching display name is running; ranking by activation policy and match kind fixes the class of bug instead of the single symptom. + +### 📁 Files Modified +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift` +- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift` diff --git a/docs/histories/2026-07/20260702-1837-compact-app-state-output.md b/docs/histories/2026-07/20260702-1837-compact-app-state-output.md new file mode 100644 index 0000000..5d2bd63 --- /dev/null +++ b/docs/histories/2026-07/20260702-1837-compact-app-state-output.md @@ -0,0 +1,32 @@ +## [2026-07-02 18:37] | Task: Compact app state output + +### Execution Context +* **Agent ID**: `Amp` +* **Base Model**: `Amp deep mode, model not exposed` +* **Runtime**: `macOS SwiftPM + Linux/Windows Go tests` + +### User Query +> Apply Codex-derived context-saving behavior to the open computer use fork, build it, and replace the local installed package with a release build. + +### Changes Overview +**Scope:** `OpenComputerUseKit` macOS MCP runtime, Linux/Windows runtimes, and release docs + +**Key Actions:** +- **[get_app_state options]**: Added `include_image`, `force_image`, `max_text_chars`, and `only_changes` to the macOS, Linux, and Windows `get_app_state` schema and dispatcher. +- **[Runtime output policy]**: Added per-app output cache so repeated screenshots can be omitted, app-state text can be capped after rendering, and unchanged state can return `There has been no change`. +- **[Regression coverage]**: Added unit tests for the output policy, integer argument parsing, and schema fields. +- **[Docs]**: Updated architecture, release notes, and plugin package metadata to record the host-facing behavior. + +### Design Intent (Why) +Installed Codex artifacts expose state-context concepts such as `screenshotNeededForContext`, `There has been no change`, and screenshot / AX text presence flags. Moving the compacting controls into `open-computer-use` keeps the MCP result smaller before host wrappers see it, while preserving explicit knobs for clients that still need full images or uncapped text. + +### Files Modified +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift` +- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift` +- `apps/OpenComputerUseLinux/main.go` +- `apps/OpenComputerUseWindows/main.go` +- `docs/ARCHITECTURE.md` +- `docs/releases/feature-release-notes.md` +- `plugins/open-computer-use/.codex-plugin/plugin.json` diff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.md index 1ba16be..5d3d6a5 100644 --- a/docs/releases/feature-release-notes.md +++ b/docs/releases/feature-release-notes.md @@ -1,5 +1,13 @@ # 功能发布记录 +## 2026-07 + +| 日期 | 功能域 | 用户价值 | 变更摘要 | +| --- | --- | --- | --- | +| 2026-07-02 | `get_app_state` 上下文输出控制 | MCP host 可以直接让 `open-computer-use` 省略截图、去重重复截图、限制 state 文本长度,减少复杂 UI 反复进上下文的成本。 | 按 Codex bundled app 中观察到的 `screenshotNeededForContext`、`There has been no change`、`had_screenshot` / `had_ax_text` 等状态线索,为 macOS、Linux 和 Windows `get_app_state` 增加 `include_image`、`force_image`、`max_text_chars` 和 `only_changes` 参数;默认保留可用 state,并把精确去重/截断放在 runtime 层。 | +| 2026-07-02 | 文本选区工具对齐 | Agent 可以像官方 Codex `computer-use` 一样在文本元素内选中指定文本,或把光标放到匹配文本前后,适合编辑长文本时做局部替换或插入。 | 按官方 `computer-use` `1.0.857` 新增 `select_text` 工具;macOS 使用 Accessibility selected text range,Linux 使用 AT-SPI text selection / caret API,Windows 使用显式 opt-in 的 UI Automation `TextPattern`,并补 fixture smoke、schema 测试和跨平台 runtime 定义。 | +| 2026-07-02 | macOS 截图上下文控制 | MCP host 可以按自己的上下文预算调小 `get_app_state` 和 action tool 返回的截图,降低复杂窗口反复返回大 PNG 对 agent 上下文的压力。 | macOS 截图捕获新增 `OPEN_COMPUTER_USE_IMAGE_CAPTURE_TIMEOUT`、`OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION`、`OPEN_COMPUTER_USE_IMAGE_MAX_BYTES`、`OPEN_COMPUTER_USE_IMAGE_MIN_SCALE` 配置;默认行为保持不变,并确保较小的 `OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION` 仍作为返回 PNG 的长边上限生效。 | + ## 2026-06 | 日期 | 功能域 | 用户价值 | 变更摘要 | diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift index 8b7b6d9..89009f3 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift @@ -39,16 +39,110 @@ enum SnapshotMode { let accessibilityTreeMaxNodeCount = 1200 let accessibilityTreeMaxDepth = 64 -let screenshotCaptureTimeout: TimeInterval = 5 -let screenshotResultMaxPNGBytes = 900_000 -let screenshotResultMaxDimension: CGFloat = 1280 -let screenshotResultMinScale: CGFloat = 0.25 let snapshotTextDefaultCharacterLimit = 500 private let windowVisibilityRecoveryDelay: TimeInterval = 0.7 private let axWebAreaRole = "AXWebArea" private let axContentsAttribute = "AXContents" private let axVisibleChildrenAttribute = "AXVisibleChildren" +struct ImageCaptureConfig: Sendable, Equatable { + let captureTimeout: TimeInterval + let maxPNGBytes: Int + let maxDimension: Int + let minScale: CGFloat + + static let defaults = ImageCaptureConfig( + captureTimeout: 5, + maxPNGBytes: 900_000, + maxDimension: 1280, + minScale: 0.25 + ) + + static let current = ImageCaptureConfig.fromEnvironment() + + static func fromEnvironment(_ environment: [String: String] = ProcessInfo.processInfo.environment) -> ImageCaptureConfig { + let minScale = imageConfigValue( + name: "OPEN_COMPUTER_USE_IMAGE_MIN_SCALE", + raw: environment["OPEN_COMPUTER_USE_IMAGE_MIN_SCALE"], + defaultValue: Double(defaults.minScale), + parse: parseImageConfigUnitInterval + ) + + return ImageCaptureConfig( + captureTimeout: imageConfigValue( + name: "OPEN_COMPUTER_USE_IMAGE_CAPTURE_TIMEOUT", + raw: environment["OPEN_COMPUTER_USE_IMAGE_CAPTURE_TIMEOUT"], + defaultValue: defaults.captureTimeout, + parse: parseImageConfigPositiveDouble + ), + maxPNGBytes: imageConfigValue( + name: "OPEN_COMPUTER_USE_IMAGE_MAX_BYTES", + raw: environment["OPEN_COMPUTER_USE_IMAGE_MAX_BYTES"], + defaultValue: defaults.maxPNGBytes, + parse: parseImageConfigPositiveInt + ), + maxDimension: imageConfigValue( + name: "OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION", + raw: environment["OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION"], + defaultValue: defaults.maxDimension, + parse: parseImageConfigPositiveInt + ), + minScale: CGFloat(minScale) + ) + } +} + +private func imageConfigValue(name: String, raw: String?, defaultValue: T, parse: (String?) -> T?) -> T { + if let value = parse(raw) { + return value + } + + if raw?.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty == false { + fputs("[open-computer-use] Ignoring invalid \(name); using default.\n", stderr) + } + + return defaultValue +} + +private func parseImageConfigPositiveInt(_ raw: String?) -> Int? { + guard let raw = raw?.trimmingCharacters(in: .whitespacesAndNewlines), + !raw.isEmpty, + let value = Int(raw), + value > 0 + else { + return nil + } + + return value +} + +private func parseImageConfigPositiveDouble(_ raw: String?) -> Double? { + guard let raw = raw?.trimmingCharacters(in: .whitespacesAndNewlines), + !raw.isEmpty, + let value = Double(raw), + value > 0, + value.isFinite + else { + return nil + } + + return value +} + +private func parseImageConfigUnitInterval(_ raw: String?) -> Double? { + guard let raw = raw?.trimmingCharacters(in: .whitespacesAndNewlines), + !raw.isEmpty, + let value = Double(raw), + value > 0, + value <= 1, + value.isFinite + else { + return nil + } + + return value +} + public struct AppSnapshot { public let app: RunningAppDescriptor public let windowTitle: String? @@ -343,7 +437,7 @@ enum SnapshotBuilder { treeLines: lines, focusedSummary: focusedSummary, focusedElement: nil, - selectedText: nil, + selectedText: state.selectedText, elements: records ) } @@ -362,6 +456,7 @@ private struct WindowCapture { let layer: Int let bounds: CGRect let image: CGImage? + let imageConfig: ImageCaptureConfig static func resolve(for pid: pid_t, titleHint: String?) -> WindowCapture? { guard let infoList = CGWindowListCopyWindowInfo([.optionOnScreenOnly], kCGNullWindowID) as? [[String: Any]] else { @@ -396,13 +491,14 @@ private struct WindowCapture { return nil } - let image = captureImage(windowID: best.windowID, bounds: best.bounds) + let imageConfig = ImageCaptureConfig.current + let image = captureImage(windowID: best.windowID, bounds: best.bounds, config: imageConfig) - return WindowCapture(windowID: best.windowID, layer: best.layer, bounds: best.bounds, image: image) + return WindowCapture(windowID: best.windowID, layer: best.layer, bounds: best.bounds, image: image, imageConfig: imageConfig) } - private static func captureImage(windowID: CGWindowID, bounds: CGRect) -> CGImage? { - try? BlockingAsyncBridge.run(timeout: screenshotCaptureTimeout) { + private static func captureImage(windowID: CGWindowID, bounds: CGRect, config: ImageCaptureConfig) -> CGImage? { + try? BlockingAsyncBridge.run(timeout: config.captureTimeout) { let shareableContent = try await SCShareableContent.current guard let window = shareableContent.windows.first(where: { $0.windowID == windowID }) else { return nil @@ -433,7 +529,12 @@ private struct WindowCapture { return nil } - return boundedScreenshotPNGData(for: image) + return boundedScreenshotPNGData( + for: image, + maxBytes: imageConfig.maxPNGBytes, + maxDimension: CGFloat(imageConfig.maxDimension), + minScale: imageConfig.minScale + ) } } @@ -480,24 +581,34 @@ func preferredWindowCaptureCandidate(_ candidates: [WindowCaptureCandidate], tit func boundedScreenshotPNGData( for image: CGImage, - maxBytes: Int = screenshotResultMaxPNGBytes, - maxDimension: CGFloat = screenshotResultMaxDimension, - minScale: CGFloat = screenshotResultMinScale + maxBytes: Int = ImageCaptureConfig.defaults.maxPNGBytes, + maxDimension: CGFloat = CGFloat(ImageCaptureConfig.defaults.maxDimension), + minScale: CGFloat = ImageCaptureConfig.defaults.minScale ) -> Data? { - guard image.width > 0, image.height > 0, maxBytes > 0 else { + guard image.width > 0, + image.height > 0, + maxBytes > 0, + maxDimension >= 1, + maxDimension.isFinite, + maxDimension.rounded(.down) == maxDimension, + minScale > 0, + minScale <= 1, + minScale.isFinite + else { return nil } let original = pngData(for: image) let largestDimension = CGFloat(max(image.width, image.height)) var scale = min(1, maxDimension / largestDimension) + let byteBudgetFloorScale = scale * minScale if scale >= 1, let original, original.count <= maxBytes { return original } var best = original - while scale >= minScale { + while scale >= byteBudgetFloorScale { guard let resized = resizedCGImage(image, scale: scale), let data = pngData(for: resized) else { @@ -521,8 +632,8 @@ private func pngData(for image: CGImage) -> Data? { } private func resizedCGImage(_ image: CGImage, scale: CGFloat) -> CGImage? { - let width = max(1, Int((CGFloat(image.width) * scale).rounded())) - let height = max(1, Int((CGFloat(image.height) * scale).rounded())) + let width = max(1, Int((CGFloat(image.width) * scale).rounded(.down))) + let height = max(1, Int((CGFloat(image.height) * scale).rounded(.down))) let colorSpace = CGColorSpaceCreateDeviceRGB() let bitmapInfo = CGImageAlphaInfo.premultipliedLast.rawValue diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift index ba97d72..88a533a 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift @@ -166,6 +166,12 @@ enum AppDiscovery { throw ComputerUseError.appNotFound(normalizedQuery) } + struct ResolutionCandidate { + let name: String + let executableName: String? + let isRegularApp: Bool + } + private static func resolvedRunningApp(in descriptors: [RunningAppDescriptor], matching query: String) -> RunningAppDescriptor? { if isBundleIdentifierQuery(query) { return descriptors.first(where: { descriptor in @@ -173,14 +179,56 @@ enum AppDiscovery { }) } - return descriptors.first(where: { descriptor in - guard !AppSafetyPolicy.isBlocked(bundleIdentifier: descriptor.bundleIdentifier) else { - return false + let allowed = descriptors.filter { descriptor in + !AppSafetyPolicy.isBlocked(bundleIdentifier: descriptor.bundleIdentifier) + } + let candidates = allowed.map { descriptor in + ResolutionCandidate( + name: descriptor.name, + executableName: descriptor.runningApplication.executableURL?.deletingPathExtension().lastPathComponent, + isRegularApp: descriptor.runningApplication.activationPolicy == .regular + ) + } + + guard let index = bestResolutionIndex(of: candidates, matching: query) else { + return nil + } + + return allowed[index] + } + + static func bestResolutionIndex(of candidates: [ResolutionCandidate], matching query: String) -> Int? { + var best: (rank: Int, index: Int)? + + for (index, candidate) in candidates.enumerated() { + let rank: Int? + if candidate.isRegularApp, candidate.name.caseInsensitiveCompare(query) == .orderedSame { + rank = 0 + } else if candidate.isRegularApp, candidate.executableName?.caseInsensitiveCompare(query) == .orderedSame { + rank = 1 + } else if candidate.name.caseInsensitiveCompare(query) == .orderedSame { + rank = 2 + } else if candidate.executableName?.caseInsensitiveCompare(query) == .orderedSame { + rank = 3 + } else { + rank = nil + } + + guard let rank else { + continue + } + + if let current = best, rank >= current.rank { + continue } - return descriptor.name.caseInsensitiveCompare(query) == .orderedSame - || descriptor.runningApplication.executableURL?.deletingPathExtension().lastPathComponent.caseInsensitiveCompare(query) == .orderedSame - }) + best = (rank, index) + if rank == 0 { + break + } + } + + return best?.index } private static func userFacingRunningApps() -> [RunningAppDescriptor] { diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift index c892773..584fe3f 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift @@ -170,6 +170,189 @@ func invalidSecondaryActionErrorMessage(action: String, elementIndex: Int) -> St "\(action) is not a valid secondary action for \(elementIndex)" } +public struct AppStateOutputOptions: Sendable, Equatable { + public let includeImage: Bool + public let forceImage: Bool + public let maxTextChars: Int? + public let onlyChanges: Bool + + public static let defaults = AppStateOutputOptions() + + public init( + includeImage: Bool = true, + forceImage: Bool = false, + maxTextChars: Int? = nil, + onlyChanges: Bool = false + ) { + self.includeImage = includeImage + self.forceImage = forceImage + self.maxTextChars = maxTextChars + self.onlyChanges = onlyChanges + } +} + +let appStateNoChangeMessage = "There has been no change" + +struct AppStateOutputCache: Equatable { + let renderedText: String + let screenshotPNGData: Data? + let imageIncluded: Bool + + init(renderedText: String, screenshotPNGData: Data?, imageIncluded: Bool = false) { + self.renderedText = renderedText + self.screenshotPNGData = screenshotPNGData + self.imageIncluded = imageIncluded + } + + func sameSnapshot(as other: AppStateOutputCache) -> Bool { + renderedText == other.renderedText && screenshotPNGData == other.screenshotPNGData + } +} + +func limitedAppStateText(_ text: String, maxCharacters: Int?) -> String { + guard let maxCharacters, maxCharacters > 0, text.count > maxCharacters else { + return text + } + + let suffix = "\n\n[truncated after \(maxCharacters) characters]" + guard maxCharacters > suffix.count else { + return String(text.prefix(maxCharacters)) + } + + return String(text.prefix(maxCharacters - suffix.count)) + suffix +} + +func appStateToolResult( + renderedText: String, + screenshotPNGData: Data?, + previous: AppStateOutputCache?, + options: AppStateOutputOptions +) -> ToolCallResult { + appStateToolResultWithCache( + renderedText: renderedText, + screenshotPNGData: screenshotPNGData, + previous: previous, + options: options + ).result +} + +func appStateToolResultWithCache( + renderedText: String, + screenshotPNGData: Data?, + previous: AppStateOutputCache?, + options: AppStateOutputOptions +) -> (result: ToolCallResult, cache: AppStateOutputCache) { + let current = AppStateOutputCache(renderedText: renderedText, screenshotPNGData: screenshotPNGData) + if options.onlyChanges, previous?.sameSnapshot(as: current) == true { + return (ToolCallResult.text(appStateNoChangeMessage), current) + } + + var content = [ToolResultContentItem.text(limitedAppStateText(renderedText, maxCharacters: options.maxTextChars))] + var imageIncluded = false + if options.includeImage, let screenshotPNGData { + let screenshotChanged = previous?.screenshotPNGData != screenshotPNGData + if options.forceImage || previous == nil || previous?.imageIncluded == false || screenshotChanged { + content.append(.pngImage(screenshotPNGData)) + imageIncluded = true + } + } + + return ( + ToolCallResult(content: content), + AppStateOutputCache(renderedText: renderedText, screenshotPNGData: screenshotPNGData, imageIncluded: imageIncluded) + ) +} + +enum TextSelectionMode: String { + case text + case cursorBefore = "cursor_before" + case cursorAfter = "cursor_after" + + init(toolValue: String) throws { + let normalized = toolValue.trimmingCharacters(in: .whitespacesAndNewlines) + if normalized.isEmpty { + self = .text + return + } + + guard let mode = TextSelectionMode(rawValue: normalized) else { + throw ComputerUseError.message("Invalid selection: \(toolValue)") + } + + self = mode + } + + func selectedRange(for match: NSRange) -> NSRange { + switch self { + case .text: + return match + case .cursorBefore: + return NSRange(location: match.location, length: 0) + case .cursorAfter: + return NSRange(location: match.location + match.length, length: 0) + } + } +} + +func textSelectionMatch(in value: String, text: String, prefix: String?, suffix: String?) throws -> NSRange { + let target = text as NSString + guard target.length > 0 else { + throw ComputerUseError.missingArgument("text") + } + + let haystack = value as NSString + let prefix = prefix ?? "" + let suffix = suffix ?? "" + let prefixLength = (prefix as NSString).length + let suffixLength = (suffix as NSString).length + var matches: [NSRange] = [] + var searchRange = NSRange(location: 0, length: haystack.length) + + while searchRange.length >= target.length { + let found = haystack.range(of: text, options: [], range: searchRange) + if found.location == NSNotFound { + break + } + + let afterStart = found.location + found.length + let prefixMatches = prefixLength == 0 || ( + found.location >= prefixLength + && haystack.compare( + prefix, + options: [], + range: NSRange(location: found.location - prefixLength, length: prefixLength) + ) == .orderedSame + ) + let suffixMatches = suffixLength == 0 || ( + haystack.length - afterStart >= suffixLength + && haystack.compare( + suffix, + options: [], + range: NSRange(location: afterStart, length: suffixLength) + ) == .orderedSame + ) + if prefixMatches && suffixMatches { + matches.append(found) + } + + let nextLocation = found.location + max(found.length, 1) + if nextLocation > haystack.length { + break + } + searchRange = NSRange(location: nextLocation, length: haystack.length - nextLocation) + } + + if matches.isEmpty { + throw ComputerUseError.message("Target text not found in element") + } + + if matches.count > 1 { + throw ComputerUseError.message("Target text is ambiguous; provide prefix or suffix") + } + + return matches[0] +} + func localClickActionPoints(frame: CGRect, isSyntheticText: Bool) -> [CGPoint] { let center = CGPoint(x: frame.midX, y: frame.midY) let leading = CGPoint( @@ -349,6 +532,7 @@ func shouldPreferContainingWebRowAXClickCandidate( public final class ComputerUseService { private var snapshotsByApp: [String: AppSnapshot] = [:] + private var appStateOutputsByApp: [String: AppStateOutputCache] = [:] public init() {} @@ -360,8 +544,17 @@ public final class ComputerUseService { ) } - public func getAppState(app query: String, showFullText: Bool = false) throws -> ToolCallResult { - snapshotResult(for: try refreshSnapshot(for: query, showFullText: showFullText), style: .fullState) + public func getAppState( + app query: String, + showFullText: Bool = false, + outputOptions: AppStateOutputOptions = .defaults + ) throws -> ToolCallResult { + appStateResult( + for: try refreshSnapshot(for: query, showFullText: showFullText), + query: query, + style: .fullState, + options: outputOptions + ) } public func click(app query: String, elementIndex: String?, x: Double?, y: Double?, clickCount: Int, mouseButton: String) throws -> ToolCallResult { @@ -550,6 +743,42 @@ public final class ComputerUseService { return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult) } + public func selectText(app query: String, elementIndex: String, text: String, prefix: String?, suffix: String?, selection: String) throws -> ToolCallResult { + let mode = try TextSelectionMode(toolValue: selection) + let snapshot = try currentSnapshot(for: query) + let record = try lookupElement(snapshot: snapshot, index: elementIndex) + + if snapshot.mode == .fixture { + guard let identifier = record.identifier else { + throw ComputerUseError.invalidArguments("fixture select_text requires an identifier-backed element") + } + + try FixtureBridge.post(FixtureCommand( + kind: "select_text", + identifier: identifier, + value: text, + prefix: prefix, + suffix: suffix, + selection: mode.rawValue + )) + Thread.sleep(forTimeInterval: 0.15) + return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult) + } + + guard let element = record.element else { + throw ComputerUseError.stateUnavailable("element \(elementIndex) has no backing accessibility object") + } + + guard let value = selectableTextValue(for: element) else { + throw ComputerUseError.message("Cannot select text for an element that does not expose text") + } + + let match = try textSelectionMatch(in: value, text: text, prefix: prefix, suffix: suffix) + try setSelectedTextRange(mode.selectedRange(for: match), on: element) + Thread.sleep(forTimeInterval: 0.1) + return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult) + } + public func drag(app query: String, fromX: Double, fromY: Double, toX: Double, toY: Double) throws -> ToolCallResult { let snapshot = try currentSnapshot(for: query) if snapshot.mode == .fixture { @@ -696,6 +925,39 @@ public final class ComputerUseService { invalidSecondaryActionErrorMessage(action: action, elementIndex: record.index) } + private func appStateResult( + for snapshot: AppSnapshot, + query: String, + style: SnapshotTextStyle, + options: AppStateOutputOptions + ) -> ToolCallResult { + let renderedText = snapshot.renderedText(style: style) + let keys = appStateCacheKeys(query: query, snapshot: snapshot) + let previous = keys.lazy.compactMap { self.appStateOutputsByApp[$0] }.first + let output = appStateToolResultWithCache( + renderedText: renderedText, + screenshotPNGData: snapshot.screenshotPNGData, + previous: previous, + options: options + ) + for key in keys { + appStateOutputsByApp[key] = output.cache + } + return output.result + } + + private func appStateCacheKeys(query: String, snapshot: AppSnapshot) -> [String] { + var seen = Set() + var keys: [String] = [] + for key in [query, snapshot.app.name, snapshot.app.bundleIdentifier ?? ""] { + let normalized = key.trimmingCharacters(in: .whitespacesAndNewlines).lowercased() + if !normalized.isEmpty, seen.insert(normalized).inserted { + keys.append(normalized) + } + } + return keys + } + private func performPreferredClick(on record: ElementRecord, button: MouseButtonKind, clickCount: Int) throws -> Bool { guard let element = record.element else { return false @@ -1232,6 +1494,40 @@ public final class ComputerUseService { } } + private func selectableTextValue(for element: AXUIElement) -> String? { + for attribute in [ + kAXValueAttribute as String, + kAXTitleAttribute as String, + kAXDescriptionAttribute as String, + kAXHelpAttribute as String, + ] { + guard let value = stringValue(of: element, attribute: attribute), !value.isEmpty else { + continue + } + + return value + } + + return nil + } + + private func setSelectedTextRange(_ range: NSRange, on element: AXUIElement) throws { + var cfRange = CFRange(location: range.location, length: range.length) + guard let value = AXValueCreate(.cfRange, &cfRange) else { + throw ComputerUseError.message("Failed to encode selected text range") + } + + let result = AXUIElementSetAttributeValue(element, kAXSelectedTextRangeAttribute as CFString, value) + switch result { + case .success: + return + case .failure, .attributeUnsupported, .actionUnsupported, .cannotComplete, .noValue, .invalidUIElement, .illegalArgument: + throw ComputerUseError.message("Cannot select text for an element that does not expose a settable text selection range") + default: + throw ComputerUseError.message("AXUIElementSetAttributeValue(\(kAXSelectedTextRangeAttribute)) failed with \(result.rawValue)") + } + } + private func canTypeTextUsingKeyboardFallback(in snapshot: AppSnapshot) throws -> Bool { guard let element = snapshot.focusedElement else { return false diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift index 892128f..dc8c966 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift @@ -36,6 +36,38 @@ private func normalizedElementIndexNumber(_ value: Double) -> String? { return String(Int(value)) } +func normalizedNonNegativeIntegerArgument(_ value: Any?) -> Int? { + if let integer = value as? Int { + return integer >= 0 ? integer : nil + } + + if let number = value as? NSNumber { + if CFGetTypeID(number as CFTypeRef) == CFBooleanGetTypeID() { + return nil + } + + return normalizedNonNegativeIntegerNumber(number.doubleValue) + } + + if let double = value as? Double { + return normalizedNonNegativeIntegerNumber(double) + } + + return nil +} + +private func normalizedNonNegativeIntegerNumber(_ value: Double) -> Int? { + guard value.isFinite, value.rounded(.towardZero) == value else { + return nil + } + + guard value >= 0, value <= Double(Int.max) else { + return nil + } + + return Int(value) +} + public final class ComputerUseToolDispatcher { private let service: ComputerUseService @@ -50,7 +82,13 @@ public final class ComputerUseToolDispatcher { case "get_app_state": return try service.getAppState( app: requireString("app", in: arguments), - showFullText: optionalBool("show_full_text", in: arguments) ?? false + showFullText: optionalBool("show_full_text", in: arguments) ?? false, + outputOptions: AppStateOutputOptions( + includeImage: optionalBool("include_image", in: arguments) ?? true, + forceImage: optionalBool("force_image", in: arguments) ?? false, + maxTextChars: try optionalNonNegativeInt("max_text_chars", in: arguments), + onlyChanges: optionalBool("only_changes", in: arguments) ?? false + ) ) case "click": return try service.click( @@ -74,6 +112,15 @@ public final class ComputerUseToolDispatcher { elementIndex: requireElementIndex(in: arguments), pages: optionalDouble("pages", in: arguments) ?? 1 ) + case "select_text": + return try service.selectText( + app: requireString("app", in: arguments), + elementIndex: requireElementIndex(in: arguments), + text: requireString("text", in: arguments), + prefix: optionalString("prefix", in: arguments), + suffix: optionalString("suffix", in: arguments), + selection: optionalString("selection", in: arguments) ?? "text" + ) case "drag": return try service.drag( app: requireString("app", in: arguments), @@ -178,6 +225,18 @@ public final class ComputerUseToolDispatcher { return nil } + + private func optionalNonNegativeInt(_ key: String, in arguments: [String: Any]) throws -> Int? { + guard arguments.keys.contains(key) else { + return nil + } + + guard let value = normalizedNonNegativeIntegerArgument(arguments[key]) else { + throw ComputerUseError.invalidArguments("\(key) must be a non-negative integer") + } + + return value + } } public struct OpenComputerUseCallSpec { diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift index 14cff5d..17ae39f 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift @@ -43,12 +43,14 @@ public struct FixtureAppState: Codable, Sendable { public let windowTitle: String public let windowBounds: FixtureRect public let focusedIdentifier: String? + public let selectedText: String? public let elements: [FixtureElementState] - public init(windowTitle: String, windowBounds: FixtureRect, focusedIdentifier: String?, elements: [FixtureElementState]) { + public init(windowTitle: String, windowBounds: FixtureRect, focusedIdentifier: String?, selectedText: String? = nil, elements: [FixtureElementState]) { self.windowTitle = windowTitle self.windowBounds = windowBounds self.focusedIdentifier = focusedIdentifier + self.selectedText = selectedText self.elements = elements } } @@ -63,6 +65,9 @@ public struct FixtureCommand: Codable, Sendable { public let toY: Double? public let direction: String? public let pages: Double? + public let prefix: String? + public let suffix: String? + public let selection: String? public init( kind: String, @@ -73,7 +78,10 @@ public struct FixtureCommand: Codable, Sendable { toX: Double? = nil, toY: Double? = nil, direction: String? = nil, - pages: Double? = nil + pages: Double? = nil, + prefix: String? = nil, + suffix: String? = nil, + selection: String? = nil ) { self.kind = kind self.identifier = identifier @@ -84,6 +92,9 @@ public struct FixtureCommand: Codable, Sendable { self.toY = toY self.direction = direction self.pages = pages + self.prefix = prefix + self.suffix = suffix + self.selection = selection } } diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift index 575bc71..6d5250c 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift @@ -7,7 +7,7 @@ Some apps might have a separate dedicated plugin or skill. You may want to use t Begin by calling `get_app_state` every turn you want to use Computer Use to get the latest state before acting. Codex will automatically stop the session after each assistant turn, so this step is required before interacting with apps in a new assistant turn. -The available tools are list_apps, get_app_state, click, perform_secondary_action, scroll, drag, type_text, press_key, and set_value. If any of these are not available in your environment, use tool_search to surface one before calling any Computer Use action tools. +The available tools are list_apps, get_app_state, click, perform_secondary_action, scroll, select_text, drag, type_text, press_key, and set_value. If any of these are not available in your environment, use tool_search to surface one before calling any Computer Use action tools. Computer Use tools allow you to use the user's apps in the background, so while you're using an app, the user can continue to use other apps on their computer. Avoid doing anything that would disrupt the user's active session, such as overwriting the contents of their clipboard, unless they asked you to! diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift index e27e14f..947bf97 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift @@ -72,6 +72,10 @@ public enum ToolDefinitions { properties: [ "app": stringProperty(description: "App name or bundle identifier"), "show_full_text": booleanProperty(description: "Return full accessibility text without the default 500 character truncation. Defaults to false."), + "include_image": booleanProperty(description: "Return a screenshot image block when one is available. Defaults to true."), + "force_image": booleanProperty(description: "Return the screenshot even when it matches the previous app state for this app. Defaults to false."), + "max_text_chars": integerProperty(description: "Maximum characters to return from the rendered accessibility text. Set 0 for no post-render cap.", minimum: 0), + "only_changes": booleanProperty(description: "Return only a no-change message when the rendered state and screenshot match the previous get_app_state result for this app. Defaults to false."), ], required: ["app"] ) @@ -121,6 +125,25 @@ public enum ToolDefinitions { required: ["app", "element_index", "direction"] ) ), + ToolDefinition( + name: "select_text", + description: "Select text inside a text element, or place the text cursor before or after it. Provide text exactly as it appears in the accessibility tree, including any Markdown formatting. If the text is not unique, provide surrounding prefix or suffix text to disambiguate it.", + annotations: defaultAnnotations(), + inputSchema: objectSchema( + properties: [ + "app": stringProperty(description: "App name or bundle identifier"), + "element_index": stringProperty(description: "Text element identifier"), + "text": stringProperty(description: "Target text as shown in the accessibility tree"), + "prefix": stringProperty(description: "Optional text immediately before the target, used to disambiguate repeated matches"), + "suffix": stringProperty(description: "Optional text immediately after the target, used to disambiguate repeated matches"), + "selection": stringProperty( + description: "Whether to select the text or place the cursor before or after it. Defaults to text.", + enumValues: ["text", "cursor_before", "cursor_after"] + ), + ], + required: ["app", "element_index", "text"] + ) + ), ToolDefinition( name: "set_value", description: "Set the value of a settable accessibility element. This tool is part of plugin `Computer Use`.", @@ -199,11 +222,17 @@ private func booleanProperty(description: String) -> [String: Any] { ] } -private func integerProperty(description: String) -> [String: Any] { - [ +private func integerProperty(description: String, minimum: Int? = nil) -> [String: Any] { + var property: [String: Any] = [ "type": "integer", "description": description, ] + + if let minimum { + property["minimum"] = minimum + } + + return property } private func numberProperty(description: String) -> [String: Any] { diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift index 4691fc9..37b3b35 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift @@ -175,8 +175,364 @@ final class OpenComputerUseKitTests: XCTestCase { XCTAssertEqual(size.height, 24) } + func testImageCaptureConfigReadsValidEnvironmentOverrides() { + let config = ImageCaptureConfig.fromEnvironment([ + "OPEN_COMPUTER_USE_IMAGE_CAPTURE_TIMEOUT": " 2.5 ", + "OPEN_COMPUTER_USE_IMAGE_MAX_BYTES": " 120000 ", + "OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION": " 640 ", + "OPEN_COMPUTER_USE_IMAGE_MIN_SCALE": "0.1", + ]) + + XCTAssertEqual(config.captureTimeout, 2.5) + XCTAssertEqual(config.maxPNGBytes, 120_000) + XCTAssertEqual(config.maxDimension, 640) + XCTAssertEqual(config.minScale, 0.1) + } + + func testImageCaptureConfigFallsBackForInvalidEnvironmentOverrides() { + let config = ImageCaptureConfig.fromEnvironment([ + "OPEN_COMPUTER_USE_IMAGE_CAPTURE_TIMEOUT": "0", + "OPEN_COMPUTER_USE_IMAGE_MAX_BYTES": "nope", + "OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION": "0", + "OPEN_COMPUTER_USE_IMAGE_MIN_SCALE": "0", + ]) + + XCTAssertEqual(config, ImageCaptureConfig.defaults) + } + + func testImageCaptureConfigFallsBackForBlankAndNonFiniteOverrides() { + let config = ImageCaptureConfig.fromEnvironment([ + "OPEN_COMPUTER_USE_IMAGE_CAPTURE_TIMEOUT": "nan", + "OPEN_COMPUTER_USE_IMAGE_MAX_BYTES": " ", + "OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION": "inf", + "OPEN_COMPUTER_USE_IMAGE_MIN_SCALE": "nan", + ]) + + XCTAssertEqual(config, ImageCaptureConfig.defaults) + } + + func testImageCaptureConfigAcceptsUnitMinScaleBoundary() { + let config = ImageCaptureConfig.fromEnvironment([ + "OPEN_COMPUTER_USE_IMAGE_MIN_SCALE": "1", + ]) + + XCTAssertEqual(config.minScale, 1) + } + + func testImageCaptureConfigFallsBackForMinScaleAboveUnitBoundary() { + let config = ImageCaptureConfig.fromEnvironment([ + "OPEN_COMPUTER_USE_IMAGE_CAPTURE_TIMEOUT": "-1", + "OPEN_COMPUTER_USE_IMAGE_MIN_SCALE": "1.01", + ]) + + XCTAssertEqual(config.captureTimeout, ImageCaptureConfig.defaults.captureTimeout) + XCTAssertEqual(config.minScale, ImageCaptureConfig.defaults.minScale) + } + + func testImageCaptureConfigFallsBackForNegativeMinScale() { + let config = ImageCaptureConfig.fromEnvironment([ + "OPEN_COMPUTER_USE_IMAGE_MIN_SCALE": "-0.1", + ]) + + XCTAssertEqual(config.minScale, ImageCaptureConfig.defaults.minScale) + } + + func testImageCaptureConfigFallsBackForFractionalDimensionAndIntegerOverflow() { + let config = ImageCaptureConfig.fromEnvironment([ + "OPEN_COMPUTER_USE_IMAGE_MAX_BYTES": "92233720368547758070", + "OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION": "80.6", + ]) + + XCTAssertEqual(config.maxPNGBytes, ImageCaptureConfig.defaults.maxPNGBytes) + XCTAssertEqual(config.maxDimension, ImageCaptureConfig.defaults.maxDimension) + } + + func testImageCaptureConfigFallsBackForUnicodeDigits() { + let config = ImageCaptureConfig.fromEnvironment([ + "OPEN_COMPUTER_USE_IMAGE_MAX_BYTES": "123", + "OPEN_COMPUTER_USE_IMAGE_MAX_DIMENSION": "480", + ]) + + XCTAssertEqual(config.maxPNGBytes, ImageCaptureConfig.defaults.maxPNGBytes) + XCTAssertEqual(config.maxDimension, ImageCaptureConfig.defaults.maxDimension) + } + + func testBoundedScreenshotPNGDataHonorsMaxDimensionBelowMinScale() throws { + let image = try makeNoisyTestImage(width: 800, height: 600) + let data = try XCTUnwrap(boundedScreenshotPNGData( + for: image, + maxBytes: 1_000_000, + maxDimension: 80, + minScale: 0.25 + )) + let size = try imageSize(in: data) + + XCTAssertEqual(size.width, 80) + XCTAssertEqual(size.height, 60) + } + + func testBoundedScreenshotPNGDataRejectsNonPositiveMaxDimension() throws { + let image = try makeNoisyTestImage(width: 800, height: 600) + + XCTAssertNil(boundedScreenshotPNGData( + for: image, + maxBytes: 1_000_000, + maxDimension: 0, + minScale: 0.05 + )) + XCTAssertNil(boundedScreenshotPNGData( + for: image, + maxBytes: 1_000_000, + maxDimension: -1, + minScale: 0.05 + )) + } + + func testBoundedScreenshotPNGDataRejectsFractionalMaxDimension() throws { + let image = try makeNoisyTestImage(width: 800, height: 600) + + XCTAssertNil(boundedScreenshotPNGData( + for: image, + maxBytes: 1_000_000, + maxDimension: 80.6, + minScale: 0.05 + )) + } + + func testBoundedScreenshotPNGDataHonorsMaxDimensionOneBoundary() throws { + let image = try makeNoisyTestImage(width: 800, height: 600) + let data = try XCTUnwrap(boundedScreenshotPNGData( + for: image, + maxBytes: 1_000_000, + maxDimension: 1, + minScale: 0.05 + )) + let size = try imageSize(in: data) + + XCTAssertEqual(size.width, 1) + XCTAssertEqual(size.height, 1) + } + + func testBoundedScreenshotByteBudgetRetriesBelowDimensionCapWhenMinScaleAllows() throws { + let image = try makeNoisyTestImage(width: 800, height: 600) + let data = try XCTUnwrap(boundedScreenshotPNGData( + for: image, + maxBytes: 1, + maxDimension: 80, + minScale: 0.25 + )) + let size = try imageSize(in: data) + let point = screenshotPixelToWindowPoint( + CGPoint(x: CGFloat(size.width) / 2, y: CGFloat(size.height) / 2), + screenshotPixelSize: CGSize(width: size.width, height: size.height), + windowBounds: CGRect(x: 0, y: 0, width: 800, height: 600) + ) + + XCTAssertLessThan(max(size.width, size.height), 80) + XCTAssertGreaterThanOrEqual(max(size.width, size.height), 20) + XCTAssertEqual(point.x, 400, accuracy: 0.0001) + XCTAssertEqual(point.y, 300, accuracy: 0.0001) + } + + func testBoundedScreenshotPNGDataRejectsInvalidMinScale() throws { + let image = try makeNoisyTestImage(width: 800, height: 600) + + XCTAssertNil(boundedScreenshotPNGData( + for: image, + maxBytes: 1_000_000, + maxDimension: 80, + minScale: 1.01 + )) + } + + func testBoundedScreenshotPNGDataRejectsNonFiniteMinScale() throws { + let image = try makeNoisyTestImage(width: 800, height: 600) + + XCTAssertNil(boundedScreenshotPNGData( + for: image, + maxBytes: 1_000_000, + maxDimension: 80, + minScale: .nan + )) + } + + func testBoundedScreenshotDimensionsMapBackToWindowCoordinatesBelowMinScale() throws { + let image = try makeNoisyTestImage(width: 800, height: 600) + let data = try XCTUnwrap(boundedScreenshotPNGData( + for: image, + maxBytes: 1_000_000, + maxDimension: 80, + minScale: 0.25 + )) + let size = try imageSize(in: data) + let point = screenshotPixelToWindowPoint( + CGPoint(x: CGFloat(size.width) / 2, y: CGFloat(size.height) / 2), + screenshotPixelSize: CGSize(width: size.width, height: size.height), + windowBounds: CGRect(x: 0, y: 0, width: 800, height: 600) + ) + + XCTAssertEqual(point.x, 400, accuracy: 0.0001) + XCTAssertEqual(point.y, 300, accuracy: 0.0001) + } + + func testBoundedScreenshotByteBudgetRetryCoordinatesUseReturnedDimensions() throws { + let image = try makeNoisyTestImage(width: 800, height: 600) + let data = try XCTUnwrap(boundedScreenshotPNGData( + for: image, + maxBytes: 50_000, + maxDimension: 800, + minScale: 0.05 + )) + let size = try imageSize(in: data) + let point = screenshotPixelToWindowPoint( + CGPoint(x: CGFloat(size.width) / 2, y: CGFloat(size.height) / 2), + screenshotPixelSize: CGSize(width: size.width, height: size.height), + windowBounds: CGRect(x: 0, y: 0, width: 800, height: 600) + ) + + XCTAssertLessThanOrEqual(data.count, 50_000) + XCTAssertLessThan(max(size.width, size.height), 800) + XCTAssertEqual(point.x, 400, accuracy: 0.0001) + XCTAssertEqual(point.y, 300, accuracy: 0.0001) + } + func testToolDefinitionCount() { - XCTAssertEqual(ToolDefinitions.all.count, 9) + XCTAssertEqual(ToolDefinitions.all.count, 10) + } + + func testAppStateToolResultCanOmitImageAndCapText() { + let result = appStateToolResult( + renderedText: String(repeating: "x", count: 120), + screenshotPNGData: Data([1, 2, 3]), + previous: nil, + options: AppStateOutputOptions(includeImage: false, maxTextChars: 80) + ) + + XCTAssertEqual(result.content.count, 1) + let text = result.primaryText ?? "" + XCTAssertLessThanOrEqual(text.count, 80) + XCTAssertTrue(text.contains("[truncated after 80 characters]")) + } + + func testAppStateToolResultTreatsZeroTextCapAsUnlimited() { + let text = String(repeating: "x", count: 120) + let result = appStateToolResult( + renderedText: text, + screenshotPNGData: nil, + previous: nil, + options: AppStateOutputOptions(maxTextChars: 0) + ) + + XCTAssertEqual(result.primaryText, text) + } + + func testLimitedAppStateTextDoesNotTruncateAtExactCap() { + XCTAssertEqual(limitedAppStateText("abc", maxCharacters: 3), "abc") + } + + func testAppStateToolResultDeduplicatesScreenshotUnlessForced() { + let image = Data([1, 2, 3]) + let previous = AppStateOutputCache(renderedText: "old", screenshotPNGData: image, imageIncluded: true) + let deduped = appStateToolResult( + renderedText: "new", + screenshotPNGData: image, + previous: previous, + options: .defaults + ) + let forced = appStateToolResult( + renderedText: "new", + screenshotPNGData: image, + previous: previous, + options: AppStateOutputOptions(forceImage: true) + ) + + XCTAssertEqual(contentTypes(in: deduped), ["text"]) + XCTAssertEqual(contentTypes(in: forced), ["text", "image"]) + } + + func testAppStateToolResultIncludesImageAfterTextOnlyResponse() { + let image = Data([1, 2, 3]) + let previous = AppStateOutputCache(renderedText: "old", screenshotPNGData: image, imageIncluded: false) + let result = appStateToolResult( + renderedText: "new", + screenshotPNGData: image, + previous: previous, + options: .defaults + ) + + XCTAssertEqual(contentTypes(in: result), ["text", "image"]) + } + + func testAppStateToolResultReturnsNoChangeWhenRequested() { + let image = Data([1, 2, 3]) + let previous = AppStateOutputCache(renderedText: "same", screenshotPNGData: image, imageIncluded: true) + let result = appStateToolResult( + renderedText: "same", + screenshotPNGData: image, + previous: previous, + options: AppStateOutputOptions(onlyChanges: true) + ) + + XCTAssertEqual(result.primaryText, appStateNoChangeMessage) + XCTAssertEqual(contentTypes(in: result), ["text"]) + } + + func testAppStateToolResultDoesNotReturnNoChangeWhenOnlyScreenshotMatches() { + let image = Data([1, 2, 3]) + let previous = AppStateOutputCache(renderedText: "old", screenshotPNGData: image, imageIncluded: true) + let result = appStateToolResult( + renderedText: "new", + screenshotPNGData: image, + previous: previous, + options: AppStateOutputOptions(onlyChanges: true) + ) + + XCTAssertEqual(result.primaryText, "new") + XCTAssertEqual(contentTypes(in: result), ["text"]) + } + + func testNonNegativeIntegerArgumentAcceptsJSONNumbers() throws { + let arguments = try readOpenComputerUseToolArguments( + json: #"{"app":"TextEdit","max_text_chars":20000}"#, + file: nil + ) + + XCTAssertEqual(normalizedNonNegativeIntegerArgument(arguments["max_text_chars"]), 20_000) + XCTAssertEqual(normalizedNonNegativeIntegerArgument(0), 0) + XCTAssertNil(normalizedNonNegativeIntegerArgument(-1)) + XCTAssertNil(normalizedNonNegativeIntegerArgument(1.5)) + XCTAssertNil(normalizedNonNegativeIntegerArgument(true)) + } + + func testTextSelectionMatchUsesPrefixAndSuffix() throws { + let value = "first target middle target end" as NSString + let range = try textSelectionMatch( + in: value as String, + text: "target", + prefix: "middle ", + suffix: " end" + ) + let cursorRange = try TextSelectionMode(toolValue: "cursor_after").selectedRange(for: range) + + XCTAssertEqual(range.location, value.range(of: "target", options: [], range: NSRange(location: 13, length: value.length - 13)).location) + XCTAssertEqual(range.length, ("target" as NSString).length) + XCTAssertEqual(cursorRange.location, range.location + range.length) + XCTAssertEqual(cursorRange.length, 0) + } + + func testTextSelectionMatchRequiresAdjacentPrefixAndSuffix() { + XCTAssertThrowsError(try textSelectionMatch(in: "set-xvalue-ok-typed", text: "value-ok", prefix: "set-", suffix: "-typed")) { error in + XCTAssertEqual((error as? ComputerUseError)?.errorDescription, "Target text not found in element") + } + XCTAssertThrowsError(try textSelectionMatch(in: "set-value-ok-x-typed", text: "value-ok", prefix: "set-", suffix: "-typed")) { error in + XCTAssertEqual((error as? ComputerUseError)?.errorDescription, "Target text not found in element") + } + } + + func testTextSelectionMatchRejectsAmbiguousTextWithoutContext() { + XCTAssertThrowsError(try textSelectionMatch(in: "target and target", text: "target", prefix: nil, suffix: nil)) { error in + XCTAssertEqual((error as? ComputerUseError)?.errorDescription, "Target text is ambiguous; provide prefix or suffix") + } } func testReadToolArgumentsAcceptsJSONObject() throws { @@ -382,6 +738,42 @@ final class OpenComputerUseKitTests: XCTestCase { XCTAssertFalse(AppDiscovery.compareListedApps(frequent, frontmost)) } + func testBestResolutionIndexPrefersRegularAppNameMatchOverAccessoryExecutableMatch() { + let candidates = [ + AppDiscovery.ResolutionCandidate(name: "Bitwarden", executableName: "safari", isRegularApp: false), + AppDiscovery.ResolutionCandidate(name: "Safari", executableName: "Safari", isRegularApp: true), + ] + + XCTAssertEqual(AppDiscovery.bestResolutionIndex(of: candidates, matching: "Safari"), 1) + } + + func testBestResolutionIndexPrefersNameMatchOverExecutableMatchAmongRegularApps() { + let candidates = [ + AppDiscovery.ResolutionCandidate(name: "Other", executableName: "Notes", isRegularApp: true), + AppDiscovery.ResolutionCandidate(name: "Notes", executableName: "Notes", isRegularApp: true), + ] + + XCTAssertEqual(AppDiscovery.bestResolutionIndex(of: candidates, matching: "Notes"), 1) + } + + func testBestResolutionIndexMatchesCaseInsensitively() { + let candidates = [ + AppDiscovery.ResolutionCandidate(name: "Safari", executableName: "Safari", isRegularApp: true), + ] + + XCTAssertEqual(AppDiscovery.bestResolutionIndex(of: candidates, matching: "sAfArI"), 0) + } + + func testBestResolutionIndexFallsBackToAccessoryMatchesWhenNoRegularAppMatches() { + let candidates = [ + AppDiscovery.ResolutionCandidate(name: "Helper", executableName: "helper", isRegularApp: false), + AppDiscovery.ResolutionCandidate(name: "Safari", executableName: "Safari", isRegularApp: true), + ] + + XCTAssertEqual(AppDiscovery.bestResolutionIndex(of: candidates, matching: "helper"), 0) + XCTAssertNil(AppDiscovery.bestResolutionIndex(of: candidates, matching: "missing")) + } + func testPreferredPermissionAppBundleURLPrefersInstalledCopyOverTransientRunningCopy() { let installed = URL(fileURLWithPath: "/opt/homebrew/lib/node_modules/open-computer-use/dist/Open Computer Use.app") let running = URL(fileURLWithPath: "/Users/example/projects/open-codex-computer-use/dist/Open Computer Use.app") @@ -570,6 +962,11 @@ final class OpenComputerUseKitTests: XCTestCase { let getAppStateSchema = tools["get_app_state"]?.inputSchema let getAppStateProperties = getAppStateSchema?["properties"] as? [String: [String: Any]] XCTAssertEqual(getAppStateProperties?["show_full_text"]?["type"] as? String, "boolean") + XCTAssertEqual(getAppStateProperties?["include_image"]?["type"] as? String, "boolean") + XCTAssertEqual(getAppStateProperties?["force_image"]?["type"] as? String, "boolean") + XCTAssertEqual(getAppStateProperties?["only_changes"]?["type"] as? String, "boolean") + XCTAssertEqual(getAppStateProperties?["max_text_chars"]?["type"] as? String, "integer") + XCTAssertEqual(getAppStateProperties?["max_text_chars"]?["minimum"] as? Int, 0) XCTAssertEqual(getAppStateSchema?["required"] as? [String], ["app"]) let scrollPages = (tools["scroll"]?.inputSchema["properties"] as? [String: [String: Any]])?["pages"] XCTAssertEqual(scrollPages?["type"] as? String, "number") @@ -577,6 +974,17 @@ final class OpenComputerUseKitTests: XCTestCase { scrollPages?["description"] as? String, "Number of pages to scroll. Fractional values are supported. Defaults to 1" ) + let selectTextSchema = tools["select_text"]?.inputSchema + let selectTextProperties = selectTextSchema?["properties"] as? [String: [String: Any]] + XCTAssertEqual( + tools["select_text"]?.description, + "Select text inside a text element, or place the text cursor before or after it. Provide text exactly as it appears in the accessibility tree, including any Markdown formatting. If the text is not unique, provide surrounding prefix or suffix text to disambiguate it." + ) + XCTAssertEqual(selectTextSchema?["required"] as? [String], ["app", "element_index", "text"]) + XCTAssertEqual( + selectTextProperties?["selection"]?["enum"] as? [String], + ["text", "cursor_before", "cursor_after"] + ) } func testDispatcherMissingArgumentsMatchOfficialToolText() { @@ -1717,6 +2125,10 @@ final class OpenComputerUseKitTests: XCTestCase { XCTAssertGreaterThan(abs(negativePose.angleOffset), 0.08) } + private func contentTypes(in result: ToolCallResult) -> [String] { + result.content.compactMap { $0.dictionary["type"] as? String } + } + private func makeSnapshot(treeLines: [String], focusedSummary: String?, selectedText: String? = nil) -> AppSnapshot { AppSnapshot( app: RunningAppDescriptor( diff --git a/plugins/open-computer-use/.codex-plugin/plugin.json b/plugins/open-computer-use/.codex-plugin/plugin.json index 182811b..56753ef 100644 --- a/plugins/open-computer-use/.codex-plugin/plugin.json +++ b/plugins/open-computer-use/.codex-plugin/plugin.json @@ -20,7 +20,7 @@ "interface": { "displayName": "Open Computer Use", "shortDescription": "Control desktop apps from Codex with the open local server", - "longDescription": "Open Computer Use packages this repository's open-source desktop automation server as a Codex plugin. It exposes the same nine desktop-control MCP tools through a local macOS app bundle or Linux / Windows native binary, with semantic accessibility paths preferred before fallback input.", + "longDescription": "Open Computer Use packages this repository's open-source desktop automation server as a Codex plugin. It exposes the same ten desktop-control MCP tools through a local macOS app bundle or Linux / Windows native binary, with semantic accessibility paths preferred before fallback input.", "developerName": "Leo", "category": "Productivity", "capabilities": [ @@ -34,7 +34,7 @@ "defaultPrompt": [ "Inspect the current UI state of this desktop app without stealing focus when possible", "Open a new local document with keyboard shortcuts first, then type content without moving my real pointer unless required", - "Drive the fixture app and verify the nine computer-use tools still behave correctly" + "Drive the fixture app and verify the ten computer-use tools still behave correctly" ], "brandColor": "#0E7490", "composerIcon": "./assets/open-computer-use-small.svg", diff --git a/skills/open-computer-use/SKILL.md b/skills/open-computer-use/SKILL.md index 42f9898..8ba4b17 100644 --- a/skills/open-computer-use/SKILL.md +++ b/skills/open-computer-use/SKILL.md @@ -11,7 +11,7 @@ Open Computer Use exposes Computer Use as a local CLI and stdio MCP server. It i It supports the same core tool surface across macOS, Linux, and Windows: `list_apps`, `get_app_state`, `click`, `perform_secondary_action`, `scroll`, -`drag`, `type_text`, `press_key`, and `set_value`. +`select_text`, `drag`, `type_text`, `press_key`, and `set_value`. ## Core Workflow @@ -31,7 +31,7 @@ It supports the same core tool surface across macOS, Linux, and Windows: - Ask before sending, deleting, purchasing, approving, uploading, or making other externally visible changes. - Do not assume Codex.app plugin helpers are available. Use the installed `open-computer-use` / `ocu` CLI or an explicit MCP config. - Always run `get_app_state` before using `element_index`; do not guess indexes across sessions or after large UI changes. -- Prefer semantic actions and `set_value` for editable controls. Use coordinate `click`, `scroll`, and `drag` only when the element tree does not expose a safer target. +- Prefer semantic actions, `select_text`, and `set_value` for editable controls. Use coordinate `click`, `scroll`, and `drag` only when the element tree does not expose a safer target. - On macOS, do not enable `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` unless the user explicitly wants diagnostic behavior that may move the real pointer. - On Windows and Linux, confirm the command is running inside the logged-in desktop session before assuming GUI automation is available. @@ -45,6 +45,7 @@ open-computer-use call list_apps ocu call list_apps open-computer-use call get_app_state --args '{"app":"TextEdit"}' open-computer-use call get_app_state --args '{"app":"TextEdit","show_full_text":true}' +open-computer-use call select_text --args '{"app":"TextEdit","element_index":"1","text":"Draft"}' open-computer-use call click --args '{"app":"TextEdit","element_index":"0"}' open-computer-use call type_text --args '{"app":"TextEdit","text":"Hello from Open Computer Use"}' ``` diff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.md index 3e39bfc..cf9c5ca 100644 --- a/skills/open-computer-use/references/usage.md +++ b/skills/open-computer-use/references/usage.md @@ -35,6 +35,7 @@ get_app_state click perform_secondary_action scroll +select_text drag type_text press_key @@ -49,6 +50,7 @@ Use `call` for one-off checks: open-computer-use call list_apps ocu call list_apps open-computer-use call get_app_state --args '{"app":"TextEdit"}' +open-computer-use call select_text --args '{"app":"TextEdit","element_index":"1","text":"Draft"}' open-computer-use call set_value --args '{"app":"TextEdit","element_index":"1","value":"Draft"}' ``` @@ -81,6 +83,16 @@ open-computer-use snapshot --show-full-text TextEdit The same `show_full_text` tool argument and `--show-full-text` snapshot flag apply on macOS, Linux, and Windows. +For context-budgeted hosts, `get_app_state` also accepts output controls on macOS, Linux, and Windows: + +```sh +open-computer-use call get_app_state --args '{"app":"TextEdit","include_image":false,"max_text_chars":20000}' +open-computer-use call get_app_state --args '{"app":"TextEdit","only_changes":true}' +open-computer-use call get_app_state --args '{"app":"TextEdit","include_image":true,"force_image":true}' +``` + +Use `include_image:false` to omit screenshot content, `force_image:true` to return a repeated screenshot, `max_text_chars:0` for no post-render cap, and `only_changes:true` to receive `There has been no change` when rendered text and screenshot match the previous `get_app_state` result for that app. + Action tools return refreshed app state with the default 500 character text limit. If full text is still needed after an action, run `get_app_state` again with `show_full_text: true`. ## Choosing Targets @@ -100,6 +112,8 @@ The macOS runtime uses Accessibility, ScreenCaptureKit, and targeted input event The Windows runtime uses UI Automation and Win32 message fallbacks. It must run in a logged-in desktop session. A detached SSH or service context may start the CLI but fail to see top-level windows. +Windows `select_text` uses UIA `TextPattern` only when `OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_SELECTION=1` is set, because that selection operation can bring the target app to the foreground. + ### Linux The Linux runtime uses AT-SPI2 through the desktop session bus. It must run in a logged-in graphical session with usable accessibility services. Wayland screenshot and coordinate input support is compositor-dependent and best-effort.