Add ui touch and ui pen commands by yeelam-gordon · Pull Request #600 · microsoft/winappCli

yeelam-gordon · 2026-07-06T03:00:16Z

Summary

add winapp ui touch for tap, swipe, pinch, stretch, and related touch gestures
add winapp ui pen for stylus tap/inking with pressure, tilt, and eraser options
regenerate the CLI schema, npm command surface, and UI automation skill/docs

Why

This expands WinApp CLI beyond mouse/keyboard input so touch-first and pen/stylus experiences can be exercised directly in automation.

Adds `winapp ui touch` and `winapp ui pen` so automation can inject touch gestures (tap, swipe, pinch, stretch) and stylus/inking input (tap, path, pressure, tilt, eraser) against running Windows apps. Also updates the generated CLI schema, npm command surface, and UI automation skill/docs to surface the new commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR extends the winapp ui automation surface beyond mouse/keyboard by adding two new OS-level input-injection commands, ui touch and ui pen, so touch-first and pen/stylus experiences can be driven directly from the CLI. It introduces a testable IPointerInput abstraction (with a native PointerInput P/Invoke implementation using the Windows synthetic-pointer and legacy touch-injection APIs), a pure PointerGesturePlanner for coordinate parsing and gesture expansion, and a new IUiAutomationService.TryGetWindowRect for bounds-checking. It follows the existing gesture-command pattern (resolve session → resolve selector/coords → foreground → stabilize → bounds-check → inject) while adding stricter safety (no_target rejection when no window resolves, and per-point window bounds checks). Schema, npm wrapper, and skill/docs are regenerated to match.

Changes:

New ui touch (tap/double-tap/long-press/swipe/pinch/stretch) and ui pen (tap/ink with pressure, tilt, eraser) commands, wired into UiCommand, DI, and the IPointerInput injection abstraction.
New helpers: PointerInput (native touch/pen injection with synthetic→legacy fallback), PointerGesturePlanner (parsing + gesture geometry + bounds), plus TryGetWindowRect, new JSON result/error models, and System.Single support in the npm generator/schema context.
Unit tests for both commands (happy + several error paths) and regenerated docs/schema/skill artifacts.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`src/winapp-CLI/WinApp.Cli/Commands/UiTouchCommand.cs`	New touch command: validation, target/foreground/bounds gating, injection, JSON output.
`src/winapp-CLI/WinApp.Cli/Commands/UiPenCommand.cs`	New pen command: pressure/tilt validation, path/at/selector targeting, injection.
`src/winapp-CLI/WinApp.Cli/Commands/UiCommand.cs`	Registers `touch`/`pen` subcommands.
`src/winapp-CLI/WinApp.Cli/Helpers/PointerInput.cs`	Native synthetic-pointer/legacy touch + pen injection with cleanup.
`src/winapp-CLI/WinApp.Cli/Helpers/PointerGesturePlanner.cs`	Coordinate/path parsing, gesture expansion, out-of-bounds detection.
`src/winapp-CLI/WinApp.Cli/Helpers/IPointerInput.cs` / `RealPointerInput.cs`	Injection abstraction + production delegate.
`src/winapp-CLI/WinApp.Cli/Helpers/HostBuilderExtensions.cs`	DI registration for `IPointerInput` and the two handlers.
`src/winapp-CLI/WinApp.Cli/Helpers/UiJsonContext.cs` / `UiJsonError.cs`	New result models (`UiTouchResult`/`UiPenResult`/`UiPointResult`) and `no_target`/`injection_unsupported` codes.
`src/winapp-CLI/WinApp.Cli/Helpers/CliSchema.cs`	Adds `float` to the schema serializer context (for `--pressure` default).
`src/winapp-CLI/WinApp.Cli/Services/IUiAutomationService.cs` / `UiAutomationService.cs`	`TryGetWindowRect` contract + `GetWindowRect` implementation.
`src/winapp-CLI/WinApp.Cli/NativeMethods.txt`	Adds touch/pen injection P/Invoke entries and structs.
`src/winapp-CLI/WinApp.Cli.Tests/*`	`FakePointerInput`, `TryGetWindowRect` fake, and new Touch/Pen command tests.
`src/winapp-npm/src/winapp-commands.ts`	Regenerated `uiTouch`/`uiPen` wrappers.
`src/winapp-npm/scripts/generate-commands.mjs`	Maps `System.Single` → TS `number`.
`scripts/generate-llm-docs.ps1`	Adds `ui touch`/`ui pen` to the ui-automation skill map.
`docs/usage.md`, `docs/ui-automation.md`, `docs/fragments/skills/...`, `docs/cli-schema.json`, `.github/plugin/.../SKILL.md`, `.claude/.../SKILL.md`	Documentation + regenerated schema/skill artifacts for the new commands.

Notes for the author (not anchorable to changed lines): The hand-written "Key subcommands" list in .github/plugin/agents/winapp.agent.md (lines 207-223) still lists ui drag/ui hover/ui send-keys but not the new touch/pen. That file is not auto-generated from the schema, so it should be updated alongside docs/usage.md to keep the shipped Copilot agent aware of the new verbs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    public static bool LooksLikeCoordinates(string token)
+    {
+        var parts = token.Split(',');
+        return parts.Length >= 2
+            && int.TryParse(parts[0].Trim(), NumberStyles.Integer, CultureInfo.InvariantCulture, out _);
+    }


+            case TouchGesture.Pinch:
+            case TouchGesture.Stretch:
+            {
+                fingers = Math.Max(2, fingers);
+                int half = Math.Max(PinchCenterGapPx + 1, distance / 2);
+
+                // Two opposing fingers along the x-axis. Pinch converges toward the center; stretch
+                // diverges away from it.
+                var leftApart = new PointerPoint(start.X - half, start.Y);
+                var rightApart = new PointerPoint(start.X + half, start.Y);
+                var leftNear = new PointerPoint(start.X - PinchCenterGapPx, start.Y);
+                var rightNear = new PointerPoint(start.X + PinchCenterGapPx, start.Y);
+
+                if (gesture == TouchGesture.Pinch)
+                {
+                    contactPaths.Add([leftApart, leftNear]);
+                    contactPaths.Add([rightApart, rightNear]);
+                }
+                else
+                {
+                    contactPaths.Add([leftNear, leftApart]);
+                    contactPaths.Add([rightNear, rightApart]);
+                }
+                break;
+            }


github-actions · 2026-07-06T03:17:28Z

Build Metrics Report

Binary Sizes

Artifact	Baseline	Current	Delta
CLI (ARM64)	31.71 MB	31.86 MB	📈 +152.0 KB (+0.47%)
CLI (x64)	32.04 MB	32.18 MB	📈 +148.5 KB (+0.45%)
MSIX (ARM64)	13.30 MB	13.37 MB	📈 +70.2 KB (+0.52%)
MSIX (x64)	14.15 MB	14.21 MB	📈 +53.5 KB (+0.37%)
NPM Package	27.77 MB	27.89 MB	📈 +119.6 KB (+0.42%)
NuGet Package	27.79 MB	27.91 MB	📈 +123.5 KB (+0.43%)
VS Code Extension	20.60 MB	20.71 MB	📈 +116.8 KB (+0.55%)

Test Results

✅ 1559 passed, 1 skipped out of 1560 tests in 472.6s (+14 tests, -5.4s vs. baseline)

Test Coverage

❌ 18.4% line coverage, 37.6% branch coverage · ✅ +0.3% vs. baseline

CLI Startup Time

34ms median (x64, winapp --version) · ✅ -9ms vs. baseline

Updated 2026-07-06 03:17:28 UTC · commit 011826d · workflow run

nmetulev

🤖 AI-generated review — produced by GitHub Copilot CLI at a maintainer's request. Every finding comes from an automated, multi-dimensional review plus a hands-on build-and-test of this branch in an isolated worktree. Treat file:line references as pointers to verify, not gospel. Nothing was pushed. Posting as a comment (no approve / request-changes).

Verdict: 🟡 SHIP-WITH-CHANGES — Solid injection safety model and the modern pointer API works without hardware; the gaps are ergonomics and agent discovery.

What I actually ran

Built (~29s, 0 warnings); 70/70 touch+pen unit tests pass. Verified by direct P/Invoke that CreateSyntheticPointerDevice(PT_TOUCH/PT_PEN) succeeds on this no-touchscreen machine (handles returned, err=0) — so the modern path is viable without a digitizer. All validation/error paths return structured JSON + exit 1 (missing_app, invalid gesture, bad coords, pinch requires --distance, fingers 1–10, pressure 0–1, tilt ±90, path parse), and the bounds-check fires before injection.

⚠️ Environmental note: actual on-screen gesture effect could not be visually confirmed — this was a remote/headless desktop and injection hit the same foreground_not_target guard that gates all input verbs (click/drag/hover/scroll/send-keys). Not a PR bug.

Must-fix (High)

H1 — agent file not updated. ui touch and ui pen are absent from .github/plugin/agents/winapp.agent.md, so Copilot agents won't autonomously choose them.
H2 — swipe has no direction. --gesture swipe --distance N always moves +X (right) (PointerGesturePlanner.cs:125-134); there's no --direction. An agent wanting swipe up/down/left must compute absolute coordinates by hand — inconsistent with ui scroll --direction. Add --direction {right|left|up|down}.

Medium / Low

M1 double-checked-lock on _touchInitialized isn't volatile (PointerInput.cs:53-56). Safe on x64 TSO but spec-noncompliant; one-word fix.
M2 --gesture long-press with the default --hold-ms 0 silently behaves like a tap. Auto-default (~500ms) or warn.
M3 pen inter-point delay is hardcoded to 10ms with no --duration-ms (asymmetric with touch); stroke timing is untested.
L1 Interpolate discards intermediate waypoints (latent — only 1–2 pt paths today). L2 pen lacks any timing control.

Fit & recommendation

This is a natural extension of the existing input-verb family and fills real gaps a mouse can't emulate — multi-touch pinch/stretch, pen pressure/tilt/eraser (Ink Workspace, OneNote, Maps zoom). The IPointerInput/Real/Fake abstraction mirrors IMouseInput cleanly and unit-tests run headless. Caveats: these are long-tail scenarios (most agent work is click/type), and — like all input verbs — the injection paths can't be exercised in headless CI. Ship after the swipe-direction (H2) and agent-file (H1) fixes.

Copilot AI review requested due to automatic review settings July 6, 2026 03:00

Copilot started reviewing on behalf of yeelam-gordon July 6, 2026 03:01 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

nmetulev reviewed Jul 6, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add ui touch and ui pen commands#600

Add ui touch and ui pen commands#600
yeelam-gordon wants to merge 1 commit into
mainfrom
yeelam/ui-touch-pen

yeelam-gordon commented Jul 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Jul 6, 2026

Uh oh!

nmetulev left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

yeelam-gordon commented Jul 6, 2026

Summary

Why

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions Bot commented Jul 6, 2026

Build Metrics Report

Binary Sizes

Test Results

Test Coverage

CLI Startup Time

Uh oh!

nmetulev left a comment

Choose a reason for hiding this comment

What I actually ran

Must-fix (High)

Medium / Low

Fit & recommendation

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants