Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .claude/skills/winapp-ui-automation/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,26 @@ winapp ui screenshot -a myapp --capture-screen --output with-popups.png
winapp ui screenshot -a myapp --focus --output focused.png
```

### Record video (H.264 MP4)
Record the window — or a single element's region — to an MP4. Frames are captured via Windows
Graphics Capture (PrintWindow fallback) and encoded incrementally with Media Foundation, so long
captures never buffer in memory.
```powershell
# Record a window for 10s at 15 fps
winapp ui record -a myapp --duration-sec 10 --fps 15 --output demo.mp4

# Record until Ctrl+C, downscaled so the longest edge is 1280px
winapp ui record -a myapp --duration-sec 0 --max-edge 1280 --output capture.mp4

# Record a single element's region
winapp ui record itm-chart-9f8e -a myapp --output chart.mp4

# Include overlays/popups (captures from screen; may include occluding windows)
winapp ui record -a myapp --capture-screen --duration-sec 5 --output with-popups.mp4
```
- `--duration-sec 0` records until Ctrl+C; the MP4 is finalized on exit.
- The `--json` envelope reports `path`, `frames`, `width`, `height`, `fileSize`, `codec` (`"h264"`), and `mode` — the capture path used (`wgc`, `printwindow`, or `screen`).

### Hover (for tooltips, flyouts, hover states)
`--dwell-time <ms>` sets how long to wait after hovering (default: 800, range: 0–10000).
```powershell
Expand Down Expand Up @@ -374,6 +394,29 @@ Capture the target window or element as a PNG image. When multiple windows exist
| `--output` | Save output to file path (e.g., screenshot) | (none) |
| `--window` | Target window by HWND (stable handle from list output). Takes precedence over --app. | (none) |

### `winapp ui record`

Record the target window (or an element's region) to an H.264 MP4 video. Captures frames via Windows Graphics Capture and encodes with Media Foundation. Use --duration-sec 0 to record until Ctrl+C. Use --capture-screen to include overlays/popups.

#### Arguments
<!-- auto-generated from cli-schema.json -->
| Argument | Required | Description |
|----------|----------|-------------|
| `<selector>` | No | Semantic slug (e.g., btn-minimize-d1a0) or text to search by name/automationId |

#### Options
<!-- auto-generated from cli-schema.json -->
| Option | Description | Default |
|--------|-------------|---------|
| `--app` | Target app (process name, window title, or PID). Lists windows if ambiguous. | (none) |
| `--capture-screen` | Capture from screen DC via BitBlt (includes popups/overlays not owned by the target). Implies --focus. | (none) |
| `--duration-sec` | Recording duration in seconds. 0 = record until Ctrl+C. | (none) |
| `--fps` | Frames per second to capture | `15` |
| `--json` | Format output as JSON | (none) |
| `--max-edge` | Downscale so the longest edge is at most this many pixels (0 = no downscale) | (none) |
| `--output` | Save output to file path (e.g., screenshot) | (none) |
| `--window` | Target window by HWND (stable handle from list output). Takes precedence over --app. | (none) |

### `winapp ui invoke`

Activate an element by slug or text search. Tries InvokePattern, TogglePattern, SelectionItemPattern, and ExpandCollapsePattern in order.
Expand Down
43 changes: 43 additions & 0 deletions .github/plugin/skills/winapp-cli/ui-automation/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,26 @@ winapp ui screenshot -a myapp --capture-screen --output with-popups.png
winapp ui screenshot -a myapp --focus --output focused.png
```

### Record video (H.264 MP4)
Record the window — or a single element's region — to an MP4. Frames are captured via Windows
Graphics Capture (PrintWindow fallback) and encoded incrementally with Media Foundation, so long
captures never buffer in memory.
```powershell
# Record a window for 10s at 15 fps
winapp ui record -a myapp --duration-sec 10 --fps 15 --output demo.mp4

# Record until Ctrl+C, downscaled so the longest edge is 1280px
winapp ui record -a myapp --duration-sec 0 --max-edge 1280 --output capture.mp4

# Record a single element's region
winapp ui record itm-chart-9f8e -a myapp --output chart.mp4

# Include overlays/popups (captures from screen; may include occluding windows)
winapp ui record -a myapp --capture-screen --duration-sec 5 --output with-popups.mp4
```
- `--duration-sec 0` records until Ctrl+C; the MP4 is finalized on exit.
- The `--json` envelope reports `path`, `frames`, `width`, `height`, `fileSize`, `codec` (`"h264"`), and `mode` — the capture path used (`wgc`, `printwindow`, or `screen`).

### Hover (for tooltips, flyouts, hover states)
`--dwell-time <ms>` sets how long to wait after hovering (default: 800, range: 0–10000).
```powershell
Expand Down Expand Up @@ -374,6 +394,29 @@ Capture the target window or element as a PNG image. When multiple windows exist
| `--output` | Save output to file path (e.g., screenshot) | (none) |
| `--window` | Target window by HWND (stable handle from list output). Takes precedence over --app. | (none) |

### `winapp ui record`

Record the target window (or an element's region) to an H.264 MP4 video. Captures frames via Windows Graphics Capture and encodes with Media Foundation. Use --duration-sec 0 to record until Ctrl+C. Use --capture-screen to include overlays/popups.

#### Arguments
<!-- auto-generated from cli-schema.json -->
| Argument | Required | Description |
|----------|----------|-------------|
| `<selector>` | No | Semantic slug (e.g., btn-minimize-d1a0) or text to search by name/automationId |

#### Options
<!-- auto-generated from cli-schema.json -->
| Option | Description | Default |
|--------|-------------|---------|
| `--app` | Target app (process name, window title, or PID). Lists windows if ambiguous. | (none) |
| `--capture-screen` | Capture from screen DC via BitBlt (includes popups/overlays not owned by the target). Implies --focus. | (none) |
| `--duration-sec` | Recording duration in seconds. 0 = record until Ctrl+C. | (none) |
| `--fps` | Frames per second to capture | `15` |
| `--json` | Format output as JSON | (none) |
| `--max-edge` | Downscale so the longest edge is at most this many pixels (0 = no downscale) | (none) |
| `--output` | Save output to file path (e.g., screenshot) | (none) |
| `--window` | Target window by HWND (stable handle from list output). Takes precedence over --app. | (none) |

### `winapp ui invoke`

Activate an element by slug or text search. Tries InvokePattern, TogglePattern, SelectionItemPattern, and ExpandCollapsePattern in order.
Expand Down
161 changes: 161 additions & 0 deletions docs/cli-schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -2765,6 +2765,167 @@
}
}
},
"record": {
"description": "Record the target window (or an element's region) to an H.264 MP4 video. Captures frames via Windows Graphics Capture and encodes with Media Foundation. Use --duration-sec 0 to record until Ctrl+C. Use --capture-screen to include overlays/popups.",
"hidden": false,
"arguments": {
"selector": {
"description": "Semantic slug (e.g., btn-minimize-d1a0) or text to search by name/automationId",
"order": 0,
"hidden": false,
"valueType": "System.String",
"hasDefaultValue": false,
"arity": {
"minimum": 0,
"maximum": 1
}
}
},
"options": {
"--app": {
"description": "Target app (process name, window title, or PID). Lists windows if ambiguous.",
"hidden": false,
"aliases": [
"-a"
],
"valueType": "System.String",
"hasDefaultValue": false,
"arity": {
"minimum": 1,
"maximum": 1
},
"required": false,
"recursive": false
},
"--capture-screen": {
"description": "Capture from screen DC via BitBlt (includes popups/overlays not owned by the target). Implies --focus.",
"hidden": false,
"valueType": "System.Boolean",
"hasDefaultValue": true,
"defaultValue": false,
"arity": {
"minimum": 0,
"maximum": 1
},
"required": false,
"recursive": false
},
"--duration-sec": {
"description": "Recording duration in seconds. 0 = record until Ctrl+C.",
"hidden": false,
"valueType": "System.Int32",
"hasDefaultValue": true,
"defaultValue": 0,
"arity": {
"minimum": 1,
"maximum": 1
},
"required": false,
"recursive": false
},
"--fps": {
"description": "Frames per second to capture",
"hidden": false,
"valueType": "System.Int32",
"hasDefaultValue": true,
"defaultValue": 15,
"arity": {
"minimum": 1,
"maximum": 1
},
"required": false,
"recursive": false
},
"--json": {
"description": "Format output as JSON",
"hidden": false,
"valueType": "System.Boolean",
"hasDefaultValue": true,
"defaultValue": false,
"arity": {
"minimum": 0,
"maximum": 1
},
"required": false,
"recursive": false
},
"--max-edge": {
"description": "Downscale so the longest edge is at most this many pixels (0 = no downscale)",
"hidden": false,
"valueType": "System.Int32",
"hasDefaultValue": true,
"defaultValue": 0,
"arity": {
"minimum": 1,
"maximum": 1
},
"required": false,
"recursive": false
},
"--output": {
"description": "Save output to file path (e.g., screenshot)",
"hidden": false,
"aliases": [
"-o"
],
"valueType": "System.String",
"hasDefaultValue": false,
"arity": {
"minimum": 1,
"maximum": 1
},
"required": false,
"recursive": false
},
"--quiet": {
"description": "Suppress progress messages",
"hidden": false,
"aliases": [
"-q"
],
"valueType": "System.Boolean",
"hasDefaultValue": true,
"defaultValue": false,
"arity": {
"minimum": 0,
"maximum": 1
},
"required": false,
"recursive": false
},
"--verbose": {
"description": "Enable verbose output",
"hidden": false,
"aliases": [
"-v"
],
"valueType": "System.Boolean",
"hasDefaultValue": true,
"defaultValue": false,
"arity": {
"minimum": 0,
"maximum": 1
},
"required": false,
"recursive": false
},
"--window": {
"description": "Target window by HWND (stable handle from list output). Takes precedence over --app.",
"hidden": false,
"aliases": [
"-w"
],
"valueType": "System.Nullable<System.Int64>",
"hasDefaultValue": false,
"arity": {
"minimum": 1,
"maximum": 1
},
"required": false,
"recursive": false
}
}
},
"screenshot": {
"description": "Capture the target window or element as a PNG image. When multiple windows exist (e.g., dialogs), captures each to a separate file. With --json, returns file path and dimensions. Use --capture-screen for popup overlays.",
"hidden": false,
Expand Down
20 changes: 20 additions & 0 deletions docs/fragments/skills/winapp-cli/ui-automation.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,26 @@ winapp ui screenshot -a myapp --capture-screen --output with-popups.png
winapp ui screenshot -a myapp --focus --output focused.png
```

### Record video (H.264 MP4)
Record the window — or a single element's region — to an MP4. Frames are captured via Windows
Graphics Capture (PrintWindow fallback) and encoded incrementally with Media Foundation, so long
captures never buffer in memory.
```powershell
# Record a window for 10s at 15 fps
winapp ui record -a myapp --duration-sec 10 --fps 15 --output demo.mp4

# Record until Ctrl+C, downscaled so the longest edge is 1280px
winapp ui record -a myapp --duration-sec 0 --max-edge 1280 --output capture.mp4

# Record a single element's region
winapp ui record itm-chart-9f8e -a myapp --output chart.mp4

# Include overlays/popups (captures from screen; may include occluding windows)
winapp ui record -a myapp --capture-screen --duration-sec 5 --output with-popups.mp4
```
- `--duration-sec 0` records until Ctrl+C; the MP4 is finalized on exit.
- The `--json` envelope reports `path`, `frames`, `width`, `height`, `fileSize`, `codec` (`"h264"`), and `mode` — the capture path used (`wgc`, `printwindow`, or `screen`).

### Hover (for tooltips, flyouts, hover states)
`--dwell-time <ms>` sets how long to wait after hovering (default: 800, range: 0–10000).
```powershell
Expand Down
29 changes: 29 additions & 0 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -1080,6 +1080,7 @@ winapp ui [command] [options]
- `get-property` - Read element properties
- `get-text` / `get-value` - Read value/text from element (TextPattern, ValuePattern, or Name)
- `screenshot` - Capture window/element as PNG (auto-captures dialogs separately)
- `record` - Record a window/element region to an H.264 MP4 video (Windows Graphics Capture + Media Foundation)
- `invoke` - Activate element (click, toggle, expand)
- `click` - Click element via mouse simulation (for controls that don't support invoke)
- `hover` - Move mouse to element to trigger tooltips, flyouts, and hover states (default dwell: 800ms)
Expand All @@ -1096,4 +1097,32 @@ winapp ui [command] [options]
- `-a, --app <app>` - Target app (name, title, or PID)
- `-w, --window <hwnd>` - Target window by HWND (stable)

#### ui record

Record the target window — or a single element's region — to an H.264 MP4 video. Frames are
captured via Windows Graphics Capture (with a PrintWindow fallback) and encoded incrementally with
Media Foundation, so long recordings never buffer in memory.

```bash
# Record a window for 10 seconds at 15 fps
winapp ui record -a Calculator --duration-sec 10 --fps 15 -o demo.mp4

# Record until Ctrl+C, downscaled so the longest edge is 1280px
winapp ui record -a "My App" --duration-sec 0 --max-edge 1280 -o capture.mp4

# Record just one element's region
winapp ui record -a "My App" btn-save-1234 -o button.mp4
```

**Record options:**
- `--duration-sec <n>` - Recording length in seconds. `0` records until Ctrl+C (default `0`).
- `--fps <n>` - Frames per second to capture (default `15`).
- `--max-edge <px>` - Downscale so the longest edge is at most this many pixels (`0` = no downscale).
- `--capture-screen` - Capture from the screen so overlays/popups are included (may capture occluding windows).
- `-o, --output <path>` - Output `.mp4` path (defaults to `recording-<timestamp>.mp4`).

With `--json`, emits a `UiRecordResult` envelope including the output `path`, `frames`, `width`,
`height`, `fileSize`, `codec` (`"h264"`), and `mode` — the capture path actually used
(`wgc`, `printwindow`, or `screen`).

For full documentation, see [docs/ui-automation.md](ui-automation.md).
2 changes: 1 addition & 1 deletion scripts/generate-llm-docs.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ $SkillCommandMap = @{
"manifest" = @("manifest generate", "manifest update-assets", "manifest add-alias")
"troubleshoot" = @("get-winapp-path", "tool", "store")
"frameworks" = @() # No auto-generated command sections — links to guides
"ui-automation" = @("ui status", "ui inspect", "ui search", "ui get-property", "ui get-value", "ui screenshot", "ui invoke", "ui click", "ui drag", "ui hover", "ui send-keys", "ui set-value", "ui focus", "ui scroll-into-view", "ui scroll", "ui wait-for", "ui list-windows", "ui get-focused")
"ui-automation" = @("ui status", "ui inspect", "ui search", "ui get-property", "ui get-value", "ui screenshot", "ui record", "ui invoke", "ui click", "ui drag", "ui hover", "ui send-keys", "ui set-value", "ui focus", "ui scroll-into-view", "ui scroll", "ui wait-for", "ui list-windows", "ui get-focused")
}

# Validate that all CLI commands are covered by at least one skill
Expand Down
17 changes: 17 additions & 0 deletions src/winapp-CLI/WinApp.Cli.Tests/FakeUiServices.cs
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,23 @@ public Task<UiElement[]> SearchAsync(UiSessionInfo session, SelectorExpression s
public Task<(byte[] Pixels, int Width, int Height)> ScreenshotAsync(UiSessionInfo session, string? elementId, bool captureScreen, bool focus, CancellationToken ct)
=> Task.FromResult(ScreenshotResult);

/// <summary>Configurable result for <see cref="RecordAsync"/>. The fake writes a tiny placeholder file to the output path.</summary>
public RecordCaptureResult RecordResult { get; set; } = new() { Frames = 3, Width = 2, Height = 2, FileSize = 0, Mode = "wgc" };

public async Task<RecordCaptureResult> RecordAsync(UiSessionInfo session, string? elementId, RecordOptions options, CancellationToken ct)
{
await File.WriteAllBytesAsync(options.OutputPath, new byte[16], ct);
var size = new FileInfo(options.OutputPath).Length;
return new RecordCaptureResult
{
Frames = RecordResult.Frames,
Width = RecordResult.Width,
Height = RecordResult.Height,
FileSize = size,
Mode = RecordResult.Mode,
};
}

public Task<string> InvokeAsync(UiSessionInfo session, UiElement element, CancellationToken ct)
=> Task.FromResult(InvokeResult);

Expand Down
Loading
Loading