Add ui record command for MP4 capture by yeelam-gordon · Pull Request #599 · microsoft/winappCli

yeelam-gordon · 2026-07-06T03:00:07Z

Summary

add winapp ui record to capture a window or element region to H.264 MP4
support duration, FPS, max-edge downscale, and --capture-screen for overlays/popups
regenerate the CLI schema and UI automation skill/docs for the new command

Why

This adds first-class video evidence to the UI automation surface so scenarios can be recorded and reviewed instead of relying only on still screenshots.

Adds a new `winapp ui record` command that captures a window or element region to H.264 MP4 using Windows Graphics Capture and Media Foundation, with duration, FPS, max-edge downscale, and optional screen capture for overlays/popups. Also updates the generated CLI schema and UI automation skill/docs to surface the new command. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR adds a new winapp ui record subcommand that captures a target window (or a single element's region) to an H.264 MP4. Frames are grabbed via Windows Graphics Capture (with PrintWindow / screen-DC fallbacks) and encoded incrementally through Media Foundation's IMFSinkWriter, so long recordings never buffer in memory. It fits into the existing UI-automation surface alongside ui screenshot, giving agents/tests video evidence rather than only stills.

Changes:

New recorder pipeline: UiRecordCommand, UiAutomationService.Record (capture/crop/scale loop), WgcCapture.Record (persistent frame grabber), and Mp4SinkWriterEncoder (Media Foundation encode), plus RecordOptions/RecordCaptureResult/UiRecordResult models and new NativeMethods.txt MF projections.
Wiring: registered in HostBuilderExtensions, UiCommand, the IUiAutomationService interface, plus test stubs/fakes and new UiCommandTests.Record unit tests.
Docs/schema: regenerated cli-schema.json and updated docs/usage.md, the hand-written skill fragment, and the two generated SKILL.md mirrors.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`src/winapp-CLI/WinApp.Cli/Commands/UiRecordCommand.cs`	New command: option parsing, input validation, JSON envelope (validation uses the wrong error code).
`src/winapp-CLI/WinApp.Cli/Commands/SharedUiOptions.cs`	Adds `--duration-sec`, `--fps`, `--max-edge` options.
`src/winapp-CLI/WinApp.Cli/Services/UiAutomationService.Record.cs`	Core capture/crop/scale/encode loop and element-rect resolution (WGC-failure fallback diverges from `screenshot`).
`src/winapp-CLI/WinApp.Cli/Services/WgcCapture.Record.cs`	Persistent free-threaded `FrameGrabber` that caches the latest frame.
`src/winapp-CLI/WinApp.Cli/Services/Mp4SinkWriterEncoder.cs`	Media Foundation H.264 encoder with deterministic COM release and partial-file cleanup.
`src/winapp-CLI/WinApp.Cli/Services/RecordModels.cs`	`RecordOptions` / `RecordCaptureResult` DTOs.
`src/winapp-CLI/WinApp.Cli/Helpers/UiJsonContext.cs`	Adds `UiRecordResult` for AOT-safe JSON.
`src/winapp-CLI/WinApp.Cli/Helpers/HostBuilderExtensions.cs`	Registers the command handler.
`src/winapp-CLI/WinApp.Cli/Commands/UiCommand.cs`	Adds `record` as a `ui` subcommand.
`src/winapp-CLI/WinApp.Cli/Services/IUiAutomationService.cs`	Adds `RecordAsync` to the interface.
`src/winapp-CLI/WinApp.Cli/NativeMethods.txt`	Adds Media Foundation P/Invoke + interface projections.
`src/winapp-CLI/WinApp.Cli.Tests/UiCommandTests.Record.cs`	Command validation + JSON envelope tests.
`src/winapp-CLI/WinApp.Cli.Tests/FakeUiServices.cs` / `UiSessionServiceTests.cs`	Fake/stub `RecordAsync` implementations.
`docs/cli-schema.json`, `docs/usage.md`, `docs/fragments/skills/.../ui-automation.md`, `.github/plugin/.../SKILL.md`, `.claude/.../SKILL.md`, `scripts/generate-llm-docs.ps1`	Schema + docs/skill updates for the new command.

Notes for the author (files outside this PR, so not inline-commentable): per repo conventions, docs/ui-automation.md should gain a ### record section under ## Commands (it documents every other ui subcommand), and .github/plugin/agents/winapp.agent.md "Key subcommands" list should include ui record.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+            if (durationSec < 0)
+            {
+                UiJsonError.Emit(json, UiJsonError.CodeInternalError, "--duration-sec must be 0 or greater.");
+                logger.LogError("{Symbol} --duration-sec must be 0 or greater.", UiSymbols.Error);
+                return 1;
+            }
+            if (fps < 1)
+            {
+                UiJsonError.Emit(json, UiJsonError.CodeInternalError, "--fps must be at least 1.");
+                logger.LogError("{Symbol} --fps must be at least 1.", UiSymbols.Error);
+                return 1;
+            }
+            if (maxEdge < 0)
+            {
+                UiJsonError.Emit(json, UiJsonError.CodeInternalError, "--max-edge must be 0 or greater.");
+                logger.LogError("{Symbol} --max-edge must be 0 or greater.", UiSymbols.Error);
+                return 1;
+            }


+                    _logger.LogDebug(ex, "WGC recorder init failed; falling back to screen-DC capture");
+                    grabber?.Dispose();
+                    grabber = null;
+                    useScreen = true;
+                    useWgc = false;
+                    mode = "screen";
+                    srcWidth = rect.right - rect.left;
+                    srcHeight = rect.bottom - rect.top;
+                    captureOriginLeft = rect.left;
+                    captureOriginTop = rect.top;


github-actions · 2026-07-06T03:15:57Z

Build Metrics Report

Binary Sizes

Artifact	Baseline	Current	Delta
CLI (ARM64)	31.71 MB	31.84 MB	📈 +136.5 KB (+0.42%)
CLI (x64)	32.04 MB	32.18 MB	📈 +141.0 KB (+0.43%)
MSIX (ARM64)	13.30 MB	13.36 MB	📈 +59.4 KB (+0.44%)
MSIX (x64)	14.15 MB	14.20 MB	📈 +44.7 KB (+0.31%)
NPM Package	27.77 MB	27.86 MB	📈 +96.2 KB (+0.34%)
NuGet Package	27.79 MB	27.89 MB	📈 +102.7 KB (+0.36%)
VS Code Extension	20.60 MB	20.69 MB	📈 +92.0 KB (+0.44%)

Test Results

✅ 1551 passed, 1 skipped out of 1552 tests in 398.0s (+6 tests, -80.1s vs. baseline)

Test Coverage

❌ 16.8% line coverage, 36.8% branch coverage · ⚠️ -1.3% vs. baseline

CLI Startup Time

41ms median (x64, winapp --version) · ✅ no change vs. baseline

Updated 2026-07-06 03:15:57 UTC · commit f7f44cb · workflow run

nmetulev

🤖 AI-generated review — produced by GitHub Copilot CLI at a maintainer's request. Every finding comes from an automated, multi-dimensional review plus a hands-on build-and-test of this branch in an isolated worktree. Treat file:line references as pointers to verify, not gospel. Nothing was pushed. Posting as a comment (no approve / request-changes).

Verdict: 🟡 SHIP-WITH-CHANGES — The core encoder (Windows.Graphics.Capture + Media Foundation, no external deps) works well; the blockers are agent-ergonomics and generated-file sync, not the capture path.

What I actually ran

Built WinApp.Cli (Debug, ~3.5s, 0 warnings); 6/6 Record unit tests pass. Recorded Notepad via -w <hwnd>:

3s @15fps → valid MP4 (ftyp/mp42, h264, 45 frames, 62 KB) ✅
--max-edge 640 downscale ✅ correct aspect (1858×1166 → 640×402)
element-region crop ("File" item, 62×48) ✅
--capture-screen → mode":"screen" ✅
error paths: missing_app ✅, bad HWND ✅, unwritable path → E_ACCESSDENIED ✅

Encoding is a pure Windows stack: WGC (Direct3D11CaptureFramePool, works while occluded) → Media Foundation IMFSinkWriter (incremental H.264, no buffering) → SkiaSharp for crop/scale. Blocks until done; partial files auto-deleted on failure; no temp litter.

Must-fix (High)

H1 — wrong error codes. --duration-sec / --fps / --max-edge validation emits CodeInternalError ("internal_error") instead of CodeInvalidArguments (UiRecordCommand.cs:61,67,73). Confirmed at runtime: --fps 0 → {"error":{"code":"internal_error",...}}. An agent that treats internal_error as transient will retry-loop on a bad argument.
H2 — agent file not updated. ui record is absent from .github/plugin/agents/winapp.agent.md (both the decision tree and the command reference). Copilot agents that rely on that file won't discover the command.
H3 — npm bindings not regenerated. src/winapp-npm/src/winapp-commands.ts has no typed uiRecord(). Run npm run generate-commands and include the result.

Medium / Low

M1 --capture-screen help says "Implies --focus" but record has no --focus option (SharedUiOptions.cs:86).
M2 mode":"screen" is used for both explicit --capture-screen and the silent WGC-init-failure fallback (UiAutomationService.Record.cs:70,102) — consumers can't detect degradation. Suggest a distinct "screen-fallback".
M3 WGC→screen fallback, element-not-found, and cancel-mid-record paths are untested.
M4 (agent safety) default --duration-sec 0 records until Ctrl+C — an agent that omits it hangs forever with no signal. Recommend a safe non-zero default (e.g. 30s) with 0 as an explicit opt-in.
L1 -o help reads "e.g., screenshot" on a video command.

Fit & recommendation

Nicely implemented and native (no bundled ffmpeg, no downloads). The honest caveat is audience: agents can't parse H.264, so record is more a human/CI-evidence tool than an agent-perception tool — for state checks, looped screenshot is usually better. It also needs an interactive desktop + WGC-capable GPU, so it's hard to cover in CI. Worth shipping for evidence / UI-regression capture once M4 and the generated-file sync (H2/H3) land.

Copilot AI review requested due to automatic review settings July 6, 2026 03:00

Copilot started reviewing on behalf of yeelam-gordon July 6, 2026 03:00 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

nmetulev reviewed Jul 6, 2026

View reviewed changes

nmetulev mentioned this pull request Jul 10, 2026

Improve pr-review skill: real validation, necessity/simplicity, and independent multi-model #613

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add ui record command for MP4 capture#599

Add ui record command for MP4 capture#599
yeelam-gordon wants to merge 1 commit into
mainfrom
yeelam/ui-record

yeelam-gordon commented Jul 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Jul 6, 2026

Uh oh!

nmetulev left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

yeelam-gordon commented Jul 6, 2026

Summary

Why

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions Bot commented Jul 6, 2026

Build Metrics Report

Binary Sizes

Test Results

Test Coverage

CLI Startup Time

Uh oh!

nmetulev left a comment

Choose a reason for hiding this comment

What I actually ran

Must-fix (High)

Medium / Low

Fit & recommendation

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants