Enhance winapp run --debug-output crash stacks with WinUI stowed-exception triage

[zateutsch]

Enhance winapp run --debug-output crash stacks with WinUI stowed-exception triage

Closes #549

What

When winapp run --debug-output captures a crash from a WinUI 3 app, the resulting stack was thin — the user's faulting frame plus a couple of cswinrt ABI shims, with no visibility into the XAML dispatcher / CoreMessaging / runtime chain that led to the failure, and no breakdown of stowed exceptions (0xC000027B), the most common WinUI crash shape.

This PR adds an automatic WinUI stowed-exception triage pass on top of the existing ClrMD analysis. When Microsoft.UI.Xaml.dll is present in the crashed process's module list, winapp now surfaces:

• The originating HRESULT and its ErrorContext chain
• The full native XAML dispatch stack (Microsoft.UI.Xaml → CXcpDispatcher → CoreMessagingXP → CLR host)
• The managed user frame that threw (from winapp's existing ClrMD pass)

The feature is auto-enabled for WinUI apps — no new flag. The standard managed/native analysis is unchanged for all other apps.

How

• Hosts DbgEng directly via the Microsoft.Debugging.Platform.DbgEng NuGet package and runs the WinUI team's winui-dbgext.js extension (!xamlstowed / !xamltriage) against the same minidump — no WinDbg install required.
• Co-locates symsrv.dll (from Microsoft.Debugging.Platform.SymSrv) next to dbgeng.dll so srv* symbol paths resolve.
• JsProvider.dll (the JS scripting host, not on NuGet) is fetched on first use directly from the official WinDbg download using ranged reads (only the few hundred KB needed), Authenticode-verified as Microsoft-signed, and cached under the winapp global directory.
• The extension script is pinned by SHA and hash-gated before load.
• Debugger component versions are pinned (kept in sync with Directory.Packages.props via a drift test) so dev/CI acquisition is deterministic.
• Everything is cached, so subsequent runs are offline. For locked-down environments, set WINAPP_DBGTOOLS_DIR to a directory containing dbgeng.dll and JsProvider.dll. When the binaries can't be obtained, triage is skipped with a clear log note and the standard analysis still runs.

--symbols additionally improves the native XAML dispatch stack for WinUI apps.

Testing

• Added unit tests covering: ZIP64 central-directory parsing (ranged extraction), stowed-exception record selection, NuGet global-cache version pinning + fallback, child-process argument construction, the pinned extension-hash gate, and a drift test tying the pinned debugger version to Directory.Packages.props.
• Full suite: 1355 passed, 0 failed (1 environment-only skip). Build clean with 0 warnings, including the Release/AOT (ILC) compile.

Docs

• Updated docs/usage.md (--debug-output / --symbols) and the setup.md skill fragment; docs/debugging.md documents the triage pass and WINAPP_DBGTOOLS_DIR override. Regenerated Copilot/Claude skills.

Notes

• !xamlstowed produces the highest-value output and works fully with the NuGet + JsProvider setup. !xamltriage's managed-frame walker depends on SOS/CLR data access; winapp instead combines its own ClrMD managed stack with the extension's native output, so managed frames are always resolved.
• Relevant new code: Services/XamlTriageService.cs, XamlTriageBinaries.cs, XamlTriageRunner.cs, WinDbgJsProviderAcquirer.cs, Helpers/ZipRangeExtractor.cs, Helpers/AuthenticodeVerifier.cs; triage slots in after CrashDumpService.AnalyzeWithClrMD.

[Copilot]

Pull request overview

This PR enriches winapp run --debug-output crash diagnostics for WinUI 3 apps (closes #549). When a captured minidump contains Microsoft.UI.Xaml.dll, winapp now runs an automatic stowed-exception triage pass that hosts DbgEng directly and executes the WinUI team's winui-dbgext.js extension (!xamlstowed / !xamltriage), surfacing the originating HRESULT + ErrorContext chain and the native XAML dispatch stack alongside the existing ClrMD managed frames. It fits into CrashDumpService.AnalyzeDumpAsync, immediately after the ClrMD pass, and is auto-enabled (no new flag). The debugger engine bits are restored from NuGet (or downloaded/cached on first use), JsProvider.dll is range-extracted from the official WinDbg bundle and Authenticode-verified, and the pinned extension is hash-gated before load.

Changes:

• Adds a WinUI triage subsystem (XamlTriageService/IXamlTriageService, XamlTriageRunner, XamlTriageBinaries, WinDbgJsProviderAcquirer, ZipRangeExtractor, AuthenticodeVerifier) run in an isolated __xaml-triage child process to avoid a dbghelp.dll loader collision.
• Extends WriteMiniDump to prefer the terminating stowed exception's record/parameters (so !xamlstowed can locate the stowed array) while keeping the first-chance thread CONTEXT for ClrMD; AnalyzeWithClrMD now reports whether the dump is WinUI.
• Adds restore-only Microsoft.Debugging.Platform.DbgEng/SymSrv package references (version pinned + drift-tested), unit tests for the new logic, and docs/skill updates.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
Services/XamlTriageService.cs	Orchestrates binary resolution/acquisition, spawns the isolated child, formats output & symbol-gap notes
Services/XamlTriageRunner.cs	Child-process DbgEng host that loads the JS provider + extension and runs the triage commands
Services/XamlTriageBinaries.cs	Resolves/acquires engine DLLs from override dir, installed Debuggers, or NuGet cache/download
Services/WinDbgJsProviderAcquirer.cs	Range-extracts JsProvider.dll from the WinDbg bundle and Authenticode-verifies it
Services/IXamlTriageService.cs	Triage service interface/contract
Services/CrashDumpService.cs	Selects stowed vs first-chance exception record; wires in triage output; detects WinUI
Services/ICrashDumpService.cs	WriteMiniDump signature extended with crash-exception code/address/parameters
Services/DebugOutputService.cs	Forwards terminating exception parameters into the dump
Helpers/ZipRangeExtractor.cs	Ranged ZIP/ZIP64 central-directory reader + IRangeReader
Helpers/AuthenticodeVerifier.cs	WinVerifyTrust-based Microsoft-signer check
Helpers/HostBuilderExtensions.cs	Registers IXamlTriageService
Program.cs	Intercepts the hidden __xaml-triage verb before host setup
WinApp.Cli.csproj / Directory.Packages.props	Restore-only debugger package references + pinned versions
WinApp.Cli.Tests/*	Tests for ZIP extraction, arg building, symbol-gap, version pinning, exception-record selection; fakes updated
docs/usage.md, docs/debugging.md, docs/fragments/skills/.../setup.md, .github/plugin/.../SKILL.md, .claude/skills/.../SKILL.md	Document the triage pass + WINAPP_DBGTOOLS_DIR
src/winapp-npm/src/winapp-commands.ts	Auto-generated schema-version comment bump

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Build Metrics Report

Binary Sizes

Artifact	Baseline	Current	Delta
CLI (ARM64)	31.71 MB	31.87 MB	📈 +167.0 KB (+0.51%)
CLI (x64)	32.04 MB	32.19 MB	📈 +159.5 KB (+0.49%)
MSIX (ARM64)	13.30 MB	13.39 MB	📈 +87.9 KB (+0.65%)
MSIX (x64)	14.15 MB	14.22 MB	📈 +66.8 KB (+0.46%)
NPM Package	27.77 MB	27.91 MB	📈 +148.3 KB (+0.52%)
NuGet Package	27.79 MB	27.94 MB	📈 +148.5 KB (+0.52%)

Test Results

✅ 1611 passed, 1 skipped out of 1612 tests in 360.0s (+66 tests, -306.2s vs. baseline)

Test Coverage

❌ 18.3% line coverage, 37.4% branch coverage · ✅ +0.3% vs. baseline

CLI Startup Time

42ms median (x64, winapp --version) · ✅ no change vs. baseline

---

Updated 2026-07-11 02:43:21 UTC · commit 9e3955a · workflow run

[nmetulev]

Review: WinUI stowed-exception triage — findings + live end-to-end verification

I reviewed this across security, correctness, CLI UX, alternative-solution, test-coverage, docs/samples, and packaging, cross-checked the top findings on a second model family (GPT-5.4), and validated the feature end-to-end against a purpose-built crashing WinUI 3 app.

End-to-end verification ✅

Built a packaged WinUI 3 app that auto-crashes ~2 s after activation with 0xC000027B, then ran the branch-built winapp run --debug-output against it:

• Happy path works: ClrMD managed stack + full !xamlstowed/!xamltriage breakdown (ErrorCode 0xc000027b, ErrorMessage "IDispatcherQueueHandler::Invoke failed.", native CoreMessagingXP!DispatcherQueueTimer::TimerCallback → RoOriginateError dispatch stack).
• Acquisition pipeline validated: engine bits copied from the NuGet cache (pinned 20260319.1511.0), JsProvider.dll (~23 MB) downloaded from the WinDbg CDN + Authenticode-verified, winui-dbgext.js (137 KB) from GitHub — all cached under ~\.winapp\dbgtools\.
• Warm-cache re-run: 8.7 s, no re-download.
• M5 reproduced live (see inline): with WINAPP_DBGTOOLS_DIR pointed at an empty dir and a fully warm cache, triage still reported "unavailable — set WINAPP_DBGTOOLS_DIR …" (the variable that was already set).

Findings

ID	Sev	Location	Issue
H1	High	CrashDumpService.cs:202 + XamlTriageService.cs:101	Triage OperationCanceledException (incl. HttpClient timeout) escapes and discards the already-computed managed crash stack → "Analysis failed." Regression on slow / first-run networks.
M1	Med	.github/plugin/agents/winapp.agent.md:139	--debug-output agent doc not updated for triage / WINAPP_DBGTOOLS_DIR / first-run download.
M2	Med	tests	No integration test asserts triage failure is non-fatal (fail-open) — would have caught H1.
M3	Med	tests	AuthenticodeVerifier (the trust gate for a DLL loaded into the debugger) has no unit tests.
M4	Med	XamlTriageBinaries.cs:287	Engine .nupkg downloaded + extracted with no package-hash/signature verification (HTTPS-only trust).
M5	Med	XamlTriageBinaries.cs:107	WINAPP_DBGTOOLS_DIR override short-circuit → wasted egress into an ignored cache dir, then false "unavailable."
M6	Med	WinDbgJsProviderAcquirer.cs:66	Non-atomic writes to final filenames (23 MB JsProvider + File.Copy sites) → concurrent / partial-file races.
L1	Low	RunCommand.cs help	--debug-output/--symbols help text doesn't mention triage or first-run downloads.
L2	Low	AuthenticodeVerifier.cs:82	Revocation checking disabled (WTD_REVOKE_NONE) — no CRL/OCSP.
L3	Low	CrashDumpService.cs:234	"triage written to the debug log" prints even when triage was skipped/unavailable; native value-add never surfaced in console.
L4	Low	XamlTriageBinaries.cs:157	Cached JsProvider.dll not re-verified on cache hit (only at download time).
L5	Low	XamlTriageBinaries.cs:275	Bespoke NuGet download duplicates INugetService (maintainability — note: NugetService also skips integrity checks, so reuse would not fix M4).

Positives

• JsProvider.dll is HTTPS + Authenticode-gated; the extension script is git-blob-SHA1-pinned and fail-closed; ZipArchive.ExtractToDirectory avoids zip-slip; the ranged ZIP64 reader uses bounded reads. Packaging is clean (exact-pinned Microsoft packages, restore-only refs, AOT end-user path falls back to flat-container download).

Inline comments below anchor the actionable code-level items (H1, M4, M5, M6, L2, L3). H1 is the one blocking-worthy item; the rest are polish/hardening.

[nmetulev]

Follow-up review: all prior findings resolved — but stress-testing the real "download on first use" path surfaced a blocking issue

Thanks for the thorough pass on 11cafc7 — every finding from my first review is addressed, several with extra hardening (SHA-512 per-.nupkg, AtomicFile, JsProvider re-verify self-heal, revocation whole-chain). Re-verification table is at the bottom.

I then did what I'd flagged I hadn't done live the first time: exercised the genuine cold-cache "download on first use" path (my earlier happy-path run reused a warm cache from an earlier session). That's where the problem is.

🔴 Blocking — JsProvider.dll / engine version drift breaks triage for every fresh user

The engine and the JS provider are pinned to different, incompatible sources:

Binary	Source	Version I got today
dbgeng.dll (engine)	NuGet Microsoft.Debugging.Platform.DbgEng 20260319.1511.0, SHA-512-gated	10.0.29547.1002
JsProvider.dll	WinDbg "current" appinstaller (WinDbgJsProviderAcquirer.cs:29)	10.0.29617.1000 (30.7 MB)

The "current" WinDbg bundle rolls forward independently of the NuGet-pinned engine. Loading the newer JS provider into the older engine makes the triage child process exit 0x80000003 (STATUS_BREAKPOINT, -2147483645), and triage is silently skipped — the exact headline feature this PR adds never runs.

Reproduced from a genuinely empty ~\.winapp\dbgtools\, both acquisition paths:

• Scenario A — dev/CI cold (NUGET_PACKAGES has the debugger packages): engine copied from the NuGet cache, JsProvider downloaded (30.7 MB "current") → triage skipped, ~16.6 s.
• Scenario B — true end-user cold (NUGET_PACKAGES empty → engine must download): engine downloaded from api.nuget.org flat-container (the live .nupkg passed the compiled SHA-512 gate ✅ — that path is solid), JsProvider downloaded (30.7 MB "current") → triage skipped, ~20.7 s.
• Control: dropping in a version-matched JsProvider.dll (10.0.29547.1002, 23 MB) → triage succeeds with the full verdict (0xc000027b — "IDispatcherQueueHandler::Invoke failed.").

So the only variable separating success from skip is the JsProvider build. Why it hides in dev: iterative dev keeps a warm cache whose JsProvider was fetched when "current" happened to equal the pinned engine build — which is exactly why my first-pass happy path looked green. A user (or CI, or me with a cleared cache) downloading today gets the mismatch.

Root cause: AppInstallerUrl always resolves to the "current" MainBundle. FallbackBundleUrl (WinDbgJsProviderAcquirer.cs:32) is only used when the appinstaller fails to parse, so a version-matched bundle is never the primary source, and nothing checks that the two binaries agree after acquisition.

Any one of these closes it:

1. Pin JsProvider to a WinDbg bundle whose build matches the pinned DbgEng, and make that bundle the primary source (not just the parse-failure fallback). Add a unit test asserting JsProvider build == engine build — mirroring the excellent .nupkg SHA-512 drift test you already added for the engine.
2. Or add a post-acquire compatibility check: compare dbgeng.dll and JsProvider.dll ProductVersion; on mismatch, reject the JsProvider (re-acquire the matched bundle, or fail-closed) so self-heal + honest messaging kick in instead of a silent child crash.
3. Either way, surface a clearer skip reason than exited with code -2147483645 — detect the mismatch and say so.

Residuals (non-blocking)

• dbgeng.dll corruption has no self-heal (Low). ResolveExisting re-verifies only JsProvider.dll's Authenticode on a cache hit; a truncated/corrupt engine DLL passes File.Exists, the child crashes, triage skips, and the cache stays poisoned across runs (JsProvider self-heals, engine doesn't). Managed stack is still preserved. Consider extending the re-verify (size or hash) to the engine bits too.
• H1 console noise on the download timeout (Low). The H1 fix correctly preserves the managed stack now (verified with a live 5-min induced timeout 👍), but the raw TaskCanceledException stack still prints to the console and the user waits the full 5-minute HttpClient.Timeout before the crash stack appears. Consider a shorter default JsProvider-acquisition timeout and suppressing the raw stack.

Re-verified fixed this pass ✅

H1 (managed stack preserved on timeout — double-guarded) · L3 (XamlTriageResult surfaces the one-line verdict in console) · M5 (override no longer wastes egress; DescribeOverrideGap is honest) · M6 (AtomicFile staged-write + publish) · L4 (cached JsProvider re-verified every resolve → truncation self-heals) · M4 (compiled-in SHA-512 per .nupkg, latest fallback dropped → fail-closed; drift test passes) · L2 (revocation whole-chain, cache-only so no offline hang; CERT_E_REVOKED hard-fails). Triage unit tests: 63/63 pass (the 5 failing E2E_NodeSubcommand_* are pre-existing and unrelated — they need npm run build).

[nmetulev]

✅ Revalidated 3711ec2 end-to-end — the version-drift fix works, and both residuals are closed.

Rebuilt at 3711ec2 and ran the triage unit suite (43/43 pass, including the new VersionsMatch cases and the PinnedJsProviderProductVersion drift guard), then re-ran the cold-cache scenarios that were failing before:

Cold first-run (the scenario that was broken): cleared ~\.winapp\dbgtools\ and forced a fresh acquire → JsProvider.dll now downloads from the pinned bundle as 22.3 MB / 10.0.29547.1002 (matches the engine) → triage SUCCEEDS with the verdict 0xc000027b — "IDispatcherQueueHandler::Invoke failed.", ~18.6 s. (Before this commit: 30.7 MB / 10.0.29617.1000 → silently skipped.) 🎯

Compat-gate self-heal: planted a valid Microsoft-Authenticode-signed but wrong-version DLL (14.44.35222.100) as JsProvider.dll → on the next run it was rejected by the version gate, the matched provider was re-downloaded, and triage succeeded (cache healed back to 10.0.29547.1002).

Engine self-heal (my earlier residual): truncated the cached dbgeng.dll to 2 KB → IsUsablePeFile + the unreadable-engine-version compat path rejected it, the engine was re-acquired, and triage succeeded.

Nice work — you took both recommendations (pin the matched bundle and add a runtime compat gate), closed the two residuals I'd left (engine PE self-heal + the H1 timeout console noise via the new OperationCanceledException fail-open), and the 0x80000003 exit now explains itself. From my side this resolves the blocking finding — no further concerns.

One tiny non-blocking thought: PinnedJsProviderProductVersion_MatchesEngineDrift_Guard asserts against a hardcoded "10.0.29547.1002" rather than reading dbgeng.dll's actual product version, so a future DbgPackageVersion bump that updates neither constant wouldn't be caught by that specific test. The runtime compat gate still fails closed regardless, so it's belt-and-suspenders either way.