GameStudio screenshot regression pipeline#3162
Merged
Merged
Conversation
Move the LPIPS / Claude vision comparator out of samples/Tests/Comparator/ into sources/tests/Stride.ScreenshotComparator/ so other test pipelines (notably the GameStudio AutoTesting fixtures in tests/editor/) can reuse it via ProjectReference. Namespace renamed Stride.SampleScreenshotComparator to Stride.Tests.ScreenshotComparator. The lpips_alex.onnx model propagates to consumers' bins via <None CopyToOutputDirectory>. samples/Tests/Stride.Samples.Tests.csproj drops its inline OnnxRuntime + ImageSharp PackageRefs and the explicit model <None> item — all come transitively from the shared lib now. SampleScreenshotTests.cs uses the new namespace and lets the comparator default to AppContext.BaseDirectory for model lookup. Stride.sln update is intentionally deferred — bundle it with the AutoTesting sln entry when those land together.
Lets projects opt into versioned OS monikers (e.g. net10.0-windows10.0.22621.0 for WinRT) without breaking platform/dependency conditions. Also surfaces NuGet restore diagnostics in NuGetAssemblyResolver and lets consumers override NuGetResolverTargetFramework.
AutoTesting hosts GS in-process and drives [UITest] fixtures through IUITestContext (queue waits, WGC capture, AvalonDock panel capture, modal drive-by). Stride.Editor.Tests adds EmptyEditor / NewGameEditor / TopDownCreate / TopDownLoad fixtures. GS gets an appHosted callback in Run() and qualified XAML pack URIs so resources resolve from a non-entry assembly.
test-windows-editor.yml runs the four EditorScreenshotTests fixtures (EmptyEditor, NewGameEditor, TopDownCreate, TopDownLoad) and uploads the captured PNGs + diag logs. Trigger is daily cron + workflow_dispatch (matches test-samples-screenshots), offset to avoid runner contention. set-display-resolution-vdd installs the IDD-based MttVDD driver via NefCon and disables Hyper-V Video so the virtual display is the only attached output (reaches 2560x1600). set-display-resolution is the simpler Set-DisplayResolution variant for workflows that only need <=1920x1080.
Adds an overrideable record-based prompt builder (ComparisonPrompt with sealed Gameplay/Editor variants, each carrying its own tolerance and regression-trigger toggles plus a curated static Default). Threads the prompt through ClaudeVisionFallback.Compare and ScreenshotComparator's new defaultPrompt parameter. Existing samples flow keeps the gameplay preset as the default; editor flow can opt into EditorComparisonPrompt.
Adds support for committing multiple acceptable variants of a frame as '<frame>.<tag>.png' siblings of the canonical '<frame>.png' baseline. LPIPS picks the closest match across the variant set (best-of); the Claude vision fallback receives all variants in one request and is asked whether the capture is consistent with the demonstrated variance range. Single-baseline path is unchanged.
[Theory] driven from the in-DLL fixture list (EmptyEditor, NewGameEditor, TopDownCreate, TopDownLoad); each entry spawns Stride.GameStudio.AutoTesting.exe in a subprocess for WPF singleton-state isolation, snapshots the per-fixture output dir, then runs ScreenshotComparator.Compare against tests/editor/baselines/dpi100/ with EditorComparisonPrompt.Default. Adds xunit + Microsoft.NET.Test.Sdk package refs and the Stride.ScreenshotComparator project ref.
New IUITestContext primitives:
- RunProject / WaitForGameWindow / WaitForGameFrames / ScreenshotHwnd /
CloseGameWindow: drive F5 from GameStudio and capture the launched game
once WGC has delivered enough frames for post-effects to settle
- AddAssetFromTemplate / QueueAssetPickerResponse / AddEntityToScene:
invoke the same VM commands the user does (RunAssetTemplate,
CreateEntityInRootCommand) plus drive intermediate dialogs
NewGameEditor fixture adds a procedural capsule (with Sphere Material picked
via the auto-resolved picker), drops it into the scene, runs the game, and
captures four artefacts: new-game-editor, scene-with-capsule, game-running,
build-log-after-run.
GraphicsCaptureClient: factor shared OpenSession / StartCaptureAndNudge /
CloseSession so CaptureToPngAsync and the new WaitForFramesAsync share the
WGC plumbing. Caller attaches FrameArrived before StartCapture so frames
produced during nudge don't slip past the unattached handler.
DebuggingViewModel: tuple-return BuildProject(Core) so RunProjectAsync can
surface the launched Process to the autotest path.
UITestHost: typed access for AvalonDock panels (Dirkster.AvalonDock pkg ref),
EditorGameController.IEditorGameAccess interface and AssetEditorsManager
.EditorViewModels accessor (both via InternalsVisibleTo) replace the
reflection-based EnumerateActiveGames + SearchTree / FloatAnchorable walks.
SelectTemplate / AddAssetFromTemplate now take Guid; built-in templates
referenced via {Type}TemplateGenerator.TemplateId.
Captured from xen2/stride CI run 25592292751: - EmptyEditor: startup - NewGameEditor: new-game-editor, scene-with-capsule, game-running, build-log-after-run - TopDownCreate: main, scene-main, 5 docked panels - TopDownLoad: main
Acissathar
added a commit
to Acissathar/stride
that referenced
this pull request
May 12, 2026
commit 776d261 Author: Virgile Bello <virgile@stride3d.net> Date: Mon May 11 14:21:02 2026 +0900 ci: checkout submodules in test-windows-editor.yml commit 7b88c7a Author: Virgile Bello <virgile@stride3d.net> Date: Mon May 11 14:18:57 2026 +0900 ci: welcome comment pointing draft PRs at /ci commit 60e3124 Author: Will B <willbentz92@outlook.com> Date: Tue May 5 09:25:51 2026 -0600 Add GetInputFiles to SpriteSheet Asset and Thumbnail compiler so changes to the source are picked up commit d9cc4a1 Author: jklawreszuk <jk.lawreszuk@gmail.com> Date: Fri Apr 24 23:33:41 2026 +0200 fix: Paths are case sensitive on Linux commit b467bdd Author: Basewq <Basewq@users.noreply.github.com> Date: Thu Apr 23 14:08:58 2026 +1200 fix: [Editor] AssetViewModel.TypeDisplayName use object type name as fallback name if DisplayAttribute.Name is not set commit 62e11df Author: Virgile Bello <virgile@stride3d.net> Date: Mon May 11 13:37:28 2026 +0900 ci: /ci chatops covers builds + tests, drafts skip auto CI commit c6d5b48 Author: Virgile Bello <virgile@stride3d.net> Date: Mon May 11 13:31:28 2026 +0900 ci: checkout submodules + warn on workflow-YAML drift in /test PR-side workflow YAML edits don't take effect via /test (repository_dispatch loads YAML from the default branch). Sdsl-rewrite needs `submodules: true` on the checkouts that build Stride.Shaders.Spirv.Core; add it now so the next /test against PR 3134 picks it up. Also surface the trap: chatops bot diffs each dispatched workflow's blob between master and the PR ref; if they differ, the dispatch comment links to the file's diff and explains that master's YAML is what actually ran. commit 480a619 Author: Virgile Bello <virgile@stride3d.net> Date: Mon May 11 12:55:53 2026 +0900 ci: chatops uses Check Runs API for in-progress state commit 15bf00f Author: Henrik Gedionsen <hg@impact.dk> Date: Thu Apr 30 15:33:49 2026 +0200 Reduce dictionary lookups commit dfd3b70 Author: Virgile Bello <virgile@stride3d.net> Date: Fri Apr 24 16:36:29 2026 +0900 sdk: strip graphics-API-specific native packages from consumer bin/ Filter out copy-local runtime assets from packages Stride pulls for graphics APIs the consumer isn't using (Silk.NET.Direct3D12, DXGI, WinPixEventRuntime, Vortice.Vulkan). Only strips when every requester of the package is a Stride contributor and the user hasn't referenced it directly — preserving user-direct refs and third-party transitive refs. Silk.NET.Direct3D11 is intentionally kept on every build: the FXC-based shader compiler uses ID3D11ShaderReflection from d3dcompiler.dll for DXBC reflection on both D3D11 and D3D12 targets. Architecture: each package owns its rules. Stride.Graphics declares base rendering rules (DXC=D3D-only); Stride.Assets ships a sibling targets file that extends DXC's APIs list to include Vulkan because the asset pipeline cross-compiles HLSL to SPIR-V. Internal editor builds pick up both via Stride.Build.Sdk.Editor. commit 3cd79b8 Author: Virgile Bello <virgile@stride3d.net> Date: Mon May 11 09:18:03 2026 +0900 ci: PR-comment chatops bot (/test ...) commit 69bae47 Author: Virgile Bello <virgile@stride3d.net> Date: Mon May 11 09:18:02 2026 +0900 ci: skip screenshots without ANTHROPIC_API_KEY commit 8915fb8 Author: Virgile Bello <virgile@stride3d.net> Date: Mon May 11 09:18:02 2026 +0900 ci/main: explicit per-API jobs + path-filter gating commit f14f466 Author: Virgile Bello <virgile@stride3d.net> Date: Mon May 11 09:18:01 2026 +0900 ci: drop duplicate pull_request triggers from build-* commit 0093d02 Merge: a197854 64e7c09 Author: xen2 <virgile@stride3d.net> Date: Sun May 10 01:17:48 2026 +0900 Merge pull request stride3d#3162 from xen2/gamestudio-screenshot-regression GameStudio screenshot regression pipeline commit 64e7c09 Author: Virgile Bello <virgile@stride3d.net> Date: Sat May 9 15:19:13 2026 +0900 editor/autotest: dpi100 baselines for editor fixtures Captured from xen2/stride CI run 25592292751: - EmptyEditor: startup - NewGameEditor: new-game-editor, scene-with-capsule, game-running, build-log-after-run - TopDownCreate: main, scene-main, 5 docked panels - TopDownLoad: main commit 6ff99fa Author: Virgile Bello <virgile@stride3d.net> Date: Sat May 9 13:50:28 2026 +0900 editor/autotest: F5 + scene-mutation primitives in NewGameEditor New IUITestContext primitives: - RunProject / WaitForGameWindow / WaitForGameFrames / ScreenshotHwnd / CloseGameWindow: drive F5 from GameStudio and capture the launched game once WGC has delivered enough frames for post-effects to settle - AddAssetFromTemplate / QueueAssetPickerResponse / AddEntityToScene: invoke the same VM commands the user does (RunAssetTemplate, CreateEntityInRootCommand) plus drive intermediate dialogs NewGameEditor fixture adds a procedural capsule (with Sphere Material picked via the auto-resolved picker), drops it into the scene, runs the game, and captures four artefacts: new-game-editor, scene-with-capsule, game-running, build-log-after-run. GraphicsCaptureClient: factor shared OpenSession / StartCaptureAndNudge / CloseSession so CaptureToPngAsync and the new WaitForFramesAsync share the WGC plumbing. Caller attaches FrameArrived before StartCapture so frames produced during nudge don't slip past the unattached handler. DebuggingViewModel: tuple-return BuildProject(Core) so RunProjectAsync can surface the launched Process to the autotest path. UITestHost: typed access for AvalonDock panels (Dirkster.AvalonDock pkg ref), EditorGameController.IEditorGameAccess interface and AssetEditorsManager .EditorViewModels accessor (both via InternalsVisibleTo) replace the reflection-based EnumerateActiveGames + SearchTree / FloatAnchorable walks. SelectTemplate / AddAssetFromTemplate now take Guid; built-in templates referenced via {Type}TemplateGenerator.TemplateId. commit 7886f76 Author: Virgile Bello <virgile@stride3d.net> Date: Thu May 7 18:31:47 2026 +0900 editor/autotest: EditorScreenshotTests xunit orchestrator [Theory] driven from the in-DLL fixture list (EmptyEditor, NewGameEditor, TopDownCreate, TopDownLoad); each entry spawns Stride.GameStudio.AutoTesting.exe in a subprocess for WPF singleton-state isolation, snapshots the per-fixture output dir, then runs ScreenshotComparator.Compare against tests/editor/baselines/dpi100/ with EditorComparisonPrompt.Default. Adds xunit + Microsoft.NET.Test.Sdk package refs and the Stride.ScreenshotComparator project ref. commit cd1cb6a Author: Virgile Bello <virgile@stride3d.net> Date: Thu May 7 18:31:34 2026 +0900 comparator: multi-baseline-per-frame variants Adds support for committing multiple acceptable variants of a frame as '<frame>.<tag>.png' siblings of the canonical '<frame>.png' baseline. LPIPS picks the closest match across the variant set (best-of); the Claude vision fallback receives all variants in one request and is asked whether the capture is consistent with the demonstrated variance range. Single-baseline path is unchanged. commit 43bf0c4 Author: Virgile Bello <virgile@stride3d.net> Date: Thu May 7 18:28:27 2026 +0900 comparator: ComparisonPrompt domain classes + Claude vision wiring Adds an overrideable record-based prompt builder (ComparisonPrompt with sealed Gameplay/Editor variants, each carrying its own tolerance and regression-trigger toggles plus a curated static Default). Threads the prompt through ClaudeVisionFallback.Compare and ScreenshotComparator's new defaultPrompt parameter. Existing samples flow keeps the gameplay preset as the default; editor flow can opt into EditorComparisonPrompt. commit ad59bed Author: Virgile Bello <virgile@stride3d.net> Date: Thu May 7 13:42:03 2026 +0900 ci/editor: editor screenshot workflow + virtual display actions test-windows-editor.yml runs the four EditorScreenshotTests fixtures (EmptyEditor, NewGameEditor, TopDownCreate, TopDownLoad) and uploads the captured PNGs + diag logs. Trigger is daily cron + workflow_dispatch (matches test-samples-screenshots), offset to avoid runner contention. set-display-resolution-vdd installs the IDD-based MttVDD driver via NefCon and disables Hyper-V Video so the virtual display is the only attached output (reaches 2560x1600). set-display-resolution is the simpler Set-DisplayResolution variant for workflows that only need <=1920x1080. commit 5843679 Author: Virgile Bello <virgile@stride3d.net> Date: Thu May 7 13:41:55 2026 +0900 editor/autotest: GameStudio screenshot regression harness AutoTesting hosts GS in-process and drives [UITest] fixtures through IUITestContext (queue waits, WGC capture, AvalonDock panel capture, modal drive-by). Stride.Editor.Tests adds EmptyEditor / NewGameEditor / TopDownCreate / TopDownLoad fixtures. GS gets an appHosted callback in Run() and qualified XAML pack URIs so resources resolve from a non-entry assembly. commit 9e735a1 Author: Virgile Bello <virgile@stride3d.net> Date: Thu May 7 13:41:37 2026 +0900 sdk: collapse versioned TFM suffix into StrideTargetFramework Lets projects opt into versioned OS monikers (e.g. net10.0-windows10.0.22621.0 for WinRT) without breaking platform/dependency conditions. Also surfaces NuGet restore diagnostics in NuGetAssemblyResolver and lets consumers override NuGetResolverTargetFramework. commit 4947898 Author: Virgile Bello <virgile@stride3d.net> Date: Wed May 6 17:35:11 2026 +0900 tests: extract ScreenshotComparator into shared sources/tests/ project Move the LPIPS / Claude vision comparator out of samples/Tests/Comparator/ into sources/tests/Stride.ScreenshotComparator/ so other test pipelines (notably the GameStudio AutoTesting fixtures in tests/editor/) can reuse it via ProjectReference. Namespace renamed Stride.SampleScreenshotComparator to Stride.Tests.ScreenshotComparator. The lpips_alex.onnx model propagates to consumers' bins via <None CopyToOutputDirectory>. samples/Tests/Stride.Samples.Tests.csproj drops its inline OnnxRuntime + ImageSharp PackageRefs and the explicit model <None> item — all come transitively from the shared lib now. SampleScreenshotTests.cs uses the new namespace and lets the comparator default to AppContext.BaseDirectory for model lookup. Stride.sln update is intentionally deferred — bundle it with the AutoTesting sln entry when those land together. commit a197854 Author: Virgile Bello <virgile@stride3d.net> Date: Sat May 9 14:44:18 2026 +0900 samples/fps screenshot test: 4 deterministic frames Replace flaky after-jump with initial / reloading / reloaded / after-fire. The 500ms held tap was triggering reload (gun spawns empty), and reload completion timing varied between runs. commit 01be2cf Author: Virgile Bello <virgile@stride3d.net> Date: Sat May 9 13:54:56 2026 +0900 docs: drop stale v145-rationale note The bullet now de-emphasizes the specific version (just '(Latest)' with v145 as a parenthetical), so the 'Why v145 specifically?' framing is awkward. commit ec551d0 Author: Virgile Bello <virgile@stride3d.net> Date: Sat May 9 13:49:46 2026 +0900 ci: offset nightly cron schedules off top-of-hour GitHub Actions cron events at the top of every hour (e.g. '0 4 * * *') are delayed or silently skipped during peak load. Move check-parallel-build to 04:17 and test-samples-screenshots to 04:37 to dodge the spike (per GitHub's documented recommendation). commit 6ee2e04 Author: Virgile Bello <virgile@stride3d.net> Date: Sat May 9 13:16:31 2026 +0900 docs: move build/docs to docs/build, simplify top-level README Move the build-internals docs (SDK-GUIDE.md, plus the new build/README.md created in this PR) under docs/. Trim the top-level README's prerequisites to the 3 essentials (Git, .NET 10 SDK, VS 2026 with C++ + .NET workloads) and link to docs/build/README.md for specifics — toolset versions, optional ARM64/iOS/Android/VSIX components, Build Tools / dotnet build paths, and troubleshooting all live there. Drop the stale C++/CLI workload requirement (no current vcxproj uses CompileAsManaged). Fix git lfs clone -> git clone (LFS auto-fetches with a modern git+LFS install). commit 55b468e Author: Virgile Bello <virgile@stride3d.net> Date: Sat May 9 12:38:04 2026 +0900 sdk: skip MSVC mode under Core MSBuild (dotnet build/msbuild) Core MSBuild can't build vcxproj end-to-end (TypeLoadException on Microsoft.Build.CPPTasks.Common which depends on Framework-only utilities). Require MSBuildRuntimeType=Full so dotnet build from a VS Developer Command Prompt picks Clang instead of failing on MSVC. Document the resulting behavior in the README. commit b9cebae Author: Virgile Bello <virgile@stride3d.net> Date: Fri May 8 23:39:30 2026 +0900 sdk: auto-detect StrideNativeBuildMode, drop redundant overrides MSVC only when targeting Windows AND the VS C++ tools are loaded in the current shell (VCINSTALLDIR set). Clang everywhere else, including cross-compile from Windows hosts. Removes the need for -p:StrideNativeBuildMode=Clang in 12 CI workflows + test-linux-gpu.ps1 (release.yml keeps the override since it runs msbuild.exe with VS env active). commit f3038b9 Author: Virgile Bello <virgile@stride3d.net> Date: Sat May 9 00:40:41 2026 +0900 ci/auditor: group by resolved OutputPath to catch same-output dispatches commit c674bfc Author: Virgile Bello <virgile@stride3d.net> Date: Fri May 8 22:49:47 2026 +0900 ci: ParallelBuildAuditor — detect property-leak duplicate builds Reads an MSBuild binlog and reports projects built more than once due to global-property leaks, after filtering legitimate multi-build axes (TargetFramework, Configuration, Platform, RuntimeIdentifier, StrideGraphicsApi) and noise from Restore + ProjectReference probing targets (GetTargetFrameworks, GetAllRuntimeIdentifiers, _StrideQueryGraphicsApis, etc.). For each duplicate, reports the parent project + target list to point at the call site triggering the duplicate build. Wires up a GitHub workflow that runs on PR + nightly + workflow_dispatch: builds Stride.sln with -bl, runs the auditor, fails the job if any leak is detected. Stride.sln currently passes with zero duplicates. commit af4dff2 Author: Nicolas Musset <musset.nicolas@gmail.com> Date: Thu May 7 18:39:07 2026 +0200 feat: Add more unit tests in Core to increase coverage (stride3d#2996) commit ee271b6 Author: Virgile Bello <virgile@stride3d.net> Date: Wed May 6 18:09:10 2026 +0900 core/packages: avoid sync-over-async deadlock in SyncLocalFolderSources
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Details
GameStudio screenshot regression tests, mirroring the samples-screenshots pattern.
4 tests (
tests/editor/):EmptyEditor— bare GameStudio startupNewGameEditor— creates project from New-Game template, adds a procedural capsule asset, drops it in the scene, runs F5, captures the running game window + build logTopDownCreate— generates the TopDownRPG sample, captures docked panels (assets, properties, references, etc.)TopDownLoad— opens a pre-checked-out TopDownRPG and captures the main editorParts:
Stride.GameStudio.AutoTestinghosts GameStudio in-process and drives[UITest]fixtures viaIUITestContext(waits, WGC capture, AvalonDock panel float-and-capture, modal drive-by, F5+game-window capture, scene mutation viaIEntityFactory).Stride.ScreenshotComparatorextracted from samples to be shared (LPIPS + Claude vision fallback, multi-baseline-per-frame variants).EditorScreenshotTestsxunit[Theory]runs each fixture in a subprocess (WPF singleton isolation) then compares againsttests/editor/baselines/dpi100/.test-windows-editor.yml, daily cron +workflow_dispatch(matches samples convention).Baselines committed for all 4 fixtures from a green CI run.
Note: tests themselves and UITestContext generated using Claude Code, I wouldn't bother too much to make it nice-looking, we can just keep using Claude to adjust tests as necessary since it's not redistributed code.
Types of changes
Checklist