Image/Video Background Refactor #831
Open
+15,549
−3,668
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Make clicks working
+ Fix UI layout resized incorrectly + Fix UI state updated incorrectly + Cleaning-up constants and unused/duplicate methods + Add option to select pip by hovering instead of clicking + Add event handlers for ItemIndex and ItemsCount changes
+ By preloading it to an empty grid, this issue can be fixed. Yeah, for no reason, even when the element has already been added to the ContentPresenter, the first frame of the element will report the offset and size to 0x0. So by tricking it to load into an empty grid, the element will force itself to load and reports the correct offset and size.
The countdown is currently using Timer for counting down the slideshow. This approach, however, uses interval event on each ticks so the duration of the count might be inaccurate (but not significant). The tick resolution is 100ms for now.
+ CodeQA + Simplify content placement on XAML + Fade-in/out countdown bar on entering and exiting + Fix content not updating while adding/removing items from Items collection
+ Implements Parallax effect on mouse hover + The image is currently using a dummy one and not yet compositing. Will be fully implemented later (ETA: at the end of this week, hopefully)
+ Bug fix on object -> double conversion, causing throw + Add IMediaCacheHandler interface for handling cache on media loading
Now we are able to perform direct frame drawing without dependency on CanvasBitmap or any SoftwareBitmap which requires back-to-back GPU->CPU->GPU frame copy. The GPU usage difference is negligible, as close as mainstream video player GPU usage.
Member
|
Fatal crash on region change |
Member
Author
As mentioned above, the code for these changes haven't been applied to main Collapse Launcher code yet and still on |
Member
|
As per @neon-nyan statement above, copilot code check is removed until PR is actually finished in Collapse side Cc @Cryotechnic |
+ This also occurs during element ``Unloaded`` event + The loader will perform media seeking to the last position if the previous source was the same
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
| UseShellExecute = true | ||
| }); | ||
|
|
||
| await process!.WaitForExitAsync(); |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
+ Add artificial delays to avoid flash frame due to incomplete initialization on Video Frame. + Fix where video background plays in background while Collapse is running on start-up. + The "Keep Play Video When Unfocused" is now enabled by default, matching HoYoPlay behavior. + Make color palette generation works if FFmpeg is used
| { | ||
| using IRandomAccessStream sourceRandomStream = stream.AsRandomAccessStream(true); | ||
| SvgImageSource svgImageSource = new(); | ||
| svgImageSource.SetSourceAsync(sourceRandomStream); |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Comment on lines
+406
to
+416
| if (!(ffmpegMediaSource.PlaybackItem?.Source.IsOpen ?? false)) | ||
| { | ||
| await Task.Delay(150); | ||
|
|
||
| // Unsubscribe frame renderer event to avoid double call, and then mark deinitialization. | ||
| Interlocked.Exchange(ref instance._isVideoInitialized, 0); | ||
| player.VideoFrameAvailable -= !instance.UseSafeFrameRenderer | ||
| ? instance.VideoPlayer_VideoFrameAvailableUnsafe | ||
| : instance.VideoPlayer_VideoFrameAvailableSafe; | ||
|
|
||
| ffmpegMediaSource.Dispose(); |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Comment on lines
+348
to
+352
| catch (Exception e) | ||
| { | ||
| Console.WriteLine(e); | ||
| await tempStream.DisposeAsync(); | ||
| } |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
|
||
| if (sourceStream != null) | ||
| { | ||
| sourceStreamRandom = sourceStream.AsRandomAccessStream(true); |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Comment on lines
+436
to
+438
| { | ||
| if (isError) sourceStreamRandom?.Dispose(); | ||
| } |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Comment on lines
238
to
239
| CanvasSvgDocument svgDoc = await CanvasSvgDocument.LoadAsync(svgDevice, svgRandomStream); | ||
| return null; |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
The Save button is already in the expanded menu, it takes up too much space here.
|
|
||
| if (sourceStream != null) | ||
| { | ||
| sourceStreamRandom = sourceStream.AsRandomAccessStream(true); |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Comment on lines
+110
to
+116
| // -- Get upscaled image file if Waifu2X is enabled | ||
| if (GlobalIsWaifu2XEnabled) | ||
| { | ||
| downloadedOverlayUri = await TryGetScaledWaifu2XImagePath(downloadedOverlayUri, token); | ||
| downloadedBackgroundUri = await TryGetScaledWaifu2XImagePath(downloadedBackgroundUri, token); | ||
| downloadedBackgroundStaticUri = await TryGetScaledWaifu2XImagePath(downloadedBackgroundStaticUri, token); | ||
| } |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Comment on lines
238
to
239
| CanvasSvgDocument svgDoc = await CanvasSvgDocument.LoadAsync(svgDevice, svgRandomStream); | ||
| return null; |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
+ Attempt to fix thumbnail generation on FFmpeg to be fetched for color palette generator. + Attempt to generate color palette for SVG background
| using VideoFrame? ffmpegFrame = await ffmpegFrameGrabber.ExtractVideoFrameAsync(ffmpegFrameDuration, true); | ||
|
|
||
| MemoryStream ffmpegFrameExtracted = new(); | ||
| IRandomAccessStream ffmpegFrameStream = ffmpegFrameExtracted.AsRandomAccessStream(); |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| const uint biBitfields = 3; | ||
|
|
||
| int stride = width * height * 4; | ||
| string tempPath = Path.Combine(Path.GetTempPath(), Path.GetTempFileName() + ".bmp"); |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Comment on lines
+110
to
+116
| // -- Get upscaled image file if Waifu2X is enabled | ||
| if (GlobalIsWaifu2XEnabled) | ||
| { | ||
| downloadedOverlayUri = await TryGetScaledWaifu2XImagePath(downloadedOverlayUri, token); | ||
| downloadedBackgroundUri = await TryGetScaledWaifu2XImagePath(downloadedBackgroundUri, token); | ||
| downloadedBackgroundStaticUri = await TryGetScaledWaifu2XImagePath(downloadedBackgroundStaticUri, token); | ||
| } |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Main Goal
As described by the title, this PR contains major changes to Collapse Launcher's Image/Video Background mechanism to match current behavior on HoYoPlay which have multiple sequence of background and layered images.
Here's the current list of what have been implemented:
Control-based element.PipsPager(Implemented as:NewPipsPager)FlipView(Implemented asPanelSlideshow)This implementation of a
Slideshowelement added some customization whichFlipViewcan't accommodate, including:Transitionanimation for navigating between elements.SlideshowDuration.LayeredBackgroundImage)This custom control has major improvements than its previous background loading mechanism, including:
Built-in external decoder support for unsupported WIC (Windows Imaging Component) formats, including
.webp,.avifand.jxrviaPhotoScalerlibrary.Using direct Bitmap copy for drawing video frames via native
ISurfaceImageSourceNativeWithD2D.Previously, Collapse Launcher has to rely on software bitmaps to draw each frames into the
UIElementto pulls of the XAML-effects and compositions, including XAML Blur.The reason why Collapse Launcher has to perform this multiple frame copy is because
MediaPlayer, by default, doesn't support XAML effects and Composition and instead, each frames were drawn directly into different pipeline ofIDXGISwapChainwhich decoupled from the main WinUI's, causing the element above it feels a bit off (including no acrylic effects for any elements above it)This causes few overhead, in-terms of memory allocation or CPU cycles as it requires to perform multiple buffer copy steps:
SoftwareBitmap(CPU)MediaPlayer's frame buffer intoSoftwareBitmap(GPU -> CPU)VirtualCanvasImageSourceas source forImageto copy fromSoftwareBitmap(GPU)SoftwareBitmapbuffer intoVirtualCanvasImageSource(CPU -> GPU)Image's surface (GPU)Now, thanks to huge changes on
Hi3Helper.Win32submodule (and documentations provided by Win2D and DirectNAot), we managed to copy the frames fromMediaPlayerdirectly into backed-VirtualSurfaceImageSource/ISurfaceImageSourceNativeWithD2Dpipeline, loaded asImage's source, with XAML-effects and Composition support and without additional buffer copy on CPU side. Now, the process only require these steps:ISurfaceImageSourceNativeWithD2D->BeginDrawto createIDirect3DSurfacedrawing session (GPU) (Must be inside of UI Thread)MediaPlayer's frame buffer intoIDirect3DSurface(GPU -> GPU) (Must be inside of UI Thread)Image's surface viaISurfaceImageSourceNativeWithD2D->EndDraw(GPU) (Must be inside of UI Thread)(Update: 2026/01/13)
The call routines are now reduced. We are now reusing
IDirect3DSurfaceviaCanvasRenderTargetfor multiple draw routines instead of creating one every single time fromISurfaceImageSourceNativeWithD2D. Due to the nature ofCanvasRenderTargetbeing multi-threaded, we can now move the frame buffer copy routine to outside of the UI Thread. This significantly reduces the amount of overhead and thus, making the UI more responsive. The following call routines changed as follows:MediaPlayer's frame buffer intoIDirect3DSurface(GPU -> GPU) (Outside of UI Thread).ICanvasImageSource.CreateDrawingSession, Invalidate and draw buffers (GPU) (Must be inside of UI Thread).TODO List:
MediaPlayer,ISurfaceImageSourceNativeWithD2Dand underlayID3D11Device + ID2D1Device[Optional] Use fully-asynchronous drawing session onISurfaceImageSourceNativeWithD2D(Detaching draw session from UI-thread).Currently not possible due to SurfaceImageSource thread model bound to the UI-Thread and not set to Multi-thread mode.(Update: 2026/01/13) frame buffer copy routine can now be executed outside of the UI Thread. Though, buffers draw and invalidation still need to be called from inside of the UI Thread.PR Status :
Templates
Changelog Prefixes