Fix black screen and inverted R/B colours when streaming via Steam Remote Play on NVIDIA

[cmspam]

Fix black screen and inverted colours when streaming via Steam Remote Play on NVIDIA

On the NVIDIA proprietary driver, two issues affect PipeWire streaming (e.g. Steam Remote Play).

Black screen: Screenshot textures were created without bSampled = true, so no sampled image view was created for the RGB-to-NV12 conversion shader, resulting in a black frame. Fixed by adding bSampled = true in vulkan_acquire_screenshot_texture(). This is safe on all hardware.

Inverted R/B channels: Sampling a VK_FORMAT_A2R10G10B10_UNORM_PACK32 storage image on the NVIDIA proprietary driver returns R and B swapped. The RGB-to-NV12 colour matrix then operates on the wrong channels, causing the remote client to display inverted red and blue. Fixed by using DRM_FORMAT_XBGR2101010 as the intermediate capture format on NVIDIA, compensating for the driver's component ordering.

The colour fix is gated on VK_DRIVER_ID_NVIDIA_PROPRIETARY and has no effect on Nouveau, NVK, or any other vendor. A new isNvidiaProprietaryDriver() accessor is added to CVulkanDevice, populated via VkPhysicalDeviceDriverProperties chained into the existing GetPhysicalDeviceProperties2 call in createDevice().

Tested with NVIDIA RTX 4080, with Steam Remote Play in Game Mode (using Jovian) to Steam Link on an iPhone (PipeWire NV12 + NVENC AV1).

[nauzethc]

I can confirm both problems are fixed (black screen and inverted colors) but introduces double inputs from client side. I tested Steam Remote Play from Steam Deck, D-Pad and buttons are send twice to remote host (in-game interactions) while the controller connected directly to the host works correctly.

[cmspam]

I can confirm both problems are fixed (black screen and inverted colors) but introduces double inputs from client side. I tested Steam Remote Play from Steam Deck, D-Pad and buttons are send twice to remote host (in-game interactions) while the controller connected directly to the host works correctly.

Are you sure this is a new problem introduced by this patch? It doesn't touch input at all, so I think you will likely have the same issue without this patch. Could you please test it without the patch (it will have a black screen, yes, but inputs still go through) to make sure this patch introduces the issue?

I don't have a steam deck to test with, but I tested with a nintendo switch controller paired to my iPhone, logged controller inputs, and didn't see any double inputs in that case, so this may be something steam deck-specific. Do you have any other clients you can try streaming to?

[DeNo64]

No, change for me at all.

Not getting the double inputs like nauzethc. But also still seeing back frame.
Potentially something different on the nvidia-open-proprietry driver?

GPU: Nvidia 2080 Super
Driver: nvidia-open-dkms 590.48.01-4
DM: Hyprland (Wayland)
Reported Encoding: Desktop Black Frame + NVENC HEVC
Tested Clients: Steam Deck & Pixel 6 Pro (Steam Link Android)

[cmspam]

No, change for me at all.

Not getting the double inputs like nauzethc. But also still seeing back frame. Potentially something different on the nvidia-open-proprietry driver?

GPU: Nvidia 2080 Super Driver: nvidia-open-dkms 590.48.01-4 DM: Hyprland (Wayland) Reported Encoding: Desktop Black Frame + NVENC HEVC Tested Clients: Steam Deck & Pixel 6 Pro (Steam Link Android)

I am on the same driver you are. Your post mentions that you are using Hyprland.
This fix targets not using a desktop environment or a traditional compositor like Hyprland, but instead using gamescope-session (Steam's Game Mode) as the Wayland compositor.

I also get a black screen when trying to stream with KDE, but I believe this is a different issue regarding wayland security, as I also have a black screen on a different PC with AMD graphics in this case. The issue of streaming from a desktop environment (which I think you are describing) can be worked around by running steam with the -pipewire flag and allowing steam to share the desktop via pipewire.

Have you tested using gamescope-session?

[DeNo64]

Okay, using your gamescope patch from a TTY does work and I don't seem to be getting the double inputs either. Which is great.

Can this patch also be applied to the "Game Vulkan NV12" encoder, for using in a Nested Gamescope session?

The encoding performance of 'Desktop PipeWire NV12' is significantly worse. Which I assume is related to the graphical flickering I'm experiencing on the host (not visible on client) relating to some combination of my setup, nvidia driver issue, and/or another gamescope issue on nvidia (not related to this patch)

[cmspam]

Can this patch also be applied to the "Game Vulkan NV12" encoder, for using in a Nested Gamescope session?

I don't think the Game Vulkan NV12 encoder has any similar issues that need to be patched. When running a game in steam it switches from the "Black Screen" encoder to the "Game Vulkan NV12" encoder, at which point I was able to see it on my phone.

But I may be misunderstanding what you're asking. Could you be really specific?

[nauzethc]

Replying to #2094 (comment)

I used Plasma Desktop under Wayland and I can confirm that it works without getting black screen and with the colors displayed correctly. I'll test with the master version from the repository to see if the input problem occurs, but I seem to recall that it didn't happen in previous tests. It's certainly strange.

[DeNo64]

Can this patch also be applied to the "Game Vulkan NV12" encoder, for using in a Nested Gamescope session?

I don't think the Game Vulkan NV12 encoder has any similar issues that need to be patched. When running a game in steam it switches from the "Black Screen" encoder to the "Game Vulkan NV12" encoder, at which point I was able to see it on my phone.

But I may be misunderstanding what you're asking. Could you be really specific?

Are you using gamescope as a launch options for the game you're streaming from steam?
This is the issue called out in #1596. I can stream games fine over Remote Play without Gamescope. But if I want to run my game in gamescope (for better HDR, as an example) and stream via Remote Play the game will stay on 'Black Frame NV12'

[cmspam]

Replying to #2094 (comment)

As far as I can tell, anything running on Wayland, including nested gamescope sessions, can't be captured by Steam unless pipewire is utilized or Steam itself is running inside gamescope. This PR only fixes an NVIDIA-specific issue with streaming when Steam is already running inside a gamescope session — it doesn't address the broader nested session capture problem.

As far as I can tell, this is a Steam/Wayland compositor limitation rather than a gamescope-specific one, and PipeWire is likely the only supported path forward for that. I could be wrong though.
I've also removed the reference to resolving #1596 from the PR — you're right that while the NVIDIA bug was discussed there, this doesn't address the original intent of that issue. Thanks for pointing that out.

That said, there is a workaround for streaming nested sessions — run Steam itself inside gamescope:

gamescope -W 3840 -H 2160 -r 120 -f --hdr-enabled -- steam

From there I was able to successfully stream games with HDR and so on.

[DeNo64]

Okay, this is the aspect I was missing. Running Steam within the gamescope window and then launching the game is now displaying the stream, with all the correct colours, and is using PipeWire NV12. (As opposed to just a black screen - still using PipeWire NV12 on v3.16.20-1 of gamescope).
So that's good.

Testing again under steam -pipewire, using the current gamescope 3.16.20-1 as the launch option for the game, the game is displayed correctly (although with the bad Pipewire performance I'm getting).
With your patched version of gamescope in the launch options, all I get is the generic game splashscreen (with the repeated game poster) which shows while it's setting up the connection + audio from the game.

I'll keep playing around with this tomorrow and see if I can figure out what's going on.

[cmspam]

Thank you for testing it out so thoroughly.

The patch really only makes two changes-- adding bSampled = true in vulkan_acquire_screenshot_texture(), and inverting the red and blue if the hardware is nvidia.

I don't think those changes would cause any technical differences in pipewire streaming, in terms of whether it works or not, so it's interesting that you are experiencing them. Please let me know if you find out anything else, I'm happy to adjust the changes in this PR if they can be made better somehow.

Although I may not have the same game or hardware, if you can list the game you're trying it with and the exact command used to run it, I can try to recreate it.

[nauzethc]

I've run several tests today with different parameters and code versions, and I think I can conclude that the double input problem is related to my setup or GameScope configuration. For some reason I can't remember, I was including the -pipewire-dmabuf parameter, and it seems to be introducing this problem (sometimes on the host and sometimes on the client).

Now everything seems to be working correctly. Thank you so much!

[lukewilde]

there is a workaround for streaming nested sessions — run Steam itself inside gamescope

Can confirm this patch is working for me if I run steam with the following params:

gamescope -W 2560 -H 1440 -e -- steam -pipewire -tenfoot