Vtk streaming encoding

[Justineantoine]

Handle video encoding using vtk-streaming lib

Changes:

• Creates protocol for RCA render schedulers
• Splits utils.py (still exposing the classes)
• Splits vtk_utils.py (still exposing the VtkWindow class)
• Splits test_rca_utils.py
• Creates tests for video encoding
• Moved tests dependencies in pyproject.toml
• Added class names to widgets enabling to overwrite CSS

[Justineantoine]

FYI @Thibault-Pelletier @jourdain I am not a fan of the "window" terminology (the interface that handles the interaction and resize events). I remember we briefly discussed this at some point last year, but if you have any new naming ideas, I'd be happy to hear them

[jourdain]

In general I like what you've done.

[Thibault-Pelletier]

Just an additional remark on this PR. vtkStreaming is currently leaking when resizing / rebuilding the encoder.
Should we wait for that to be resolved before merging this PR @jourdain ?

[Justineantoine]

Just an additional remark on this PR. vtkStreaming is currently leaking when resizing / rebuilding the encoder. Should we wait for that to be resolved before merging this PR @jourdain ?

with NVENC, VP9 seems to be ok

[jourdain]

Just an additional remark on this PR. vtkStreaming is currently leaking when resizing / rebuilding the encoder. Should we wait for that to be resolved before merging this PR @jourdain ?

with NVENC, VP9 seems to be ok

Do we know the origin of the leak?

[jourdain]

RCA should not force VTK as a dependency.
We should have an easy path to handle vtkRenderWindow but it should not be forced if we plan to use RCA with a custom image generator.

[Justineantoine]

I've changed some imports on this matter. I now can run the vanilla example without having vtk installed. But if you can test on your side that nothing is missing it would be nice

[AlexyPellegrini]

Just an additional remark on this PR. vtkStreaming is currently leaking when resizing / rebuilding the encoder. Should we wait for that to be resolved before merging this PR @jourdain ?

with NVENC, VP9 seems to be ok

Do we know the origin of the leak?

Working on it, but couldn't find anything useful for now. I would recommend not to use NVENC at all until I fix this...

New available vtk-streaming wheel, v0.3.2, fixes the leak.

[Justineantoine]

Ready to merge @jourdain @Thibault-Pelletier

[Thibault-Pelletier]

LGTM!

[jourdain]

Do you want to provide a default in case nothing is provided?

self.display = kwargs.get("display", "image")

[jourdain]

Also do you know that self.display already exist because of that kwargs.

So a better way of writting it could be

def __init__(self, display="image", **kwargs):
        super().__init__(
            "remote-controlled-area",
           display=display,
            **kwargs,
        )

[jourdain]

Looks good, but I guess I'll try it out before merging.

[Thibault-Pelletier]

On dual screen configuration, using video-decoder generates an image that can overflow (probably due to DPI or scaling ) screenshot below with screen at 125% scaling showing off center cone and overlow scroll PB:

[jspanchu]

yup, the JS code for VideoDecoder needs reviewed.

[Thibault-Pelletier]

Follow up on this, I think the problem on my side is that we apply the pixel ratio twice (size sent by the browser is already the scaled up version). Commenting out the pixel ratio apply solves the problem and it was just hidden by the JS scaling for the image encoder.

@jourdain do you concur or is there an OS specific thing that may be happening here?

    def update_size(self, origin, size):
        # Resize to ten pixel min to avoid rendering problems
        width = max(10, int(size.get("w", 300)))
        height = max(10, int(size.get("h", 300)))
        pixel_ratio = size.get("p", 1)
        self._current_size = (width, height, pixel_ratio)
        width = int(width * pixel_ratio * self._scale) # <-- incriminated lines
        height = int(height * pixel_ratio * self._scale)

[jspanchu]

I think the fps should not be hard-coded to 30. It gives an awful first impression for first-time users.

Instead, we could be smart about it by getting the client's estimated refresh rate and transmit that to the server.

let last = performance.now(), samples = [];
let fpsEstId;
function tick(now) {
  samples.push(now - last); last = now;
  if (samples.length >= 60) {
    cancelAnimationFrame(fpsEstId);
    const avg = samples.reduce((a,b)=>a+b)/samples.length;
    // ~1000/avg is the refresh rate; send it to the server via state?
  }
  fpsEstId = requestAnimationFrame(tick);
}
fpsEstId = requestAnimationFrame(tick);

[Thibault-Pelletier]

I think the fps should not be hard-coded to 30. It gives an awful first impression for first-time users.

Instead, we could be smart about it by getting the client's estimated refresh rate and transmit that to the server.

The FPS can be manually set by the application to the desired value. It's actually something we do in trame-slicer with the snapshot RCA to limit the bandwidth for non interactive views to 10 FPS (switch is done interactively and can be changed through parameters).

From my tests, 30 FPS works well in deployed applications on AWS with a reasonable bandwidth, but aiming higher may severely slow down the application as the application is not doing rendering only but also computation (although this bottleneck may be removed now that we have vtk streaming / NVENC).

For first time users, we can easily increase the values in the examples. We can also add an option to have a smart FPS, but I would like to also be able to set a fixed target FPS and being able to toggle that from the application layer.

EDIT:
Since RCA is one of many ways to have a trame app running, we can orient users wanting smoother feel to trame-vtk local / VTK WASM.

RCA will remain a bit more sluggish and add in some latency as you would have using any VNC / cloud streaming plateform but from my tests it remains a much nicer experience than using a desktop application (3D Slicer) through an AWS Appstream service.