pygfx backgrounds in multiple scenes

[gselzer]

Suppose you want to display a Scene through two different Views, each with a different background color. This is currently not possible, because the pygfx View adaptor adds a background to the underlying scene. Thus if that scene is part of two different views it will have two different backgrounds.

The fix is probably for each pygfx view to have its own pygfx scene which has as children:

• a background
• the Scene adaptor it is supposed to display

[gselzer]

Okay, so I tried this:

import numpy as np

import scenex as snx
from scenex.utils.projections import zoom_to_fit

# --- Build the shared scene ---
rng = np.random.default_rng(0)
image = snx.Image(data=rng.random((64, 64), dtype=np.float32), clims=(0.0, 1.0))
shared_scene = snx.Scene(children=[image])

# --- Create two views that share the same scene ---
view_left = snx.View(
    scene=shared_scene,
    camera=snx.Camera(controller=snx.PanZoom(), interactive=True),
)
view_right = snx.View(
    scene=shared_scene,
    camera=snx.Camera(controller=snx.PanZoom(), interactive=True),
)

# Give tcontroller=snx.PanZoomhe views different background colors
view_left.layout.background_color = "navy"   # should be dark blue
view_right.layout.background_color = "darkred"  # should be dark red

# Split the canvas in two halves
view_left.layout.x_end = view_right.layout.x_start = "50%"

canvas = snx.Canvas(width=800, height=400, views=[view_left, view_right])

print(__doc__)
print("Left view background:  navy    (expected dark blue)")
print("Right view background: darkred (expected dark red)")
print(
    "Bug: both views will share the same background color because they share "
    "the same pygfx Scene."
)

snx.show(canvas)
zoom_to_fit(view_left)
zoom_to_fit(view_right)
snx.run()

On main, both views show dark red backgrounds, as stated in the issue.

The fix is probably for each pygfx view to have its own pygfx scene which has as children:

* a background

* the `Scene` adaptor it is supposed to display

So, when I try this, it doesn't work, because a Scene can only have one parent (and this "fix" would ask for the Scene to have two). So, with the current state of pygfx, I don't think it's possible to display two different scenes with different backgrounds, because the background is a part of the scene?

@almarklein any thoughts on how we could support something like this?

[almarklein]

I don't see how it would be possible with the current state of PyGfx.

Out of interest, is there a special use-case for wanting to show the same scene with different backgrounds? I do think it represents a more general case where you want a certain (sub)scene to be shown in different conditions.

This does touch on a very fundamental design choice. You can allow multiple parents (as IIRC vispy allows), but this adds a lot of complexity. I've also experimented with parent-less objects, where an object can be a child of multiple parents, because the child objects are simply unaware of their parents. This is pretty slick, but would not allow for a transform system like we have in PyGfx.

One possible solution (that I just made up, and have not idea how well it would work), is to have something like gfx.SceneGroup, that can contain multiple scenes, but each scene by itself is considered toplevel.

Maybe @Korijn also has ideas

[Korijn]

When all you have is a scene graph, everything looks like a nail :)

This is typically solved with render passes. You set up three scenes. One for each background (let's call them A and B) and one for the scene (let's call it S).

Then you render A, S, B, S. Vary the camera and renderer settings between passes to set up the views.

[gselzer]

Thank you @almarklein and @Korijn for responding!

Out of interest, is there a special use-case for wanting to show the same scene with different backgrounds? I do think it represents a more general case where you want a certain (sub)scene to be shown in different conditions.

Nope! I just realized this would be a limitation of the current design in scenex. We do want to show the same scene with different cameras (for things like volume orthoviews), but probably not with different background colors.

One possible solution (that I just made up, and have not idea how well it would work), is to have something like gfx.SceneGroup, that can contain multiple scenes, but each scene by itself is considered toplevel.

Hmm, so how would that work in this situation? Scene 1 has the scene I want to display (including the camera), and Scene 2 has the background (and another Camera?)

This is typically solved with render passes.

Right, this was the approach I was thinking I'd take if I need this down the line. Mostly wanted to make sure that there wasn't an obvious better way I was missing. Should there be much of a performance hit when rendering double the scenes?

I think for the moment I'll just log a warning if the user wants to do this type of thing. Can consider the multi-pass rendering later if there's a real need.

[Korijn]

Should there be much of a performance hit when rendering double the scenes?

Well, there is some overhead to having multiple render passes because you have to wait for the GPU to finish a pass before you can issue the next. Ideally you avoid as much CPU time as you can. There are many ways to achieve that (e.g. "indirect drawing", or "GPU rendering") but pygfx does not offer a mechanism to "prepare" render passes for sequential execution on the GPU at the moment.

[almarklein]

[...] you have to wait for the GPU to finish a pass before you can issue the next.

I think you actually can. The CPU simply submits commands to the GPU, and there are just a few places when it syncs, flushing to the screen is one (potential, depending on backend) sync-point. So rendering twice before doing a renderer.flush() has some overhead, but should not have any extra waiting for the GPU.

Hmm, so how would that work in this situation? Scene 1 has the scene I want to display (including the camera), and Scene 2 has the background (and another Camera?)

So to entertain the SceneGroup idea:

sg1 = gfx.SceneGroup(background1, shared_scene)
sg2 = gfx.SceneGroup(background2, shared_scene)

# Then use each scene group in the different views.

In contrast to using multiple render passes, all objects would be ordered appropriately, so you could use overlays, mix transparent objects, etc.