Relightable 3DGS for Unity

This repository is a Unity implementation of Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing (ECCV 2024), built on top of UnityGaussianSplatting.

Usage

1. Train a relightable 3DGS asset using the upstream repository. On a 3090 Ti, this step usually takes several hours.

   • Supplemental scripts for data preparation and conversion are available in train_utils.
   • You can use either real-world photos or images captured from a fully synthetic scene.
   • Run COLMAP and remove_bg.py to prepare the training data.
   • See train_utils/img_to_r3dg_cmd.md for an example workflow. In practice, taking 100+ images from a wide range of well-lit viewpoints usually works best.
   • Here’s a [link][https://drive.google.com/file/d/1Ha3MQ8T2Abda9H3Jh8WYkmU9YBi5bOsq/view?usp=sharing] to four relightable gaussians I trained on Synthetic4Relight dataset.

2. A relightable 3DGS stores the following additional PBR-related attributes:

   • Base color
   • Roughness
   • Incident SH
   • Visibility SH

3. In Unity, import the relightable Gaussian file via Tools -> Gaussian Splats -> Create GaussianSplatAssets.

   • Enable export relight data .

In the original implementation, visibility is computed on the fly using real-time ray tracing. In this Unity port, I bake the visibility term offline for simpler integration and better runtime performance. The baking script is available in train_utils, although the current implementation still has known issues.

4. In the Inspector, you can assign an arbitrary environment map to preview the relighting result. I also added a few heuristics to make the renderer more stable in low-light scenes.

A Gaussian chair under a pink-sky HDRI. At the moment, the HDRI affects only the Gaussian splats.

5. To inspect different lighting components, edit the GS_DEBUG_LIGHT_MODE macro in package/Shaders/SplatUtilities.compute.

This image shows the baked incident SH visualized on the Gaussian splats. Because it is baked per Gaussian, it does not change under different lighting conditions.

6. You can also generate a proxy mesh for the Gaussian splats, which is mainly useful for shadow casting. This is done through stride-based downsampling. In practice, a budget of around 30k primitives usually provides a reasonable approximation.

Known Limitations

• This project uses a baked visibility term instead of the ray-traced visibility used in the original implementation, so approximation errors are expected.
• Incident SH is baked per Gaussian, so the result is not fully physically accurate.

References

• aras-p/UnityGaussianSplatting
• Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing (ECCV 2024)