EndoLRMGS

📝 Abstract

Complete reconstruction of surgical scenes is crucial for robot-assisted surgery (RAS). Deep depth estimation is promising but existing works struggle with depth discontinuities, resulting in noisy predictions at object boundaries and do not achieve complete reconstruction omitting occluded surfaces.

To address these issues we propose EndoLRMGS, that combines Large Reconstruction Modelling (LRM) and Gaussian Splatting (GS), for complete surgical scene reconstruction. GS reconstructs deformable tissues and LRM generates 3D models for surgical tools while position and scale are subsequently optimized by introducing orthogonal perspective joint projection optimization (OPjPO) to enhance accuracy.

🏗️ Architecture

Supplementary results

📋 Table of Contents

• Prerequisites
• Installation
• Dataset Preparation
• Usage
  • Inference
  • Training
  • Evaluation
• Visualization
• Citation
• Acknowledgements

🔧 Prerequisites

• Python 3.10
• CUDA 12.1 compatible GPU (tested on NVIDIA RTX A6000)
• Docker (recommended)
• Conda or Miniconda

📦 Installation

# Clone the repository
git clone https://github.com/your-username/EndoLRMGS.git
cd EndoLRMGS

# Create and activate Python environment
conda create -n endolrmgs python=3.10
conda activate endolrmgs

# Install PyTorch with CUDA 12.1 support
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu121

# Install project dependencies
pip install -r requirements.txt

# Install Gaussian Splatting submodules
cd FMGaussianSplatting
pip install submodules/depth-diff-gaussian-rasterization
pip install submodules/simple-knn
cd ..

# If you encountered error, like "ModuleNotFoundError: No module named 'torch'",
# please try this : pip install submodules/depth-diff-gaussian-rasterization --no-build-isolation

📂 Dataset Preparation

1. Download Datasets

Download the required surgical datasets:

• EndoNeRF Dataset
• SCARED Dataset
• StereoMIS Dataset
• Curated Hamlyn Dataset

2. Prepare Annotations

Follow DEVA to generate the required annotations and binary masks for surgical instruments. Use text-prompted :

python demo/demo_with_text.py --chunk_size 4 --img_path ./stereomis/p2_6/left_finalpass --amp --temporal_setting semionline --size 480 --output ./output/stereomis/p2_6 --prompt instruments.tools

3. Dataset Structure

Organize your dataset with the following structure:

endonerf/               # Root directory
├── pulling/
│   ├── Annotations/    # Surgical instruments color masks
│   ├── binary_mask_deva/ # Surgical instruments binary masks
│   ├── depth/         # Depth maps
│   └── images/        # Left rectified images
├── cutting_tissues_twice/
├── scared/
└── stereomis/

4. Pre-processed EndoNeRF Dataset

For reproducing our work, we provide a pre-processed EndoNeRF dataset that includes:

• Surgical instrument mask annotations
• Binary segmentation masks
• Depth maps
• Our reconstruction results

Download from: EndoLRMGS_endonerf

🚀 Usage

Inference

For EndoNerf pulling dataset:

python -m openlrm.launch infer.lrm \
    --infer ./configs/infer-b-endonerf-pulling.yaml \
    --model_name zxhezexin/openlrm-mix-base-1.1 \
    --image_input /workspace/datasets/endolrm_dataset/endonerf/pulling/images \
    --export_mesh true \
    --gaussian_config ./FMGaussianSplatting/arguments/endonerf/pulling.py

For EndoNerf cutting dataset:

python -m openlrm.launch infer.lrm \
    --infer ./configs/infer-b-endonerf-cutting.yaml \
    --model_name zxhezexin/openlrm-mix-base-1.1 \
    --image_input /workspace/datasets/endolrm_dataset/endonerf/cutting_tissues_twice/images \
    --export_mesh true \
    --gaussian_config ./FMGaussianSplatting/arguments/endonerf/cutting.py

For Stereomis dataset:

python -m openlrm.launch infer.lrm \
    --infer ./configs/infer-b-stereomis.yaml \
    --model_name zxhezexin/openlrm-mix-base-1.1 \
    --image_input /workspace/datasets/endolrm_dataset/stereomis/p2_6/left_finalpass \
    --export_mesh true \
    --gaussian_config ./FMGaussianSplatting/arguments/stereomis/stereomis_2_6.py

For Scared dataset:

python -m openlrm.launch infer.lrm \
    --infer ./configs/infer-b-scared.yaml \
    --model_name zxhezexin/openlrm-mix-base-1.1 \
    --image_input /workspace/datasets/endolrm_dataset/scared/dataset_6/data/left_finalpass \
    --export_mesh true \
    --gaussian_config ./FMGaussianSplatting/arguments/scared/d6k4.py

Training

To access LRM pretrained model "model.safetensors", please refer to https://huggingface.co/zxhezexin/openlrm-mix-small-1.1/tree/main and save the model in EndoLRMGS/checkpoint/mix-small For EndoNerf pulling dataset:

accelerate launch --config_file ./configs/accelerate-train.yaml \
    -m openlrm.launch train.lrm \
    --config ./configs/train-sample.yaml \
    --no-freeze_gaussian \
    --gaussian_config ./FMGaussianSplatting/arguments/endonerf/pulling.py

For EndoNerf cutting dataset:

accelerate launch --config_file ./configs/accelerate-train.yaml \
    -m openlrm.launch train.lrm \
    --config ./configs/train-sample.yaml \
    --no-freeze_gaussian \
    --gaussian_config ./FMGaussianSplatting/arguments/endonerf/cutting.py

For Stereomis dataset:

accelerate launch --config_file ./configs/accelerate-train.yaml \
    -m openlrm.launch train.lrm \
    --config ./configs/train-sample.yaml \
    --no-freeze_gaussian \
    --gaussian_config ./FMGaussianSplatting/arguments/stereomis/stereomis_2_6.py

For Scared dataset:

accelerate launch --config_file ./configs/accelerate-train.yaml \
    -m openlrm.launch train.lrm \
    --config ./configs/train-sample.yaml \
    --no-freeze_gaussian \
    --gaussian_config ./FMGaussianSplatting/arguments/scared/d6k4.py

Optimize surgical instruments via OPjPO

python postprocess_sequences_stereomis.py

Evaluation

To evaluate the reconstructed tissues and surgical instruments, use the following scripts:

Evaluate tissues

python ./evaluation/metrics.py

Evaluate surgical instruments

python postprocess_sequences_stereomis.py
python ./evaluation/tools_reprojection_loss.py

Visualization

To generate a spinning point cloud, please refer to another repo: https://github.com/MichaelWangGo/3D_spinning_visualization.git

Frequency-Modulated Gaussian Splatting

If you are only interested in FMGS, please refer to another repo: https://github.com/MichaelWangGo/Frequency-Modulated-Gaussian-Splatting.git

🙏 Acknowledgements

This project builds upon the following works:

• EndoGaussian
• Deformable-3D-Gaussians
• Mip-Splatting
• LRM
• OpenLRM
• DEVA