Clarification and improvement request regarding missing RGB videos in the depth camera modality

[GJCav]

Hello authors, thank you for releasing OctoNet, which is a very valuable multi-modal sensing dataset, and we appreciate the effort put into making it public.

While exploring the dataset and running the provided demo.ipynb, we noticed that RGB video data from the depth camera is not included, and only the depth channel is published. As a result, the demo code raises warnings/errors such as:

[ WARN:0@1035.404] global cap.cpp:175 open VIDEOIO(CV_IMAGES): raised OpenCV exception:

OpenCV(4.13.0) /io/opencv/modules/videoio/src/cap_images.cpp:293:
error: (-215:Assertion failed) !_filename.empty() in function 'open'

The warnings/errors originate from dataset_loader.py, line 811-812:

# 3) Read the video into a list of frames
cap = cv2.VideoCapture(video_path)
original_fps = cap.get(cv2.CAP_PROP_FPS)

where video_path is None as the previous stage 1) Gather PNG depth files + an MP4 file could not find any .mp4 file (same Python file, line 781-789):

# 1) Gather PNG depth files + an MP4 file
depth_image_files = []
video_path = None
for root, dirs, files in os.walk(data_path):
    for file in files:
        if file.endswith('.png') and 'depth' in file:
            depth_image_files.append(os.path.join(root, file))
        elif file.endswith('.mp4'):
            video_path = os.path.join(root, file)

This makes the demo somewhat confusing for first-time users and introduces unnecessary friction when reproducing the examples.

We fully understand that RGB videos may contain personal or sensitive information and that releasing them could violate IRB or privacy regulations. That said, we kindly suggest the following improvements to enhance clarity and usability:

1. Explicit documentation:
   Please explicitly state in the README (both on GitHub and Hugging Face, if applicable) that:

   • RGB videos from the depth camera are intentionally not released
   • The reason (e.g., IRB/privacy constraints)

2. Demo code adjustment
   Updating relevant codes to:

   • Gracefully handle the absence of RGB videos, or
   • Clearly warn users that RGB data is unavailable

   would greatly improve the user experience.

3. Optional: privacy-preserving RGB alternatives
   If feasible, we would kindly encourage providing a privacy-preserving version of the RGB videos, for example, Face-blurred or body-silhouetted videos. Many public datasets successfully release RGB data in this manner, such as, NTU RGB+D, EPIC-KITCHENS, and Ego4D. Even a limited or processed RGB subset would significantly improve the dataset’s usability and reproducibility.

Finally, we hope this issue can also serve as a progress tracker for the availability (or planned availability) of RGB video data, so that interested users can follow updates in one place.

Thank you again for your excellent work and for considering these suggestions!