generate_dataset Fails After Successful Teleoperation and Annotation With Custom Robot Model

[Yuchili2022]

I encountered some issues while working on Teleoperation and Imitation Learning with Isaac Lab Mimic, and I would like to discuss possible improvements.

1. Custom robot model configuration
   Since I am using my own robot model, I modified the following environment configurations to match my setup:
   • stack_joint_pos_env_cfg
   • stack_ik_real_env_cfg
   • stack_ik_real_mimic_env
   • stack_ik_real_mimic_env_cfg
   Additionally, because my gripper has been replaced, I also updated the "observation" settings in the stack task accordingly.

2. Teleoperation and demonstration recording are working correctly
   The following steps can all be completed successfully:
   • I can run teleop_se3_agent.py and the robot performs correct Cartesian motions at the TCP, including both translation and rotation.
   • record_demos works and I can collect 10 successful demonstrations that satisfy the success conditions.
   • annotate_demos also runs without issues.

3. The issue occurs when running generate_dataset
   When I run generate_dataset, it always results in failures.
   Based on the documentation, high quality demonstrations should follow these principles:
   • Keep demonstrations short
   • Use direct motion paths
   • Avoid pauses and maintain smooth, continuous movement

Based on this, I suspect the following issues in my demonstrations:
a. Keyboard input makes smooth motions difficult
Using the keyboard often results in discrete, non smooth movements or unintended pauses.
b. I need to open two fixed views (front + right) to perform accurate demonstrations
Currently, I open both a front view and a right view simultaneously to better align with objects.
Although this helps with accuracy, it complicates the teleoperation process and may reduce overall smoothness.

Questions I would like to ask:

1. How well does the SpaceMouse work for Mimic demonstrations?
   Does it significantly improve the smoothness and quality of the teleoperation process?
2. The official documentation also mentions CloudXR Teleoperation.
   Would you recommend directly using CloudXR for this type of task?
3. Are there recommended camera/view configurations in Isaac Lab Mimic that can reduce visual difficulties during demonstrations?

Thank you!