Loads real2sim asset in locomanipulation SDG

[dengyuchenkit]

Description

This PR adds support to load a USD background in locomanipulation SDG pipeline.

1. Load NuRec asset in locomanipulation SDG script.
2. Instantiates the first table in the freespace of the background occupancy map. It resets robot/object position according to table position. It also writes the default state for robot/object state, so these states are correct when resetting scene due to termination.
3. Termination conditions:
   • Add a termination condition object_too_far_from_robot, to check if robot picks up the wheel.
   • Add success termination check on task completion, and write successful episodes. Use object relative pose to the table for success check.

It was tested by:

Generate locomanipulation SDG data with the hand_hold-endeavor-wormhole-2 dataset at: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/tree/main

./isaaclab.sh -p scripts/imitation_learning/locomanipulation_sdg/generate_data.py \
   --device cpu \
   --kit_args="--enable isaacsim.replicator.mobility_gen" \
   --task="Isaac-G1-SteeringWheel-Locomanipulation" \
   --dataset ./datasets/generated_dataset_g1_locomanip.hdf5 \
   --num_runs 1 \
   --lift_step 60 \
   --navigate_step 130 \
   --enable_pinocchio \
   --output_file ./datasets/generated_dataset_g1_locomanipulation_sdg.hdf5 \
   --enable_cameras \
   --background_usd_path <PATH_TO_USD_ASSET>/stage.usdz \
   --background_occupancy_yaml_file <PATH_TO_USD_ASSET>/occupancy_map.yaml

Type of change

• New feature (non-breaking change which adds functionality)

Screenshots

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Summary

This PR adds support for loading USD background scenes in the locomanipulation SDG pipeline. It enables instantiating packing tables within background freespace and projecting initial robot/object states from input episodes into the environment.

Key changes:

• Added command-line arguments for --background_usd_path, --background_occupancy_yaml_file, and --high_res_video
• Refactored scene setup to conditionally handle background assets vs. default environment
• Implemented state projection functions to transform robot and object poses into the new environment
• Enhanced navigation scene setup with fixture randomization in background freespace
• Added termination condition for detecting failed object pickups (object_too_far)
• Improved occupancy map handling to treat unknown areas as occupied
• Added bounds checking for path planning endpoints

Critical Issues Found:

• Multiple functions (project_robot_state_into_env, project_object_state_into_env) reference global variable input_episode_data without it being passed as a parameter, causing NameError at runtime
• NavigationScene class lacks @dataclass decorator but is instantiated with keyword arguments
• Argparse type=bool doesn't work correctly - should use action="store_true"
• Background path check uses != "" instead of is not None for None default values

Confidence Score: 0/5

• This PR contains critical bugs that will cause runtime failures
• Multiple NameError exceptions will occur due to missing function parameters, NavigationScene instantiation will fail without @dataclass, and argparse boolean handling is broken. These are blocking issues that prevent the feature from working.
• Pay close attention to scripts/imitation_learning/locomanipulation_sdg/generate_data.py which contains all the critical bugs

Important Files Changed

Filename	Overview
scripts/imitation_learning/locomanipulation_sdg/generate_data.py	Multiple critical bugs: global variable usage without parameters, incorrect argparse type, missing @dataclass decorator, and wrong None check
source/isaaclab_mimic/isaaclab_mimic/locomanipulation_sdg/envs/g1_locomanipulation_sdg_env.py	Added background asset loading and camera configuration methods, restructured scene setup to be conditional on background asset presence
source/isaaclab_mimic/isaaclab_mimic/locomanipulation_sdg/occupancy_map_utils.py	Changed occupancy mask logic to treat unknown areas as occupied and refactored bounds checking into separate method
source/isaaclab_mimic/isaaclab_mimic/locomanipulation_sdg/path_utils.py	Added bounds checking for end point pixels before path planning to prevent index out of bounds errors

Sequence Diagram

sequenceDiagram
    participant User
    participant Main
    participant Env as G1LocomanipulationSDGEnv
    participant Scene as setup_navigation_scene
    participant Fixtures as SceneFixtures
    participant Occupancy as OccupancyMap
    participant PathPlanner as plan_path
    participant StateProj as project_*_state_into_env
    
    User->>Main: Run with --background_usd_path & --background_occupancy_yaml_file
    Main->>Env: Create environment with cfg
    Env->>Env: add_background_asset(background_usd_path)
    Env->>Env: add_robot_pov_cam(height, width)
    Env->>Env: _setup_background_mesh()
    Note over Env: Hide mesh, enable collision
    
    Main->>Scene: setup_navigation_scene(env, input_episode_data)
    Scene->>Env: get_background_fixture()
    Env-->>Scene: background_fixture with occupancy_map
    
    alt Background exists
        Scene->>Occupancy: Get background occupancy map
        Scene->>Fixtures: Randomize start, end, obstacles in freespace
        loop For each fixture
            Fixtures->>Fixtures: place_randomly()
            Fixtures->>Scene: sync_simulation_state()
            Scene->>Occupancy: merge_occupancy_maps()
        end
    else No background
        Scene->>Occupancy: Create empty occupancy map
        Scene->>Fixtures: Randomize end and obstacles only
    end
    
    Scene->>StateProj: project_robot_state_into_env()
    StateProj->>Env: Transform and write robot pose/joints
    Scene->>StateProj: project_object_state_into_env()
    StateProj->>Env: Transform and write object pose
    
    Scene->>PathPlanner: plan_path(start, end, occupancy_map)
    PathPlanner-->>Scene: base_path (waypoints)
    
    alt Path valid (>2 points)
        Scene-->>Main: NavigationScene
        Main->>Main: Execute state machine with navigation
    else Path invalid
        Scene-->>Main: None (failure)
        Main->>Main: Retry with different demo
    end

Loading

[greptile-apps]

_{9 files reviewed, 6 comments}

_{Edit Code Review Agent Settings | Greptile}

[kellyguo11]

please run the formatter with ./isaaclab.sh -f

[dengyuchenkit]

Done

[jaybdub]

Hi @dengyuchenkit ,

Thanks for the PR! It looks great.

Have you tested the existing G1 locomanipulation data generation to ensure that it still works with these added changes?

Best,
John

[dengyuchenkit]

Hi @jaybdub , I tested the existing workflow and it works fine!

[gavrielstate]

It feels like this should already be set as a collision mesh in the background USD, rather than having code here to force it.

A background USD might or might not include a collision mesh, and if it does, it could be somewhere other than under /volume/mesh depending on how it was generated. We should take care with assumptions about the specific background USD that gets used here.

[dengyuchenkit]

I removed "forcing collision mesh" logic and harcoded /volume/mesh path from the code. The huggingface assets are updated to have collision mesh enabled.
I added in the document that, if using a custom USD file, it should include neural reconstruction for rendering and collision mesh for interactions. Is it accurate to say that?

[jaybdub]

This PR looks great to me and adds features that we need for IsaacLab 3.0

• Quaternion re-ordering
• Proper task terminations

@kellyguo11 @gavrielstate Would it be possible for this PR to be merged before the 02/26 cutoff? I had an existing PR #4725 but encountered issues while rebasing, so was wondering if we can instead use this.

[peterd-NV]

The changes to the termination here is significant and looks targeted towards the G1 locomanip env specifically.

The pickplace base env is shared with the GR1 for a variety of other tasks. Can we leave the existing termination and GR1 envs alone and create a separate termination event with these changes for the G1?

[dengyuchenkit]

sure, I separated the termination event for G1 in the latest patch. The changes don't affect GR1 tasks now.

[dengyuchenkit]

Actually, there are some reverting errors. Let me fix those

[dengyuchenkit]

Fixed the reverting errors.

[peterd-NV]

See comment in terminations.py. It would be good if we can leave this alone for the GR1 as there are a number of existing envs and datasets that are already published using the existing termination function.

[dengyuchenkit]

Done