added scene cameras for cosmos by LJones-alt · Pull Request #6 · aarch-uol/IsaacLab-Laura

LJones-alt · 2026-03-25T11:45:44Z

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
List any dependencies that are required for this change.

Fixes # (issue)

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (existing functionality will not work without user modification)
Documentation update

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist

I have read and understood the contribution guidelines
I have run the pre-commit checks with ./isaaclab.sh --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
I have updated the changelog and the corresponding version in the extension's config/extension.toml file
I have added my name to the CONTRIBUTORS.md or my name already exists there

Copilot

Pull request overview

This PR appears aimed at enabling “cosmos”-style visuomotor observations by adding/expanding scene camera outputs (segmentation / normals / depth) and aligning some config imports.

Changes:

Reformats/extends camera observation terms for cosmos-style inputs (segmentation, normals, depth).
Updates several dev config files’ import paths and removes some commented imports.
Adjusts termination behavior/logging (disables a joint-limit termination in one env cfg; adds a debug print in joint-limit termination).

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/config/franka/stack_ik_rel_visuomotor_cosmos_env_cfg.py	Reformats camera normals ObsTerm params (but currently not pre-commit clean).
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/cube_lift/lift_env_cfg.py	Comments out `joint_violation` termination (behavior change).
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_vial_insert_top_down.py	Removes a commented import (but leaves unused imports).
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_vial_insert.py	Removes a commented import (but leaves unused imports).
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_skillgen.py	Switches UR10 import path (but import block contains unused imports).
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_env_place_vismot.py	Adds camera datatypes + attempts to add additional image ObsTerms for cosmos.
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_env_place.py	Removes a commented import.
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_env_insert_vismot.py	Switches UR10 import path (but import block contains unused imports).
source/isaaclab/isaaclab/envs/mdp/terminations.py	Adds debug `print()` when joint limits are violated.

Copilot · 2026-03-25T11:52:08Z

+        self.observations.table_cam_segmentation= ObsTerm(
+            func = mdp.image,
+            params = {
+                "sensor_cfg": SceneEntityCfg("table_cam"),
+                "data_type": "semantic_segmentation",
+                "normalize" : True,
+            }
+        )
+
+        self.observations.table_cam_normals= ObsTerm(
+            func=mdp.image,
+            params={
+                "sensor_cfg": SceneEntityCfg("table_cam"),
+                "data_type" : "normals",
+                "normalize" : True,
+            }
+        )
+
+        self.observations.table_cam_depth = ObsTerm(
+            func=mdp.image,
+            params={
+                "sensor_cfg": SceneEntityCfg("table_cam"),
+                "data_type": "distance_to_image_plane",
+                "normalize" : True,
+            }


These new image ObsTerms are assigned onto self.observations instead of the policy observation group (e.g., self.observations.policy.<term>). As written they won’t be part of the policy observation group (and may raise an attribute error depending on how configclass is implemented). Please attach them under self.observations.policy like the existing table_cam / wrist_cam terms.

Suggested change

self.observations.table_cam_segmentation= ObsTerm(

func = mdp.image,

params = {

"sensor_cfg": SceneEntityCfg("table_cam"),

"data_type": "semantic_segmentation",

"normalize" : True,

}

)

self.observations.table_cam_normals= ObsTerm(

func=mdp.image,

params={

"sensor_cfg": SceneEntityCfg("table_cam"),

"data_type" : "normals",

"normalize" : True,

}

)

self.observations.table_cam_depth = ObsTerm(

func=mdp.image,

params={

"sensor_cfg": SceneEntityCfg("table_cam"),

"data_type": "distance_to_image_plane",

"normalize" : True,

}

self.observations.policy.table_cam_segmentation = ObsTerm(

func=mdp.image,

params={

"sensor_cfg": SceneEntityCfg("table_cam"),

"data_type": "semantic_segmentation",

"normalize": True,

},

)

self.observations.policy.table_cam_normals = ObsTerm(

func=mdp.image,

params={

"sensor_cfg": SceneEntityCfg("table_cam"),

"data_type": "normals",

"normalize": True,

},

)

self.observations.policy.table_cam_depth = ObsTerm(

func=mdp.image,

params={

"sensor_cfg": SceneEntityCfg("table_cam"),

"data_type": "distance_to_image_plane",

"normalize": True,

},

Copilot · 2026-03-25T11:52:09Z

+    if torch.any(out_of_upper_limits | out_of_lower_limits):
+        print("Joint position out of limit", asset.data.joint_pos[:, asset_cfg.joint_ids])


Debug print() inside this termination will spam logs and can significantly slow down large batched envs (and forces a device sync for the if torch.any(...)). Prefer structured logging (e.g., at debug level) and log only the violating env indices / min-max values, or remove this side-effect entirely.

Suggested change

if torch.any(out_of_upper_limits | out_of_lower_limits):

print("Joint position out of limit", asset.data.joint_pos[:, asset_cfg.joint_ids])

Copilot · 2026-03-25T11:52:09Z

 from isaaclab.assets import AssetBaseCfg
 from isaaclab.sim.spawners.from_files import UsdFileCfg
 from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR
-from source.isaaclab_assets.isaaclab_assets.robots.universal_robots import UR10_CFG
+from isaaclab_assets.robots.universal_robots import UR10_CFG
 from . import dev_env_cfg


Unused imports (AssetBaseCfg, UsdFileCfg, UR10_CFG) will be flagged by flake8 F401 now that this file is modified. Please remove them or add # noqa: F401 where truly needed.

Copilot · 2026-03-25T11:52:09Z

 from isaaclab.assets import AssetBaseCfg
 from isaaclab.sim.spawners.from_files import UsdFileCfg
 from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR, NVIDIA_NUCLEUS_DIR
-from source.isaaclab_assets.isaaclab_assets.robots.universal_robots import UR10_CFG
+from isaaclab_assets.robots.universal_robots import UR10_CFG
 from . import dev_env_cfg


Unused imports (AssetBaseCfg, UsdFileCfg, UR10_CFG) will be flagged by flake8 F401 once this file is changed. Please remove them or add # noqa: F401 if they are intentionally kept for future use.

Copilot · 2026-03-25T11:52:09Z

 from isaaclab.assets import AssetBaseCfg
 from isaaclab.sim.spawners.from_files import UsdFileCfg
 from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR, NVIDIA_NUCLEUS_DIR
-#from source.isaaclab_assets.isaaclab_assets.robots.universal_robots import UR10_CFG
 from isaaclab_assets.robots.universal_robots import UR10_CFG
 from . import dev_env_cfg


Unused imports (AssetBaseCfg, UsdFileCfg, UR10_CFG) will be flagged by flake8 F401 now that this file is modified in this PR. Please remove them or mark with # noqa: F401.

Copilot · 2026-03-25T11:52:10Z

 from isaaclab.assets import AssetBaseCfg
 from isaaclab.sim.spawners.from_files import UsdFileCfg
 from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR, NVIDIA_NUCLEUS_DIR
-from source.isaaclab_assets.isaaclab_assets.robots.universal_robots import UR10_CFG
+from isaaclab_assets.robots.universal_robots import UR10_CFG
 from . import dev_env_cfg


This module imports AssetBaseCfg, UsdFileCfg, and UR10_CFG but they are unused in the file, which will trigger flake8 F401 once this file is modified. Please remove unused imports or mark them with the appropriate # noqa: F401.

Copilot · 2026-03-25T11:52:10Z

 ##
 from isaaclab_assets.robots.franka import FRANKA_PANDA_HIGH_PD_CFG  # isort: skip
-from isaaclab_assets.robots.universal_robots import UR10e_ROBOTIQ_GRIPPER_CFG
+#from isaaclab_assets.robots.universal_robots import UR10e_ROBOTIQ_GRIPPER_CFG


The commented-out import #from isaaclab_assets... violates flake8 E265 (block comments should start with # ). Either remove it or change to # from ... (with a space) to keep pre-commit passing.

Suggested change

#from isaaclab_assets.robots.universal_robots import UR10e_ROBOTIQ_GRIPPER_CFG

# from isaaclab_assets.robots.universal_robots import UR10e_ROBOTIQ_GRIPPER_CFG

Copilot · 2026-03-25T11:52:10Z

 from isaaclab.assets import AssetBaseCfg
 from isaaclab.sim.spawners.from_files import UsdFileCfg
 from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR, NVIDIA_NUCLEUS_DIR
-#from source.isaaclab_assets.isaaclab_assets.robots.universal_robots import UR10_CFG
 from isaaclab_assets.robots.universal_robots import UR10_CFG
 from . import dev_env_cfg


Unused imports (AssetBaseCfg, UsdFileCfg, UR10_CFG) will be flagged by flake8 F401 now that this file is modified in this PR. Please remove them or mark with # noqa: F401.

Copilot · 2026-03-25T11:52:11Z

+                "sensor_cfg": SceneEntityCfg("table_cam"), 
+                "data_type": "normals", 
+                "normalize": True},


params dict for table_cam_normals has trailing whitespace and is not Black-formatted (closing brace on same line, missing trailing comma). This will fail the trailing-whitespace / black hooks; please reformat (e.g., run pre-commit / black).

Suggested change

"sensor_cfg": SceneEntityCfg("table_cam"),

"data_type": "normals",

"normalize": True},

"sensor_cfg": SceneEntityCfg("table_cam"),

"data_type": "normals",

"normalize": True,

},

Copilot · 2026-03-25T11:52:11Z

    time_out = DoneTerm(func=mdp.time_out, time_out=True)

-    joint_violation = DoneTerm(func=mdp.joint_pos_out_of_limit)
+    #joint_violation = DoneTerm(func=mdp.joint_pos_out_of_limit)



This change disables the joint_violation termination by commenting it out, which changes task behavior and is not mentioned in the PR title/description. If this is intentional, consider moving it to a dev-only env cfg or guarding it behind a config flag; otherwise please restore the termination. Also add a space after # (# joint_violation) to satisfy flake8 E265.

added scene cameras for cosmos

53d8099

Copilot AI review requested due to automatic review settings March 25, 2026 11:45

github-actions Bot added the isaac-lab label Mar 25, 2026

LJones-alt merged commit 471fd73 into aarch-uol:vision_based Mar 25, 2026
4 of 7 checks passed

Copilot started reviewing on behalf of LJones-alt March 25, 2026 11:46 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

added scene cameras for cosmos#6

added scene cameras for cosmos#6
LJones-alt merged 1 commit into
aarch-uol:vision_basedfrom
LJones-alt:vision_based

LJones-alt commented Mar 25, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		if torch.any(out_of_upper_limits \| out_of_lower_limits):
		print("Joint position out of limit", asset.data.joint_pos[:, asset_cfg.joint_ids])

	#from isaaclab_assets.robots.universal_robots import UR10e_ROBOTIQ_GRIPPER_CFG
	# from isaaclab_assets.robots.universal_robots import UR10e_ROBOTIQ_GRIPPER_CFG

Uh oh!

Conversation

LJones-alt commented Mar 25, 2026

Description

Type of change

Screenshots

Checklist

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants