Skip to content

added scene cameras for cosmos #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
LJones-alt merged 1 commit into from
Mar 25, 2026
Merged

added scene cameras for cosmos #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions source/isaaclab/isaaclab/envs/mdp/terminations.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@ def joint_pos_out_of_limit(env: ManagerBasedRLEnv, asset_cfg: SceneEntityCfg = S
limits = asset.data.soft_joint_pos_limits[:, asset_cfg.joint_ids]
out_of_upper_limits = torch.any(asset.data.joint_pos[:, asset_cfg.joint_ids] > limits[..., 1], dim=1)
out_of_lower_limits = torch.any(asset.data.joint_pos[:, asset_cfg.joint_ids] < limits[..., 0], dim=1)
if torch.any(out_of_upper_limits | out_of_lower_limits):
print("Joint position out of limit", asset.data.joint_pos[:, asset_cfg.joint_ids])
Comment on lines +92 to +93

Copilot AI Mar 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug print() inside this termination will spam logs and can significantly slow down large batched envs (and forces a device sync for the if torch.any(...)). Prefer structured logging (e.g., at debug level) and log only the violating env indices / min-max values, or remove this side-effect entirely.

Suggested change
if torch.any(out_of_upper_limits | out_of_lower_limits):
print("Joint position out of limit", asset.data.joint_pos[:, asset_cfg.joint_ids])

Copilot uses AI. Check for mistakes.
return torch.logical_or(out_of_upper_limits, out_of_lower_limits)


Expand Down
2 changes: 1 addition & 1 deletion ..._tasks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_env_insert_vismot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
from isaaclab.assets import AssetBaseCfg
from isaaclab.sim.spawners.from_files import UsdFileCfg
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR, NVIDIA_NUCLEUS_DIR
from source.isaaclab_assets.isaaclab_assets.robots.universal_robots import UR10_CFG
from isaaclab_assets.robots.universal_robots import UR10_CFG
from . import dev_env_cfg
Comment on lines 12 to 16

Copilot AI Mar 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused imports (AssetBaseCfg, UsdFileCfg, UR10_CFG) will be flagged by flake8 F401 once this file is changed. Please remove them or add # noqa: F401 if they are intentionally kept for future use.

Copilot uses AI. Check for mistakes.
import isaaclab.sim as sim_utils
from isaaclab.sensors import CameraCfg
Expand Down
1 change: 0 additions & 1 deletion ...isaaclab_tasks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_env_place.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from isaaclab.assets import AssetBaseCfg
from isaaclab.sim.spawners.from_files import UsdFileCfg
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR
#from source.isaaclab_assets.isaaclab_assets.robots.universal_robots import UR10_CFG
from . import dev_env_cfg
import math
from isaaclab.managers import SceneEntityCfg
Expand Down
46 changes: 40 additions & 6 deletions ...b_tasks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_env_place_vismot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
from isaaclab.assets import AssetBaseCfg
from isaaclab.sim.spawners.from_files import UsdFileCfg
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR, NVIDIA_NUCLEUS_DIR
from source.isaaclab_assets.isaaclab_assets.robots.universal_robots import UR10_CFG
from isaaclab_assets.robots.universal_robots import UR10_CFG
from . import dev_env_cfg
Comment on lines 12 to 16

Copilot AI Mar 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This module imports AssetBaseCfg, UsdFileCfg, and UR10_CFG but they are unused in the file, which will trigger flake8 F401 once this file is modified. Please remove unused imports or mark them with the appropriate # noqa: F401.

Copilot uses AI. Check for mistakes.
import isaaclab.sim as sim_utils
from isaaclab.sensors import CameraCfg
Expand All @@ -27,11 +27,11 @@
# Pre-defined configs
##
from isaaclab_assets.robots.franka import FRANKA_PANDA_HIGH_PD_CFG # isort: skip
from isaaclab_assets.robots.universal_robots import UR10e_ROBOTIQ_GRIPPER_CFG
#from isaaclab_assets.robots.universal_robots import UR10e_ROBOTIQ_GRIPPER_CFG

Copilot AI Mar 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commented-out import #from isaaclab_assets... violates flake8 E265 (block comments should start with # ). Either remove it or change to # from ... (with a space) to keep pre-commit passing.

Suggested change
#from isaaclab_assets.robots.universal_robots import UR10e_ROBOTIQ_GRIPPER_CFG
# from isaaclab_assets.robots.universal_robots import UR10e_ROBOTIQ_GRIPPER_CFG

Copilot uses AI. Check for mistakes.

## add some cameras in


## this is fully configured for cosmos now!

@configclass
class FrankaDevEnvVMCfg(dev_env_cfg.FrankaDevEnvCfg):
Expand Down Expand Up @@ -70,12 +70,15 @@ def __post_init__(self):
self.scene.table_cam = CameraCfg(
prim_path="{ENV_REGEX_NS}/table_cam",
update_period=0.0,
height=84,
width=84,
data_types=["rgb", "distance_to_image_plane"],
height=200,
width=200,
data_types=["rgb", "semantic_segmentation", "normals","distance_to_image_plane"],
colorize_semantic_segmentation=True,
semantic_segmentation_mapping=SEMANTIC_MAPPING,
spawn=sim_utils.PinholeCameraCfg(
Comment on lines +75 to 78

Copilot AI Mar 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SEMANTIC_MAPPING is referenced in the CameraCfg but is not defined/imported anywhere in this module, so importing this config will raise NameError. Define the mapping (as done in other cosmos/blueprint env cfgs) or import it from a shared location.

Copilot uses AI. Check for mistakes.
focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 2)
),
### Where is this in the scene?
offset=CameraCfg.OffsetCfg(
pos=(1.0, 0.0, 0.4), rot=(0.35355, -0.61237, -0.61237, 0.35355), convention="ros"
),
Expand Down Expand Up @@ -154,6 +157,37 @@ def __post_init__(self):
self.observations.policy.table_cam = ObsTerm(
func=mdp.image, params={"sensor_cfg": SceneEntityCfg("table_cam"), "data_type": "rgb", "normalize": False}
)

#### for cosmos add the segmentation, normals and depth data
self.observations.table_cam_segmentation= ObsTerm(
func = mdp.image,
params = {
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type": "semantic_segmentation",
"normalize" : True,
}
)

self.observations.table_cam_normals= ObsTerm(
func=mdp.image,
params={
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type" : "normals",
"normalize" : True,
}
)

self.observations.table_cam_depth = ObsTerm(
func=mdp.image,
params={
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type": "distance_to_image_plane",
"normalize" : True,
}
Comment on lines +162 to +186

Copilot AI Mar 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These new image ObsTerms are assigned onto self.observations instead of the policy observation group (e.g., self.observations.policy.<term>). As written they won’t be part of the policy observation group (and may raise an attribute error depending on how configclass is implemented). Please attach them under self.observations.policy like the existing table_cam / wrist_cam terms.

Suggested change
self.observations.table_cam_segmentation= ObsTerm(
func = mdp.image,
params = {
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type": "semantic_segmentation",
"normalize" : True,
}
)
self.observations.table_cam_normals= ObsTerm(
func=mdp.image,
params={
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type" : "normals",
"normalize" : True,
}
)
self.observations.table_cam_depth = ObsTerm(
func=mdp.image,
params={
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type": "distance_to_image_plane",
"normalize" : True,
}
self.observations.policy.table_cam_segmentation = ObsTerm(
func=mdp.image,
params={
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type": "semantic_segmentation",
"normalize": True,
},
)
self.observations.policy.table_cam_normals = ObsTerm(
func=mdp.image,
params={
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type": "normals",
"normalize": True,
},
)
self.observations.policy.table_cam_depth = ObsTerm(
func=mdp.image,
params={
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type": "distance_to_image_plane",
"normalize": True,
},

Copilot uses AI. Check for mistakes.
)

## wrist cam? do we have one?

self.observations.policy.wrist_cam = ObsTerm(
func=mdp.image, params={"sensor_cfg": SceneEntityCfg("wrist_cam"), "data_type": "rgb", "normalize": False}
)
Expand Down
2 changes: 1 addition & 1 deletion .../isaaclab_tasks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_skillgen.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
from isaaclab.assets import AssetBaseCfg
from isaaclab.sim.spawners.from_files import UsdFileCfg
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR
from source.isaaclab_assets.isaaclab_assets.robots.universal_robots import UR10_CFG
from isaaclab_assets.robots.universal_robots import UR10_CFG
from . import dev_env_cfg
Comment on lines 12 to 16

Copilot AI Mar 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused imports (AssetBaseCfg, UsdFileCfg, UR10_CFG) will be flagged by flake8 F401 now that this file is modified. Please remove them or add # noqa: F401 where truly needed.

Copilot uses AI. Check for mistakes.
import math
from isaaclab.managers import SceneEntityCfg
Expand Down
1 change: 0 additions & 1 deletion ...aaclab_tasks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_vial_insert.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from isaaclab.assets import AssetBaseCfg
from isaaclab.sim.spawners.from_files import UsdFileCfg
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR, NVIDIA_NUCLEUS_DIR
#from source.isaaclab_assets.isaaclab_assets.robots.universal_robots import UR10_CFG
from isaaclab_assets.robots.universal_robots import UR10_CFG
from . import dev_env_cfg
Comment on lines 12 to 16

Copilot AI Mar 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused imports (AssetBaseCfg, UsdFileCfg, UR10_CFG) will be flagged by flake8 F401 now that this file is modified in this PR. Please remove them or mark with # noqa: F401.

Copilot uses AI. Check for mistakes.
import isaaclab.sim as sim_utils
Expand Down
1 change: 0 additions & 1 deletion ...sks/manager_based/manipulation/cube_lift/config/franka/dev_ik_rel_vial_insert_top_down.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from isaaclab.assets import AssetBaseCfg
from isaaclab.sim.spawners.from_files import UsdFileCfg
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR, NVIDIA_NUCLEUS_DIR
#from source.isaaclab_assets.isaaclab_assets.robots.universal_robots import UR10_CFG
from isaaclab_assets.robots.universal_robots import UR10_CFG
from . import dev_env_cfg
Comment on lines 12 to 16

Copilot AI Mar 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused imports (AssetBaseCfg, UsdFileCfg, UR10_CFG) will be flagged by flake8 F401 now that this file is modified in this PR. Please remove them or mark with # noqa: F401.

Copilot uses AI. Check for mistakes.
import isaaclab.sim as sim_utils
Expand Down
2 changes: 1 addition & 1 deletion source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/cube_lift/lift_env_cfg.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,7 @@ class TerminationsCfg:

time_out = DoneTerm(func=mdp.time_out, time_out=True)

joint_violation = DoneTerm(func=mdp.joint_pos_out_of_limit)
#joint_violation = DoneTerm(func=mdp.joint_pos_out_of_limit)

Comment on lines 295 to 298

Copilot AI Mar 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change disables the joint_violation termination by commenting it out, which changes task behavior and is not mentioned in the PR title/description. If this is intentional, consider moving it to a dev-only env cfg or guarding it behind a config flag; otherwise please restore the termination. Also add a space after # (# joint_violation) to satisfy flake8 E265.

Copilot uses AI. Check for mistakes.
#object_orientation = DoneTerm(func=mdp.bad_orientation)

Expand Down
5 changes: 4 additions & 1 deletion .../manager_based/manipulation/stack/config/franka/stack_ik_rel_visuomotor_cosmos_env_cfg.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,10 @@ class PolicyCfg(ObsGroup):
)
table_cam_normals = ObsTerm(
func=mdp.image,
params={"sensor_cfg": SceneEntityCfg("table_cam"), "data_type": "normals", "normalize": True},
params={
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type": "normals",
"normalize": True},
Comment on lines +48 to +50

Copilot AI Mar 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

params dict for table_cam_normals has trailing whitespace and is not Black-formatted (closing brace on same line, missing trailing comma). This will fail the trailing-whitespace / black hooks; please reformat (e.g., run pre-commit / black).

Suggested change
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type": "normals",
"normalize": True},
"sensor_cfg": SceneEntityCfg("table_cam"),
"data_type": "normals",
"normalize": True,
},

Copilot uses AI. Check for mistakes.
)
table_cam_depth = ObsTerm(
func=mdp.image,
Expand Down
Loading