Adds Shadow Hand Environments Newton Implementation#4616
Adds Shadow Hand Environments Newton Implementation#4616nv-rgresia wants to merge 16 commits intoisaac-sim:dev/newtonfrom
Conversation
|
Too many files changed for review. ( |
nv-rgresia
requested review from
ClemensSchwarke,
Mayankm96,
Toni-SM,
hhansen-bdai,
jtigue-bdai,
kellyguo11,
masoudmoghani,
matthewtrepte,
ooctipus,
ossamaAhmed,
pascal-roth,
peterd-NV and
xyao-nv
as code owners
github-actions
bot
added
documentation
kellyguo11
left a comment
kellyguo11
left a comment
There was a problem hiding this comment.
Thanks, looks good overall! Does the Newton renderer actually provide segmentation data? or that's just an empty buffer for now?
I also wonder if the difference in training performance compared to PhysX might be from the tendon setup or missing sensor data?
|
The segmentation data is indeed empty buffers; I wonder if the lack of color data is impacting it as well. I'll have to do some benchmarking between all the environments to determine whats going on. The current allegro hand checkpoint looks to be underperforming as well and I can compare it against the basic environment, but i'll need to do the training first. Also I've added another commit the OpenAI FF environment, the EventCfg has some limitations due to Newton currently, but its able to start training. The OpenAI LSTM should be working as well, although I get a key error in the rl_games train script, is this a known issue? |
source/isaaclab_newton/isaaclab_newton/actuators/actuator_base.py
Outdated
Show resolved
Hide resolved
source/isaaclab_newton/isaaclab_newton/assets/articulation/articulation.py
Outdated
Show resolved
Hide resolved
AntoineRichard
left a comment
AntoineRichard
left a comment
There was a problem hiding this comment.
Thanks for PR! I've left a couple comments. I think there is an issue with the joint_pos_limits, but i'm working on making that homogeneous across PhysX and Newton.
source/isaaclab_newton/isaaclab_newton/assets/articulation/articulation.py
Show resolved
Hide resolved
| ), | ||
| # collision_props=sim_utils.CollisionPropertiesCfg(contact_offset=0.005, rest_offset=0.0), | ||
| joint_drive_props=sim_utils.JointDrivePropertiesCfg(drive_type="force"), | ||
| # fixed_tendons_props=sim_utils.FixedTendonPropertiesCfg(limit_stiffness=30.0, damping=0.1), |
There was a problem hiding this comment.
is this setting still being applied in Newton?
| self.fingertip_pos.view(self.num_envs, self.num_fingertips * 3), | ||
| self.fingertip_rot.view(self.num_envs, self.num_fingertips * 4), | ||
| self.fingertip_velocities.view(self.num_envs, self.num_fingertips * 6), | ||
| # self.cfg.force_torque_obs_scale * |
There was a problem hiding this comment.
I think this API should be available in Newton as well?
| "distribution": "gaussian", | ||
| }, | ||
| ) | ||
| # robot_tendon_properties = EventTerm( |
There was a problem hiding this comment.
are we able to randomize the tendon parameters in Newton?
There was a problem hiding this comment.
No because tendons aren't part of this MR, will update once Vidur's PR makes it in.
| ) | ||
|
|
||
| # -- scene | ||
| # reset_gravity = EventTerm( |
There was a problem hiding this comment.
I think this should also be supported
| SHADOW_HAND_CFG = ArticulationCfg( | ||
| spawn=sim_utils.UsdFileCfg( | ||
| # newton requires implicitactuators be specified in usd and there's a bug with physx tendons | ||
| usd_path=f"{ISAAC_NUCLEUS_DIR}/Robots/ShadowRobot/ShadowHand/shadow_hand_instanceable_newton.usd", |
There was a problem hiding this comment.
can we do this in a new cfg that extends from SHADOW_HAND_CFG? I'm worried we might miss this when we merge this with develop
kellyguo11
changed the title
|
@AntoineRichard - could you give these changes another look? @nv-rgresia - pls also run |
3 participants
Description
Imports the shadow hand environments to Newton. Currently the working environments are Isaac-Repose-Cube-Shadow-Direct-v0 and 'Isaac-Repose-Cube-Shadow-Vision-Direct-v0', but all around model performance is lacking during testing.
Type of change
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there