diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 6a96c1bf6bef..095b3536cd2a 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -56,6 +56,7 @@ Guidelines for modifications: * Bingjie Tang * Brayden Zhang * Brian Bingham +* Brian Dilinila * Brian McCann * Calvin Yu * Cameron Upright diff --git a/docs/index.rst b/docs/index.rst index 96f593f1d982..5a0e484e41ff 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -94,6 +94,7 @@ Table of Contents :titlesonly: source/setup/quickstart + source/setup/physx_warp_training source/overview/own-project/index source/setup/walkthrough/index source/tutorials/index diff --git a/docs/source/overview/core-concepts/sensors/camera.rst b/docs/source/overview/core-concepts/sensors/camera.rst index 6a34c6fab30c..dd9da7592bef 100644 --- a/docs/source/overview/core-concepts/sensors/camera.rst +++ b/docs/source/overview/core-concepts/sensors/camera.rst @@ -23,6 +23,17 @@ The Tiled Rendering APIs provide a vectorized interface for collecting data from Isaac Lab provides tiled rendering APIs for RGB, depth, along with other annotators through the :class:`~sensors.TiledCamera` class. Configurations for the tiled rendering APIs can be defined through the :class:`~sensors.TiledCameraCfg` class, specifying parameters such as the regex expression for all camera paths, the transform for the cameras, the desired data type, the type of cameras to add to the scene, and the camera resolution. +The renderer backend (Isaac RTX vs. Newton Warp) can be selected at run time via the config's ``renderer_type`` +(``"isaac_rtx"`` or ``"newton_warp"``). When using Hydra (e.g. in ``train.py``), use the top-level override +``render=isaac_rtx`` or ``render=newton_warp`` to apply the selected preset to all cameras; or override a specific +camera—e.g. ``env.scene.base_camera.renderer_type=isaac_rtx`` when the scene exposes ``base_camera``, or +``env.tiled_camera.renderer_type=isaac_rtx`` when the camera is on the env config. See **Hydra Configuration System** (Features) for override paths and examples. + +Renderer backends +~~~~~~~~~~~~~~~~~ + +By default, tiled cameras use **Omniverse RTX** (Replicator annotators). You can optionally use the **Newton Warp** backend for Warp-based ray tracing while keeping PhysX for simulation: set :attr:`~sensors.TiledCameraCfg.renderer_type` to ``"newton_warp"`` in the camera config. With Newton Warp, PhysX rigid-body state is synced to a Newton model each frame before rendering; the combined sync and render step is reported in the training script's timing summary as ``newton_warp_sync_plus_render`` when using the RSL-RL train script with ``--renderer_backend newton_warp``. For how to set up and run with the Warp renderer, see :ref:`physx-warp-training`. + .. code-block:: python tiled_camera: TiledCameraCfg = TiledCameraCfg( diff --git a/docs/source/setup/physx_warp_training.rst b/docs/source/setup/physx_warp_training.rst new file mode 100644 index 000000000000..0fcc5bf289fe --- /dev/null +++ b/docs/source/setup/physx_warp_training.rst @@ -0,0 +1,155 @@ +.. _physx-warp-training: + +PhysX + Warp Training (Isaac Sim 6.0) +===================================== + +This document gives step-by-step instructions to set up the environment and run RL training (e.g., Cartpole RGB Camera, Direct) using PhysX simulation and the Isaac RTX or Newton Warp renderer. + +1. Isaac Sim (6.0) from source +------------------------------ + +#. Clone and build Isaac Sim: + + .. code-block:: bash + + cd ~/git # or your preferred parent directory + git clone https://gitlab-master.nvidia.com/omniverse/isaac/omni_isaac_sim.git + cd omni_isaac_sim + git checkout b69c05612c11ee0bafe15ea9e8d0189fab3e07f4 + +#. If needed for the build, add Jinja2 to ``deps/pip_cloud.toml`` (e.g. ``"Jinja2==3.1.5"`` in the packages list). +#. Build: + + .. code-block:: bash + + ./build.sh -r + +2. Clone IsaacLab-Physx-Warp and symlink Isaac Sim +-------------------------------------------------- + +#. Clone the IsaacLab-Physx-Warp repo (if not already), e.g.: + + .. code-block:: bash + + cd ~/git + git clone git@github.com:bdilinila/IsaacLab.git IsaacLab-Physx-Warp + cd IsaacLab-Physx-Warp + + Use the appropriate branch or fork URL for your setup. + +#. Create the ``_isaac_sim`` symlink to the built Sim: + + .. code-block:: bash + + rm -f _isaac_sim + ln -sfn /path/to/omni_isaac_sim/_build/linux-x86_64/release _isaac_sim + + Replace ``/path/to/omni_isaac_sim`` with the actual path to your ``omni_isaac_sim`` clone. + +3. Conda environment +-------------------- + +#. From the **IsaacLab-Physx-Warp** repo root, create the conda env (Python 3.12): + + .. code-block:: bash + + ./isaaclab.sh -c physx_dextrah + +#. Activate it: + + .. code-block:: bash + + source "$(conda info --base)/etc/profile.d/conda.sh" + conda activate physx_dextrah + +4. Install IsaacLab and dependencies +----------------------------------- + +#. Install all IsaacLab extensions (this installs the ``isaaclab`` package from ``source/isaaclab`` and other packages under ``source/``): + + .. code-block:: bash + + ./isaaclab.sh -i + + If you only need to reinstall the ``isaaclab`` package (e.g. after editing code in ``source/isaaclab``), you can run from the repo root: + + .. code-block:: bash + + pip install -e source/isaaclab + +#. Remove Newton and Warp from the Isaac Sim build so the app uses the pip-installed Newton (and avoids version conflicts). In the ``omni_isaac_sim`` build tree, rename or remove the prebundle folders so they are not loaded, e.g.: + + .. code-block:: bash + + cd /path/to/omni_isaac_sim/_build/linux-x86_64/release + mv pip_prebundle/newton pip_prebundle/newton_bak # or remove + mv pip_prebundle/warp pip_prebundle/warp_bak # if needed + + Replace ``/path/to/omni_isaac_sim`` with your clone path. + +#. Install Newton via pip (required for the Newton Warp renderer). Either from the Git commit in ``source/isaaclab/setup.py``: + + .. code-block:: bash + + pip install "newton @ git+https://github.com/newton-physics/newton.git@35657fc" + +5. Verify installation +---------------------- + +From the IsaacLab-Physx-Warp root with the conda env activated: + +.. code-block:: bash + + python -c "import newton; import warp; print('Newton, Warp OK')" + python -c "import isaacsim; print('Isaac Sim:', isaacsim.__file__)" + python -c "from isaaclab.app import AppLauncher; print('IsaacLab OK')" + +Possible Newton / Warp conflict +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Isaac Sim's build ships its own Newton and Warp under ``_build/.../pip_prebundle/``. If you skip the steps in section 4 (removing those folders and pip-installing Newton), the app may load the prebundle and you can see version or API conflicts. Follow the steps in section 4 to remove or rename the ``newton`` and ``warp`` folders in the Isaac Sim build and install Newton via pip. + +6. Run training +--------------- + +Renderer selection +~~~~~~~~~~~~~~~~~~ + +Camera renderer is chosen via the Hydra ``render`` config group. Supported values are ``isaac_rtx`` (default) and ``newton_warp``. +The selected preset is applied to all cameras (see ``isaaclab_tasks.utils.render_config_store`` and ``process_hydra_config`` in ``isaaclab_tasks.utils.hydra``). + +Use the top-level override ``render=isaac_rtx`` or ``render=newton_warp``. If omitted, it defaults to ``isaac_rtx``. + +Example command +~~~~~~~~~~~~~~~ + +From the **IsaacLab-Physx-Warp** repo root, with the conda env activated: + +.. code-block:: bash + + ./isaaclab.sh -p scripts/reinforcement_learning/rl_games/train.py \ + --task=Isaac-Cartpole-RGB-Camera-Direct-v0 \ + --enable_cameras \ + --headless \ + render=isaac_rtx + +Use ``render=newton_warp`` to use the Newton Warp renderer instead. For longer training runs, increase ``--num_envs`` and ``--max_iterations`` (e.g. ``--num_envs=2048 --max_iterations=32``); redirect stdout/stderr if desired, e.g. ``2>&1 | tee train.log``. + +Summary +------ + ++------+----------------------------------------------------------------------------------------------------------------------------------+ +| Step | Action | ++======+==================================================================================================================================+ +| 1 | Clone and build Isaac Sim 6.0; checkout commit ``b69c05612c11ee0bafe15ea9e8d0189fab3e07f4`` | ++------+----------------------------------------------------------------------------------------------------------------------------------+ +| 2 | Clone IsaacLab-Physx-Warp and set ``_isaac_sim`` symlink | ++------+----------------------------------------------------------------------------------------------------------------------------------+ +| 3 | ``./isaaclab.sh -c physx_dextrah`` then ``conda activate physx_dextrah`` | ++------+----------------------------------------------------------------------------------------------------------------------------------+ +| 4 | ``./isaaclab.sh -i``; remove/rename ``newton`` and ``warp`` in omni_isaac_sim ``pip_prebundle``; ``pip install`` Newton (git or local) | ++------+----------------------------------------------------------------------------------------------------------------------------------+ +| 5 | Run the verification commands | ++------+----------------------------------------------------------------------------------------------------------------------------------+ +| 6 | Run ``./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-RGB-Camera-Direct-v0 --enable_cameras --headless ... render=isaac_rtx`` as above | ++------+----------------------------------------------------------------------------------------------------------------------------------+ diff --git a/source/isaaclab/isaaclab/sensors/camera/tiled_camera.py b/source/isaaclab/isaaclab/sensors/camera/tiled_camera.py index 9802f263ae20..b07b5d20b648 100644 --- a/source/isaaclab/isaaclab/sensors/camera/tiled_camera.py +++ b/source/isaaclab/isaaclab/sensors/camera/tiled_camera.py @@ -11,6 +11,7 @@ from typing import TYPE_CHECKING, Any import torch +import warp as wp from pxr import UsdGeom diff --git a/source/isaaclab_tasks/isaaclab_tasks/utils/hydra.py b/source/isaaclab_tasks/isaaclab_tasks/utils/hydra.py index 525b425917fa..ba227bd24f55 100644 --- a/source/isaaclab_tasks/isaaclab_tasks/utils/hydra.py +++ b/source/isaaclab_tasks/isaaclab_tasks/utils/hydra.py @@ -20,6 +20,46 @@ from isaaclab.utils import replace_slices_with_strings, replace_strings_with_slices from isaaclab_tasks.utils.parse_cfg import load_cfg_from_registry +from isaaclab_tasks.utils.render_config_store import register_render_configs + + +def process_hydra_config( + hydra_cfg: DictConfig | dict, + env_cfg: ManagerBasedRLEnvCfg | DirectRLEnvCfg, + agent_cfg: dict | object, +) -> tuple[ManagerBasedRLEnvCfg | DirectRLEnvCfg, dict | object]: + """Process composed Hydra config and update env/agent configs in place. + + Shared by hydra_task_config and tests. Applies render config to cameras, + updates env/agent from dict, restores gymnasium spaces. + """ + if not isinstance(hydra_cfg, dict): + hydra_cfg = OmegaConf.to_container(hydra_cfg, resolve=True) + hydra_cfg = replace_strings_with_slices(hydra_cfg) + + if "render" in hydra_cfg and hydra_cfg["render"]: + renderer_dict = hydra_cfg["render"] + if isinstance(renderer_dict, dict): + env_dict = hydra_cfg.get("env", {}) + + def apply_to_cameras(d: dict) -> None: + for v in d.values(): + if isinstance(v, dict): + if "renderer_cfg" in v: + v["renderer_cfg"] = renderer_dict + apply_to_cameras(v) + + apply_to_cameras(env_dict) + + env_cfg.from_dict(hydra_cfg["env"]) + env_cfg = replace_strings_with_env_cfg_spaces(env_cfg) + + if isinstance(agent_cfg, dict) or agent_cfg is None: + agent_cfg = hydra_cfg["agent"] + else: + agent_cfg.from_dict(hydra_cfg["agent"]) + + return env_cfg, agent_cfg def register_task_to_hydra( @@ -54,7 +94,8 @@ def register_task_to_hydra( cfg_dict = {"env": env_cfg_dict, "agent": agent_cfg_dict} # replace slices with strings because OmegaConf does not support slices cfg_dict = replace_slices_with_strings(cfg_dict) - # store the configuration to Hydra + cfg_dict["defaults"] = ["_self_", {"render": "isaac_rtx"}] + register_render_configs() ConfigStore.instance().store(name=task_name, node=cfg_dict) return env_cfg, agent_cfg @@ -82,21 +123,7 @@ def wrapper(*args, **kwargs): # define the new Hydra main function @hydra.main(config_path=None, config_name=task_name.split(":")[-1], version_base="1.3") def hydra_main(hydra_env_cfg: DictConfig, env_cfg=env_cfg, agent_cfg=agent_cfg): - # convert to a native dictionary - hydra_env_cfg = OmegaConf.to_container(hydra_env_cfg, resolve=True) - # replace string with slices because OmegaConf does not support slices - hydra_env_cfg = replace_strings_with_slices(hydra_env_cfg) - # update the configs with the Hydra command line arguments - env_cfg.from_dict(hydra_env_cfg["env"]) - # replace strings that represent gymnasium spaces because OmegaConf does not support them. - # this must be done after converting the env configs from dictionary to avoid internal reinterpretations - env_cfg = replace_strings_with_env_cfg_spaces(env_cfg) - # get agent configs - if isinstance(agent_cfg, dict) or agent_cfg is None: - agent_cfg = hydra_env_cfg["agent"] - else: - agent_cfg.from_dict(hydra_env_cfg["agent"]) - # call the original function + env_cfg, agent_cfg = process_hydra_config(hydra_env_cfg, env_cfg, agent_cfg) func(env_cfg, agent_cfg, *args, **kwargs) # call the new Hydra main function @@ -105,3 +132,4 @@ def hydra_main(hydra_env_cfg: DictConfig, env_cfg=env_cfg, agent_cfg=agent_cfg): return wrapper return decorator + diff --git a/source/isaaclab_tasks/isaaclab_tasks/utils/render_config_store.py b/source/isaaclab_tasks/isaaclab_tasks/utils/render_config_store.py new file mode 100644 index 000000000000..bc6636b11e9f --- /dev/null +++ b/source/isaaclab_tasks/isaaclab_tasks/utils/render_config_store.py @@ -0,0 +1,28 @@ +# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause + +"""Renderer config presets for Hydra ConfigStore. + +Register renderer backend configs that can be selected via the ``render`` config +group (e.g. ``render=isaac_rtx`` or ``render=newton_warp``). The selected config +is applied to all cameras in the scene. +""" + +from hydra.core.config_store import ConfigStore + +from isaaclab_physx.renderers import IsaacRtxRendererCfg + +try: + from isaaclab_newton.renderers import NewtonWarpRendererCfg +except ImportError: + NewtonWarpRendererCfg = None + + +def register_render_configs() -> None: + """Register renderer config presets in Hydra ConfigStore.""" + cs = ConfigStore.instance() + cs.store(name="isaac_rtx", group="render", node=IsaacRtxRendererCfg()) + if NewtonWarpRendererCfg is not None: + cs.store(name="newton_warp", group="render", node=NewtonWarpRendererCfg()) diff --git a/source/isaaclab_tasks/test/test_hydra.py b/source/isaaclab_tasks/test/test_hydra.py index 5c81cb3e650f..096b5ec4b896 100644 --- a/source/isaaclab_tasks/test/test_hydra.py +++ b/source/isaaclab_tasks/test/test_hydra.py @@ -21,39 +21,22 @@ import hydra from hydra import compose, initialize -from omegaconf import OmegaConf - -from isaaclab.utils import replace_strings_with_slices +import pytest import isaaclab_tasks # noqa: F401 -from isaaclab_tasks.utils.hydra import register_task_to_hydra +from isaaclab_tasks.utils.hydra import process_hydra_config, register_task_to_hydra def hydra_task_config_test(task_name: str, agent_cfg_entry_point: str) -> Callable: - """Copied from hydra.py hydra_task_config, since hydra.main requires a single point of entry, - which will not work with multiple tests. Here, we replace hydra.main with hydra initialize - and compose.""" + """Mirrors hydra_task_config: register task, compose, process_hydra_config, then run test.""" def decorator(func): @functools.wraps(func) def wrapper(*args, **kwargs): - # register the task to Hydra env_cfg, agent_cfg = register_task_to_hydra(task_name, agent_cfg_entry_point) - - # replace hydra.main with initialize and compose with initialize(config_path=None, version_base="1.3"): hydra_env_cfg = compose(config_name=task_name, overrides=sys.argv[1:]) - # convert to a native dictionary - hydra_env_cfg = OmegaConf.to_container(hydra_env_cfg, resolve=True) - # replace string with slices because OmegaConf does not support slices - hydra_env_cfg = replace_strings_with_slices(hydra_env_cfg) - # update the configs with the Hydra command line arguments - env_cfg.from_dict(hydra_env_cfg["env"]) - if isinstance(agent_cfg, dict): - agent_cfg = hydra_env_cfg["agent"] - else: - agent_cfg.from_dict(hydra_env_cfg["agent"]) - # call the original function + env_cfg, agent_cfg = process_hydra_config(hydra_env_cfg, env_cfg, agent_cfg) func(env_cfg, agent_cfg, *args, **kwargs) return wrapper @@ -103,3 +86,61 @@ def main(env_cfg, agent_cfg): # clean up sys.argv = [sys.argv[0]] hydra.core.global_hydra.GlobalHydra.instance().clear() + + +def test_render_config_default(): + """No override: default render config (isaac_rtx) is applied.""" + sys.argv = [sys.argv[0]] + + @hydra_task_config_test("Isaac-Cartpole-RGB-Camera-Direct-v0", None) + def main(env_cfg, agent_cfg): + assert env_cfg.tiled_camera.renderer_cfg is not None + assert env_cfg.tiled_camera.renderer_cfg.renderer_type == "isaac_rtx" + + main() + sys.argv = [sys.argv[0]] + hydra.core.global_hydra.GlobalHydra.instance().clear() + + +def test_render_config_override(): + """Override render=isaac_rtx is applied to cameras.""" + sys.argv = [sys.argv[0], "render=isaac_rtx"] + + @hydra_task_config_test("Isaac-Cartpole-RGB-Camera-Direct-v0", None) + def main(env_cfg, agent_cfg): + assert env_cfg.tiled_camera.renderer_cfg is not None + assert env_cfg.tiled_camera.renderer_cfg.renderer_type == "isaac_rtx" + + main() + sys.argv = [sys.argv[0]] + hydra.core.global_hydra.GlobalHydra.instance().clear() + + +def test_render_config_override_newton_warp(): + """Override render=newton_warp sets renderer_type on cameras. Skip if Newton not installed.""" + try: + from isaaclab_newton.renderers import NewtonWarpRendererCfg # noqa: F401 + except ImportError: + pytest.skip("Newton Warp renderer not installed") + sys.argv = [sys.argv[0], "render=newton_warp"] + + @hydra_task_config_test("Isaac-Cartpole-RGB-Camera-Direct-v0", None) + def main(env_cfg, agent_cfg): + assert env_cfg.tiled_camera.renderer_cfg is not None + assert env_cfg.tiled_camera.renderer_cfg.renderer_type == "newton_warp" + + main() + sys.argv = [sys.argv[0]] + hydra.core.global_hydra.GlobalHydra.instance().clear() + + +def test_render_config_invalid_raises(): + """Invalid render= choice raises.""" + sys.argv = [sys.argv[0], "render=invalid_renderer"] + register_task_to_hydra("Isaac-Cartpole-RGB-Camera-Direct-v0", None) + + with pytest.raises(Exception): # Hydra/OmegaConf error for unknown config group choice + with initialize(config_path=None, version_base="1.3"): + compose(config_name="Isaac-Cartpole-RGB-Camera-Direct-v0", overrides=sys.argv[1:]) + sys.argv = [sys.argv[0]] + hydra.core.global_hydra.GlobalHydra.instance().clear()