Add Option for Warp Renderer backend with PhysX simulation engine

CONTRIBUTORS.md

@@ -56,6 +56,7 @@ Guidelines for modifications:
 * Bingjie Tang
 * Brayden Zhang
 * Brian Bingham
+* Brian Dilinila
 * Brian McCann
 * Calvin Yu
 * Cameron Upright

docs/index.rst

@@ -94,6 +94,7 @@ Table of Contents
    :titlesonly:
 
    source/setup/quickstart
+   source/setup/physx_warp_training
    source/overview/own-project/index
    source/setup/walkthrough/index
    source/tutorials/index

docs/source/overview/core-concepts/sensors/camera.rst

@@ -23,6 +23,17 @@ The Tiled Rendering APIs provide a vectorized interface for collecting data from
 
 Isaac Lab provides tiled rendering APIs for RGB, depth, along with other annotators through the :class:`~sensors.TiledCamera` class. Configurations for the tiled rendering APIs can be defined through the :class:`~sensors.TiledCameraCfg` class, specifying parameters such as the regex expression for all camera paths, the transform for the cameras, the desired data type, the type of cameras to add to the scene, and the camera resolution.
 
+The renderer backend (Isaac RTX vs. Newton Warp) can be selected at run time via the config's ``renderer_type``
+(``"isaac_rtx"`` or ``"newton_warp"``). When using Hydra (e.g. in ``train.py``), use the top-level override
+``render=isaac_rtx`` or ``render=newton_warp`` to apply the selected preset to all cameras; or override a specific
+camera—e.g. ``env.scene.base_camera.renderer_type=isaac_rtx`` when the scene exposes ``base_camera``, or
+``env.tiled_camera.renderer_type=isaac_rtx`` when the camera is on the env config. See **Hydra Configuration System** (Features) for override paths and examples.
+
+Renderer backends
+~~~~~~~~~~~~~~~~~
+
+By default, tiled cameras use **Omniverse RTX** (Replicator annotators). You can optionally use the **Newton Warp** backend for Warp-based ray tracing while keeping PhysX for simulation: set :attr:`~sensors.TiledCameraCfg.renderer_type` to ``"newton_warp"`` in the camera config. With Newton Warp, PhysX rigid-body state is synced to a Newton model each frame before rendering; the combined sync and render step is reported in the training script's timing summary as ``newton_warp_sync_plus_render`` when using the RSL-RL train script with ``--renderer_backend newton_warp``. For how to set up and run with the Warp renderer, see :ref:`physx-warp-training`.
+
 .. code-block:: python
 
     tiled_camera: TiledCameraCfg = TiledCameraCfg(

docs/source/setup/physx_warp_training.rst

@@ -0,0 +1,155 @@
+.. _physx-warp-training:
+
+PhysX + Warp Training (Isaac Sim 6.0)
+=====================================
+
+This document gives step-by-step instructions to set up the environment and run RL training (e.g., Cartpole RGB Camera, Direct) using PhysX simulation and the Isaac RTX or Newton Warp renderer.
+
+1. Isaac Sim (6.0) from source
+------------------------------
+
+#. Clone and build Isaac Sim:
+
+   .. code-block:: bash
+
+      cd ~/git   # or your preferred parent directory
+      git clone https://gitlab-master.nvidia.com/omniverse/isaac/omni_isaac_sim.git
+      cd omni_isaac_sim
+      git checkout b69c05612c11ee0bafe15ea9e8d0189fab3e07f4
+
+#. If needed for the build, add Jinja2 to ``deps/pip_cloud.toml`` (e.g. ``"Jinja2==3.1.5"`` in the packages list).
+#. Build:
+
+   .. code-block:: bash
+
+      ./build.sh -r
+
+2. Clone IsaacLab-Physx-Warp and symlink Isaac Sim
+--------------------------------------------------
+
+#. Clone the IsaacLab-Physx-Warp repo (if not already), e.g.:
+
+   .. code-block:: bash
+
+      cd ~/git
+      git clone git@github.com:bdilinila/IsaacLab.git IsaacLab-Physx-Warp
+      cd IsaacLab-Physx-Warp
+
+   Use the appropriate branch or fork URL for your setup.
+
+#. Create the ``_isaac_sim`` symlink to the built Sim:
+
+   .. code-block:: bash
+
+      rm -f _isaac_sim
+      ln -sfn /path/to/omni_isaac_sim/_build/linux-x86_64/release _isaac_sim
+
+   Replace ``/path/to/omni_isaac_sim`` with the actual path to your ``omni_isaac_sim`` clone.
+
+3. Conda environment
+--------------------
+
+#. From the **IsaacLab-Physx-Warp** repo root, create the conda env (Python 3.12):
+
+   .. code-block:: bash
+
+      ./isaaclab.sh -c physx_dextrah
+
+#. Activate it:
+
+   .. code-block:: bash
+
+      source "$(conda info --base)/etc/profile.d/conda.sh"
+      conda activate physx_dextrah
+
+4. Install IsaacLab and dependencies
+-----------------------------------
+
+#. Install all IsaacLab extensions (this installs the ``isaaclab`` package from ``source/isaaclab`` and other packages under ``source/``):
+
+   .. code-block:: bash
+
+      ./isaaclab.sh -i
+
+   If you only need to reinstall the ``isaaclab`` package (e.g. after editing code in ``source/isaaclab``), you can run from the repo root:
+
+   .. code-block:: bash
+
+      pip install -e source/isaaclab
+
+#. Remove Newton and Warp from the Isaac Sim build so the app uses the pip-installed Newton (and avoids version conflicts). In the ``omni_isaac_sim`` build tree, rename or remove the prebundle folders so they are not loaded, e.g.:
+
+   .. code-block:: bash
+
+      cd /path/to/omni_isaac_sim/_build/linux-x86_64/release
+      mv pip_prebundle/newton pip_prebundle/newton_bak   # or remove
+      mv pip_prebundle/warp pip_prebundle/warp_bak     # if needed
+
+   Replace ``/path/to/omni_isaac_sim`` with your clone path.
+
+#. Install Newton via pip (required for the Newton Warp renderer). Either from the Git commit in ``source/isaaclab/setup.py``:
+
+   .. code-block:: bash
+
+      pip install "newton @ git+https://github.com/newton-physics/newton.git@35657fc"
+
+5. Verify installation
+----------------------
+
+From the IsaacLab-Physx-Warp root with the conda env activated:
+
+.. code-block:: bash
+
+   python -c "import newton; import warp; print('Newton, Warp OK')"
+   python -c "import isaacsim; print('Isaac Sim:', isaacsim.__file__)"
+   python -c "from isaaclab.app import AppLauncher; print('IsaacLab OK')"
+
+Possible Newton / Warp conflict
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Isaac Sim's build ships its own Newton and Warp under ``_build/.../pip_prebundle/``. If you skip the steps in section 4 (removing those folders and pip-installing Newton), the app may load the prebundle and you can see version or API conflicts. Follow the steps in section 4 to remove or rename the ``newton`` and ``warp`` folders in the Isaac Sim build and install Newton via pip.
+
+6. Run training
+---------------
+
+Renderer selection
+~~~~~~~~~~~~~~~~~~
+
+Camera renderer is chosen via the Hydra ``render`` config group. Supported values are ``isaac_rtx`` (default) and ``newton_warp``.
+The selected preset is applied to all cameras (see ``isaaclab_tasks.utils.render_config_store`` and ``process_hydra_config`` in ``isaaclab_tasks.utils.hydra``).
+
+Use the top-level override ``render=isaac_rtx`` or ``render=newton_warp``. If omitted, it defaults to ``isaac_rtx``.
+
+Example command
+~~~~~~~~~~~~~~~
+
+From the **IsaacLab-Physx-Warp** repo root, with the conda env activated:
+
+.. code-block:: bash
+
+   ./isaaclab.sh -p scripts/reinforcement_learning/rl_games/train.py \
+     --task=Isaac-Cartpole-RGB-Camera-Direct-v0 \
+     --enable_cameras \
+     --headless \
+     render=isaac_rtx
+
+Use ``render=newton_warp`` to use the Newton Warp renderer instead. For longer training runs, increase ``--num_envs`` and ``--max_iterations`` (e.g. ``--num_envs=2048 --max_iterations=32``); redirect stdout/stderr if desired, e.g. ``2>&1 | tee train.log``.
+
+Summary
+------
+
++------+----------------------------------------------------------------------------------------------------------------------------------+
+| Step | Action                                                                                                                            |
++======+==================================================================================================================================+
+| 1    | Clone and build Isaac Sim 6.0; checkout commit ``b69c05612c11ee0bafe15ea9e8d0189fab3e07f4``                                     |
++------+----------------------------------------------------------------------------------------------------------------------------------+
+| 2    | Clone IsaacLab-Physx-Warp and set ``_isaac_sim`` symlink                                                                          |
++------+----------------------------------------------------------------------------------------------------------------------------------+
+| 3    | ``./isaaclab.sh -c physx_dextrah`` then ``conda activate physx_dextrah``                                                            |
++------+----------------------------------------------------------------------------------------------------------------------------------+
+| 4    | ``./isaaclab.sh -i``; remove/rename ``newton`` and ``warp`` in omni_isaac_sim ``pip_prebundle``; ``pip install`` Newton (git or local) |
++------+----------------------------------------------------------------------------------------------------------------------------------+
+| 5    | Run the verification commands                                                                                                     |
++------+----------------------------------------------------------------------------------------------------------------------------------+
+| 6    | Run ``./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-RGB-Camera-Direct-v0 --enable_cameras --headless ... render=isaac_rtx`` as above |
++------+----------------------------------------------------------------------------------------------------------------------------------+

source/isaaclab/isaaclab/sensors/camera/tiled_camera.py

@@ -11,6 +11,7 @@
 from typing import TYPE_CHECKING, Any
 
 import torch
+import warp as wp
 
 from pxr import UsdGeom
 

source/isaaclab_tasks/isaaclab_tasks/utils/hydra.py

@@ -20,6 +20,46 @@
 from isaaclab.utils import replace_slices_with_strings, replace_strings_with_slices
 
 from isaaclab_tasks.utils.parse_cfg import load_cfg_from_registry
+from isaaclab_tasks.utils.render_config_store import register_render_configs
+
+
+def process_hydra_config(
+    hydra_cfg: DictConfig | dict,
+    env_cfg: ManagerBasedRLEnvCfg | DirectRLEnvCfg,
+    agent_cfg: dict | object,
+) -> tuple[ManagerBasedRLEnvCfg | DirectRLEnvCfg, dict | object]:
+    """Process composed Hydra config and update env/agent configs in place.
+
+    Shared by hydra_task_config and tests. Applies render config to cameras,
+    updates env/agent from dict, restores gymnasium spaces.
+    """
+    if not isinstance(hydra_cfg, dict):
+        hydra_cfg = OmegaConf.to_container(hydra_cfg, resolve=True)
+    hydra_cfg = replace_strings_with_slices(hydra_cfg)
+
+    if "render" in hydra_cfg and hydra_cfg["render"]:
+        renderer_dict = hydra_cfg["render"]
+        if isinstance(renderer_dict, dict):
+            env_dict = hydra_cfg.get("env", {})
+
+            def apply_to_cameras(d: dict) -> None:
+                for v in d.values():
+                    if isinstance(v, dict):
+                        if "renderer_cfg" in v:
+                            v["renderer_cfg"] = renderer_dict
+                        apply_to_cameras(v)
+
+            apply_to_cameras(env_dict)
+
+    env_cfg.from_dict(hydra_cfg["env"])
+    env_cfg = replace_strings_with_env_cfg_spaces(env_cfg)
+
+    if isinstance(agent_cfg, dict) or agent_cfg is None:
+        agent_cfg = hydra_cfg["agent"]
+    else:
+        agent_cfg.from_dict(hydra_cfg["agent"])
+
+    return env_cfg, agent_cfg
 
 
 def register_task_to_hydra(
@@ -54,7 +94,8 @@ def register_task_to_hydra(
     cfg_dict = {"env": env_cfg_dict, "agent": agent_cfg_dict}
     # replace slices with strings because OmegaConf does not support slices
     cfg_dict = replace_slices_with_strings(cfg_dict)
-    # store the configuration to Hydra
+    cfg_dict["defaults"] = ["_self_", {"render": "isaac_rtx"}]
+    register_render_configs()
     ConfigStore.instance().store(name=task_name, node=cfg_dict)
     return env_cfg, agent_cfg
 
@@ -82,21 +123,7 @@ def wrapper(*args, **kwargs):
             # define the new Hydra main function
             @hydra.main(config_path=None, config_name=task_name.split(":")[-1], version_base="1.3")
             def hydra_main(hydra_env_cfg: DictConfig, env_cfg=env_cfg, agent_cfg=agent_cfg):
-                # convert to a native dictionary
-                hydra_env_cfg = OmegaConf.to_container(hydra_env_cfg, resolve=True)
-                # replace string with slices because OmegaConf does not support slices
-                hydra_env_cfg = replace_strings_with_slices(hydra_env_cfg)
-                # update the configs with the Hydra command line arguments
-                env_cfg.from_dict(hydra_env_cfg["env"])
-                # replace strings that represent gymnasium spaces because OmegaConf does not support them.
-                # this must be done after converting the env configs from dictionary to avoid internal reinterpretations
-                env_cfg = replace_strings_with_env_cfg_spaces(env_cfg)
-                # get agent configs
-                if isinstance(agent_cfg, dict) or agent_cfg is None:
-                    agent_cfg = hydra_env_cfg["agent"]
-                else:
-                    agent_cfg.from_dict(hydra_env_cfg["agent"])
-                # call the original function
+                env_cfg, agent_cfg = process_hydra_config(hydra_env_cfg, env_cfg, agent_cfg)
                 func(env_cfg, agent_cfg, *args, **kwargs)
 
             # call the new Hydra main function
@@ -105,3 +132,4 @@ def hydra_main(hydra_env_cfg: DictConfig, env_cfg=env_cfg, agent_cfg=agent_cfg):
         return wrapper
 
     return decorator
+

source/isaaclab_tasks/isaaclab_tasks/utils/render_config_store.py

@@ -0,0 +1,28 @@
+# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""Renderer config presets for Hydra ConfigStore.
+
+Register renderer backend configs that can be selected via the ``render`` config
+group (e.g. ``render=isaac_rtx`` or ``render=newton_warp``). The selected config
+is applied to all cameras in the scene.
+"""
+
+from hydra.core.config_store import ConfigStore
+
+from isaaclab_physx.renderers import IsaacRtxRendererCfg
+
+try:
+    from isaaclab_newton.renderers import NewtonWarpRendererCfg
+except ImportError:
+    NewtonWarpRendererCfg = None
+
+
+def register_render_configs() -> None:
+    """Register renderer config presets in Hydra ConfigStore."""
+    cs = ConfigStore.instance()
+    cs.store(name="isaac_rtx", group="render", node=IsaacRtxRendererCfg())
+    if NewtonWarpRendererCfg is not None:
+        cs.store(name="newton_warp", group="render", node=NewtonWarpRendererCfg())