Update RSL-RL configs to work with new version 4.0

[ClemensSchwarke]

This PR updates the rsl-rl config classes to be compatible with version 4.0 and 4.1. It also adds a configuration handling function to ensure that old configs work with version 4.0/4.1 and adapts train.py and play.py accordingly. Lastly, it updates the config classes for ANYmal D locomotion for reference and adds recurrent configs.

The main change for 4.0 is that the policy config is split into seperate actor and critic / student and teacher configs. For more details see the Release Notes.

The main change for 4.1 is that the noise distribution is now configured via the distribution_cfg to allow for arbitrary distributions.

Checklist

• I have run the pre-commit checks with ./isaaclab.sh --format
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file

[greptile-apps]

Greptile Overview

Greptile Summary

• Updates IsaacLab’s RSL-RL config schema for rsl-rl v4.0.0 by splitting legacy policy into explicit model configs (actor/critic, student/teacher) and adding recurrent variants.
• Adds a compatibility shim (handle_deprecated_rsl_rl_cfg) to translate old/new config fields depending on installed rsl-rl version, and wires it into train.py/play.py.
• Updates ANYmal-D locomotion reference configs and gym registry kwargs to expose recurrent PPO/distillation entry points.
• Pins rsl-rl-lib extra dependency to 4.0.0 and adapts play-time export to use new runner export APIs when available.

Confidence Score: 2/5

• This PR is not safe to merge until a couple of runtime-breaking issues are fixed.
• Two changes are likely to cause immediate runtime failures: (1) handle_deprecated_rsl_rl_cfg uses isinstance(x, type(MISSING)) to detect missing config fields, which will evaluate true for almost any value and breaks the compatibility logic; (2) play.py unconditionally calls policy.reset() for rsl-rl >= 4.0.0, which will fail for non-recurrent inference policies that don’t implement reset(). Additionally, RslRlCNNModelCfg lacks the @configclass decorator, which can break config serialization if used.
• scripts/reinforcement_learning/rsl_rl/cli_args.py, scripts/reinforcement_learning/rsl_rl/play.py, source/isaaclab_rl/isaaclab_rl/rsl_rl/rl_cfg.py

Important Files Changed

Filename	Overview
scripts/reinforcement_learning/rsl_rl/cli_args.py	Adds handle_deprecated_rsl_rl_cfg() to translate deprecated runner/policy fields across rsl-rl <4 and >=4; likely has incorrect MISSING detection and may mis-handle optimizer removal.
scripts/reinforcement_learning/rsl_rl/play.py	Adds rsl-rl version check, runs deprecated-config handler, and uses new runner export APIs for >=4; changes recurrent reset to call policy.reset() which may not exist for non-recurrent policies.
scripts/reinforcement_learning/rsl_rl/train.py	Invokes deprecated-config handler before setting env seed; otherwise unchanged; relies on installed_version defined earlier in file.
source/isaaclab_rl/isaaclab_rl/rsl_rl/distillation_cfg.py	Introduces explicit student/teacher model cfgs in distillation runner while keeping deprecated policy cfg for backward compatibility; structural reordering only.
source/isaaclab_rl/isaaclab_rl/rsl_rl/rl_cfg.py	Splits old policy cfg into new model cfgs (MLP/RNN/CNN) and adds optimizer to PPO algo; keeps deprecated ActorCritic configs; potential missing @configclass on CNN model.
source/isaaclab_rl/setup.py	Bumps rsl-rl extra dependency pin from 3.1.2 to 4.0.0.
source/isaaclab_tasks/isaaclab_tasks/manager_based/locomotion/velocity/config/anymal_d/init.py	Registers additional gym kwargs for recurrent RSL-RL PPO/distillation runner entry points.
source/isaaclab_tasks/isaaclab_tasks/manager_based/locomotion/velocity/config/anymal_d/agents/rsl_rl_distillation_cfg.py	Updates ANYmal-D distillation runner to new student/teacher model cfgs and adds recurrent variant; obs_groups keys updated to student/teacher.
source/isaaclab_tasks/isaaclab_tasks/manager_based/locomotion/velocity/config/anymal_d/agents/rsl_rl_ppo_cfg.py	Updates ANYmal-D PPO runner to new actor/critic model cfgs and adds recurrent runner config; adjusts obs_groups accordingly.

Sequence Diagram

sequenceDiagram
    participant U as User CLI
    participant P as play.py / train.py
    participant CA as cli_args.py
    participant H as Hydra/Registry
    participant RL as isaaclab_rl.rsl_rl cfgs
    participant R as rsl_rl Runner

    U->>P: launch with --task/--agent
    P->>H: load env_cfg + agent_cfg (entry_point)
    P->>CA: update_rsl_rl_cfg(agent_cfg, args)
    P->>CA: handle_deprecated_rsl_rl_cfg(agent_cfg, installed_version)
    CA->>RL: map legacy policy<->model cfgs
    P->>R: construct Runner(env, agent_cfg.to_dict())
    R->>R: load checkpoint (play) / learn (train)
    alt rsl-rl >= 4.0.0 (play)
        P->>R: export_policy_to_jit / export_policy_to_onnx
    else rsl-rl < 4.0.0 (play)
        P->>RL: export_policy_as_jit / export_policy_as_onnx
    end

Loading

[greptile-apps]

_{9 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Incorrect MISSING sentinel check

dataclasses.MISSING is a sentinel object, but the code checks isinstance(x, type(MISSING)), which is effectively isinstance(x, object) and will always be true for any normal value. This makes the deprecated-config logic behave incorrectly (e.g., treating policy as missing even when it's set) and can raise the "policy required" ValueError for valid configs. Use identity checks instead (e.g., agent_cfg.policy is MISSING) consistently throughout this function.

[greptile-apps]

Unconditional reset call

For rsl-rl >= 4.0.0, this calls policy.reset(dones) unconditionally, but policy = runner.get_inference_policy(...) may return a non-recurrent policy callable that does not implement reset(). That will crash during play for non-recurrent checkpoints. Consider guarding with hasattr(policy, "reset") (or only resetting when using a recurrent runner/config) to avoid AttributeError.

[greptile-apps]

Additional Comments (1)

source/isaaclab_rl/isaaclab_rl/rsl_rl/rl_cfg.py
Missing @configclass decorator

RslRlCNNModelCfg is declared without @configclass, unlike the other config dataclasses. If this is intended to be used like the other cfg types (and nested CNNCfg already is), it won't get the same dataclass/configclass behavior and may break .to_dict()/Hydra/config serialization paths when selected. Add @configclass to RslRlCNNModelCfg.

[Wenkai-Dong]

Missing share_cnn_encoders in config dataclass

The code uses cfg["algorithm"].pop("share_cnn_encoders", None) to configure shared encoders, but share_cnn_encoders is not defined in the RslRlPpoAlgorithmCfg dataclass.
This causes validation errors with strict config systems (e.g., Hydra/OmegaConf) when users attempt to set this parameter in YAML, as it's treated as an unexpected argument.
Consider adding share_cnn_encoders: bool = False to RslRlPpoAlgorithmCfg to support this feature properly and ensure type safety.