fix(isaaclab): stabilize G1 locomotion when training past 2000 iterations#9
Open
Abhishek21g wants to merge 1 commit into
Open
fix(isaaclab): stabilize G1 locomotion when training past 2000 iterations#9Abhishek21g wants to merge 1 commit into
Abhishek21g wants to merge 1 commit into
Conversation
…ar#4) Auto-apply long-run hyperparam overrides when training G1/H1 past the tuned 2000-iter budget. Add repro script, training observability tools, and root-cause documentation for the reward cliff in issue amazon-far#4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses reward cliff at ~4k iterations when extending G1 locomotion training to 5k (#4).
Root cause: G1/H1 hyperparams are tuned for ≤2000 iterations (
ema_decay=0.95, 32 learning epochs, advantage normalization). Continuing training past convergence causes unstable CFM ratio updates.Fix: Auto-apply long-run stability overrides in
train.pywhen--max_iterations > 2000on humanoid locomotion tasks (skipped if user passes explicitagent.algorithm.*overrides so the cliff remains reproducible).Also adds:
scripts/reproduce_g1_cliff.shtools/fpo_training_receipts— per-run cliff doctortools/fpo_run_observatory— multi-run HTML dashboard vs paper baselinestools/docs/FPO_CLIFF_RC.md— full root-cause writeupGPU verification status
Paste W&B / TensorBoard screenshots here after GPU run.
Test plan
pytest tools/fpo_training_receipts/tests/(3 passed)pytest tools/fpo_run_observatory/tests/(3 passed)bash isaaclab_experiments/scripts/reproduce_g1_cliff.shon Linux + NVIDIA GPUfpo-observatory scan logs/isaaclab_fpo/g1_flat_flow/<run>returns no red grades for fix runWorkaround (no code change)
Stop at 2000 iterations, or manually set
ema_decay=0.99,num_learning_epochs=16,normalize_advantage=falsefor longer runs.Fixes #4