You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now, the only way to specify when we want training to automatically end is by specifying the maximum environment steps. This is an not an intuitive number for us humans to reason about in the age of massive parallel environment RL (in practice, nobody would want just part of a rollout if max-steps isn't a multiple of env_count x rollout_steps).
Iteration count is much easier to reason about, and is often supported in other RL frameworks. This PR adds this as an option via CLI flag, as an alternative to max-steps, while leaving max-steps in place for those coding agents that like to show off they can reason about bigger numbers!
add --training-max-iterations as an alternative to --training-max-steps
define one iteration as a complete rollout across every environment and rank,
followed by the configured optimization updates
reject commands that specify both duration flags
preserve the existing step-based default and whole-iteration floor behavior
forward iteration limits through the Slurm launcher and document resume
semantics
Behavior
--training-max-iterations N runs to an absolute target of exactly N complete
trainer epochs; it never stops partway through a rollout. When omitted, --training-max-steps retains its current behavior:
Thanks! Only ask is to avoid backward compatibility whenever possible.
Usually, we can simply recreate the config file, which allows us to avoid messy fallbacks for backward compat logic.
In each release, we simply re-create all the configs for all the released models. Ensuring it keeps working, but also properly aligned with the latest codebase.
The logic in training_limits.py can be moved directly into BaseAgentConfig. Since it's just 1 function, we can have BaseAgentConfig support a resolve_max_epochs(total_envs) function.
Then BaseAgent can simply call self.max_epochs = self.config.resolve_max_epochs(self._total_envs).
We can then also avoid backward compatibility clauses such as training_max_iterations = getattr(config, "training_max_iterations", None)
Also in train_agent.py we should revert from max_iterations = getattr(args, "training_max_iterations", None) to max_iterations = args.training_max_iterations.
Documentation:
Can you add a short note to configuration.rst or developer_tips.rst:
--training-max-iterations N sets the absolute number of rollout/optimization iterations for the experiment. Each iteration collects one num_steps rollout from every environment and performs the configured optimization updates. If an experiment is resumed at iteration 20 with a limit of 100, training continues to iteration 100; it does not run 100 additional iterations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Right now, the only way to specify when we want training to automatically end is by specifying the maximum environment steps. This is an not an intuitive number for us humans to reason about in the age of massive parallel environment RL (in practice, nobody would want just part of a rollout if max-steps isn't a multiple of env_count x rollout_steps).
Iteration count is much easier to reason about, and is often supported in other RL frameworks. This PR adds this as an option via CLI flag, as an alternative to max-steps, while leaving max-steps in place for those coding agents that like to show off they can reason about bigger numbers!
--training-max-iterationsas an alternative to--training-max-stepsfollowed by the configured optimization updates
semantics
Behavior
--training-max-iterations Nruns to an absolute target of exactlyNcompletetrainer epochs; it never stops partway through a rollout. When omitted,
--training-max-stepsretains its current behavior:max_iterations = training_max_steps // total_parallel_envs // agent.num_stepsOn resume, the selected limit remains the absolute total target rather than an
additional number of iterations.