feat: add DP memory controls and physical batch sizing#518
Draft
binaryaaron wants to merge 5 commits into
Draft
Conversation
Collaborator
binaryaaron
commented
May 26, 2026
- feat: add DP memory training controls
- fix: estimate preflight VRAM from quantization state
- feat: add physical batch size constraints
- fix: cover DP batching and progress edge cases
Add ghost clipping support and diagnostic loss controls so DP training can reduce and inspect memory pressure on larger models. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Use the model quantization flag rather than PEFT mode to estimate base-weight memory so preflight warnings match actual loading behavior. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Resolve runtime Trainer batch arguments from a physical microbatch cap while preserving the configured effective batch for training and DP accounting. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Preserve validated privacy and batching config paths while fixing ghost clipping adapter saves and generation progress rate rendering. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Tighten the updated tests around resolver contracts, DP adapter saves, and SDK validation so they assert behavior rather than implementation details. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.