Skip to content

Tkurth/improved state handling#78

Open
azrael417 wants to merge 46 commits intomainfrom
tkurth/improved-state-handling
Open

Tkurth/improved state handling#78
azrael417 wants to merge 46 commits intomainfrom
tkurth/improved-state-handling

Conversation

@azrael417
Copy link
Copy Markdown
Collaborator

@azrael417 azrael417 commented Apr 17, 2026

This branch will contain some major refactoring of preprocessor.py and noise.py. Those two rely on some states with implicitly assumptions and for some features implement order contracts. This MR will clean up all of this because it can lead to semantic problems and data corruption if not aware.

  • rewrites noise.py and preprocessor.py to not accidentally hold on to dead references
  • folds local ensemble dimension into batch dimension for forward pass, simplifies a lot of the noise handling
  • added more data loader tests related to number of samples in epochs and cycles.
  • added external source fix for split file DALI loader which silently duplicated samples. Invalid samples are excluded from the file list now

Do not merge before #77

@azrael417 azrael417 requested a review from bonevbs April 17, 2026 09:24
@azrael417 azrael417 self-assigned this Apr 17, 2026
@azrael417 azrael417 marked this pull request as ready for review April 17, 2026 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant