[2186] Scaling utils#2231
Open
florianscheidl wants to merge 86 commits into
Open
Conversation
- Track startup_time_seconds: time from run() start to training loop - Track total_training_time_seconds: time in training/validation cycles - Track overall_time_seconds: total wall-clock time from launch to finish - All metrics logged only on root rank to avoid file contention - Metrics written to metrics.json, automatically uploaded to MLflow - Console logs show timing summaries for quick monitoring
- Created .hermes/ directory with skills/, tasks/, docs/ subfolders - Added skills overview (README.md) with task-type skills - Implemented 'planning' and 'metrics' skills - Documented timing metrics task in tasks/2026-04-17-timing-metrics/ - Added agent structure documentation - Updated .gitignore with optional .hermes/ entry
- Added 2-3 month review cycle recommendation - Defined criteria for skill consolidation - Included usage frequency thresholds - Documented when to merge or remove skills
…dl/WeatherGenerator into ekfs/scaling-plots-20260417
…dl/WeatherGenerator into ekfs/scaling-plots-20260417
ekouts
suggested changes
May 8, 2026
ekouts
approved these changes
May 8, 2026
Contributor
Author
|
@clessig, anything blocking? |
clessig
reviewed
Jun 8, 2026
| stddev_all: dict, | ||
| avg_loss: list[float] = None, | ||
| lr: float = None, | ||
| elapsed_training_time_seconds: float | None = None, |
Collaborator
There was a problem hiding this comment.
Can you open a PR that we change plot_train so that we can plot losses against training time.
- Replace --x_type ('step'/'reltime') with --x-axis column selector
('samples', 'elapsed_training_time')
- Add x_axis param to plot_loss_avg (previously hardcoded num_samples)
- Add friendly x-axis labels: 'elapsed training time [s]' when
plotting against elapsed_training_time_seconds
- plot_lr, plot_loss_avg, plot_loss_per_stream, plot_loss_per_run all
now respect x_axis; xlabel is auto-derived from column name
- Remove dead x_type parameter from plot_loss_per_stream
…dl/WeatherGenerator into ekfs/scaling-plots-20260417
Contributor
Author
|
I've implemented the changes discussed here and in #2501. The latter will only have the plotting changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Contains utility we used for scaling experiments, namely to capture elapsed time metrics in the trainer and run_train.
Small code modifications became necessary as we trained on more than 16 nodes. Specifically:
The data extraction and plot generation script are addressed in https://gitlab.jsc.fz-juelich.de/esde/WeatherGenerator-private/-/merge_requests/180.
Example plots and tables:
Issue Number
Fixes #2186
Checklist before asking for review
./scripts/actions.sh lint./scripts/actions.sh unit-test./scripts/actions.sh integration-testlaunch-slurm.py --time 60