Conversation
anagainaru
left a comment
There was a problem hiding this comment.
Looks good, I recommend the two changes so you pass the CI. I cannot run this since I don't have your data/model but I changed it to work with my model and it seems like it's running fine.
In addition to the changes requested, you need types-pyyaml in pyproject.
diff --git a/pyproject.toml b/pyproject.toml
index 8b252e9..e3657bf 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -20,7 +20,8 @@ dependencies = [
"transformers (>=4.57.1,<5.0.0)",
"river (>=0.21.0,<0.24.0)",
"evidently (>=0.4.0,<0.8.0)",
- "matplotlib (>=3.10.7,<4.0.0)"
+ "matplotlib (>=3.10.7,<4.0.0)",
+ "types-pyyaml (>=6.0.12.20250915,<7.0.0.0)"
]
[dependency-groups]
diff --git a/poetry.lock b/poetry.lock
index 426ce3d..c232f3e 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -4874,6 +4874,18 @@ files = [
{file = "types_pytz-2025.2.0.20251108.tar.gz", hash = "sha256:fca87917836ae843f07129567b74c1929f1870610681b4c92cb86a3df5817bdb"},
]
+[[package]]
+name = "types-pyyaml"
+version = "6.0.12.20250915"
+description = "Typing stubs for PyYAML"
+optional = false
+python-versions = ">=3.9"
+groups = ["main"]
+files = [
+ {file = "types_pyyaml-6.0.12.20250915-py3-none-any.whl", hash = "sha256:e7d4d9e064e89a3b3cae120b4990cd370874d2bf12fa5f46c97018dd5d3c9ab6"},
+ {file = "types_pyyaml-6.0.12.20250915.tar.gz", hash = "sha256:0f8b54a528c303f0e6f7165687dd33fafa81c807fcac23f632b63aa624ced1d3"},
+]
+
[[package]]
name = "types-requests"
version = "2.32.4.20250913"Let me know when it's ready to be merged and I will approve.
examples/slac_fel/utils.py
Outdated
| X_scaled: Tensor = input_scaler.transform(X_raw) | ||
| y_scaled: Tensor = output_scaler.transform(y_raw) |
There was a problem hiding this comment.
diff --git a/examples/slac_fel/utils.py b/examples/slac_fel/utils.py
index e52df9c..59d347d 100644
--- a/examples/slac_fel/utils.py
+++ b/examples/slac_fel/utils.py
@@ -181,8 +181,8 @@ def load_fel_data(
X_raw = torch.tensor(df[input_cols].values, dtype=torch.float32)
y_raw = torch.tensor(df[output_cols].values, dtype=torch.float32)
- X_scaled: Tensor = input_scaler.transform(X_raw)
- y_scaled: Tensor = output_scaler.transform(y_raw)
+ X_scaled: Tensor = input_scaler(X_raw)
+ y_scaled: Tensor = output_scaler(y_raw)
return X_scaled, y_scaled, df.indexThere was a problem hiding this comment.
these are botorch transformers right now (from the model source). I can save and load them differently if we don't want the added dependency
There was a problem hiding this comment.
The CI was complaining, it's prob not clear to mypy what those are.
|
I updated the way the model is being loaded (there was a bug before), and the config for the drift detector (not confident that these make sense though). There seems to be something off with the logging, so I don't want to mark it as ready to merge yet. |
I will try to run it as well tomorrow before our meeting |
|
I updated the docstrings to be in Google style, which seems to be the dominant/intended format across the codebase (although config/configuration.py and some example modules are using NumPy style) |
|
After I rebase this on main you can use in the toml file the following option (introduced in #88): to save checkpoints after each CL loop and only keep the most recent one. |
Summary
This pull request introduces support for the SLAC FEL continuous-learning experiment, including configuration, data utilities, and integration with the main example harness. The changes add a new TOML config file, a data utility module, and update the configuration and dependency management to support the new experiment.
Approach
examples/slac_fel/slac-fel.tomlconfiguration file with model, data, training, continual learning, drift detection, logging, and visualization settings for the SLAC FEL experiment.examples/slac_fel/utils.pywith data-loading and preprocessing utilities, including dataset, dataloader, feature config, scaler loading, window discovery, and legacy window splitting for SLAC FEL data.examples/utils.pyto support the SLAC FEL experiment by returningSLAC_FELharness whencfg.data.nameisslac-fel.Configuration and dependency updates:
ModelCfgandDataCfgdataclasses insrc/config/configuration.pyto support optional model config paths and window size parameters for time-windowed datasets.pyproject.tomlto include new dependencies (types-pyyaml,botorch) required for SLAC FEL experiment and added theexamplesdirectory to the package list.This PR also includes a fix for #85 by @Zilinghan
Dependencies
botorchnote: can work on implementation to remove the need for it
Testing Plan
python -m app --helpDocumentation
Checklist
ruff format --checkruff check .mypy srcpytest -q