Skip to content

Simplify the canonical data loading boundary#823

Draft
jder wants to merge 2 commits into
fomo/delete-static-data-correctors-816from
codex/canonical-reader-llc
Draft

Simplify the canonical data loading boundary#823
jder wants to merge 2 commits into
fomo/delete-static-data-correctors-816from
codex/canonical-reader-llc

Conversation

@jder

@jder jder commented Jul 24, 2026

Copy link
Copy Markdown
Member

Summary

  • rebase the canonical-reader refactor needed by Add opt-in Rust OM4 data loader #800 onto the LLC support landed in Add LLC CPU data canonicalization #670
  • replace the overlapping DatasetSpec / CanonicalDataset / OceanData abstractions with DataLayout, CanonicalSource, BatchPreprocessor, HostBatch, and ModelBatch
  • make the storage-independent reader accept explicit time indices and canonical channel names
  • push prognostic-versus-boundary selection into read requests instead of maintaining sliced source objects
  • move OM4-specific canonicalization into Om4DataSourceConfig.canonicalize_datasets
  • make OM4 and LLC canonicalizers derive their channel selections from variable keys and return the resulting DataLayout
  • keep raw source conventions such as LLC staggered masks and OM4 mask names out of DataLayout
  • preserve the grid-geometry, analysis-ready writer output, and training-progress changes currently on main

This intentionally does not add an LLC Rust loader. It establishes the narrower CanonicalReader seam that the Rust loader can implement later without exposing xarray or source-specific naming to the training pipeline.

Impact

Callers now request ordered canonical channels directly. Canonicalization owns source-specific naming, dimension, mask, and variable-selection rules, while DataLayout describes only the canonical/model-facing result. This removes repeated variable filtering and normalization wrappers from the hot data-loading path.

Validation

  • uvx pre-commit run --all-files
  • CUDA_VISIBLE_DEVICES='' uv run pytest -q -m 'not manual and not cuda'
    • 323 passed, 2 skipped, 10 xfailed

@oa-jder-bot
oa-jder-bot force-pushed the codex/canonical-reader-llc branch from e19e931 to 0272d29 Compare July 24, 2026 20:10
@oa-jder-bot
oa-jder-bot changed the base branch from main to fomo/delete-static-data-correctors-816 July 24, 2026 20:11
@oa-jder-bot
oa-jder-bot force-pushed the codex/canonical-reader-llc branch from 0272d29 to adac072 Compare July 24, 2026 20:19
@oa-jder-bot
oa-jder-bot force-pushed the codex/canonical-reader-llc branch from adac072 to 2703fcf Compare July 24, 2026 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants