[WIP] Add ERA5-Land dataset support to DataWrangling#400
Draft
xkykai wants to merge 2 commits into
Draft
Conversation
Introduce ERA5HourlyLand and ERA5MonthlyLand as subtypes of ERA5Dataset, reusing the entire ERA5 download/split/read/regrid pipeline and overriding only what differs for the land-surface reanalysis product: - 0.1° native grid (3600, 1801, 1) with half-cell longitude offset (-0.05, 359.95), overriding the inherited 0.25° single-level size. - Land-surface variables: skin/soil temperature (4 levels), volumetric soil water (4 layers), 2m temperature/dewpoint, snow depth and snow water equivalent, with verified netCDF short names. - No unit conversion (instantaneous analysis fields) and no inpainting, so ocean cells stay masked as NaN via nan_convert_missing. In the CDS extension, add cds_product / cds_varnames / nc_varnames / coord_vars dispatch for the land datasets, mapping to the reanalysis-era5-land and reanalysis-era5-land-monthly-means products. Refactor build_era5_request to set product_type through a dispatched set_product_type! hook: ERA5 (single/pressure) keeps ["reanalysis"], while ERA5-Land omits the key entirely (the land products reject it). Verified against live CDS downloads: all netCDF short names match, the 0.1° coordinate spacing is correct, and ocean cells load as NaN while land cells carry physical values. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xkykai
changed the title
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
The ERA5-Land native size used Ny = 1801 (the file's latitude row count), but `construct_native_grid` builds the LatitudeLongitudeGrid with `size=(Nx,Ny)` over `latitude=(-90,90)`. With Ny=1801 the cells span 180/1801 ≈ 0.09994°, misregistered from the true 0.1° latitudes by up to half a cell near the equator. Following the single-level convention where the cell count is one less than the file row count, set Ny=1800: the 1800 cells then sit at exactly -89.95:0.1:89.95, and the extra file row folds in through `AverageNorthSouth` mangling (data_lat_count 1801 == Ny + 1). This touches neither the variable dicts nor the verified netCDF short names. Add a network-free "ERA5-Land dataset, metadata, and dispatch" testset to test/test_cds_downloading.jl covering: construction and supported_datasets membership, the 0.1° size/interfaces with exact cell spacing and the AverageNorthSouth mangling, the variable-name dicts, trait dispatch, all_dates, the CDSAPIExt dispatch helpers, product_type omission via the set_product_type! hook, and the padded BoundingBox area (which is only clean because Δφ = 180/1800 = 0.1, pinning the Ny=1800 choice). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the ERA5-Land reanalysis (
reanalysis-era5-land) as a first-class dataset inDataWrangling, so land-surface state (soil temperature, soil moisture, skin temperature, snow) can be loaded asFieldTimeSeries/ single-snapshotFields on the native grid or regridded onto a target land grid.The motivating use is observational ground truth for calibrating land models (e.g.
SlabLand) online against reanalysis — the data layer that Phases 2–3 of the land-training effort depend on.Approach
ERA5-Land is implemented as subtypes of the existing
ERA5Dataset(ERA5HourlyLand,ERA5MonthlyLand), reusing the entire CDS download / NetCDF-split /retrieve_data/ region-crop / regrid pipeline. Only what genuinely differs from ERA5 single-levels is overridden:Base.size→(3600, 1801, 1)and specialized longitude/latitude interfaces (the inherited 0.25° values would be silently wrong).conversion_units = nothing(no accumulation→flux conversion).default_inpainting = nothingso masked ocean cells stayNaN(to be masked in a loss, not smeared into coastal land).reanalysis-era5-landproduct rejectsproduct_type, sobuild_era5_requestnow sets it via a dispatchedset_product_type!hook (ERA5 →["reanalysis"]; ERA5-Land → omit). The landUnionmethod is strictly more specific than the abstractERA5Dataset, so it resolves unambiguously.Files
src/DataWrangling/ERA5/ERA5_land.jl(new)src/DataWrangling/ERA5/ERA5.jl,src/DataWrangling/DataWrangling.jl,src/NumericalEarth.jl— includes + exportsext/NumericalEarthCDSAPIExt.jl—cds_product/cds_varnames/nc_varnames/coord_varsdispatch +set_product_type!refactorTesting
Verified against a live CDS download (Borneo bbox), not just static checks:
supported_datasets().Metadata/Metadatumbuild;size(ERA5HourlyLand(), :skin_temperature) == (3600, 1801, 1); native grid is 0.1° (N→S).product_type; the single-level request still includes it.skt, stl1, swvl1, t2m, d2m, sde, sdall matched the dict exactly (no corrections needed).NaNocean cells, no inpainting.Notes / follow-ups
reanalysis-era5-land-monthly-means) shares the code path (only itscds_productstring differs) and was not exercised with a live download.PrescribedLand-style coupling type is added: ERA5-Land targets are plainFieldTimeSeriesconsumed by a loss.🤖 Generated with Claude Code