Skip to content

On-the-fly nested-atmosphere boundary conditions + Davies relaxation#388

Open
glwagner wants to merge 73 commits into
mainfrom
glw/nested-atmosphere-bc
Open

On-the-fly nested-atmosphere boundary conditions + Davies relaxation#388
glwagner wants to merge 73 commits into
mainfrom
glw/nested-atmosphere-bc

Conversation

@glwagner

Copy link
Copy Markdown
Member

Follow-up to #386. Derives a Breeze nested child's lateral boundary conditions and interior Davies relaxation on the fly from a coarse parent atmosphere's raw state — eliminating the materialized breeze_prognostic_state parent series. Builds on the ERA5PrescribedAtmosphere / PrescribedAtmosphere refactor from #386 (now on main).

The abstraction

  • ParentStateBoundary (NestedSimulations) — a multi-field boundary condition: at each boundary-face node it interpolates a NamedTuple of parent FieldTimeSeries (each with its own interpolation transform — log for the strictly-positive p/T, identity otherwise) and applies a pointwise transform to the resulting state. Carrier-only; knows nothing about which prognostic or dataset.
  • ParentStateTarget — the interior-relaxation analogue: a Relaxation target(x, y, z, t) using the same _query_source machinery at an arbitrary node.
  • Both use Oceananigans' interpolate(f, …) (Add interpolate(f, …): map source values through f before blending CliMA/Oceananigans.jl#5726) for the log-space interpolation; the local shim is gone.

Breeze wiring (in NumericalEarthBreezeExt)

  • Prognostic transforms (isbits callables over the pointwise maps): AirDensity, MomentumDensity, EnergyDensity, MoistureDensity (density-weighted, for the BCs); LiquidIcePotentialTemperature, TotalWater (specific, for the relaxation).
  • atmosphere_simulation(grid; parent_atmosphere, relaxation_rate, relaxation_mask):
    • lateral BCs for ρ, ρu, ρv, ρe, <moisture> via ParentStateBoundary (ρu/ρv → NormalFlowBC, the rest → ValueBC), merged per-side with the coupling bottom-flux BCs;
    • interior Davies relaxation keyed (u, v, θ, qᵉ) for Breeze's SpecificForcing (u/v toward the raw parent velocities, θ/qᵉ toward on-the-fly specific targets).
    • The parent kwarg was named parent_atmosphere to avoid shadowing Base.parent / Oceananigans parent(field).

Workflow: build parent → atmosphere_simulation(grid; parent_atmosphere)NestedModel.

Dependency

interpolate(f, …) is merged to Oceananigans main but not yet tagged, so this PR points Oceananigans at main via [sources] (root/test/docs). Swap to a [compat] bump once a release is tagged (TODOs in place).

Status / TODO (draft)

  • Rewrite examples/breeze_downscaling_era5.jl to use atmosphere_simulation(grid; parent_atmosphere, relaxation_rate, …), deleting the parent_series materialization loop + single-source parent_boundary_conditions.
  • Tests for ParentStateBoundary / ParentStateTarget (CDS-free, toy parent FTS).
  • Benchmark the on-the-fly cost (vs. a precomputed parent series) — precomputing is a possible later optimization (would move the conversion into parent construction).
  • Not yet exercised locally (dev env can't precompile here); relying on CI against Oceananigans main.

🤖 Generated with Claude Code

glwagner and others added 6 commits June 27, 2026 02:39
…ognostic maps

Extracts air_density / virtual_temperature / potential_temperature /
liquid_ice_potential_temperature / total_water_specific_humidity as @inline scalar maps (GPU-safe),
so the same definitions serve both breeze_prognostic_state (broadcast over Fields) and the upcoming
on-the-fly nesting boundary conditions (per boundary-face node). Renames qᶜ/qⁱ → qᶜˡ/qᶜⁱ. Prep for
the state-derived nesting BC; no behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ng boundary value

Internal carrier (not user-facing): holds a NamedTuple of parent-state FieldTimeSeries + per-source
interpolation transforms (log for p/T, identity otherwise) + a prognostic map. getbc interpolates each
source at the boundary-face node and applies the map → a child prognostic computed on the fly.

Interpolation uses the #5726-style interpolate(func,…) (func-space, no inverse) and un-maps locally
via inverse_transform (log→exp, identity→identity), so log-space pressure interpolation is exp(blend of
log p) — faithful ln-p. Reuses the node/boundary-index/clock helpers. The model-specific maps (Breeze
ρ/ρu/ρθˡⁱ/ρqᵉ) come next, in the ext. CPU smokes: a 2-field transform reconstructs the right face
values; a log-space single source recovers the geometric interpolation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The on-the-fly nesting BC (ParentStateBoundary) uses Oceananigans' interpolate(f, …)
(merged in CliMA/Oceananigans.jl#5726) for log-space interpolation of parent state,
so the local shim was dropped (in #386). Point Oceananigans at main via [sources]
until a release containing #5726 is tagged, then bump [compat] and drop [sources].

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AirDensity / MomentumDensity / EnergyDensity / MoistureDensity: isbits callables
carrying ThermodynamicConstants that map an interpolated parent-state NamedTuple to one
CompressibleDynamics density-weighted prognostic, reusing the pointwise maps. These are
the ParentStateBoundary 'transform' for each child prognostic (ρ, ρu/ρv/ρw, ρθ, ρqᵗ),
replacing the materialized breeze_prognostic_state parent series.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… the fly

Add nested_lateral_boundary_conditions(parent, constants, moisture_name): for each child
prognostic (ρ, ρu, ρv, ρe, <moisture>) build a ParentStateBoundary over the parent's raw
sources (velocity, T, qᵛ, qᶜˡ, qᶜⁱ, p), interpolating p/T in log space and applying the
matching density-weighted transform. ρu/ρv use NormalFlowBC, ρ/ρe/moisture use ValueBC.

atmosphere_simulation gains a `parent` kwarg: when given, these lateral BCs are merged
(per-side) with the coupling bottom-flux BCs. Workflow: parent -> child(parent) -> NestedModel.
Replaces the materialized breeze_prognostic_state parent series for the lateral BCs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…osphere

- ParentStateTarget (NestedSimulations): the interior-relaxation analogue of
  ParentStateBoundary — a Relaxation target(x,y,z,t) that interpolates parent state at
  the node and applies the prognostic transform (reuses _query_source).
- Specific transforms LiquidIcePotentialTemperature / TotalWater for the Davies targets
  (Breeze SpecificForcing applies the rho-weight at kernel time).
- nested_relaxation_forcings(parent_atmosphere, constants; rate, mask): forcings keyed
  (u, v, theta, q_e); u/v relax toward the raw parent velocities, theta/q_e toward
  on-the-fly specific targets. Replaces the materialized breeze_prognostic_state targets.
- atmosphere_simulation gains relaxation_rate / relaxation_mask; with parent_atmosphere it
  merges Davies forcings into `forcing`. Renamed the `parent` kwarg/arg to
  `parent_atmosphere` (avoids shadowing Base.parent / Oceananigans parent(field)).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
glwagner and others added 2 commits June 27, 2026 06:29
parent_state_boundary.jl imported instantiated_location from Oceananigans.Fields but
uses the qualified Oceananigans.instantiated_location, so the import was both stale
(unused) and not-via-owner (owner is Oceananigans). Remove it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment out the other examples so the docs build exercises only
breeze_downscaling_era5 (the on-the-fly nesting BC case); restore before merge.
Add it via the 'build all examples' label (NUMERICAL_EARTH_BUILD_ALL_EXAMPLES=true).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@glwagner glwagner added the build all examples add this label to build all the examples in the PR label Jun 27, 2026
glwagner and others added 20 commits June 27, 2026 17:57
…_grid; ...)

Per the cleaner API: build the child and the NestedModel together rather than threading
the parent through atmosphere_simulation. The relaxation rate/mask are now nesting-level
config (on nested_atmosphere_model) yet still baked into the child's forcing at construction
(tendency-integrated, no rebuild/step-nudge).

- atmosphere_simulation reverts to the generic child builder (drop parent_atmosphere /
  relaxation_rate / relaxation_mask kwargs).
- New `nested_atmosphere_model(parent_atmosphere, child_grid; relaxation_rate, relaxation_mask,
  child_kw...)` (declared in NestedSimulations, method in the Breeze extension): derives the
  lateral BCs + Davies forcings from the parent on the fly, hands them to atmosphere_simulation
  as boundary_conditions/forcing (per-side merge), and wraps in NestedModel(parent, child).
- Exported nested_atmosphere_model from NumericalEarth.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Establishes the rule (AGENTS.md): never unpack a property right after a constructor
(foo(...).model / .child) — provide a constructor that returns what's needed.

- atmosphere_model(grid; ...) returns the Breeze AtmosphereModel.
- atmosphere_simulation(grid; Δt, kw...) = Simulation(atmosphere_model(grid; kw...); Δt).
- nested_atmosphere_model builds the child via atmosphere_model (no .model unpack) and
  returns NestedModel(parent, child) — used directly (no .child unpack).
- Declared/exported atmosphere_model from Atmospheres / NumericalEarth.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enabling layer for the example rewrite onto the nested_atmosphere_model API:
- NestedModel forwards set!(nm; kwargs...) to the child, so the IC can be set on the
  nested model directly (no .child unpack).
- Export nested_lateral_boundary_conditions (stub in NestedSimulations, method in the
  Breeze ext) so a dynamical-init twin can share the parent-derived lateral BCs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the ~125-line parent-derivation block (parent grid + materialized
parent_series + per-snapshot breeze_prognostic_state loop + hydrostatic-pressure
integration) into a single `ERA5PrescribedAtmosphere(region, dates; …)`, and drive
the child through `nested_atmosphere_model(parent, grid; relaxation_rate, relaxation_mask,
…)` — lateral BCs + Davies relaxation are now derived on the fly from the parent's raw
state, no materialized parent prognostic series.

- model: `atmosphere_simulation(grid; …).model` → `nested_atmosphere_model(parent, grid; …)`
  returning a `NestedModel` (no `.model`/`.child` unpack); run via `Simulation(model)`.
- surface BCs: pass bottom-only `FieldBoundaryConditions`, merged per-side with the
  parent-derived lateral BCs inside `nested_atmosphere_model`.
- reference_θ: domain-mean θˡⁱ of the child IC (computed once, reused by `set!`), not
  the (removed) parent_series.
- DFI twin: `atmosphere_model(grid; …)` + shared `nested_lateral_boundary_conditions`.
- animation parent row: reconstruct (ρ, θˡⁱ, qᵗ) on the fly via breeze_prognostic_state
  on the parent's native PressureLevelGrid, sampled with the same `cut_plane`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…gans main)

Oceananigans main renamed `NormalFlowBoundaryCondition` to `OpenBoundaryCondition`
(classification `Open`, scheme `PerturbationAdvection`). Since this branch depends on
Oceananigans main (the `interpolate(f, …)` [sources] pin), the old name no longer exists:
Julia 1.12 emits only a "binding undeclared at import time" warning at precompile, but any
call site (the nested lateral-BC builder) would throw at runtime — i.e. the example's
nested_atmosphere_model path. Rename all imports, the OpenBoundaryCondition(condition; scheme)
calls, and stale comments. Verified: precompiles with the warning gone and loads cleanly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nscaling_era5

The example (built in docs CI under the "build all examples" label) does
`using CloudMicrophysics` (OneMomentCloudMicrophysics via BreezeCloudMicrophysicsExt),
`using JLD2` (jldsave), and `using NaturalEarth` — none were in docs/Project.toml, so the
docs build failed at `using CloudMicrophysics` ("Package not found in current path").
Verified locally: docs resolves cleanly with the three added and BreezeCloudMicrophysicsExt
loads (OneMomentCloudMicrophysics accessible).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the GeoInterface/NaturalEarth border-flattening out of the breeze_downscaling_era5
example into an extension. `natural_earth_lines(name; scale=50)` returns NaN-separated
(lons, lats) vectors for a Natural Earth feature layer, ready for `lines!`. Backend-agnostic
(no Makie dependency) — drawing stays in the caller.

- Core: `function natural_earth_lines end` stub + export in NumericalEarth.jl.
- ext/NumericalEarthNaturalEarthExt.jl: triggered by ["NaturalEarth", "GeoInterface"]
  (using NaturalEarth pulls GeoInterface transitively, so it fires on `using NaturalEarth`).
- Project.toml: NaturalEarth + GeoInterface weakdeps/extensions/compat.
- Example: drop `import GeoInterface as GI` + the inline append_border!/natural_earth_lines
  block; call the exported function. docs/Project.toml gains GeoInterface.

Verified locally (docs project): the ext loads and natural_earth_lines returns the US
state + country border vectors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
construct_native_grid mapped the bbox into the dataset's native longitude convention and labeled
the grid there, dropping the caller's convention. For ERA5 ([0,360] native) driven from a [-180,180]
bbox (e.g. a US-centered nest), the parent grid came out in [0,360] while the child LAM is [-180,180],
so the on-the-fly nesting BCs failed the "source brackets child" check in longitude. Relabel the grid
longitudes back to the bbox convention (shift is 0 when they already match, ±360 when they differ;
data is array-indexed so ordering is unchanged). Verified on a GPU ERA5→Breeze nest: the longitude
bracket now passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…'s model build

nested_atmosphere_model now supplies overridable defaults for a nested limited-area atmosphere:
- microphysics = default_nested_microphysics() — 1-moment mixed-phase when Breeze's CloudMicrophysics
  extension is loaded (resolved at call time via Base.get_extension), else SaturationAdjustment
- momentum_advection = WENO(order = 9)
- coriolis = SphericalCoriolis()
- a compressible split-explicit `dynamics` (UpperSponge over the top `damping_depth` m at `damping_rate`,
  no divergence damping) built from `surface_pressure`/`reference_potential_temperature`, plus a matching
  ρw Rayleigh lid sponge added to `forcing`.

The example's model build drops to the IC-derived args (surface_pressure, reference_potential_temperature,
relaxation, surface BCs); the DFI twin takes its moisture key from `model.microphysics`. Construction
validated on CPU (the defaults build a NestedModel; ext-detected microphysics resolves correctly).

Note: set_to_mean!/compute_reference_state recompute only the anelastic ReferenceState, not the
split-explicit ExnerReferenceState used here, so the perturbation reference is set explicitly via
reference_potential_temperature (reference_θ).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ρ = p / (Rᵐ T) on the grid of `temperature`, with the moist mixture gas constant
Rᵐ = qᵈ Rᵈ + qᵛ Rᵛ, qᵈ = 1 − qᵛ − qᶜ − qⁱ — the same EOS as hydrostatic_pressure_from_surface,
so a density built from that pressure is mutually consistent. Optional qᵛ/qᶜ/qⁱ (dry if omitted);
returns a CenterField. A focused utility for initializing a compressible model's density from an
analysis temperature + (hydrostatic) pressure: `set!(model; ρ = density_from_pressure(T, p; …))`,
replacing the IC use of breeze_prognostic_state. Verified: dry/moist values match p/(Rᵐ T).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…on't bracket)

The on-the-fly nesting BCs require the parent to bracket the child in x/y, but the child
legitimately extends below the parent's lowest level in z (ERA5 pressure-level data doesn't reach
the surface) — the strict z-bracket check (inherited from Relaxation-on-FTS) wrongly rejected this,
blocking the ERA5→Breeze example. Two changes:

- validate_source_bracket: enforce bracketing in x/y only (a too-small horizontal parent region is a
  real error); the vertical is allowed to be exceeded.
- clamp_to_source_z: clamp the query's z into the source's center-z range before interpolating, so an
  out-of-range child node returns the parent's edge value (constant extrapolation). This is explicit
  and deterministic — relying on Oceananigans' out-of-grid interpolation was halo-dependent and gave
  the wrong value above the top level (a test showed the bottom clamped correctly but the top didn't).

Test (CPU): a parent T linear in z, child spanning below and above; in-range nodes recover T exactly,
out-of-range nodes return the clamped edge value (102 below, 108 above).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…raction

- breeze_nested_atmosphere: extract parent_state_fields helper shared by
  nested_lateral_boundary_conditions and nested_relaxation_forcings
- default_nested_dynamics: drop the redundant reference_potential_temperature
  guard (CompressibleDynamics stores nothing unconverted); guard only
  surface_pressure
- hydrostatic_pressure: factor the shared interior temperature+moisture array
  extraction into interior_temperature_and_moisture, used by both
  hydrostatic_pressure_from_surface and density_from_pressure

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The breeze_downscaling_era5 docs example builds the nested model on a
terrain-following LatitudeLongitudeGrid on the GPU, which exposed three
host-side scalar-indexing / GPU-IR failures in the nesting machinery:

- lid_sponge_mask: znode(1,1,Nz+1, grid, …) reads the device z-array on a
  terrain-following grid; compute it on a CPU mirror of the grid (as Breeze's
  reference-state build does) so the one-off top-of-domain lookup doesn't
  scalar-index a GPU array.

- nested_relaxation_forcings: the u/v Davies targets were raw parent
  FieldTimeSeries, taking Oceananigans' FTS-target Relaxation path whose
  validate_fts_target_extent does extrema(znodes(grid)) — scalar-indexing the
  child's 3D terrain-following z-nodes on GPU. Route u/v through
  ParentStateTarget like θ/qᵉ; this also z-clamps to the parent range instead
  of reading unfilled FTS halos.

- _parent_state / ParentStateTarget: mapping a closure over the (sources,
  interpolations) NamedTuples is a dynamic invocation rejected by GPU IR
  validation. Interpolate via explicit tuple recursion (interpolate_sources),
  rebuilding the state NamedTuple from compile-time keys.

Validated on GPU: the nested model now builds successfully.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Breeze 0.7)

Rework the on-the-fly nested relaxation/lid-sponge as concrete per-variable discrete
forcings and simplify breeze_downscaling_era5.jl. NOTE: GPU time-stepping is blocked by
in-kernel interpolation of the parent ERA5 FieldTimeSeries (DatasetBackend) on the
PressureLevelGrid — InvalidIRError (gpu_gc_pool_alloc + throw_methoderror). Model build,
IC, and all tendency kernels compile when that interpolate is stubbed.

ext (breeze_nested_atmosphere.jl):
- Replace ParentStateTarget relaxation with concrete `<: NestedForcing` callables
  (PotentialTemperatureRelaxation, MoistureRelaxation, U/VRelaxation, LidSponge), each a
  discrete `(i,j,k,grid,clock,fields)` returning the full ρᵈ-weighted tendency.
- Bypass ContinuousForcing/DiscreteForcing via
  `materialize_atmosphere_model_forcing(::NestedForcing) = f` + `is_density_tendency_forcing`.
- Add `terrain` kwarg (materialize_terrain! before model build); lid-sponge znode via @allowscalar.

src (parent_state_boundary.jl): rename clamp_to_source_z → z_clamp; remove ParentStateTarget.

example: terrain into constructor; IC below model build; ρ via density_from_pressure; T set
directly with compute_reference_state=true (drops reference_θ); BoundingBox auto-snap (drop
snap_out); Field{Center,Center,Nothing} orography; type-stable lateral_mask (oftype).

Project/docs/test .toml: temporary [sources] pin Breeze → glw/set-reference-state (0.7) for
compute_reference_state (#812); compat 0.7.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lw/nested-atmosphere-bc

# Conflicts:
#	ext/NumericalEarthBreezeExt/breeze_nested_atmosphere.jl
#	src/EarthSystemModels/NestedSimulations/parent_state_boundary.jl
…variable types

Replace the generic ParentStateBoundary (sources/interpolations/transform + interpolate_sources)
with concrete per-variable lateral-BC samplers, mirroring the discrete relaxation forcings. Both
layers now share one set of parent-state samplers.

- src/parent_state_boundary.jl: thin `ParentBoundary{Dim,Side,LX,LY,LZ, V, G}` carrier wrapping a
  concrete `value(X,t)` sampler; getbc supplies the boundary-face node, sampler z_clamps + interpolates.
  Removes ParentStateBoundary/ParentStateTarget/interpolate_sources/inverse_transform/_query_source.
- ext: shared `air_density_at`/`liquid_ice_θ_at`/`total_water_at`/`velocity_at` samplers (interpolate
  raw parent fields + combine, log-space T/p); concrete BC samplers AirDensityBoundary/
  MomentumDensityBoundary/EnergyDensityBoundary/MoistureDensityBoundary; relaxation forcings
  refactored onto the same samplers (store `constants`); `tuple(elem)` notation; `field_name` (no `_`).

Note: still GPU-WIP — the in-kernel parent-FTS interpolate blocker is unresolved (next).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- metadata FTS: match `times` to the grid float type. Float64 times on a Float32
  grid made `interpolate`'s time weight Float64, so the value was Union{Float32,
  Float64} — boxed inside GPU tendency/halo kernels (the InvalidIRError root cause).
- ERA5 `pressure_level_field`: drop the redundant `.* 100`. `ERA5_all_pressure_levels`
  is already Pa (via the `hPa` factor), so the field was 100× too high ([1e4, 1e7] Pa),
  making sampled θ unphysical (~85 K).
- nested samplers interpolate linearly (`identity`) for all parent fields; thread the
  potential-temperature reference pressure pˢᵗ from the child `dynamics.standard_pressure`
  instead of a hardcoded 1e5 (which also leaked Float64 into the combine).
- remove dead prognostic-transform structs (superseded by the discrete samplers).

Validated on CPU: sampler is Body::Float32 and returns a physical θ (~318 K).
Requires the Oceananigans method `interpolate(func, X, Time, ::AbstractField, …)`
(drops the time index for a static `Field` — e.g. the pressure-level coordinate);
pending upstream.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… on the parent grid

Replace the per-node, in-kernel thermodynamic combine (which never compiled on GPU) with a
precomputed derived-state design that compiles + steps cleanly on GPU (validated on an H100).

- StateExchanger (ext/breeze_state_exchanger.jl): a fused `_compute_child_prognostics!` kernel
  computes the child's Breeze prognostics — dry density ρᵈ = ρ(1−qᵗ), dry-weighted ρθ/ρu/ρv, and
  total-weighted ρqᵛ — from the raw ERA5 state ON THE PARENT GRID, stored as FieldTimeSeries.
  Density weighting matches Breeze `establish_densities!`/`set!`. Condensate inputs are configurable
  (parent / another source / ZeroField ⇒ qᵗ = qᵛ). `exchange_state!` refreshes them from the parent.

- NestedModel gains an `exchanger` field; `time_step!(nested, Δt)` / `update_state!` call
  `exchange_state!(exchanger, clock.time)` before the child steps, so the child's boundary conditions
  and forcings see current parent-derived prognostics.

- nested_atmosphere_model now builds those FTS and drives the child through the generic FTS machinery:
  `parent_boundary_conditions` (Interpolated BCs; energy keyed :ρe) + `parent_forcings` (Relaxation).
  The tendency/halo kernels only interpolate a plain FieldTimeSeries — no combine, no z_clamp, no
  MappedData — so they specialize on the same type Breeze compiles for any FTS forcing. Removed the
  per-node samplers, the NestedForcing bypass, and the now-unused nested_lateral_boundary_conditions.

Requires (Oceananigans, pending upstream): FTS `Relaxation` vertical-clamp
(`validate_fts_target_extent` horizontal-only), mirroring the Interpolated BC — a nested child
extends below the parent's lowest level.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
glwagner and others added 6 commits July 3, 2026 00:46
…he convective layer)

Default damping_depth was a fixed 3 km, too thin — vertically-propagating gravity/acoustic waves
reflected off the rigid top and grew. Default it instead to default_lid_depth(grid) = grid.Lz/4
(~5 km for a ~19.5 km column, sponge base ~15 km): deep enough to absorb genuine top reflection now
that the consistent exchanger-derived IC removes the wall-mode source, yet thin enough to leave the
deep-convective layer (~12–16 km) undamped. Uses the grid's vertical extent Lz — not the top node
height, which on a terrain-following grid carries the terrain offset — so it scales with any model top.
Lifts both the dynamics UpperSponge and the ρw Relaxation. CPU smoke on defaults (no kwargs): max|w|
stays bounded (~4–6 m/s); the remaining ~6 m/s is a lateral wall residual (ρᵈ relaxation, tracked
separately), not a top runaway.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
C2 (thin Lz/4 sponge, defaults-only) showed max|w| creeping to ~6 from a
persistent LATERAL-WALL residual (viz: edge striping up the column), which a
thin top sponge can't damp and the old thick sponge only masked. Root cause is
the un-relaxed near-wall mass/density field: the Davies zone relaxed ρθ/ρu/ρv/
moisture but NOT the density. Add ρᵈ = prognostic.ρᵈ to the relaxed var set,
matching WRF (nudges dry mass μ) and MPAS (nudges ρ). parent_forcings already
handles any var set; ρᵈ is the compressible-core density prognostic, so the
Relaxation targets it directly. Expected: max|w| tightens back toward ~4 with a
thin (convection-protecting) sponge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A rebase re-introduced `include("parent_state_boundary.jl")` into
NestedModels.jl while the file itself stayed deleted (from fdf6e63),
leaving HEAD unloadable (include of a missing file). Remove the stray
include so NumericalEarth loads again; ParentBoundary stays gone.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Documents the 12 h convection-permitting downscaling runs across the 12→0.5 km resolution sweep
(gpu-prod H100s + cpu partition): the validated stable configuration (option-A exchanger IC,
domain-scaled Lz/4 lid sponge, per-side momentum BCs + ρᵈ Davies relaxation, WENO-5 momentum +
bounded per-tracer moisture advection, no explicit closure, adaptive-Δt wizard), the fixed-domain
resolution refinement (one shared ERA5 region), the compute allocation, the animation/slice outputs,
and the harness/batch/launcher files in scratch_runenv/. Includes the explicit-nudge over-relaxation
constraint (rΔt ≲ 2) motivating the Δt-scaled Davies rate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three CI failures, all in our nested-atmosphere code:

- `fill_halo_regions!` was imported from Oceananigans.Fields in
  breeze_state_exchanger.jl; its owner (Base.which) is
  Oceananigans.BoundaryConditions. Import it from there so
  check_all_explicit_imports_via_owners passes.

- NestedModels carried stale `znode`/`Center` imports (in
  interpolated_fts_boundary.jl) left over from the removed
  parent_state_boundary.jl. Drop them.

- The per-side bc_types refactor dropped the guard that rejected a
  `schemes` entry for a field that is NormalFlow on no side (the scheme
  would be silently ignored). Restore it, adapted to the per-side design:
  error only when none of the field's sides is a NormalFlowBoundaryCondition.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment on lines +43 to +55
# Cosine-ramp Davies mask: 1 at the lateral walls, ramping to 0 over the outermost `width` cells.
# Captures plain numbers so the closure is GPU-compilable.
function davies_relaxation_mask(grid, width)
λ₁, λ₂ = extrema(λnodes(grid, Face(), Center(), Center()))
φ₁, φ₂ = extrema(φnodes(grid, Center(), Face(), Center()))
Nx, Ny, _ = size(grid)
w = width * max((λ₂ - λ₁) / Nx, (φ₂ - φ₁) / Ny)
return (λ, φ, z) -> begin
d = min(λ - λ₁, λ₂ - λ, φ - φ₁, φ₂ - φ)
d >= w && return zero(λ)
return oftype(λ, (1 + cos(π * d / w)) / 2)
end
end

@ewquon ewquon Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Cosine-ramp Davies mask: 1 at the lateral walls, ramping to 0 over the outermost `width` cells.
# Captures plain numbers so the closure is GPU-compilable.
function davies_relaxation_mask(grid, width)
λ₁, λ₂ = extrema(λnodes(grid, Face(), Center(), Center()))
φ₁, φ₂ = extrema(φnodes(grid, Center(), Face(), Center()))
Nx, Ny, _ = size(grid)
w = width * max((λ₂ - λ₁) / Nx, (φ₂ - φ₁) / Ny)
return (λ, φ, z) -> begin
d = min- λ₁, λ₂ - λ, φ - φ₁, φ₂ - φ)
d >= w && return zero(λ)
return oftype(λ, (1 + cos* d / w)) / 2)
end
end
# Ramp shapes (isbits callables) for a nudging zone: weight vs. normalized distance from the wall, s ∈ [0, 1].
# Contract: ramp(0)=1, ramp(1)=0, monotone between.
struct CosineRamp end
struct SmoothStepRamp end
@inline (::CosineRamp)(s) = (1 + cos* s)) / 2
@inline (::SmoothStepRamp)(s) = 1 - s^2 * (3 - 2s)
# Davies mask: 1 at the lateral walls, ramping to 0 over the outermost `width` cells.
function davies_relaxation_mask(grid, width; ramp = CosineRamp())
λ₁, λ₂ = extrema(λnodes(grid, Face(), Center(), Center()))
φ₁, φ₂ = extrema(φnodes(grid, Center(), Face(), Center()))
Nx, Ny, _ = size(grid)
w = width * max((λ₂ - λ₁) / Nx, (φ₂ - φ₁) / Ny)
return (λ, φ, z) -> begin
d = min- λ₁, λ₂ - λ, φ - φ₁, φ₂ - φ)
s = clamp(d / w, zero(d), one(d))
return oftype(d, ramp(s))
end
end

Is now a good time to change the ramping function to something more computationally efficient?

@glwagner glwagner Jul 3, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so! I doubt the precise structure of the ramping matters? So efficiency is good.

Comment on lines +60 to +64
function lid_sponge_mask(grid, depth)
z_top = @allowscalar znode(1, 1, size(grid, 3) + 1, grid, Center(), Center(), Face())
d = convert(eltype(grid), depth)
return (λ, φ, z) -> (s = clamp((z - (z_top - d)) / d, zero(z), one(z)); s * s * (3 - 2s))
end

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
function lid_sponge_mask(grid, depth)
z_top = @allowscalar znode(1, 1, size(grid, 3) + 1, grid, Center(), Center(), Face())
d = convert(eltype(grid), depth)
return (λ, φ, z) -> (s = clamp((z - (z_top - d)) / d, zero(z), one(z)); s * s * (3 - 2s))
end
function lid_sponge_mask(grid, depth; ramp = SmoothStepRamp())
z_top = @allowscalar znode(1, 1, size(grid, 3) + 1, grid, Center(), Center(), Face())
d = convert(eltype(grid), depth)
return (λ, φ, z) -> (s = clamp((z - (z_top - d)) / d, zero(z), one(z)); ramp(s))
end

with the change above

On Breeze ≥0.7 (#812), a bare `set!(model; θ, u)` no longer derives the
prognostic density from the reference state — the returned model carried
dry_density ≡ total_density ≡ 0. Anything that divides by density then
NaNs; concretely the MOST surface-flux coupling (AtmosphereOceanModel /
AtmosphereLandModel) produced NaN stress and sensible heat.

Initialize the model to a resting, hydrostatically balanced state at the
reference potential temperature via
`set!(model; θ, ρ = HydrostaticallyBalancedDensity(; surface_pressure))`,
so the returned atmosphere has a valid density. A later user `set!` of
θ/velocities (without a `ρ` entry) preserves it.

Verified: dry/total density now 0.45–1.20 (was 0), momentum ρu = ρᵈ·u
(was 0), and the coupling fluxes are finite (test_breeze_coupling
Construction 7/7 and the MOST-stress feedback tests pass). The nested
child path is unaffected — initialize_nested_child! fully overwrites this
state with the ERA5-derived IC.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- **Advection (implicit LES, no explicit closure):** `momentum_advection = WENO(order=5)`; per-tracer
`scalar_advection = (ρθ = WENO(5), ρqᵉ = WENO(5, bounds=(0,1)), ρqʳ = …, ρqˢ = …)` — moisture/precip
are positivity-bounded, `ρθ` must **not** be bounded to `[0,1]`; `closure = nothing`.
- **Time stepping:** adaptive `conjure_time_step_wizard!(cfl = 0.7, max_Δt = 20 s)` on the (slower)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

max dt = 20 s is pretty conservative (small) for dx ~ 12 km

glwagner and others added 2 commits July 3, 2026 05:28
On Breeze 0.7 the coupled compressible solver is unstable at the test's
Δt = 10 s (max|w| blows up → θ^γ DomainError); the stable Δt dropped
below 5 s, whereas Breeze 0.6 ran this stably at Δt = 10. The DFI balancer
that would settle the initial state also crashes on SaturationAdjustment
(FieldError: no field qˡ). Both are tracked upstream in
NumericalEarth/Breeze.jl#827.

Wrap the `Time stepping` and `SST responds to fluxes` subtests so the
run! is guarded and the outcome is `@test_broken` — recording them as
broken (not suite errors) and flipping to an error (prompting re-enable)
once the run completes cleanly again. The MOST-stress AtmosphereLandModel
subtest passes (6/6) with the atmosphere_model density-init fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
examples/breeze_downscaling_era5.jl `using CopernicusClimateDataStore` (activates the ERA5 download
extension) but the package was missing from the docs environment, so the docs build errored
"Package CopernicusClimateDataStore not found". The example is build_always=false (built only under
the "build all examples" label), so normal docs CI is unaffected; this lets the labeled build resolve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread ext/NumericalEarthBreezeExt/breeze_state_exchanger.jl
Comment on lines +18 to +19
# ρθ = ρᵈ · θˡⁱ, ρu = ρᵈ · u, ρv = ρᵈ · v ← DRY-weighted (energy + momentum)
# ρqᵛ = ρ · qᵛ ← TOTAL-weighted (moisture mass density)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent use of ρ vs ρᵈ here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the dycore does actually evolve ρ · qᵛ = ρᵛ so this may be correct?

Comment thread ext/NumericalEarthBreezeExt/breeze_state_exchanger.jl Outdated
Comment thread src/NestedModels/nested_model.jl Outdated
glwagner and others added 10 commits July 3, 2026 12:05
…lization plan

Compute allocation moves to the gpua100x4 (4×A100-40GB) and gpua100 partitions
with gpuprod H100 nodes reserved for CPU-side rendering; adds a Visualization
section (per-run CairoMakie render jobs chained via afterok, mp4 + final panel,
cross-run gallery) and documents the recreated scratch_runenv (now gitignored).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gpua100 node failed two consecutive boot probes (16+ min silent vs
gpuprod's 1m35s spin-up), so the fifth sweep case rides an idle H100 node
instead; the four A100s keep the 6–0.5 km cases.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Eliot Quon <eliot@aeolus.earth>
…tecture

on_architecture has no Base.ReshapedArray method (unlike SubArray/OffsetArray),
so host data arriving reshaped — e.g. a 2-D NetCDF variable reshaped to
(Nx, Ny, 1) — fell through the generic identity fallback and reached GPU
kernels as CPU memory, failing _set_region_kernel! compilation inside
mean_surface_pressure on any GPU nested build. Rewrap the transferred parent
instead; TODO upstream the method to Oceananigans.Architectures.

Verified: 12 km ERA5→Breeze GPU smoke (previously dying at kernel compilation)
now builds and completes 30 iterations with max|w| ≈ 2 m/s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed opts out

Adds an `initialize` keyword (default true, preserving the 92dbf96 MOST-coupling
density fix for every other caller); nested_atmosphere_model passes
initialize = false so the child enters initialize_nested_child! bare, exactly as
before 92dbf96 — the construction default should not imprint 285 K / 101325 Pa
anchored state on a child whose full state and reference are derived from the
parent.

Honest scoping: this is NOT a fix for the balance-twin DomainError blowup seen
on this branch — that failure reproduces with and without the resting init and
is nondeterministic on identical configs (uninitialized-read suspect, tracked
toward Breeze#827; evidence table in the run notes). It is a semantic
correction that also makes 92dbf96's "nested path unaffected" promise
structurally true.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ires

The branch tip carries a nondeterministic adiabatic-balance-twin blowup on the
CPU path (toward Breeze#827; evidence dossier in scratch_runenv). The sweep
pins Breeze to a local branch with only the #821 acoustic-substep reland
reverted, and is guarded by the smoke max|w| gate plus a per-run blowup
tripwire. Also records the two same-day fixes this campaign surfaced.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Δt=10 instability (Breeze #827) fails differently by architecture:
CPU throws a θ^γ DomainError, but GPU's CUDA `pow` returns NaN and the
run completes. The prior form keyed the broken marker on "did run! throw"
(`stepped`), so on GPU `stepped=true` → @test_broken recorded an
unexpected-pass error and the follow-on NaN assertions failed.

Fold the full would-be-success check into a single @test_broken block:
it records broken on either arch (CPU throw or GPU NaN→false) and flips to
an error (prompting re-enable) only once the run integrates cleanly to a
finite, changed state again. Verified: CPU test_breeze_coupling is 9 pass
/ 2 broken / 0 errors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`time_step!` refreshed the exchanger at the pre-step time `t`. Because the
exchanger positions a 2-level window around the passed time, refreshing at
`t` leaves the whole step extrapolated whenever `t` lands on a parent time
level — i.e. at every hourly ERA5 crossing (12 of them in a 12 h run).

Refresh at `t + Δt` so the window brackets the step end and the step's
late RK stages stay in-window. A 2-level window still cannot span a parent
interval, so on a crossing step a ≤Δt start-side extrapolation remains —
acceptable, and far smaller than the prior whole-step extrapolation.
`update_state!` is unchanged (no Δt; not the per-step path).

Verified: test_nested_simulation StateExchanger window-cycling 8/8 and
parent_boundary_conditions 9/9 still pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The child's post-initialization adiabatic (DFI) balance was hardcoded to
`set!(nested_model; balancer = true)`. On the ERA5-derived nested IC the
default balancer (fully-explicit SSP-RK3 twin at 0.85·Δz_min/c) can drive
a pathological cell's pressure negative → Exner DomainError at init,
independent of the acoustic-substepper reland (the twin doesn't use it).

Thread a `balancer` kwarg (default `true`, preserving behavior) through
`nested_atmosphere_model(child_grid, dataset; …)` → `initialize_nested_child!`
→ the `set!` call. `balancer = false` skips the balance (to isolate whether
the interpolated IC steps stably on its own); an `AdiabaticBalancer(Δt=…)`
runs a gentler excursion. Enables debugging the init blowup without editing
library internals.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread src/NestedModels/interpolated_fts_boundary.jl Outdated
kaiyuan-cheng and others added 2 commits July 6, 2026 14:33
The derived 2-level window is a pure function of the two resident parent
levels, so its values change only when the bracket moves. Recomputing on
every intra-interval child step just reproduces identical values, so gate
compute_child_prognostics! on `moved || force` and spare the hot path two
parent-grid kernels + halo fills per step. `force=true` fills the window
once at construction.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The nested child blew up at every hourly parent-data seam (t=3600, 7200, ...).
Root cause: the StateExchanger held the parent-derived child prognostics in a
2-level windowed FieldTimeSeries, but NestedModel.time_step! brackets t+Δt — so
on a step crossing a parent time node the child's start-side sub-stage query
(t < node) fell one interval BELOW the resident window and read a stale/wrong
boundary + Davies target. In single precision the marginal compressible
dynamics amplified that wrong-target kick into a domain-wide NaN at the seam.

Fix: use a 3-level window positioned start = clamp(n1-1, 1, N-2) so it spans the
crossing and every sub-stage query stays resident. A synthetic 12 h ERA5-nested
run now crosses all hourly seams and reaches t=7200 with no relaxation/sponge
crutches (base Davies rate, no density ramp, no frame sponge).

Also make Interpolated getbc's boundary-evaluation time type-stable by threading
the grid float type through clock_time (avoids a Union at Float32/GPU), and drop
three stale explicit imports flagged by test_quality_assurance.

Adds a deterministic temporal-seam test (windowed crossing-step query is finite,
physical, and exact against a linear-in-time parent) and updates the
window-cycling test for the 3-level window.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build all examples add this label to build all the examples in the PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants