ERA5 population study of buoyancy topologies and LNB-convention sensitivity#4
Open
maresb wants to merge 20 commits into
Open
ERA5 population study of buoyancy topologies and LNB-convention sensitivity#4maresb wants to merge 20 commits into
maresb wants to merge 20 commits into
Conversation
Population study over 569,478 random ERA5 ocean columns (1980-2024) of lifted-parcel buoyancy topologies and LNB-convention sensitivity for the three parcels used by potential intensity, companion to discontinuity_analysis/ (issue dgilford#77). Contents: the sample builder (build_tcpypi_era5_sample.py, reads public ARCO-ERA5, reproduces the published S3 parquets bit-for-bit from its seeds), the scan pipeline (topology/crossings/partial sums under four LNB conventions; full pi() under each convention including the wild-population legacy non-convergence census of 8-in-476k; three-parcel scan at converged P_M), figure generators, and the Sounding Scope interactive flipbook template. Paths are portable via ERA5_SCRATCH; tcpyPI imports resolve against this repo's src/. Headline numbers in era5_analysis/README.md. Generated-with: Claude Fable 5
The artifact silently restricted all three parcels to the SST >= 26 C subset and hardcoded the map to +/-38 latitude, clamping out-of-range points into edge bins. Parcel A is defined on all 569,478 columns (-78..+90 lat) and B/C on the 476,102 with a converged storm pressure; the payload now carries per-parcel populations with sentinels where B/C are undefined, the map spans the data, out-of-range points are dropped rather than clamped, and the scope draws unaltered curves through a clipped viewport instead of clamping values to the axis range. Generated-with: Claude Fable 5
Highest observed buoyancy crossing across all 1.52M parcel evaluations is 49.6 hPa (saturated core), so retaining levels with P > 10 hPa (top 20 hPa) leaves a 2.5x pressure buffer: zero clipped topologies remain, 21/569k parcel-A columns fail the entropy solve aloft and are flagged. Buoyancy x-ranges now come from the deepest interior dip (most negative value before a curve's last positive level) with 10% margin, so the terminal stratospheric plunge no longer compresses the plots. Figures and the Sounding Scope template/payload regenerated accordingly. Generated-with: Claude Fable 5
Click any marginal to filter the flipbook samples: the map selects a position (haversine, +/-1500 km), the day-of-year ring a +/-15-day circular window, the new hour-of-day ring a +/-3 h UTC window, and the year track a single year. Each marginal displays the density under all the OTHER active filters (crossfilter convention) with its own selection drawn dashed; active selections appear as dismissable chips with a live matching-sample count, and Escape clears them. Payload gains hour-of-day for the player pool and map layer. Generated-with: Claude Fable 5
… filter scan_final now records the LNB pressure under each of the four conventions (columns 8-11). The artifact shows T/M/F/R lanes on the scope marking each convention's LNB for the current profile in the document's semantic colors (red topmost, aqua max-work, yellow first, violet ballistic); the readout lists CAPE @ LNB per convention and VMAX under legacy vs max-work (from pi_conv, ptop=50), flagging disagreements. A condition chip row crossfilters everything to disagreement populations: clamp-flips, |dE|>10 J/kg, |dVMAX|>1 m/s, and the eight real legacy non-convergent columns, which are force- included in the player pool and map layer. Generated-with: Claude Fable 5
The highest buoyancy crossing anywhere in the sample is 47.6 hPa, so retaining levels down to 30 hPa (ptop=20) observes every crossing with zero clipping and, unlike the 20 hPa ceiling, zero entropy-solver failures. Conditions are now top!=max and reach!=max (>1 J/kg) as primaries, keeping clamp-flip, |dVMAX|>1 m/s, and legacy-non-convergent as the mechanistically distinct cases; the first-crossing convention is dropped from the display (materially wrong too often to be interesting). Readout gives pos and SST their own cells with figure-space-padded coordinates so the layout no longer thrashes with coordinate width. Generated-with: Claude Fable 5
The chip row is itself a marginal: with any position/season/hour/year selection or disagreement condition active, each topology's share is recomputed from the 40k map layer under all the OTHER filters (its own selection excluded, crossfilter convention), so e.g. clicking the Gulf of Mexico in September immediately shows that region-season's topology mix. With no filters active the exact population shares are shown. Generated-with: Claude Fable 5
Under active filters the topology chips re-sort by filtered frequency and empty classes are masked off (the currently selected class stays visible so it can be deselected); with no filters the canonical global-frequency order is shown. Generated-with: Claude Fable 5
The filter-status chip now shows in every state (not only with marginal selections active): flip-sample count, matching map points, and the retained share of the current parcel's population with adaptive precision (>= 0.1% one decimal, below that two significant figures, so e.g. the 8 legacy-failure columns read 0.020% instead of rounding to zero). Also fixes a latent guard bug where hour-only selections showed no count (selHour was missing from the visibility condition). Generated-with: Claude Fable 5
The single top!=max condition mixed two populations with different physics: genuine value disagreements (both conventions positive) and the zero-flips where legacy returns exactly 0 (deep-valley clamps plus launch-level surface slivers). Conditions are now 0!=top!=max and 0=top!=max (both at >1 J/kg), expressed from the existing bitmask client-side with no payload change. Generated-with: Claude Fable 5
New panel: scatter of VMAX under the legacy topmost-LNB convention (x) against max-work and lifted-ballistic (y, aqua/violet), over every map point passing the full crossfilter, with a light dashed identity diagonal, cached offscreen and invalidated with the other densities. The current flipbook profile is highlighted with rings per convention. Map layer payload gains Vtop/Vmax/Vreach. Generated-with: Claude Fable 5
Replace the combined PI scatter with two panels: dVMAX = max - top and dVMAX = reach - top vs VMAX(top), each with a horizontal zero reference instead of the diagonal and a symmetric y-axis computed from the FILTERED deltas so points are never clipped (the axis rescales). Convention naming is now uniform everywhere -- top / max / reach -- with the long definitions stated once in the footer; the condition chip reads "top non-convergent". Generated-with: Claude Fable 5
Clicking either scatter selects a window in (VMAX(top), dVMAX) space (+/-5 m/s in x, +/-10% of the y-scale at click time in delta, min 0.5), one PI selection at a time, shown as a dashed rectangle and a dismissable chip. The selection filters the flipbook pool, map, rings, year track, and chip shares; the scatters themselves skip their own dimension per crossfilter convention. Clicking the panel padding or pressing Escape clears it. Generated-with: Claude Fable 5
VMAX^2 is proportional to (SST/T_O)(E_C - E_B), so when parcel B or C is selected the other of the pair is drawn for the same profile as a thin dashed curve in its own topology-class color: the PI winds live in the gap between the two curves. Footer states the relation. Generated-with: Claude Fable 5
…ters The two-panel scatter refactor removed the scatKey variable but invalidate() still assigned to it -- a ReferenceError under strict mode that silently aborted selChanged() after the throw, so live chip shares, matching counts, and the resample tick were skipped or only partially updated on marginal clicks (some handlers ran buildChips before invalidate, making the staleness look erratic). invalidate() now clears the per-panel cache keys. Also: clicking inside the current dVMAX selection rectangle clears it (documented in the footer), alongside the existing chip-x / margin-click / Escape paths. Generated-with: Claude Fable 5
The storm outflow level (parcel C's max-work LNB, the one that sets T_O in the PI formulas) is now marked with a labeled arrow on the scope's pressure axis for every profile, whichever parcel is selected -- keeping parcel B (where the topological variety lives) as the default view without hiding the LNB that actually bills into PI. Generated-with: Claude Fable 5
Counts, class shares, densities and the dVMAX scatters previously ran on a 40k map subsample, so marginal selections sparsified fast. The payload now carries every column (569,478 rows) as a binary-packed statistics table: little-endian int16/int8 columns (deci/centi quantization, -1 sentinels preserved exactly), gzipped and base64-embedded, decoded in the browser via DecompressionStream -- 12.5 MB raw -> 7.0 MB in-page. Player curves move to int16 centi-J/kg rectangles (same 0.01 precision as the old JSON), letting the flip-book pool grow 5.5k -> 24.5k within the 16 MB artifact cap. Dead "first"-convention player fields dropped. To keep 569k rows interactive, the per-frame day/hour/year histogram scans now sit behind selection-keyed caches (invalidated with the other caches), and the scatter panels switch to a two-pass dense rendering (low-alpha mass + opaque |dVMAX| > 1 m/s outliers) above 20k points. Count chip now reads "N of TOTAL columns"; footer documents that only the animated flip-book remains a sample. Generated-with: Claude Fable 5
…nges Switching parcel A/B/C or adding a marginal/topology/condition filter used to force an immediate resample even when the displayed profile was still inside the new subset. The per-index predicate is factored out of pool() as passPlayer(), and every filter-mutation path now goes through retick(): if the current profile still passes, only the scope and readout redraw (so the same sounding can be compared across parcels); resampling happens only when the profile falls outside the new subset. The play timer's cadence is unchanged. Generated-with: Claude Fable 5
The single |dVMAX| > 1 m/s condition becomes two signed chips on the gap dv = VMAX(max) - VMAX(top): dv > +8 m/s (1,822 columns; the legacy clamp suppressing real PI) and dv < -2 m/s (71 columns; top exceeding max). gpi codes: 1 = top non-convergent, 2 = dv > 8, 3 = dv < -2. Up to 400 columns per extreme are force-included in the flip-book pool so both conditions animate. Generated-with: Claude Fable 5
Each condition chip now shows its share of the population passing all OTHER active filters (parcel, topology, marginals, dVMAX selection), computed in one full-table pass mirroring condPassMap, with adaptive precision so rare conditions (e.g. the 8 non-convergent columns, 0.0014%) stay visible. Shares refresh on every chip/marginal/parcel/ theme change. Generated-with: Claude Fable 5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
era5_analysis/: a reproducible pipeline measuring, over 569,478 random ERA5 ocean columns (1980–2024), how often multi-crossing buoyancy structures occur for the three tcpyPI parcels (A environmental, B eyewall, C saturated core) and how the LNB convention (legacy topmost-level "top" vs max-work "max" vs lifted-ballistic "reach") affects CAPE and PI. Companion to thediscontinuity_analysis/issue-dgilford#77 root-cause study (#1): that PR shows why the legacy convention destabilizes the pressure iteration; this one shows how often it matters at climatological scale.Headline results (TC-relevant subset: SST ≥ 26 °C)
+-+-32%,-+-17%,+-+-+-16%,+-14%, …); parcel B is 84% single-hump+-; parcel C is ~98%+-— convention sensitivity is inversely proportional to parcel energy.E_topclamps to 0 whileE_max> 0 on 15.4% (A) / 1.75% (B) / 0.00% (C) of columns.Contents
convert.py→scan_final.py→pi_conv.py→ …), documented inera5_analysis/README.mdwith the full result summary.build_tcpypi_era5_sample.py— builds the input sample from public ARCO-ERA5 (no credentials needed).prep_artifact_data.py+sounding_scope_template.html): flip-book of profiles with map/seasonal/diurnal/year marginals, crossfilter selection, per-convention CAPE/LNB/PI readout, ΔVMAX scatters, and conditioning on convention disagreements.Test plan
validate_scan.pycross-checks scan CAPEs against the realcape()kernels on random profiles.pi_conv.pyreproduces stock tcpyPI outputs under the legacy convention.🤖 Generated with Claude Code