Stream frame interpolation for lazy DataArrays#28
Merged
Conversation
animate() no longer materializes the full dataset (da.values) nor the upsampled array. Frames are interpolated on the fly by a generator that holds at most two source frames, and dispatched lazily to the worker pool. Quantile-based color scaling streams per-frame percentiles for non-numpy inputs and skips the data pass entirely when vmin/vmax/norm are provided. Same treatment for quiver animations. Peak RAM on a 512 MB lazily-opened netCDF: 2.7 GB -> 205 MB. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Needed for pooch-backed sample dataset fetching used by the new lazy-frame streaming tests.
The mamba-solved conda-forge environment pinned an incompatible rasterio/libjxl pair (ImportError: libjxl.so.0.11), breaking test collection. Installing everything from PyPI wheels via uv sidesteps conda-forge's binary pinning entirely (rasterio/pyogrio/pyproj wheels bundle their own GDAL/GEOS/PROJ), and is much faster to boot. ffmpeg is a system binary, not a Python package, so it's still installed separately via apt.
Ubuntu's system Python is PEP 668 externally-managed, so `uv pip install --system` refuses to touch it. uv sync creates an isolated project venv instead, sidestepping that restriction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
animate()so it no longer materializes the full dataset (da.values) or the temporally upsampled array — frames are interpolated on the fly by a generator holding at most two source frames, and dispatched lazily to the worker pool._norm_streaming) that aggregates per-frame quantiles instead of computing them over the full 3D array, and skips the data pass entirely whenvmin/vmax/normare already provided. Same treatment applied to quiver animations.poochto thetestoptional-dependency group for the new lazy-loading tests.Test plan
pytest tests/test_lazy.pypytest