Follow-up from the review on #3556.
#3556 exempts dask grids from the 500-million-cell cap, since a lazy da.full grid never materializes. The data stays lazy, but the task graph does not: with a fixed chunk size, the chunk count grows with the cell count. A typo-level resolution can build a graph big enough to bog down the client even though no data is computed.
# ~13e12 cells / 512^2 ~= 50 million chunks; graph construction alone hangs
from_template('conus', resolution=1, chunks=512)#3556 documents the failure mode in the chunks docstring. This issue is for an actual guard.
Open questions (why this is a separate design decision, not part of #3556):
- Guard on estimated chunk count rather than cell count, so a legitimate large grid with sensible chunks still passes.
- Pick a threshold. ~1e6 chunks is already a heavy graph; the new_england@10m repro is ~26k, so there is wide headroom.
- Handle
chunks='auto', where dask picks the chunk shape, so the count is not known from the args alone.
- Decide raise vs warn. A warning keeps the "huge lazy grid just works" intent; a raise is safer but reintroduces a gate.
Follow-up from the review on #3556.
#3556 exempts dask grids from the 500-million-cell cap, since a lazy
da.fullgrid never materializes. The data stays lazy, but the task graph does not: with a fixed chunk size, the chunk count grows with the cell count. A typo-level resolution can build a graph big enough to bog down the client even though no data is computed.#3556 documents the failure mode in the
chunksdocstring. This issue is for an actual guard.Open questions (why this is a separate design decision, not part of #3556):
chunks='auto', where dask picks the chunk shape, so the count is not known from the args alone.