cost_distance: route bounded large-radius dask path to iterative Dijkstra (#3342)#3345
Merged
brendancol merged 2 commits intoJun 15, 2026
Merged
Conversation
…stra (#3342) The map_overlap vs iterative branch in _cost_distance_dask compared the pad radius against the full array dimensions instead of the chunk size. When the radius exceeded the chunk size but stayed below the full dimension, the path still used map_overlap with depth greater than the chunk size, so dask silently rechunked the inputs toward larger (eventually single) blocks. That reintroduces the #880-class OOM on large dask rasters with a finite max_cost. Compare pad against the chunk size and route to the iterative tile Dijkstra when it would otherwise overflow a chunk, matching the dask+cupy guard. Adds a regression test asserting the bounded large-radius path keeps more than one chunk and matches the numpy result.
Contributor
Author
Review: route bounded large-radius dask path to iterative DijkstraReviewed from a worktree on the head branch. The fix is correct and narrowly scoped. No blockers. BlockersNone. Suggestions
Nits
What looks good
Checklist
|
The ASV CostDistance benchmark only ran cost_distance with the default max_cost=inf, so it always exercised the unbounded iterative path and never the bounded map_overlap branch this PR touches. Add a time_cost_distance_bounded case whose pixel radius stays inside one chunk so the dask path runs map_overlap, giving that branch perf coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3342.
Problem
cost_distanceon a dask+numpy raster with a finitemax_costcould rechunkthe input toward a single monolithic chunk and OOM.
_cost_distance_daskgated the
map_overlapvs iterative-tile-Dijkstra branch on the pad radiusversus the full array dimensions:
map_overlap'sdepthmust stay within the chunk size. Whenchunk_size <= pad < full_dim, this still took themap_overlapbranch withdepth > chunk_size, and dask silently rechunked toward larger blocks. With alarge radius it collapsed to one chunk, the same OOM class as #880 (fixed in
#888 for the unbounded path only).
The dask+cupy path already guarded correctly against chunk size.
Fix
Compare
padagainst the max chunk dimension and route to the iterative tileDijkstra when it would overflow a chunk, matching the dask+cupy guard.
Verification
test_bounded_large_radius_no_chunk_collapse: chunks (10,10),max_cost=95(pad 96). Asserts the result keeps >1 partition and matches thenumpy result. Fails on
main(collapses to 1 chunk, no iterative warning),passes with the fix.
OOM verdict for the bounded large-radius case: was RISKY, now bounded.
Bottleneck: memory-bound. Affected backend: dask+numpy.