Skip to content

BatchedFetchStore for shard index caching and range batching#41

Open
wietzesuijker wants to merge 1 commit into
carbonplan:mainfrom
wietzesuijker:perf/batched-fetch-store
Open

BatchedFetchStore for shard index caching and range batching#41
wietzesuijker wants to merge 1 commit into
carbonplan:mainfrom
wietzesuijker:perf/batched-fetch-store

Conversation

@wietzesuijker

Copy link
Copy Markdown
Contributor

Wraps all fetch stores with BatchedFetchStore to cut HTTP requests when rendering sharded Zarr v3 datasets.

zarrita.js creates a fresh shard-index cache per open(array) call, so for a 3-band RGB layer the same shard index is fetched 3× per shard per viewport. BatchedFetchStore caches suffix-length range reads (always shard-index fetches) in an LRU so bands 2 and 3 are served from memory. It also batches concurrent getRange calls within the same event-loop tick: calls are accumulated, sorted by offset, grouped into merged ranges (32 KB gap threshold), fetched as fewer HTTP requests, then sliced back to callers — modelled on geotiff.js BlockedSource.

For a 3-band sharded Zarr v3 (e.g. Sentinel-2 EOPF, 9-tile viewport):

Before After
Shard index requests 3× per shard per viewport 1× (cache hit for bands 2+3)
Chunk range requests N sequential GETs per band ≤ ⌈N/group⌉ merged GETs per path

BatchedFetchStore is adapted from the zarrita.js development branch (MIT, same org) and should be removed from this repo once @zarrita/storage ships it.

AI (Claude) supported my development of this PR.

@vercel

vercel Bot commented Mar 16, 2026

Copy link
Copy Markdown

@wietzesuijker is attempting to deploy a commit to the carbonplan Team on Vercel.

A member of the Team first needs to authorize it.

@wietzesuijker wietzesuijker force-pushed the perf/batched-fetch-store branch from ed7071a to faeef32 Compare March 16, 2026 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant