Skip to content

feat(encoding): opt-in lazy page metadata initialization for point reads#9

Merged
ryantqiu merged 1 commit into
mainfrom
devin/lazy-page-metadata-init
Jul 14, 2026
Merged

feat(encoding): opt-in lazy page metadata initialization for point reads#9
ryantqiu merged 1 commit into
mainfrom
devin/lazy-page-metadata-init

Conversation

@ryantqiu

Copy link
Copy Markdown

Summary
Cold point reads currently amplify into one small S3 GET per page: the v2 structural reader eagerly initializes page metadata (miniblock chunk metadata, dictionaries, repetition indices) for every page of every projected column at scheduler creation, before the requested rows are known. Measured on a real cosmos file: a 1-row read costs ~43 GETs / 5.4MB where ~9 GETs / 13KB suffice. Upstream acknowledges the cost (lance-format/lance@126e16da called eager init across fragments "catastrophic" and only parallelized it).

This adds an opt-in DecoderConfig::lazy_page_metadata_init (default false — default behavior unchanged) that restricts metadata init to pages overlapping the requested rows:

// decoder.rs
pub struct DecoderConfig {
...
pub lazy_page_metadata_init: bool, // default false
}

// requested rows are converted up front and threaded through init:
trait StructuralFieldScheduler {
fn initialize<'a>(&'a mut self, filter, context,
init_ranges: Option<&'a [Range]>) -> BoxFuture<'a, Result<()>>;
}
RequestedRows::to_ranges() (sorts/dedups indices) is computed in create_scheduler_decoder / schedule_and_decode_blocking only when the flag is set; otherwise None → eager parallel init exactly as today.
StructuralPrimitiveFieldScheduler computes page overlap from footer row counts and initializes only overlapping pages. Initialized pages are cached per page (PageDataCacheKey { column_index, page_index }); the whole-column CachedFieldData entry is written only when all pages are initialized, so partial state can never masquerade as a complete column.
Scheduling a page that wasn't covered by init_ranges returns an internal error instead of decoding with missing metadata.
struct/list/map forward parent ranges to children; fixed-size-list scales ranges by dimension.
Testing
New test_lazy_page_metadata_init (lance-file reader tests): asserts lazy == eager results for a single-page point read, multi-page index reads, small/multi ranges, and a full scan, on both V2_1 and V2_2 files.
cargo test --workspace: all pass, 0 failures.
Note: cargo clippy --all -- -D warnings fails on this repo without this change due to a pre-existing clippy::cargo error (package arrow-scalar is missing package.readme metadata); clippy with that lint group allowed is clean.
Context
Enables Exa's wormhole service (random point reads across ~3.5M cosmos Lance fragments on S3) to opt in; wormhole will bump its pinned rev and set the flag in a separate monorepo PR. Scan-shaped consumers (atlas/jata2) are unaffected by default.

@github-actions github-actions Bot added the enhancement New feature or request label Jul 11, 2026
@ryantqiu
ryantqiu merged commit 026c040 into main Jul 14, 2026
10 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant