feat(encoding): opt-in lazy page metadata initialization for point reads#9
Merged
Merged
Conversation
Co-Authored-By: rqiu <ryantqiu@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cold point reads currently amplify into one small S3 GET per page: the v2 structural reader eagerly initializes page metadata (miniblock chunk metadata, dictionaries, repetition indices) for every page of every projected column at scheduler creation, before the requested rows are known. Measured on a real cosmos file: a 1-row read costs ~43 GETs / 5.4MB where ~9 GETs / 13KB suffice. Upstream acknowledges the cost (lance-format/lance@126e16da called eager init across fragments "catastrophic" and only parallelized it).
This adds an opt-in DecoderConfig::lazy_page_metadata_init (default false — default behavior unchanged) that restricts metadata init to pages overlapping the requested rows:
// decoder.rs
pub struct DecoderConfig {
...
pub lazy_page_metadata_init: bool, // default false
}
// requested rows are converted up front and threaded through init:
trait StructuralFieldScheduler {
fn initialize<'a>(&'a mut self, filter, context,
init_ranges: Option<&'a [Range]>) -> BoxFuture<'a, Result<()>>;
}
RequestedRows::to_ranges() (sorts/dedups indices) is computed in create_scheduler_decoder / schedule_and_decode_blocking only when the flag is set; otherwise None → eager parallel init exactly as today.
StructuralPrimitiveFieldScheduler computes page overlap from footer row counts and initializes only overlapping pages. Initialized pages are cached per page (PageDataCacheKey { column_index, page_index }); the whole-column CachedFieldData entry is written only when all pages are initialized, so partial state can never masquerade as a complete column.
Scheduling a page that wasn't covered by init_ranges returns an internal error instead of decoding with missing metadata.
struct/list/map forward parent ranges to children; fixed-size-list scales ranges by dimension.
Testing
New test_lazy_page_metadata_init (lance-file reader tests): asserts lazy == eager results for a single-page point read, multi-page index reads, small/multi ranges, and a full scan, on both V2_1 and V2_2 files.
cargo test --workspace: all pass, 0 failures.
Note: cargo clippy --all -- -D warnings fails on this repo without this change due to a pre-existing clippy::cargo error (package arrow-scalar is missing package.readme metadata); clippy with that lint group allowed is clean.
Context
Enables Exa's wormhole service (random point reads across ~3.5M cosmos Lance fragments on S3) to opt in; wormhole will bump its pinned rev and set the flag in a separate monorepo PR. Scan-shaped consumers (atlas/jata2) are unaffected by default.