bench(freqhb): Generate datasets lazily to cut peak RSS#27
Merged
Conversation
Addresses Copilot review feedback on PR #26 (merged before this fix was pushed). The bench previously materialized all shapes' 1M-row datasets up front and held them resident simultaneously (~190MB in bench_build), when each benchmark only needs one shape at a time. Datasets are now produced by a shapes() table of boxed generator closures and built inside the loop, so a single ~1M-row Vec<Vec<u8>> is resident at a time. bench_merge additionally drops the source dataset after building the partial tables (the partials own their keys), trimming peak further. Matches the lazy-generation pattern already used in benches/sortcmp.rs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #26, which was merged before this Copilot-review fix could be pushed.
What
Copilot flagged that
benches/freqhb.rseagerly materialized every shape's 1M-row dataset and kept them all resident at once (~190MB peak inbench_build), even though each benchmark only needs one shape at a time.This restructures both
bench_buildandbench_mergeto pull from ashapes()table of boxed generator closures, building each dataset inside the loop so only a single ~1M-rowVec<Vec<u8>>is resident at a time.bench_mergeadditionallydrops the source dataset after the partial tables are built (the partials own their keys), trimming peak further.Matches the lazy-generation pattern already used in
benches/sortcmp.rs.Notes
Test plan
cargo bench --bench freqhb --no-run— compiles clean🤖 Generated with Claude Code