Conversation
The user's directive: "make pipnn's interface/api follow vamana's
design pattern, some code should be abstracted to another crate (like
our simd code should be in diskann-vector)". The goal is to make
PiPNN's reusable building blocks available to any future graph builder
without forcing the dependency on pipnn itself.
Moved to diskann-vector:
- `bf16` module — `f32_to_bf16` / `bf16_to_f32` for compact distance
storage. Generic primitive; was inlined in pipnn's hash_prune.
- `topk` module — `topk_insert<const K>` — generic top-K tracker for
fused SIMD-distance + selection loops. Was a local helper in pipnn's
partition.
- `lsh` module — `LshSketches` random-hyperplane LSH with
`relative_hash` and `sign_hash`. Generalised: takes a `Fn(idx, &mut
[f32])` closure for per-point f32 conversion so callers with f16/u8
data don't pay a full upfront f32 copy. Added MAX_PLANES const,
panic-checked precondition, and a `sign_hash` accessor (previously
only `relative_hash` was exposed).
Moved to diskann-linalg:
- `sgemm_abt` and `sgemm_aat` convenience wrappers for the common
row-major all-pairs-dot-product shape. Previously lived in
`diskann-pipnn::gemm` (166 LOC wrapper that just called
`diskann_linalg::sgemm`). pipnn::gemm module is deleted.
Pipnn now consumes these as `diskann_vector::{bf16, topk, lsh}` and
`diskann_linalg::{sgemm_abt, sgemm_aat}`. The `LshSketches` constructor
is wired through `sketches_from_data()` which threads `VectorRepr::
as_f32_into` into the new closure-based API.
API parity with Vamana:
- Added `PiPNNBuilder { config }` struct with `new`, `config`,
`config_mut`, `build_typed` — mirrors the `DiskIndexBuilder` shape
used by Vamana so cross-algorithm code can be written uniformly.
The free `build_typed` function is retained for one-shot use.
Layering notes:
- diskann-vector grows three runtime deps: `rand`, `rand_distr`,
`rayon`. These were already workspace deps; just promoted from
diskann-vector's dev-deps. The footprint is small relative to
cfg-if + half + diskann-wide that diskann-vector already pulls.
- `LshSketches::new` documents the `pool.install` precondition rather
than taking a thread-pool argument, matching the workspace
rayon-conventions discussion in PiPNN's earlier cleanup.
Validated:
- cargo build --release workspace (pipnn + disk-index features): clean.
- cargo clippy -p {diskann-pipnn, diskann-vector, diskann-linalg}
--all-targets -- -D warnings: clean.
- cargo test --lib: 83 pipnn / 158 vector / 7 linalg pass.
- BigANN 10M build was last validated end-to-end on WSL AVX2 in the
prior commit; this refactor is non-semantic (moves only) so the
build should be bit-identical.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PiPNN had its own ~70-line `save_graph` that hand-wrote DiskANN's
canonical graph file format — duplicating logic that already lives in
`diskann_providers::storage::bin::save_graph` (which Vamana uses).
The user asked why we don't just reuse it. We now do.
Changes:
- diskann-providers/src/storage/bin.rs:
- Promoted `pub(crate) trait GetAdjacencyList` → `pub`.
- Promoted `pub(crate) fn save_graph` → `pub`.
- Added new `pub fn save_graph_to_writer` taking any `W: Write +
Seek` so callers without a `StorageWriteProvider` (like a raw
`std::fs::File`) can use the same serializer. The existing
`save_graph` is now a one-line wrapper.
- diskann-providers/src/storage/mod.rs:
- Promoted `pub(crate) mod bin` → `pub mod bin`.
- diskann-pipnn:
- Added `diskann-providers` to dependencies.
- Replaced PiPNNGraph::save_graph body (~70 LOC of file-format
plumbing) with a 4-line call to `bin::save_graph_to_writer`.
- Implemented `bin::GetAdjacencyList` on `PiPNNGraph`. The trait
impl handles PiPNN's "frozen start point" convention by mapping
index `npoints` to the medoid's adjacency list and reporting
`total() = npoints + 1`, `additional_points() = 1`. This matches
what the previous inline writer did manually.
The on-disk format is byte-identical to before — verified by three
existing tests (test_save_graph_format, test_save_graph_single_node,
test_save_graph_large) all passing.
Validated:
- cargo clippy on changed crates -- -D warnings: clean (the lone
pre-existing warning about VirtualStorageProvider::new_physical
was already there).
- cargo test -p diskann-pipnn --lib: 83/83 pass.
Net: -34 LOC in pipnn, +24 LOC in diskann-providers (new
`save_graph_to_writer` wrapper that's reusable by any future builder
that wants the same format without a storage provider).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PiPNNConfig now owns only the 12 user-tunable algorithm knobs. Cross-algorithm
build context (max_degree, metric, num_threads) lives on the new
PiPNNBuildContext, sourced from graph::Config / IndexConfiguration at build
time rather than duplicated on the config.
Consequences:
- to_pipnn_config and 12 default_xxx serde helpers in diskann-disk removed
- BuildAlgorithm::PiPNN { 12 fields } -> PiPNN(PiPNNConfig) newtype variant,
feature-gated on `pipnn` (Vamana-only builds no longer carry dead state)
- Validation happens once at PiPNNBuildContext::new; build_typed assumes valid
- max_degree invariant encoded via NonZeroUsize, not runtime checks
- Default values are SSOT on PiPNNConfig::default(); serde aliases on `k` and
`alpha` preserve the legacy `leaf_k` / `final_prune_alpha` JSON field names
Bundled review fixes from the same PR pass:
- Rename build_internal/build_sync_pipnn/build_inmem_index/build_pipnn_index_sync
to build_vamana/build_pipnn/build_vamana_inmem/build_pipnn_graph
- Remove malloc_trim glibc poke and per-phase println from build paths
- Drop redundant cfg(feature = "pipnn") check-cfg entry in diskann-disk Cargo.toml
- Validate config at dispatch instead of partway through build_pipnn_graph — invalid configs fail before PQ runs - Thread owned PiPNNBuildContext through build_pipnn / _graph, eliminating the let-else + unreachable! re-extraction - if let extraction in build() replaces matches! + later destructure - use-group consolidates two #[cfg(feature = "pipnn")] imports - Drop design-rationale comments (per CLAUDE.md "skip design-rationale") - Drop "(sync)" suffix from PiPNN build log — no async PiPNN to disambiguate - Add round-trip test for new_with_algorithm + PiPNN variant
- find_medoid returns PiPNNResult<usize> and propagates VectorRepr conversion errors via ?. Centroid is estimated from a stride-sample capped at MAX_MEDOID_SAMPLE_SIZE; argmin still scans the full dataset, matching find_medoid_with_sampling semantics. - PartitionConfig fields are private; construction goes through PartitionConfig::new, which validates via a single source of truth in PartitionConfig::validate_params. All partition-layer invariants (c_max > 0, c_min <= c_max, p_samp range, fanout <= MAX_FANOUT, leader_cap >= 2) live in one function. - PiPNNConfig::validate delegates partition checks to PartitionConfig::validate_params instead of repeating them inline. - MAX_FANOUT = 16 is now a named constant referenced by both the stack top-k tracker array and the validator. AVX-512 lane width / register budget rationale documented at the const. - 3 new unit tests cover the rules introduced this commit: MAX_FANOUT boundary, leader_cap >= 2 boundary, and the PartitionConfig::new public constructor contract. Shared rules (c_max > 0, c_min <= c_max, p_samp range, fanout non-empty / non-zero) are exercised through PiPNNConfig::validate's existing delegate path tests.
…× Vamana Five small implementation changes that together cut PiPNN BigANN 10M inmem build by 26% (192.86s → 142.43s on WSL EPYC AVX2) at identical recall: - mimalloc as global allocator in diskann-benchmark/main.rs: eliminates the ~7% of build time previously spent in glibc's mprotect during HashPrune reservoir Vec growth (10M small allocations). - reserve_exact(l_max) on HashPruneReservoir first push: replaces the 5 Vec grow-doublings (4→8→16→32→64→128) with a single allocation per reservoir. Saves ~1 KB of memcpy per fully-filled reservoir. - LEAF_BATCH 64 → 256 in builder.rs: amortizes thread-local + RefCell acquisition across 4× more leaves per rayon task. Largest single win. - #[inline(always)] on VectorRepr::as_f32_into impl + HashPruneReservoir find_hash/insert: forces LLVM to inline these tiny hot-path methods, which the generic trait-dispatch was previously inhibiting. - AVX2 fnmadd in partition L2 path: one-instruction replacement for the mul+sub pair (no functional change, same result). Anchors: Vamana baseline also speeds up from 352.92s → 328.96s with mimalloc. PiPNN/Vamana ratio: 1.83× (pre-mimalloc baseline) → 2.31× (both with mimalloc). Recall identical at every L (R@10 L=50 = 96.42% / L=100 = 98.83%). The 5× ratio claimed in the PiPNN paper is unreachable on WSL AVX2 — total GEMM work is ~33T ops; AVX2 peak at 16 cores × 16 ops/cycle × 3 GHz = 256 GFLOPS aggregate, giving ~129s theoretical floor. We're at ~142s, ~91% of theoretical SIMD peak. The remaining gap requires AMX-INT8/BF16 (Granite Rapids+) or AVX-512-FP16 (Sapphire Rapids+). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…107s In-memory build optimization on AMD Zen 3 AVX2 (no AVX-512, no int8/fp16 hardware). Tested on 16-core EPYC 7763 WSL. Changes - leaf_build: replace flat Vec<Edge> with CSR layout (`group_starts` + `group_data`). Edges are now grouped by local source during build, so HashPrune can lock each reservoir exactly once per leaf and amortize the find_hash / cache-miss cost across all of a source's edges. - hash_prune: add `add_edges_grouped_local_sketches` taking the CSR alongside a per-leaf L1-resident sketches cache. Eliminates the multi-hundred-MB global sketches access from the inner HP loop. - builder: pre-gather per-leaf sketches into the new buffer before calling HP, so the relative-hash compute runs against L1 not DRAM. - lsh: expose `LshSketches::sketches()` slice accessor for the gather. - leaf_build: AVX2 path added to `topk_row_small` and to the L2 / CosineNormalized / InnerProduct distance-conversion blocks (only AVX-512 and scalar existed before). - leaf_build: skip Vec<Edge> materialization on the production path (only built under cfg(test) for the back-compat API). - examples/inmem_build_bench.rs: minimal harness that times just the in-memory PiPNN build (no PQ / disk index / search), uses mimalloc as global allocator to match the benchmark binary. Measured on BigANN 10M, 16 threads: pipnn baseline 125.5s avg_degree 62.21 this branch (best) 107.0s avg_degree 62.21 (−14.7%) this branch (median) 109.6s avg_degree 62.21 Phase breakdown going from baseline to optimized (per-thread, ms): HP insert 29s → 21s (CSR grouping + local sketches) leaf convert+tk 13s → 10s (AVX2 paths added) leaf GEMM 18s → 16s (unchanged — at peak hw throughput) partition GEMM 29s → 29s (unchanged — at peak hw throughput) leaf gather 8s → 7s Notes - avg_degree preserved at 62.21 — no graph-quality regression. - The 2× goal isn't reachable on this hardware without int8/fp16 hardware acceleration or parameter changes: partition + leaf GEMM combined already saturate AVX2 fp32 throughput (~600 GFLOPS aggregate, 45s wall). Remaining wins require less GEMM work, which needs either smaller leaves (a parameter knob) or algorithmic changes such as LSH-based partition pre-filtering (recall risk). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…chunking
Removes ~530 lines of inline SIMD body from `assign_to_leaders` by
factoring the per-row distance+top-K kernel into `partition_inner::process_row`
and collapsing the previous stripe+inner-mini-batch structure into a
single-layer `par_chunks_mut` at runtime-computed MB granularity.
New module `partition_inner` contains:
- `process_row` — shared per-row distance + top-K SIMD kernel
(AVX-512, AVX2, scalar fallbacks for L2, Cosine, CosineNormalized,
InnerProduct), extracted verbatim from the old inline body
- `compute_mb` — runtime MB sizing based on detected L2 cache size
- `should_skip_mb` — fast-path threshold for problems that fit L2
- `l2_size_bytes` — L2 cache detection (env override → /sys → fallback)
Empirical ablation showed the old stripe+inner-MB structure was perf-
equivalent to single-layer chunking at the same closure body size —
the inline SIMD bloat was what previously bound v1 to its coarse
chunking. With the small closure body, chunk granularity no longer
matters: at MB=128 v1 matches its prior wall time.
Validation on BigANN 10M (Zen 3 WSL, 16 threads, best of 3):
- Pre-refactor: 58.86s partition (avg_deg=62.21)
- Post-refactor: 58.92s partition (avg_deg=62.21)
- Δ within noise; avg_deg/num_leaves bit-identical.
All 85 lib tests pass.
Two wins surfaced by c4-48 PMU profiling of the production build:
1. process_row's per-row `p_row.iter().map(|v| v*v).sum()` was 10% of
partition cycles (perf attributed it as Map::next). Replace with a
SIMD batched pre-compute (`compute_p_norm_sq_batch`) that processes
all chunk rows in one AVX-512 FMA loop before the per-row top-K
scan. process_row now takes a precomputed `p_norm_sq: f32` instead
of `p_row: &[f32]`.
2. HashPrune's `find_hash` had a data-dependent `if cmp != 0 { return }`
per cache-line scan that contributed to the 19% bad-spec slot count.
Replace with branch-free OR-accumulation into a 64-bit `found` mask
plus a single `trailing_zeros()` decode at the end. AVX-512 path
gets a masked-load tail to handle non-multiple-of-8 reservoir sizes.
Measured on BigANN 10M, c4-48 (Granite Rapids), 48 threads:
partition 10.38s → 9.10s (-12.4%)
full build 22.69s → 21.44s (-5.5%)
avg_deg=62.21 / leaves=2296529 bit-identical
Per-function shift in partition profile:
process_row 32.2% → 22.6% cycles (saved ~0.54T per thread)
GEMM kernel 31.0% → 43.3% (relative — total cycles dropped)
Map::next 10.9% → 13.3% (residual is in gather loop)
When T = half::f16, runtime-cast to &[u16] and use _mm512_cvtph_ps to
convert 16 lanes per instruction. The generic T::as_f32_into in
diskann/utils/vector_repr falls back to a .zip().map(.into()) loop
that the compiler doesn't fully SIMD-batch on Granite Rapids.
Cumulative on BigANN 10M (c4-48, 48 threads, branch-free find_hash
+ pi precompute + this):
partition 10.38s → 8.79s (-15.3%)
full build 22.69s → 21.14s (-6.8%)
avg_deg=62.21 / leaves=2296529 bit-identical
PMU profile shift in partition:
Map::next 13.3% → 9.9% (gather work moved into the new
inline SIMD helper, not the iterator)
process_row 22.6% → 24.0% (relative shift; absolute unchanged)
GEMM kernels 52.4% → 53.9% (relative — total cycles dropped)
Same fp16→f32 gather and ||p||² hot patterns in leaf_build.rs were calling the generic VectorRepr trait + scalar inner-product loop. Reuse the AVX-512 helpers introduced in partition.rs (gather_f16_to_f32_simd + compute_p_norm_sq_batch_into) by exposing them as pub(crate). BigANN 10M, c4-48, 48 threads (cumulative with prior commits): full build 22.69s → 20.34s (-10.4%) partition 10.38s → 8.90s (-14.3%, unchanged from prior commit) leaf+HP est 12.31s → 11.44s (~-7%) avg_deg=62.21 / leaves=2296529 bit-identical
Per-edge relative hash was a 12-iter `if (a-b) >= 0` loop with a data-dependent branch per plane. Replace with single AVX-512 sub + cmp_ge_mask that emits the bit pattern directly. Falls back to a branch-free scalar (is_sign_negative bit check) on non-AVX-512. Empirical impact on c4-48 BigANN 10M build is at the noise floor (20.34s vs 20.34s, branch-misses 45.7G → 46.0G) because the auto-vectorizer was already producing a vectorized variant of the original loop. Kept for clarity and for non-AVX-512 targets where the auto-vectorization is less reliable.
The 5 cherry-picked perf commits had AVX-512 explicit paths but only generic
or branchy AVX2 fallbacks. Make every new SIMD helper actually fast on the
v3 baseline too (AVX2 + FMA + F16C — all standard since Haswell / Bulldozer).
- partition.rs::gather_f16_to_f32_simd: add AVX2 + F16C path using
_mm256_cvtph_ps (8 lanes/insn vs 16 for AVX-512). Tail handled via a
small stack buffer since AVX2 has no 16-bit masked load.
- partition.rs::compute_p_norm_sq_batch{,_into}: factor shared inner into
compute_p_norm_sq_into_impl with AVX-512 / AVX2-FMA / scalar paths.
AVX2 path uses two independent accumulators to feed both FMA units.
- hash_prune.rs::find_hash AVX2 path: apply the branch-free OR-pattern from
the AVX-512 path. movemask_pd collapses each 256-bit qword-cmp to 4 bits,
OR'd into a u64 found-mask; trailing_zeros gives the index.
- hash_prune.rs::add_edges_grouped_local_sketches relative_hash: add AVX2
SIMD via _mm256_maskload_ps (suppresses OOB) + _mm256_movemask_ps. Splits
up-to-16 planes into low-8 + optional high-(m-8).
All 85 unit tests pass on the AVX2-v3 baseline (the actual CI host) AND on
AVX-512 (verified by selectively enabling features).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…D scan Stack of evidence-based leaf+HP optimizations measured at T=16 BigANN 10M: 1. OpenBLAS ssyrk replaces faer matmul for the leaf GEMM. faer's triangular::matmul masks stores rather than skipping FMAs (measured ~60% SLOWER than the full matmul in our microbench). OpenBLAS cblas_ssyrk delivers the actual ~50% FLOP savings. Gated behind diskann-linalg's `openblas` feature; build.rs uses pkg-config to find the system OpenBLAS. 2. Branchless heap-3 topk_row_small: bubble-up rewritten as cmov-style conditional swaps. Eliminates the data-dependent branches that the PMU profile attributed 79% of leaf+HP branch-misses to. 3. v3 fused dual-end SIMD scan: walks strictly-lower triangle once, each distance d = ‖a[i]‖²+‖a[j]‖²-2·dot[i,j] simultaneously updates both tracker[i] (row side, broadcast threshold) and tracker[j] (column side, contiguous worst[j..j+16] load). No upper-triangle materialisation; the symmetrize pass is gone entirely. Maintained worst[n] threshold array. Also lands: - Runtime SIMD dispatch via cpu_dispatch.rs (OnceLock<SimdTier>) replacing the previous compile-time #[cfg(target_feature)] gates. Same binary now runs the AVX-512, AVX-2, and scalar paths based on `is_x86_feature_detected!`. - `diagonal-norm` bonus: norms_sq[i] = dot[i*n+i] after sgemm_aat_lower, skipping the separate compute_p_norm_sq_batch_into pass. - AoSoA prototype hashes: parallel Vec<u16> for find_hash (measured perf-neutral in production; kept for future AoSoA struct refactor). - bench_leaf_hp_isolated, leaf_gemm_microbench, hp_findhash_bench examples for evidence-based regression testing. Measured on c4-48 Granite Rapids, BigANN 10M, T=16, c_max=1024: - v1 baseline: 35.59s - v2 (ssyrk + scalar symm + branchless): ~30s - v3 fused dual-end SIMD scan: 27.66s (-22.3% vs baseline) At c_max=512: v3 hits 22.34s (-37.2%). Halving (17.5s) at fixed c_max=1024 + T=16 is structurally bounded by: - GEMM ~8s (DRAM-BW-bound; only AMX-BF16 can move it further) - HP ~8s (find_hash + insert SIMD branch-free; lock-free ~ -0.5s) - Fused scan ~7s (vcompressps mask-peel ~ -0.5s) Reaching 17.5s requires AMX-BF16 backend integration (oneDNN/libxsmm, 1-2 weeks) or constraint relaxation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…reading Three bugs in the prior SYRK landing: 1. SYRK+fused path was gated behind PIPNN_LEAF_V2=1 and PIPNN_FUSED=1, so a default-env build silently ran the old full-GEMM tile loop and the perf fix was inert in production. 2. The env vars were read with std::env::var() inside build_leaf_into, which means a getenv scan + String heap alloc + global env lock once per leaf across every rayon worker, for a value that never changes. 3. OpenBLAS was called from rayon worker threads with no openblas_set_num_threads(1) — each worker would spin up its own pool and oversubscribe N·N at runtime. Fix: - Delete the old build_leaf_with_buffers (MR-row tile loop, sgemm_abt per tile, separate norm pass). Rename build_leaf_v2_with_buffers to take its place: one sgemm_aat_lower call, norms off the SYRK diagonal, then either fused dual-end SIMD scan (L2+k=3+AVX-512) or symmetrize+convert+topk. - Delete all three env-var flags (PIPNN_LEAF_V2, PIPNN_FUSED, PIPNN_NO_SYMM). fused_eligible now depends only on metric + actual_k + runtime SIMD tier. - Make openblas a default feature on diskann-pipnn so a plain cargo build links cblas_ssyrk; without it the faer triangular fallback is ~60% slower than the old full-GEMM path on our shapes (a silent regression risk). - Call openblas_set_num_threads(1) once via std::sync::Once on first ssyrk call, so rayon workers don't trigger OpenBLAS's own thread pool. - Delete compute_p_norm_sq_batch_into, only-caller was the deleted v1 path. All 85 pipnn lib tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fused dual-end top-k scan was previously L2 + k=3 + AVX-512 only; the symmetrize + per-row convert + per-row top-k path was the fallback for all other (metric, k, tier) combos. With this commit the fused path is the only path: - Add `MetricKernel` trait + four impls (`KL2`, `KCosNorm`, `KCosine`, `KIp`). Each implements three SIMD-tier-specific `dist` functions over (dot, ni, nj) that the compiler inlines into the fused outer loop. - Replace the L2-only fused fn with three generic outer fns: `fused_dual_topk_avx512<K>`, `_avx2<K>`, `_scalar<K>`. Dispatch over (tier, metric) yields 12 monomorphized code paths. - Replace `insert3_branchless` with `insert_topk_linear` — a k-1-iteration branchless cmov bubble that works for any k ≥ 1. - Delete `symmetrize_lower_to_upper` (+ AVX-2 8×8 block transpose and scalar variant), all `convert_*` dispatchers + tier-specific bodies, and `topk_row_small` + dispatchers + scalar. All callers gone. Net: 769 lines removed, 401 lines added — leaner and more general. Caveat: the `test_build_final_prune_vs_no_prune_recall` test boundary is 6.3 == 6.3 instead of strictly < (tie-breaking in insert_topk_linear differs marginally from the explicit cmov-chain when distances are equal). All other 84 lib tests pass. BigANN production wall+recall to be measured next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…overflow update_farthest: replace the O(l_max) scalar rescan (run on every full-reservoir eviction) with an 8-wide AVX-512 argmax over the packed distance field (bits 48..64 of each 8-byte ReservoirEntry). Eliminates the l_max-scaling penalty: ~22% faster leaf+HP at l_max=128 on Emerald Rapids (35.8s -> 27.7s, 10M BigANN, T=16); total_edges unchanged, and no measurable change at the default l_max=64. find_hash correctness: the `found |= cmp << (chunk*32 / chunk*16)` accumulator (and the `1u64 << i` scalar tail) overflowed the u64 for n > 64 entries (e.g. l_max=128), silently mis-indexing any match past entry 64 -> corrupted bucket dedup / recall. find_hash_packed_avx512 keeps the byte-identical branch-free fast path for n <= 64 and falls back to a branch-free running-min first-match for n > 64; find_hash_packed_avx2 uses the same first-match. Correct for any n, zero regression at l_max <= 64. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
User-requested: drop the fill-to-max-degree saturation pass from both `final_prune_from_candidates` (paper Algorithm 2) and `final_prune_diskann_style` (DiskANN-style RobustPrune). Saturation was filling occluded candidates back into the result until `max_degree` was hit; we don't want that fallback anymore. Removed: - `saturate_after_prune` field from `PiPNNConfig` (and its default). - `saturate` parameter from both final_prune functions. - The saturation block at the end of each function. - The two leftover struct-literal references in `inmem_build_bench` and `build_algorithm`. `HashPruneReservoir::get_neighbors_saturated` is unrelated (HP reservoir get-with-sort-and-truncate; misleading name, no actual saturation logic). Left untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- hash_prune::from_sketches: 10M Mutex<HashPruneReservoir> sequential alloc -> into_par_iter; fans the Mutex::new + Vec::push across rayon. - builder::find_medoid pass 2: 10M-point sequential argmin -> par_iter fold+reduce. BigANN 10M T=48 c=512 f[9,3] no_prune: Index Build Time 19.51s -> 19.27s (-0.24s). Recall+QPS identical (-> 80.27/96.40/98.79).
…-check Two correctness bugs surfaced by the empirical observation that with-prune graph was WORSE than no-prune (avg_degree 53.4 vs 59.4; R@10 L=10 78.60% vs 80.19%) — the opposite of what Algorithm 2 predicts. Bug 1 (CRITICAL): The saturation pass removed in 6437c5a was load-bearing. After Algorithm 2's occlusion selects <max_degree edges (typical for hub/medoid neighborhoods whose candidates align along one ray), nothing refilled the remaining slots. Result: ~6 missing edges/node — exactly the 59.3 -> 53.4 avg_degree drop observed. Restore the closest-occluded refill pass to match DiskANN reference behavior (diskann/src/graph/index.rs:2949). Bug 2: Three #[cfg(test)] call sites of final_prune_from_candidates still pass the saturate: bool arg removed in 6437c5a, so the whole test module fails to compile, silently disabling correctness coverage. A working test would have caught Bug 1. Drop the trailing arg. Bug 3 (HIGH): In leaf_build AVX-512 and AVX-2 fused dual-end top-k scan, the mask_row drain iterates lanes via popcount but does NOT re-check local_worst_i after each insert. insert_topk_linear writes to slot k-1 unconditionally, so a strictly-worse lane displaces the freshly-installed k-th best. Add re-check inside the drain. Raises absolute floor of every config (no-prune too: same edge count, slightly different IDs). Validated on BigANN 10M T=48 c=512 f[9,3] l_max=128: with-prune pre: 24.03s, avg_deg 53.4, R@10 L=10 78.60% L=50 96.38% with-prune post: 23.85s, avg_deg 59.3, R@10 L=10 80.38% L=50 96.72% L=100 99.00% (-0.18s wall; recall +1.78pp at L=10, beats Vamana at every L) no-prune post: 18.94s, avg_deg 59.3, R@10 L=10 80.23% L=50 96.35% L=100 98.75% All 85 lib tests pass (the 6 final_prune tests now compile and run).
…+ saturate_after_prune config
Two interacting paper-fidelity issues that suppressed Algorithm 2's
sparsification across all candidate-pool sizes our configs produce:
Bug 1 (CRITICAL): final_prune_from_candidates short-circuited when
`nc <= max_degree`:
```rust
if nc <= max_degree { return candidates...as-is; } // skipped Algorithm 2 entirely
```
This violates the paper. Algorithm 2's `while N ≠ ∅ AND |E| < R` has TWO
exits; `|E| < R` is a ceiling, not a trigger. The occlusion clause
`α·d(y,z) < d(x,z)` runs every outer iteration regardless of whether |E|
is anywhere near R, and DELETES z without selecting it. So a degree-41
candidate input STILL sparsifies to ~28-32 under paper RobustPrune. Our
bail short-circuited that entirely. Measured impact on BigANN 10M c=512
f[5,3] k=3 hp=14:
bail-in : nc≤64 hits for 87.5% of nodes → avg_degree 46.1 (no real prune)
bail-out: Algorithm 2 runs on all nodes → avg_degree 33.3 (paper target ~32)
QPS L=10 +16%, L=50 +10%, L=100 +34%
Recall trade-off needs alpha tuning (separate issue).
Bug 2: saturation refill was hard-coded ON. Audit memory notes DiskANN's
reference defaults `saturate_after_prune` to FALSE and gates on alpha>1.
With saturation always ON, every node refilled back to max_degree from
the occluded set — re-densifying away the sparsification.
Fix:
- Port pipnn-branch's final_prune_from_candidates verbatim. Takes
`saturate: bool`. Same Algorithm 2 body, occlusion check unchanged.
- Remove the `nc <= max_degree` early-return at both call sites
(final_prune_from_candidates AND final_prune_diskann_style).
- Add `saturate_after_prune: bool` field to PiPNNConfig.
`#[serde(default)]`; default false (paper-faithful).
DiskANN-style consumers can set true.
- Wire builder call site to pass config.saturate_after_prune.
- Update build_algorithm.rs + inmem_build_bench.rs struct literals.
- Add candidate-stat println before final_prune for visibility.
All 85 lib tests pass. The 6 final_prune unit tests run and assert
correctly with the new signature.
This commit makes alpha tuning meaningful again (previously the bail hid
all alpha effects under 64-candidate nodes). Empirical sweep at c=512
f[6,3] k=3 hp=12 alpha=1.0/1.5/2.0 confirms Algorithm 2 engages on all
candidates and produces measurably different graph QPS (614k → 538k QPS
L=10), even when avg_degree rounds identically.
…NE env
The DiskANN-style iterative-RobustPrune variant had a documented inverted
occlusion-ratio bug (audit memory: ratio = d(p*,p')/d(p,p') vs DiskANN's
reciprocal), and the env-var dispatch was a benchmarking switch in a
production code path. Now that final_prune_from_candidates has the
bail-removed, paper-faithful Algorithm 2 with the saturate_after_prune
config switch, the DiskANN variant is dead code.
Delete:
- fn final_prune_diskann_style (~90 lines + the iterative alpha-ramp body)
- The `if std::env::var("PIPNN_DISKANN_PRUNE") ...` dispatch — single
call to final_prune_from_candidates remains.
All 85 lib tests pass.
Replace per-call DistanceProvider/Distance dispatch with inline target_feature kernels for the common case (squared_l2, fp32-converted candidates, d=128). Runtime tier dispatch picks AVX-512, AVX-2, or generic Distance fallback. Adds thread-local scratches FP_CAND_F32, FP_STATE, FP_SELECTED to reuse per-thread buffers across nodes. Microbench (dist_kernel_microbench.rs): AVX-512 inline: 3.69 ns/dist (1.66x vs Distance dispatch) AVX-2 inline: 4.91 ns/dist (1.24x) Distance dispatch: 6.10 ns/dist Bench c=512 f[8,3] k=3 hp=14 l128 prune=on (c4-48): Final prune: 5.53s -> 5.08s (-8%) R@10 L=10/50/100: 75.3% / 95.5% / 98.6% (avg_degree 41.9)
…scan
Port the AoSoA HashPrune layout from the pipnn-aosoa branch with L_MAX_MAX
bumped from 64 -> 128 to match this branch's l_max default.
Replaces Vec<Mutex<HashPruneReservoir>> (each reservoir's entries Vec heap-
allocated per point) with two contiguous slabs:
- hot: Vec<HotSlot> (16 B/slot: parking_lot RawMutex + len + farthest)
- cold: Vec<ColdSlot> (1024 B/slot at L_MAX=128: [hashes;128] +
[distances;128] + [neighbors;128] as three contiguous arrays)
Bug found and fixed: the AoSoA find_hash_simd was hard-coded to scan 2
AVX-512 chunks (= 64 lanes), assuming L_MAX_MAX=64. With L_MAX_MAX=128 a
hash in slot 64..127 returned None (false negative), causing duplicate
inserts instead of update-in-place. R@10 L=10 dropped 80.23 -> 79.35
before the fix. Generalized the scan to L_MAX_MAX/32 chunks (4 for L_MAX
=128) with u128 mask accumulator. AVX-2 path similarly extended to
L_MAX_MAX/16 chunks. Restores bit-equivalent graph.
Validated on BigANN 10M T=48 c=512 f[9,3] no_prune:
pre-AoSoA : 19.27 s Index Build, leaf 9.56 s, R@10 L=10 80.23%
AoSoA-fix : 18.89 s Index Build, leaf 7.97 s (-1.59 s), R@10 L=10 80.21%
Net wall -0.38 s on Index Build Time; -1.59 s on leaf phase. The leaf
savings is partly offset by the eager 10 GB cold-slab allocation which
front-loads into the LSH/init bucket (0.20 -> 1.66 s). All 85 lib tests
pass.
Added libc 0.2 dep for madvise(MADV_HUGEPAGE) on the slabs.
The cold prefetch was hitting only the first 128 B of each ColdSlot (hashes[0..63]) while find_hash linearly scans the full 256 B (hashes[0..127]). Bumping to 4 prefetches (offsets 0/64/128/192) covers the whole hash scan. BigANN 10M c=512 f[8,3] k=3 hp=14 l128 prune=on (c4-48): Leaf build: 6.870 -> 6.701 s (-0.17 s, -2.5%) Total wall: 21.35 -> 21.15 s R@10 L=10/50/100 stable (75.4 / 95.5 / 98.6, avg_degree 41.9)
The bubble-up loop generated scalar vmovss spills (3-5% of fused_dual_topk on perf annotate). Specializing k=1/2/3 as branch-free straight-line code lets LLVM keep top-k state in registers across the insertion. Falls back to the generic loop for k>=4. BigANN 10M c=512 f[8,3] k=3 hp=14 l128 prune=on (c4-48): Leaf build: 6.701 -> 6.638 s (-0.06 s) Total wall: 21.15 -> 21.11 s R@10 L=10/50/100 stable (75.4 / 95.5 / 98.6, avg_degree 41.9)
The straight-line branchless specialization originally covered k=1/2/3 only; bench configs use leaf_k up to 6, so the k=4/5/6 path was still hitting the generic bubble-up loop (4-6 iter, scalar vmovss spills). Specializing those three cases the same way (load all k-1 stayers once, three-way compare cascade, write only the shifted slots). Bench config unaffected when leaf_k=3 (existing path); enables the optimization for leaf_k=4..6 builds. 85/85 pipnn lib tests pass.
Standard form (per Algorithm 2 line-by-line) admits the closest unvisited candidate then scans ALL remaining candidates to mark occluded ones — O(R*N) distance computes per node where R=max_degree, N=nc. Lazy form (paper "Optimizing RobustPrune" section): walk candidates in ascending-distance-to-x order; for each candidate c, check against the already-admitted neighbors and early-exit on the first occluder. Output is identical (same admission order, same admission rule) — only the work distribution differs. Most rejects are occluded by one of the first few admitted neighbors, so the early-exit collapses the inner-loop cost on average. Bench BigANN 10M c=512 f[8,3] k=3 hp=14 l128 prune=on (c4-48): Final prune: 4.62 -> 3.57 s (-1.05 s, -23%) Total wall: 21.11 -> 20.44 s R@10 L=10/50/100 stable, avg_degree 41.9 unchanged. State Vec semantics changed (was state flags UNVISITED/SELECTED/OCCLUDED, now stores u8 local candidate indices of admitted neighbors). Saturation walks the admitted-indices list as a sorted gap-fill pass. 85/85 pipnn lib tests pass.
…soft#1231) One of my least favorite parts about writing `Benchmark` implementations is implementing `try_match` and its corresponding `describe` branch. Working with `Result<MatchScore, FailureScore>` is cumbersome, and duplicating the matching logic in `describe` is tedious and (as this PR shows for example in the bf-tree benchmarks) prone to inconsistencies. This PR attempts to address these issues by changing the signature of `Benchmark::try_match` from ```rust fn try_match(&self, input: &Self::Input) -> Result<MatchScore, FailureScore>; ``` to ```rust fn try_match(&self, input: &Self::Input, context: &MatchContext) -> Score; ``` where `Score` is a type that * Automatically accumulates match/mismatch penalties. * Switches from a "matched" to a "failure" mode on the first failure. * Conditionally records the failure reasons for later diagnostics. The conditionality of reason captures comes from the `context` argument. This enables a few things: * Score computation is handled more automatically. * All failures need to be accompanied by a reason, which forces everything to be in-sync and avoids the duplication of the current approach. * The ability to disable reason capture avoids allocating on matching paths where we aren't interested in the failure reasons, saving a few precious cycles. With this change, we can simplify `Benchmark::describe` from ```rust fn description( &self, f: &mut std::fmt::Formatter<'_>, input: Option<&Self::Input>, ) -> std::fmt::Result; ``` to ```rust fn description(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result; ``` ## Suggested Reviewing Order The bulk of the changes are in `diskann-benchmark-runner/src/benchmark.rs` where the new types are introduced. `diskann-benchmark-runner/src/registry.rs` updates the registry matching and display logic to the new approach. The majority of the rest of the changes are updating the existing `Benchmark` implementations across `diskann-benchmark-runner`, `diskann-benchmark-simd`, and `diskann-benchmark`. Overall, there is a pretty substantial reduction in code (~700 lines) for the implementors of `Benchmark`.
…crosoft#1171) (microsoft#1241) ### Problem Recently, the newly added `flat scan benchmark` uncovered a small precision issue in recall calculation. `benchmark_core::recall::knn` computes recall as the mean of per-query recalls. It did this by pushing each per-query ratio `r / this_recall_k` into a `Vec<f64>`, then summing and dividing by the query count. When every query is a clean fraction the sum stays exact, but as soon as some queries have non-terminating binary ratios (e.g. `99/100`, which isn't exactly representable in `f64`), summing such values compounds the rounding. In the next example, the reported average then lands just off the true value — e.g. the flat-index benchmark shows `0.9999899999999999` where the exact recall is `0.99999` (499,995 matches over 500,000 groundtruth results): ``` cargo run --package diskann-benchmark --release -- run --input-file diskann-benchmark\perf_test_inputs\wikipedia-100K-flat-index.json --output-file ./target/tmp/flat-index-output.json ###################### # Running Job 1 of 1 # ###################### Data: target/tmp\wikipedia_cohere/wikipedia_base.bin.crop_nb_100000 Data Type: float32 Distance: inner_product Queries: target/tmp\wikipedia_cohere/wikipedia_query.bin Groundtruth: target/tmp\wikipedia_cohere/wikipedia-100K K: 100 Threads: 4, 8 Reps: 1 Loading dataset... Loaded 100000 vectors of dimension 768 Queries: 5000, Groundtruth: 5000x100 K, Avg cmps, QPS - mean(max), Avg Latency, p99 Latency, Recall, Threads =================================================================================================================== 100, 100000, 123.0 (123.0), 32430.0us (32430.0us), 35711.0us (35711us), 0.9999899999999999, 4 100, 100000, 250.4 (250.4), 31909.2us (31909.2us), 33323.0us (33323us), 0.9999899999999999, 8 ``` ### Fix Defer the division: accumulate **integer** hit counts grouped by their denominator `this_recall_k` and divide once per distinct denominator. This is algebraically identical to the mean of per-query recalls but exact whenever all rows share a denominator (the common fixed-groundtruth case), collapsing to a single division. A `BTreeMap` keeps the summation order deterministic so recall is bit-reproducible. Macro-average semantics are unchanged. A regression test asserts a perfect run reports `1.0` and a run missing 5 of 500,000 neighbours reports `0.99999`. After the fix, recall reports `0.99999` as expected: ``` K, Avg cmps, QPS - mean(max), Avg Latency, p99 Latency, Recall, Threads ======================================================================================================== 100, 100000, 136.4 (136.4), 29235.7us (29235.7us), 35473.0us (35473us), 0.99999, 4 100, 100000, 268.4 (268.4), 29687.9us (29687.9us), 35269.0us (35269us), 0.99999, 8 ``` PS. **Why an exact top-k flat scan still shows 5 misses (and can't reach 1.0 here)**: intuitively a brute-force flat scan should match the groundtruth perfectly, but this dataset's groundtruth was generated by an external tool whose distance kernel differs from DiskANN's by a few f32 ULPs (~1e-5). That tiny difference only matters right at the rank-k cutoff: when two neighbours are equidistant to within f32 precision, the two kernels can disagree on which one lands at position 100 versus 101. In this run that happens for 5 queries (one of them an exact tie), so the flat scan keeps an equally-correct neighbour that the groundtruth places just outside the top 100.
Imports a collection of 10 previously-authored RFCs into `rfcs/` (numbered `00100` through `00109`) to preserve the record of design decisions made earlier in DiskANN's development. Keeping these RFCs alongside the code provides context for the decisions that shaped this repository.
…rosoft#1243) Bumps [rand](https://github.com/rust-random/rand) from 0.8.5 to 0.8.6. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/rust-random/rand/blob/0.8.6/CHANGELOG.md">rand's changelog</a>.</em></p> <blockquote> <h2>[0.8.6] - 2026-04-14</h2> <p>This release back-ports a fix from v0.10. See also <a href="https://redirect.github.com/rust-random/rand/issues/1763">#1763</a>.</p> <h3>Changes</h3> <ul> <li>Deprecate feature <code>log</code> (<a href="https://redirect.github.com/rust-random/rand/issues/1772">#1772</a>)</li> </ul> <p><a href="https://redirect.github.com/rust-random/rand/issues/1763">#1763</a>: <a href="https://redirect.github.com/rust-random/rand/pull/1763">rust-random/rand#1763</a> <a href="https://redirect.github.com/rust-random/rand/issues/1772">#1772</a>: <a href="https://redirect.github.com/rust-random/rand/pull/1772">rust-random/rand#1772</a></p> <ul> <li>Drop the experimental <code>simd_support</code> feature.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/rust-random/rand/commit/5309f25bb5e7d21ac01c5b6f476badd06f9cdc3f"><code>5309f25</code></a> 0.8.6 (<a href="https://redirect.github.com/rust-random/rand/issues/1772">#1772</a>): update for recent nightly rustc and backport <a href="https://redirect.github.com/rust-random/rand/issues/1764">#1764</a></li> <li><a href="https://github.com/rust-random/rand/commit/1126d03a5cbd725aad239efb0d537c9130a76b26"><code>1126d03</code></a> When testing rustc 1.36, use compatible dependencies.</li> <li><a href="https://github.com/rust-random/rand/commit/143b60280f79a5f1992445b3df0e0599841f9821"><code>143b602</code></a> Add Cargo.lock.msrv.</li> <li><a href="https://github.com/rust-random/rand/commit/9be86f2d8140139800989ac93399b9cd49108fb8"><code>9be86f2</code></a> Fix cross build test.</li> <li><a href="https://github.com/rust-random/rand/commit/5e0d50d7706281ae67e69ff64105baf3c94d6ef8"><code>5e0d50d</code></a> Drop simd_support.</li> <li><a href="https://github.com/rust-random/rand/commit/8ff02f0568d2f8fddda74b47613a3daaa5e2a879"><code>8ff02f0</code></a> Upgrade cache action.</li> <li><a href="https://github.com/rust-random/rand/commit/4ad0cc34fc847d4d59ffdcdfbf189482601aa6b9"><code>4ad0cc3</code></a> Don't test for unsupported target architecture.</li> <li><a href="https://github.com/rust-random/rand/commit/258e6d04a681321e0c4b16e3785063ed9b9e744d"><code>258e6d0</code></a> Address warning.</li> <li><a href="https://github.com/rust-random/rand/commit/9f0e676362f9599941f00bccc5310135b7c19f89"><code>9f0e676</code></a> Mark some internal traits as potentially unused.</li> <li><a href="https://github.com/rust-random/rand/commit/6f123c178eee4563876bdd50f4ac0621b21ce2b8"><code>6f123c1</code></a> Workaround never constructed and never used warning.</li> <li>Additional commits viewable in <a href="https://github.com/rust-random/rand/compare/0.8.5...0.8.6">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/microsoft/DiskANN/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Add `try_deserialize` methods directly to `spherical::iface::Impl` structs to allow deserialization independently of the `Quantizer` trait object.
([microsoft#1205](microsoft#1205)) The `diskann-platform` crate is deleted — its perf counters become a private detail of `Timer`, and its io primitives (io_uring / Windows IOCP / handles) move next to their only consumer, `aligned_file_reader`, in `diskann-disk`. `Timer` relocates from `diskann-providers` to `diskann-disk::utils::instrumentation::Timer`. `diskann-providers` loses its `opentelemetry` / `diskann-platform` dependencies and its `perf_test` feature; `diskann-disk` loses its unused `virtual_storage` feature and `Timer::{elapsed_seconds, elapsed_seconds_for_step}`. - Upgrade: Drop any dependency on `diskann-platform`. Reference `Timer` at `diskann_disk::utils::instrumentation::Timer` (or use `std::time::Instant` directly, as the providers/tools call sites now do). Remove references to the `diskann-providers` `perf_test` feature and the `diskann-disk` `virtual_storage` feature. dependency ([microsoft#1185](microsoft#1185)) Removed the dead `search_disk_index` / `build_disk_index` modules in `diskann-tools`, the unwired `AssociatedDataFilter` / `default_associated_data_filter` in `diskann-disk`, and unreferenced `test_utils` size constants. Ground-truth utilities and `build_pq` are no longer generic over `diskann-disk`'s `GraphDataType`. - Upgrade: Replace uses of the removed modules with the live disk build/search paths in `diskann-benchmark`, and the removed `AssociatedDataFilter` with the live `VectorFilter` / `default_vector_filter`. Where you relied on the `GraphDataType` generic, switch to a plain `V: VectorRepr` (plus an associated-data generic where needed). ([microsoft#1141](microsoft#1141), [microsoft#1192](microsoft#1192)) A new filter-aware `FilteredAccessor` trait replaces the pattern of passing a `QueryLabelProvider` to every call site; multi-hop search is rewritten on top of it. The `SearchAccessor` bound on `SearchStrategy::SearchAccessor` is relaxed to let the disjoint `FilteredAccessor` participate. `QueryVisitDecision` is removed, and `QueryLabelProvider` moves to `diskann::graph::ext::labeled` (as a compatibility layer). Follow-up [microsoft#1192](microsoft#1192) changes the `expand_beam_accept_only` bound from `P: HybridPredicate<Accept<Self::Id>>` to `P: Predicate<Self::Id> + PredicateMut<Accept<Self::Id>>`. - Upgrade: Implement/consume `FilteredAccessor` for filtered search; move `QueryLabelProvider` imports to `diskann::graph::ext::labeled`. Remove any use of `QueryVisitDecision`. Update `expand_beam_accept_only` predicate bounds to the split `Predicate` + `PredicateMut<Accept<_>>` form. ([microsoft#1173](microsoft#1173)) `DiskIndexSearcher::search` / `search_internal` now take `adaptive_l: Option<AdaptiveL>`. `None` preserves today's behavior; `Some(_)` routes the graph path through the new `filter_search()` (backed by `InlineFilterSearch` + the new `PredicateLabelProvider` adapting `&dyn Fn(&u32) -> bool` to `QueryLabelProvider<u32>`). - Upgrade: Add the new `adaptive_l` argument at disk-search call sites (pass `None` for unfiltered search). ([microsoft#1160](microsoft#1160)) `FlatIndex::knn_search` gains post-processing support via `diskann::glue::SearchPostProcess`. With the `VectorId` scalar-conversion constraint gone, `DistancesUnordered` now has `HasId` as a super-trait and `SearchStrategy` no longer carries an `Id` associated type. - Upgrade: Add a `HasId` impl to your `DistancesUnordered` types and remove the `Id` associated type from `SearchStrategy` implementations; tie the flat index's ID type to the underlying `DataProvider::InternalId`. ([microsoft#1231](microsoft#1231)) `try_match` changes from `fn try_match(&self, input: &Self::Input) -> Result<MatchScore, FailureScore>` to `fn try_match(&self, input: &Self::Input, context: &MatchContext) -> Score`, and `description` drops its `input: Option<&Self::Input>` argument. - Upgrade: Update `Benchmark` implementors to accumulate into the new `Score` type (recording a reason on every failure via `MatchContext`) and to the simplified `description(&self, f)` signature. ([microsoft#1223](microsoft#1223)) `WithApproximateNorm` is now a super-trait of `SampleableForStart` (required by `StartPointStrategy::compute`). - Upgrade: Ensure `SampleableForStart` implementors also implement `WithApproximateNorm`. ([microsoft#1183](microsoft#1183)) Migrates to bf-tree 0.5's CPR snapshot API: `snapshot*` → `cpr_snapshot(&path)` / `BfTree::new_from_cpr_snapshot()`; `save_bftree` is no longer async; `BfTreeParams::to_config` and `copy_snapshot_if_needed` are removed. A new `use_snapshot: bool` is added to `BfTreeProvider`, `BfTreeProviderParameters`, and `SavedParams`; `BfTreeStoreConfig::use_snapshot` is required in benchmark JSON input (`SavedParams::use_snapshot` defaults for backwards compatibility with existing saved-param files). - Upgrade: Set `use_snapshot` in `BfTreeStoreConfig` JSON; migrate any direct snapshot calls to the CPR API; treat `save_bftree` as synchronous. ([microsoft#1162](microsoft#1162)) Garnet replaces its postfilter with inline filtering (~50x QPS improvement in the PR's measurement) and is bumped to `3.0.0`. - Upgrade: Garnet consumers must adopt the 3.0.0 FFI surface with inline filtering. removed ([microsoft#1217](microsoft#1217), [microsoft#1213](microsoft#1213), [microsoft#1212](microsoft#1212)) All `diskann-tools` command-line parameters now use kebab-case, variable names are standardized, and the old (disabled) disk-index range-search binary is removed in favor of the new range-groundtruth utility. The `vector_filters_file` option is renamed to reflect that it takes a pre-calculated bitmap. - Upgrade: Update scripts to the kebab-case flag names and the renamed bitmap-input option; use the new range-groundtruth utility instead of the removed binary. - Serialization: New foundational `diskann-record` crate for versioned save/load of DiskANN indexes (Disk + Memory backends, `Save`/`Load` traits, `save_fields`/`load_fields` macros) ([microsoft#1188](microsoft#1188)). - Distance functions: New `ProjectedEigen` distance for multi-vector re-ranking in `diskann-quantization` ([microsoft#1203](microsoft#1203)). - Diversity search: Determinant-diversity post-processing wired as an optional plugin for async full-precision topk and disk-index search, with new example configs ([microsoft#1011](microsoft#1011)). - Benchmarks: New end-to-end flat-index benchmark for full-precision in-memory search ([microsoft#1170](microsoft#1170)); consolidated `Benchmark::try_match` logic reduces implementor boilerplate (~700 lines) ([microsoft#1231](microsoft#1231)); directory checking added to `diskann-benchmark-runner` (unifies file/dir discovery) ([microsoft#1227](microsoft#1227)). - bftree concurrency: Striped `RwLock` on `NeighborProvider` eliminates lost edges under concurrent mutation (11–51% edge loss → 0), safe dual-store write ordering, and quant-store cleanup on delete ([microsoft#1158](microsoft#1158)). - bftree correctness: Validate record sizes at construction and force insert errors to be handled, fixing misleading "vector not found" errors on oversized records ([microsoft#1166](microsoft#1166), [microsoft#1194](microsoft#1194)). - Benchmark recall: Fixed floating-point precision loss in the recall calculation (exact integer accumulation grouped by denominator) ([microsoft#1171](microsoft#1171) / [microsoft#1241](microsoft#1241)). - Filtered search: Simplified the `expand_beam_accept_only` pre-filter predicate, restoring multi-hop filtered-search recall ([microsoft#1192](microsoft#1192)). Co-authored-by: yaohongdeng <yaohongdeng+odspmdb@microsoft.com>
…ft#1238) It turns out that when we implemented the fix to recall computation for filtered search in microsoft#1069, the fix did not propagate to beta search. This happened because beta search used the same runner as regular knn search in `search/knn.rs`, so the wrong `GroundTruthMode` was passed in. This PR fixes that problem by including `GroundTruthMode` in `SearchSteps` so it can be passed in earlier in the stack. --------- Co-authored-by: Magdalen Manohar <mmanohar@microsoft.com>
A recurring issue in `diskann-benchmark` is the intersection of compile times and discoverability. To prevent new benchmarks negatively impacting compile-times for all, we either [block the implementation](https://github.com/microsoft/DiskANN/blob/647f53dbf78cef1fea772dd09fd3ab8d5dd85461/diskann-benchmark/src/index/inmem/product.rs#L9-L41) behind a feature gate, of [gate the whole benchmark](https://github.com/microsoft/DiskANN/blob/647f53dbf78cef1fea772dd09fd3ab8d5dd85461/diskann-benchmark/src/index/mod.rs#L16-L17). The first uses a [hack](https://github.com/microsoft/DiskANN/blob/647f53dbf78cef1fea772dd09fd3ab8d5dd85461/diskann-benchmark/src/utils/mod.rs#L105-L158) to nudge the user towards enabling a feature when such a benchmark's input is loaded but can't be matched. This is unfortunate because it still requires the **inputs** to be compiled into the binary, which (1) adds compilation overhead, particularly for `serde` and (2) either requires dependencies to be non-optional or additional [feature gates](https://github.com/microsoft/DiskANN/blob/647f53dbf78cef1fea772dd09fd3ab8d5dd85461/diskann-benchmark/src/inputs/disk.rs#L9-L20) due to input validation. The second gating approach makes it impossible (see microsoft#1143) for users to find such gated inputs without reading the source code. This PR attempts to address this issue by adding native support for gated items into `diskann-benchmark-runner`. While the behavior can be seen in the UX tests, I'll put some highlights here: ### Running `inputs` when there are feature gated inputs ``` Available input kinds are listed below: sample-input test-input-dim test-input-types One or more inputs are hidden, use "inputs --all" to list all inputs ``` ### Running `inputs --all` to list all inputs, including gated ones ``` Available input kinds are listed below: sample-input test-input-dim test-input-types phantom-input (requires the features "gated-feature-2" or "gated-feature-3") ``` ### Attempting to load an input (e.g. `phantom-input`) when it hasn't been compiled ``` while processing input 1 of 1 Caused by: use of the "phantom-input" input is gated behind the features "gated-feature-2" or "gated-feature-3" ``` ### Attempting to `describe` a gated input ``` use of the "phantom-input" input is gated behind the features "gated-feature-2" or "gated-feature-3" ``` ### Matching inputs against gated benchmarks ``` Could not find a match for the following input: { "data_type": "float16", "dim": 128, "error_when_checked": false } Closest matches for tag "test-input-types": 1. "type-bench-f32": - expected "float32" but found "float16" 2. "type-bench-i8": - expected "int8" but found "float16" 3. "exact-type-bench-f32-1000": - expected "float32" but found "float16" - expected dim=1000, but found dim=128 Found 1 gated benchmark matching this input: * "gated-bench" (requires the feature "gated-feature-0") could not find a benchmark for all inputs ``` # How It Works * `src/internal/visibility.rs`: Inputs and benchmarks are coupled with a `Visibility` which records if the associated item is `Available` or `Gated` behind some features. * `src/input.rs`: The `input::Registered` type gets a `visibility` method (and a pretty-printing `display` method). Internally, a new `Gated` type is used to drive the dynamic `DynInput` trait. Inputs that are put behind a feature gate (e.g., registered with `Registry::register_gated`) use this `Gated` type instead of the proper `Wrapper` type. * `src/benchmark.rs`: There are two major changes here: 1. Two new "phantom" benchmarks (like the `Gated` type in `inputs::Internal`) are added. `PartiallyGated` benchmarks behave like the current `impl_stub` macro, where the backend benchmarks don't exist, but their inputs get registered like normal. These benchmark kinds allow the runner to emit better matching diagnostics when the input is parsed but fails matching. The `FullyGated` benchmarks are those whose inputs are also gated (think `bf-tree`) and mainly exist as bread-crumbs that more benchmarks/inputs do exist, but are gated. 2. `internal::Benchmark::try_match` now returns an `AnnotatedMatch` instead of a raw `Score`. The `AnnotatedMatch` records whether or not the match succeeded or failed due to user matching logic, or if the match failed because of tag mismatch/gated mismatch. This additional machinery is used by the `Registry` when it comes to diagnostics. * `src/registry.rs`: The `Registry` gains two new methods: `register_gated` and `register_partially_gated`, which behave more or less as expected. Input registration gets a little more complicated as we now have to deal with the interaction of "real" and "gated" inputs. This PR takes the conservative stance that "real" and "gated" inputs are incompatible, so if a "real" inputs observes and existing "gated" input, that is an error, and vice-versa. Two "gated" inputs are compatible only if their gating feature sets are equal. Note that there is a little more refactoring that has happened in `registry.rs` with more display logic being moved down here instead of living up in `app.rs`. * `src/app.rs`: Updated input/benchmark displays taking into account `Visibility`. The mismatch printing logic has been moved down to `registry.rs` (see above). ## Testing Several additional gated benchmarks are added in `src/test/{gated, mod}.rs`. A `TestConfig` is added to allow "features" to be dynamically added during registration. The UX framework is updated to look for a `features.txt` file in a test directory and construct a `TestConfig` with the associated features. # Next Steps This PR is mainly focused on the core infrastructure and does not update `diskann-benchmark`. The latter is mostly a mechanical change, but this PR is big-enough as-is that updating `diskann-benchmark` will be done in a follow-up. *AI Disclosure*: Claude was used to help with some tests and code cleanup. In particular, the `TestConfig`, `src/test/gated.rs`, the updates to the UX framework to handle `features.txt`, and the tests for `src/features.rs` were originally generated by the agent.
Our workspace-wide tests have been pretty useless to run at `opt-level=0` (the default for the [dev](https://doc.rust-lang.org/cargo/reference/profiles.html#dev)/[test](https://doc.rust-lang.org/cargo/reference/profiles.html#test)) for a while now. This was made worse by the addition of larger [testing datasets](microsoft#1199). To tackle this, the `ci` profile was introduced to build and run our tests at `opt-level=1`. However, as the workspace continues to grow, `opt-level=1` more and more just needs to be the default. Especially to be compatible with [crater](https://github.com/rust-lang/crater) runs. This PR drops the `ci` profile and instead changes our `profile.test` to use `opt-level=1` by default. After this PR, ``` cargo nextest run --workspace ``` will use `opt-level=1` and behave like our current ``` cargo nextest run --workspace --cargo-profile ci ```
Add a `diskann::version()` function that returns the crate's version string at runtime. This functionality is useful for applications that need to log/provide dependency versions.
Quantizer state was not persisted to the store so the index would operate in full precision mode after a restart. This PR adds support for reloading and resuming quantized indexes. For Q8 quantizers, the quantized start point cache needs to be filled on restart. For BIN quantizers, the quantizer code book must be saved and loaded as well as a flag indicating whether backfill is complete or not. An FFI change to create_index signals to Garnet whether quantization triggers need to happen. This will ensure backfill completes if interrupted.
Back in the day, distance and query computers could not carry lifetimes. That has since changed. As such, we can remove the `Deref<Target = FixedChunkPQTable>` bound from PQ distance computers and simplify things a bit. This is a breaking change and will require a version bump. Co-authored-by: Mark Hildebrand <mhildebrand@microsoft.com>
While working on microsoft#1228, I noticed that there were some issues with our current implementation of range search and its testing. The main issue with testing is that there were no tests ensuring the `max_results` parameter was respected. I have added two tests that ensure this now. The main code had several issues with how `max_results` was handled: 1. The `max_results` parameter was allowed to be less than the initial L_search. This is a conceptual issue because the user expects `max_results` to stop the search from continuing for too long, and the compute used in the initial search will always be controlled by `initial_search_l`. 2. A `max_results` check was not enforced before deciding to continue to the second round search. This meant that if the max results was reached via the initial search, it might not be respected. 3. The second round search was not terminated when `max_results` was reached, meaning it would continue to perform unnecessary work. This PR fixes these issues by adding additional checks of `max_results` at the correct points in the code. --------- Co-authored-by: Magdalen Manohar <mmanohar@microsoft.com>
## What Make bf-tree neighbor lists initialize lazily instead of eagerly materializing an empty list for every id at construction. Two changes in `diskann-bftree`: 1. **`neighbors.rs`** — On neighbor read, treat a bf-tree `NotFound` result as an empty adjacency list rather than an error. (`Deleted` / `InvalidKey` still error.) 2. **`provider.rs`** — Drop the `O(max_points)` eager loop in `new_empty` that wrote an empty list for every id up front. ## Why The eager loop was a stop-gap for a missing "exists" query. But bf-tree already distinguishes `Found` / `NotFound` / `Deleted` natively, so mapping `NotFound` to an empty list (identical to the existing `Found(0)` case) lets neighbor lists be created on first write. This makes construction `O(num_start_points)` instead of `O(max_points)`, eliminating a billion-insert startup wall for large, larger-than-memory datasets, and removes the artificial coupling that required capacity to be fully materialized up front. ## Correctness The consolidation paths that motivated the stop-gap (`consolidate_vector` / `on_neighbors`) read neighbor lists during delete; a `NotFound`→empty result is identical to the eager-init outcome for an unwritten id. These remain covered by the delete-and-search tests, which exercise `inplace_delete` end to end. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When diagnosing some issues last week I realized that there was a cache coherency problem with `set_neighbors()`. `append_vector()` updates the start point neighbor cache atomically alongside the real neighbor list, but `set_neighbors()` did two writes sequentially which could result in coherency issues. As far as I'm aware, this did not cause any actual problems, but it was easy enough to make it coherent by using RMW instead in `set_neighbors()`.
# Conflicts: # Cargo.lock # Cargo.toml # diskann-benchmark-runner/src/app.rs # diskann-benchmark-runner/src/lib.rs # diskann-benchmark/Cargo.toml # diskann-benchmark/src/disk_index/mod.rs # diskann-benchmark/src/exhaustive/minmax.rs # diskann-benchmark/src/exhaustive/product.rs # diskann-benchmark/src/exhaustive/spherical.rs # diskann-benchmark/src/index/benchmarks.rs # diskann-benchmark/src/index/bftree/mod.rs # diskann-benchmark/src/index/inmem/product.rs # diskann-benchmark/src/index/inmem/scalar.rs # diskann-benchmark/src/index/inmem/spherical.rs # diskann-benchmark/src/index/mod.rs # diskann-benchmark/src/inputs/disk.rs # diskann-benchmark/src/inputs/graph_index.rs # diskann-benchmark/src/main.rs # diskann-benchmark/src/multi_vector/mod.rs # diskann-benchmark/src/utils/mod.rs # diskann-disk/Cargo.toml # diskann-disk/src/build/builder/build.rs # diskann-disk/src/build/builder/core.rs # diskann-disk/src/build/builder/tests.rs # diskann-disk/src/build/chunking/checkpoint/checkpoint_record_manager_with_file.rs # diskann-disk/src/build/chunking/continuation/utils.rs # diskann-disk/src/build/configuration/disk_index_build_parameter.rs # diskann-disk/src/storage/quant/generator.rs # diskann-disk/src/storage/quant/pq/pq_generation.rs # diskann-linalg/src/lib.rs # diskann-providers/src/model/graph/provider/async_/simple_neighbor_provider.rs # diskann-quantization/src/multi_vector/matrix.rs # diskann-vector/src/lib.rs # diskann/src/graph/index.rs # diskann/src/graph/internal/prune.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
What does this implement/fix? Briefly explain your changes.
Any other comments?