vector-store: create an in memory diskann provider#526
Merged
Conversation
Akvear
force-pushed
the
diskann-in-mem-provider
branch
from
July 20, 2026 14:35
a0b3de0 to
c82c143
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces an initial in-memory DiskANN-backed index factory for vector-store, enabling basic (blocking) vector insert/delete operations and plumbing it into configuration and runtime engine selection.
Changes:
- Add a new DiskANN index implementation (
vs_index::diskann) and expose anew_index_factory_diskannconstructor. - Add
DiskannAlphaconfiguration + env var (VECTOR_STORE_DISKANN_ALPHA) and use it to select DiskANN at runtime. - Add a DiskANN integration test for the
/infoendpoint and wire DiskANN deps into the workspace.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents the new VECTOR_STORE_DISKANN_ALPHA environment variable and cleans up command snippets formatting. |
| crates/vector-store/tests/integration/info.rs | Adds an integration test asserting DiskANN engine version is reported by /info. |
| crates/vector-store/src/vs_index/mod.rs | Exposes the new diskann index module. |
| crates/vector-store/src/vs_index/diskann.rs | Implements an in-memory DiskANN provider + actor handling Add/Remove (search/count currently unimplemented). |
| crates/vector-store/src/main.rs | Selects DiskANN factory when diskann_alpha is set in config. |
| crates/vector-store/src/lib.rs | Adds DiskannAlpha, adds it to Config, and exposes new_index_factory_diskann. |
| crates/vector-store/src/config_manager.rs | Parses VECTOR_STORE_DISKANN_ALPHA into config and adds a config loading test. |
| crates/vector-store/Cargo.toml | Adds DiskANN crates to the vector-store crate deps. |
| Cargo.toml | Adds git-based workspace deps for DiskANN crates. |
| Cargo.lock | Locks DiskANN and its transitive dependencies. |
Akvear
force-pushed
the
diskann-in-mem-provider
branch
from
July 21, 2026 10:50
c82c143 to
1eb5b79
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 10 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (1)
crates/vector-store/src/vs_index/diskann.rs:193
primary_id_to_u32performs a lossyu64 -> u32cast. This can cause ID collisions (and incorrect deletes/search results) wheneverPrimaryIdexceedsu32::MAXor when epochs differ, leading to silent index corruption.
// TODO: DiskANN's provider hardcodes `ExternalId = u32`, while vector-store keys are
// `u64` [`PrimaryId`]s (a 16-bit epoch packed with a row index). For this PoC we
// simply cast the `PrimaryId` down to `u32`, which silently truncates the upper bits
// (including the epoch) and can make distinct primary ids collide. Replace with a proper
// dense `u32 <-> PrimaryId` mapping before this is used beyond a baseline.
fn primary_id_to_u32(primary_id: PrimaryId) -> u32 {
u64::from(primary_id) as u32
}
Akvear
force-pushed
the
diskann-in-mem-provider
branch
4 times, most recently
from
July 22, 2026 11:46
e68d4e9 to
9cc4c17
Compare
Akvear
marked this pull request as ready for review
July 22, 2026 11:51
knowack1
reviewed
Jul 23, 2026
Akvear
force-pushed
the
diskann-in-mem-provider
branch
2 times, most recently
from
July 23, 2026 08:48
9cc4c17 to
7635092
Compare
Contributor
Author
|
Changes:
|
QuerthDP
requested changes
Jul 23, 2026
diskann-vector: that's where distance metrics can be imported from diskann-providers: that's where we import the default provider and it's traits/functions from
Akvear
force-pushed
the
diskann-in-mem-provider
branch
from
July 23, 2026 12:30
baf2316 to
a19a80e
Compare
Contributor
Author
|
Changes:
|
Akvear
force-pushed
the
diskann-in-mem-provider
branch
2 times, most recently
from
July 23, 2026 13:12
52602fc to
cc777c3
Compare
QuerthDP
previously approved these changes
Jul 23, 2026
ewienik
previously approved these changes
Jul 23, 2026
For the sake of the PoC we construct a specific full precision provider with: - a FastMemoryVectorProvider - the default high-performance store for f32 vectors - NoStore - specifies that we don't store auxilary vectors next to full precision vectors - TableDeleteProvider - default when deletes are needed A thread-hint can be passed in DiskANNIndex::new() Right now None is passed which means that diskann scratch object will be created on demand, this value is the number of maximum concurrent operations on a singular index Mappings are specified in the SEP and go as follows: - connectivity <-> R (pruned_degree) - expansion_add <-> l_build - expansion_search <-> l_value / beam_width (not needed yet) - space_type <-> distance function (metric) The space_type <-> metrics mappings are natural Unit test for mapping were added to verify correctness
Akvear
force-pushed
the
diskann-in-mem-provider
branch
from
July 24, 2026 08:10
cc777c3 to
db91f30
Compare
Contributor
Author
|
Changes:
|
QuerthDP
approved these changes
Jul 24, 2026
ewienik
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In this PR a DiskANNIndex is created in a in-memory
DefaultProvidermode using the parameters mapped fromVsIndexConfigurationFixes: VECTOR-717