vector-store: minimal DiskANN integration by Akvear · Pull Request #504 · scylladb/vector-store

Akvear · 2026-07-06T15:17:18Z

This PR are the first steps towards integrating DiskANN.
There is 1 new imported crate: diskann.
Implemented end-to-end getter for version of diskann reported via GET.

The USearch and OpenSearch code paths behave unchanged.

Fixes: VECTOR-713
Fixes: VECTOR-716
Fixes: VECTOR-718
Fixes: VECTOR-712

Copilot

Pull request overview

This PR introduces an initial (non-wired) DiskANN index-engine skeleton into the vector-store crate and adds the DiskANN dependency set to the workspace, as a first step toward future DiskANN integration.

Changes:

Adds a new vs_index::diskann module with a minimal DiskannIndexFactory implementation and a stub actor loop.
Registers the new module in vs_index/mod.rs (currently suppressed as unused).
Adds DiskANN-related crates (diskann, diskann-providers, diskann-quantization, diskann-vector) to workspace and vector-store dependencies.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
crates/vector-store/src/vs_index/mod.rs	Exposes the new DiskANN module (currently marked as unused).
crates/vector-store/src/vs_index/diskann.rs	Adds a stub `DiskannIndexFactory` + actor loop with placeholder behavior.
crates/vector-store/Cargo.toml	Adds DiskANN crates as dependencies of the vector-store crate.
Cargo.toml	Adds DiskANN crates to workspace dependencies (currently pinned exactly).
Cargo.lock	Captures the resolved dependency graph updates for the new crates.

Copilot

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated 6 comments.

Copilot

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.

QuerthDP · 2026-07-15T14:25:54Z

Please provide a changelog and rerequest review once you finish responding to the round of commits.

knowack1 · 2026-07-16T07:39:01Z

@@ -0,0 +1,83 @@
+/*
+ * Copyright 2026-present ScyllaDB
+ * SPDX-License-Identifier: LicenseRef-ScyllaDB-Source-Available-1.0


A new licence is LicenseRef-ScyllaDB-Source-Available-1.1. Also in other places.

Should we update it treewide?

I think yes. But of course in dedicated PR.

Akvear · 2026-07-16T09:16:30Z

Changes:

split up a big commit into 3 logically separate commits
deleted a commit with unnecessary StorageProvider implementation, instead used a DiskANN FileStorageProvider
squashed mapping tests into the previous commit
fixed commit history to not mislead about the quantization
kept one line per import style
put new imports and new constants in appropriate commits
changed commit order to make all of them compile

knowack1

Please find more comments from my side.

Given the scope, I'd suggest splitting this into two separate PRs:

Scaffolding for the DiskANN index instance — everything needed to create the index where all methods return "not implemented" (or similar stub). This can be verified via the info.version integration test already included in this PR.
Actual DiskANN index implementation — the collector, build pipeline, serving state, etc.

knowack1 · 2026-07-16T10:05:39Z

+    dimensions: Dimensions,
+    vectors: &[(PrimaryId, Vector)],
+) -> anyhow::Result<()> {
+    let dataset = File::create(dataset_path).map_err(|e| {


I'm not sure whether we should use the blocking or async versions of the I/O functions. Using the blocking versions might make sense here for performance reasons. @ewienik , could you please advise?

We should use async I/O functions

Akvear · 2026-07-16T11:17:39Z

Changes:

Changed the scope of this PR

Comments regarding the part delegated to a follow up will be addressed in the follow up.

ewienik · 2026-07-16T11:24:54Z

+    fn try_from(
+        (cfg, alpha, max_points): (&VsIndexConfiguration, f32, usize),
+    ) -> Result<Self, Self::Error> {


Could it be better to have DiskannParams::new() -> anyhow::Result<_>?

I see there is still TryFrom implemented for a tuple of three types - IMHO it is a sign that there is a need for a ctor method instead of TryFrom.

Akvear · 2026-07-16T14:04:58Z

Changes:

removed quantization from the scope
fixed newlines between imports
changed usize and f32 to more restricitve types
changed std::fs to tokio::fs
updated env variable name in README

knowack1 · 2026-07-17T06:21:31Z

                debug!("starting");

+                async {
+                    if tokio::fs::try_exists(&index_dir).await.unwrap_or(false) {


I think we need to wipe the existing index data at bootup - as they can be out of sync. However, this can't be done at the individual index level because we also need to wipe data for indices that no longer exist.

QuerthDP · 2026-07-17T07:21:11Z

+        .unwrap();
+
+        assert_eq!(
+            params.config.pruned_degree(),


What is a pruned_degree() and how does it apply to the MAX_POINTS?

Pruned degree is the R in VECTOR-717, it has nothing to do with MAX_POINTS. MAX_POINTS is a limit of how big the dataset the index is built around is, so it can preallocate resources - i am not sure where to get this value from.

using a git commit sha because the required version function is not in the latest released crate

VECTOR_STORE_DISKANN_ROOT_PATH - path to root DiskANN indexes directory VECTOR_STORE_DISKANN_ALPHA - DiskANN parameter for index quality/build time tradeoff add unit test verifying diskANN config variables correctness verify info_diskann returns correct information

Akvear · 2026-07-17T11:05:03Z

Changes:

removed parameters mapping from the scope as they are not needed yet
fixed commit history to read better
removed unused crates
added directory wipe both in new_diskann and on index creation
added unit tests for dir wipe

I am not sure whether it's needed to have all functions be async in the last commit, or if tokio has API that allows for something different.

Akvear · 2026-07-17T11:08:04Z

+    let mut dir = fs::read_dir(root)
+        .await
+        .context("failed to read DiskANN root directory")?;
+
+    while let Some(entry) = dir
+        .next_entry()
+        .await
+        .context("failed to read directory entry in DiskANN root")?
+    {
+        let path = entry.path();
+
+        if path.is_dir() {
+            fs::remove_dir_all(&path)
+                .await
+                .context(format!("failed to remove DiskANN directory: {:?}", path))?;
+        } else {
+            fs::remove_file(&path)
+                .await
+                .context(format!("failed to remove DiskANN file: {:?}", path))?;
+        }
+    }


Is it better/allowed to remove the directory entirely and recreate it?

We could assume that the parent directory provided by env variable exists and the content we can remove/create how we want.

What about when the directory we want to wipe is the diskann root path from the env var?

IMHO we shouldn't remove directory provided by the env - it is our parent. We can modify all its content, but not directory itself. I think we shouldn't create this directory also - it should be created for vector-store before.

Ok, that's the way it is now.

github-actions Bot added P3 area/vector labels Jul 6, 2026

Akvear requested a review from Copilot July 7, 2026 08:08

Copilot started reviewing on behalf of Akvear July 7, 2026 08:09 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread crates/vector-store/src/vs_index/mod.rs Outdated

Comment thread crates/vector-store/src/vs_index/diskann.rs

Comment thread crates/vector-store/src/vs_index/diskann.rs

Comment thread crates/vector-store/Cargo.toml Outdated

Comment thread Cargo.toml Outdated

Akvear force-pushed the diskann-index-factory-skeleton branch 6 times, most recently from bf0f8f7 to ce0f617 Compare July 8, 2026 10:01

Akvear requested a review from Copilot July 8, 2026 10:02

Copilot started reviewing on behalf of Akvear July 8, 2026 10:02 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Comment thread crates/vector-store/src/vs_index/diskann.rs Outdated

Comment thread crates/vector-store/src/vs_index/diskann.rs Outdated

Comment thread Cargo.toml Outdated

Comment thread crates/vector-store/Cargo.toml Outdated

Comment thread crates/vector-store/src/vs_index/diskann.rs Outdated

Akvear force-pushed the diskann-index-factory-skeleton branch from ce0f617 to a32c707 Compare July 9, 2026 08:30

Akvear changed the title ~~vector-store: minimal diskann index factory skeleton~~ vector-store: minimal DiskANN integration Jul 9, 2026

Akvear force-pushed the diskann-index-factory-skeleton branch from a32c707 to 1864678 Compare July 9, 2026 08:50

Akvear requested a review from Copilot July 9, 2026 08:57

Copilot started reviewing on behalf of Akvear July 9, 2026 08:58 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Akvear force-pushed the diskann-index-factory-skeleton branch 4 times, most recently from 94028e5 to 5a7a7f2 Compare July 13, 2026 12:53

Akvear requested a review from Copilot July 13, 2026 13:09

Copilot started reviewing on behalf of Akvear July 13, 2026 13:10 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Akvear force-pushed the diskann-index-factory-skeleton branch from 5a7a7f2 to 703ce8f Compare July 13, 2026 15:07

Akvear marked this pull request as ready for review July 13, 2026 15:11

Akvear force-pushed the diskann-index-factory-skeleton branch from 703ce8f to cb407ba Compare July 14, 2026 10:16

knowack1 reviewed Jul 16, 2026

View reviewed changes

Akvear force-pushed the diskann-index-factory-skeleton branch 2 times, most recently from 7d669ef to 633f1d9 Compare July 16, 2026 09:13

Akvear requested a review from QuerthDP July 16, 2026 09:21

Akvear force-pushed the diskann-index-factory-skeleton branch from 633f1d9 to 7a589de Compare July 16, 2026 11:14

knowack1 reviewed Jul 16, 2026

View reviewed changes

ewienik requested changes Jul 16, 2026

View reviewed changes

Akvear force-pushed the diskann-index-factory-skeleton branch 2 times, most recently from 70987c3 to 5a09dc4 Compare July 16, 2026 14:03

Akvear force-pushed the diskann-index-factory-skeleton branch from 5a09dc4 to 520004a Compare July 16, 2026 14:10

Akvear mentioned this pull request Jul 16, 2026

vector-store: create an actual index and materialize index files #525

Draft

knowack1 reviewed Jul 17, 2026

View reviewed changes

QuerthDP reviewed Jul 17, 2026

View reviewed changes

ewienik requested changes Jul 17, 2026

View reviewed changes

Akvear force-pushed the diskann-index-factory-skeleton branch from 520004a to f7342ee Compare July 17, 2026 10:58

Akvear added 6 commits July 17, 2026 13:01

chore: add diskann crates

4c103f3

using a git commit sha because the required version function is not in the latest released crate

vector-store: add skeleton for DiskannIndexFactory

fff79bb

README: document new diskANN env variables

cc512e9

vector-store: refine DiskannIndexFactory

43c5323

vector-store: wipe existing index data on startup

c94cb77

Akvear force-pushed the diskann-index-factory-skeleton branch from f7342ee to c94cb77 Compare July 17, 2026 11:01

Akvear commented Jul 17, 2026

View reviewed changes

Akvear requested review from QuerthDP and knowack1 July 17, 2026 11:11

Uh oh!

Conversation

Akvear commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

QuerthDP commented Jul 15, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Akvear commented Jul 16, 2026

Uh oh!

knowack1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Akvear commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Akvear commented Jul 6, 2026 •

edited

Loading

Akvear commented Jul 16, 2026 •

edited

Loading