vector-store: distinguish 503 from ann/bm25 when node is bootstrapping by knowack1 · Pull Request #524 · scylladb/vector-store

knowack1 · 2026-07-16T14:39:02Z

When an ANN or BM25 request is made while the index is not yet serving, the 503 response body now contains a JSON object with a 'reason' field that lets clients differentiate between two distinct cases without making a separate /api/v1/status call (which would introduce a TOCTOU race condition):

{"reason":"NODE_BOOTSTRAPPING"}
The whole node has not finished its startup sequence and is not
ready to serve any traffic. Clients should treat this as a
node-level outage and trigger high-availability failover.

{"reason":"INDEX_BUILDING","progress":P}
The node is healthy (SERVING) but this specific index is still
being constructed after creation. Clients should surface this
error to the caller rather than failing over.

Reference: SCYLLADB-3209

Copilot

Pull request overview

This PR updates the vector-store HTTP API so that ANN/BM25 requests returning 503 Service Unavailable can include a structured JSON body describing why the index isn’t ready (NODE_BOOTSTRAPPING vs INDEX_BUILDING with progress), avoiding a separate /status call and the associated TOCTOU race.

Changes:

Introduces httpapi::IndexNotReadyReason (tagged JSON enum) for 503 “index not ready” responses.
Updates ANN integration tests to assert the new 503 JSON body and adds coverage for the “node serving but index building” case.
Updates OpenAPI/utoipa docs to document the new 503 JSON schema for ANN and BM25.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
crates/vector-store/tests/integration/usearch.rs	Updates ANN 503 assertions to parse `IndexNotReadyReason` and adds a “node serving” test case.
crates/vector-store/tests/integration/fts.rs	Updates BM25 503 assertions to parse `IndexNotReadyReason` and adds a “node serving” test case.
crates/vector-store/src/httproutes.rs	Implements JSON 503 for ANN when index is not serving; updates OpenAPI annotations; adds helper to map node status to reason.
crates/httpapi/src/lib.rs	Adds `IndexNotReadyReason` enum with serde + utoipa schema derivations.
api/openapi.json	Updates the generated OpenAPI schema to reference `IndexNotReadyReason` for 503 responses.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

QuerthDP

I really like this change. Feels helpful and smart. Leaving some comments before I approve.

QuerthDP · 2026-07-17T06:27:47Z

+            description = "Service Unavailable. The index is not ready to serve requests. \
+The body is a JSON object with a 'reason' field: \
+'NODE_BOOTSTRAPPING' when the node has not yet finished its startup sequence; \
+'INDEX_BUILDING' (with a 'progress' field) when the node is healthy but this index is still being constructed.",


Not related to this PR, but genuinely curious should we keep the API definitions duplicated for both ANN/BM25? Couldn't the error codes be defined ones and reused?

I put this into IndexNotReadyResponse. So now there is no duplicate. Unfortunately I put this into same commit. I hope now its not too much in the single commit.

QuerthDP · 2026-07-17T06:31:51Z

+    let config = Config {
+        vector_store_addr: SocketAddr::from(([127, 0, 0, 1], 0)),
+        fulltext_indexes: true,
+        ..Default::default()
+    };
+
+    let node_state = vector_store::new_node_state().await;
+    let internals = vector_store::new_internals();
+    let (db_actor, db) = db_basic::new(node_state.clone());
+    let (receivers, senders) = crate::create_config_channels(config).await;
+    let index_factory = vector_store::new_index_factory_usearch(receivers.config.clone()).unwrap();
+    let (server, _mtls) = vector_store::run(
+        node_state,
+        db_actor,
+        internals,
+        index_factory,
+        receivers,
+        vector_store::new_metrics(),
+    )
+    .await
+    .unwrap();
+    let addr = (*server.address().await.borrow()).unwrap();
+    let client = HttpClient::new(addr);
+    let _keep = (server, senders);


Why is the setup so huge? Don't we have any common utility for that?

I've made some improvements, so the setup is now a bit more compact. Instead of doing all these initialization steps manually, I use a generic test utility that creates the table/index and waits until it's ready. After that, I create a second index for bootstrapping and run queries against it. This makes the test setup a little smaller without significantly changing or refactoring the setup utilities.

QuerthDP · 2026-07-17T06:32:50Z

+    let node_state = vector_store::new_node_state().await;
+    let internals = vector_store::new_internals();
+    let (db_actor, db) = db_basic::new(node_state.clone());
+    let (receivers, senders) = crate::create_config_channels(test_config()).await;
+    let index_factory = vector_store::new_index_factory_usearch(receivers.config.clone()).unwrap();
+    let (server, _mtls) = vector_store::run(
+        node_state,
+        db_actor,
+        internals,
+        index_factory,
+        receivers,
+        vector_store::new_metrics(),
+    )
+    .await
+    .unwrap();
+    let addr = (*server.address().await.borrow()).unwrap();
+    let client = HttpClient::new(addr);
+    let _keep = (server, senders);


Just like with FTS, I need to create a new column for the second index here. Otherwise, the best index routing will always pick up the serving index.

QuerthDP · 2026-07-17T06:41:29Z

+            let indexes = state.indexes.read().unwrap();
+            let Some(entry) = indexes.get_fts(&index_key) else {
+                timer.observe_duration();
+
+                let msg = format!("missing index: {keyspace}.{index_name}");
+                debug!("post_index_bm25: {msg}");
+                return (StatusCode::NOT_FOUND, msg).into_response();
+            };


It is duplicated above. Why do we need this 2nd time?

QuerthDP · 2026-07-17T06:51:37Z

    let index_key = IndexKey::new(&keyspace, &index_name);

-    let (fts_sender, primary_key_columns) = {
+    let not_ready_progress = {


One more thing, I think the not_ready_progress is a bit counterintuitive. I see from the code what we want to achieve and that None means we're all good, but it's not easy to think about what exactly "None not ready progress" or "Some not ready progress" means. I feel like the if was slightly better in this case.

Fixed. I renamed this serving_or_progress and removed the None.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

When an ANN or BM25 request is made while the index is not yet serving, the 503 response body now contains a JSON object with a 'reason' field that lets clients differentiate between two distinct cases without making a separate /api/v1/status call (which would introduce a TOCTOU race condition): {"reason":"NODE_BOOTSTRAPPING"} The whole node has not finished its startup sequence and is not ready to serve any traffic. Clients should treat this as a node-level outage and trigger high-availability failover. {"reason":"INDEX_BUILDING","message":M} The node is healthy (SERVING) but this specific index is still being constructed after creation. The 'message' field carries a human-readable description including the build progress. Reference: SCYLLADB-3209

knowack1 requested a review from Copilot July 16, 2026 14:40

Copilot started reviewing on behalf of knowack1 July 16, 2026 14:40 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread crates/vector-store/src/httproutes.rs Outdated

knowack1 force-pushed the scylladb-3209-distinguish-between-index-not-ready-status-when-node-is-bootstrapping branch from 6d6c81a to 6f1ac3b Compare July 16, 2026 16:11

knowack1 requested a review from Copilot July 16, 2026 16:11

Copilot started reviewing on behalf of knowack1 July 16, 2026 16:12 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread crates/vector-store/src/httproutes.rs Outdated

knowack1 force-pushed the scylladb-3209-distinguish-between-index-not-ready-status-when-node-is-bootstrapping branch from 6f1ac3b to e09aeeb Compare July 16, 2026 16:23

knowack1 requested a review from ewienik July 17, 2026 05:14

knowack1 marked this pull request as ready for review July 17, 2026 05:14

knowack1 requested a review from QuerthDP July 17, 2026 05:14

QuerthDP reviewed Jul 17, 2026

View reviewed changes

knowack1 force-pushed the scylladb-3209-distinguish-between-index-not-ready-status-when-node-is-bootstrapping branch 3 times, most recently from df7d347 to 2379748 Compare July 17, 2026 15:18

knowack1 requested a review from Copilot July 17, 2026 15:25

Copilot started reviewing on behalf of knowack1 July 17, 2026 15:26 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread crates/vector-store/tests/integration/fts.rs

Comment thread crates/vector-store/tests/integration/fts.rs

knowack1 force-pushed the scylladb-3209-distinguish-between-index-not-ready-status-when-node-is-bootstrapping branch from 2379748 to cdc336f Compare July 17, 2026 16:26

knowack1 requested a review from QuerthDP July 17, 2026 18:20

knowack1 force-pushed the scylladb-3209-distinguish-between-index-not-ready-status-when-node-is-bootstrapping branch from cdc336f to 3031bc1 Compare July 19, 2026 18:46

Uh oh!

Conversation

knowack1 commented Jul 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

QuerthDP left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants