Support Alternator in Vector Store by QuerthDP · Pull Request #394 · scylladb/vector-store

QuerthDP · 2026-03-18T12:17:19Z

Summary

This PR adds Alternator (DynamoDB-compatible API) support to the Vector Store, enabling it to index and search vectors stored through ScyllaDB's Alternator interface alongside the existing native CQL path.

Motivation

Alternator stores data differently from CQL. Vectors live as DynamoDB JSON blobs inside a map<bytes, bytes> column named :attrs, and the schema system has no native VECTOR<float, N> type. This means the Vector Store needed changes in dimension discovery, embedding extraction, full-scan queries, and CDC consumption to handle Alternator-backed tables.

Fixes: VECTOR-580

Fixes: [VECTOR-580](https://scylladb.atlassian.net/browse/VECTOR-580) [VECTOR-580]: https://scylladb.atlassian.net/browse/VECTOR-580?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Closes scylladb#394 * github.com:scylladb/vector-store: db: read dimensions from index options for Alternator validator: Alternator API test

QuerthDP · 2026-03-19T13:44:06Z

Changelog:

added initial really messy, hugely ai-assisted, WIP full-scan & CDC implementation

It passes all @m-szymon's Validator tests (except the query tests).
There exists a known bug with timestamps not being handled correctly.

Run @nyh's tests
Full results:

test_vector.py::test_vector_value PASSED                                 [  2%]
test_vector.py::test_numeric_list_precision_range PASSED                 [  4%]
test_vector.py::test_createtable_vectorindexes PASSED                    [  6%]
test_vector.py::test_createtable_vectorindexes_missing_fields PASSED     [  8%]
test_vector.py::test_createtable_vectorindexes_same_name PASSED          [ 10%]
test_vector.py::test_createtable_vectorindexes_same_name_gsi PASSED      [ 12%]
test_vector.py::test_createtable_vectorindexes_same_name_lsi PASSED      [ 14%]
test_vector.py::test_createtable_vectorindexes_vnodes_forbidden PASSED   [ 16%]
test_vector.py::test_createtable_vectorindexes_indexname_rules PASSED    [ 18%]
test_vector.py::test_createtable_vectorindexes_dimensions_rules PASSED   [ 20%]
test_vector.py::test_createtable_vectorindexes_attributename_key PASSED  [ 22%]
test_vector.py::test_createtable_vectorindexes_attributename_len PASSED  [ 25%]
test_vector.py::test_createtable_vectorindexes_multiple PASSED           [ 27%]
test_vector.py::test_describetable_vectorindexes PASSED                  [ 29%]
test_vector.py::test_createtable_vectorindexes_returned PASSED           [ 31%]
test_vector.py::test_updatetable_vectorindex_create PASSED               [ 33%]
test_vector.py::test_updatetable_vectorindex_delete PASSED               [ 35%]
test_vector.py::test_updatetable_vectorindex_delete_nonexistent PASSED   [ 37%]
test_vector.py::test_updatetable_vectorindex_missing_fields PASSED       [ 39%]
test_vector.py::test_updatetable_vectorindex_taken_name_or_attribute PASSED [ 41%]
test_vector.py::test_updatetable_vectorindex_vnodes_forbidden PASSED     [ 43%]
test_vector.py::test_updatetable_vectorindex_indexname_bad PASSED        [ 45%]
test_vector.py::test_updatetable_vectorindex_dimensions_bad PASSED       [ 47%]
test_vector.py::test_updatetable_vectorindex_attributename_bad_len PASSED [ 50%]
test_vector.py::test_updatetable_vectorindex_just_one_update PASSED      [ 52%]
test_vector.py::test_putitem_vectorindex_createtable PASSED              [ 54%]
test_vector.py::test_putitem_vectorindex_updatetable PASSED              [ 56%]
test_vector.py::test_query_vectorsearch_missing_indexname PASSED         [ 58%]
test_vector.py::test_query_vectorsearch_wrong_indexname PASSED           [ 60%]
test_vector.py::test_query_vectorindex_no_vectorsearch PASSED            [ 62%]
test_vector.py::test_query_vectorsearch_missing_queryvector PASSED       [ 64%]
test_vector.py::test_query_vectorsearch_queryvector_bad PASSED           [ 66%]
test_vector.py::test_query_vectorsearch_limit_bad PASSED                 [ 68%]
test_vector.py::test_query_vectorsearch_consistent_read PASSED           [ 70%]
test_vector.py::test_query_vectorsearch_exclusive_start_key PASSED       [ 72%]
test_vector.py::test_query_vector_prefill FAILED                         [ 75%]
test_vector.py::test_query_vector_with_ck_prefill PASSED                 [ 77%]
test_vector.py::test_query_vector_cdc FAILED                             [ 79%]
test_vector.py::test_query_vector_multiple_results FAILED                [ 81%]
test_vector.py::test_query_vector_with_ck_multiple_results FAILED        [ 83%]
test_vector.py::test_putitem_vectorindex_bad_vector XFAIL (write ope...) [ 85%]
test_vector.py::test_updateitem_vectorindex_bad_vector XFAIL (write ...) [ 87%]
test_vector.py::test_batchwriteitem_vectorindex_bad_vector XFAIL (wr...) [ 89%]
test_vector.py::test_deleteitem_vectorindex[no_ck] FAILED                [ 91%]
test_vector.py::test_deleteitem_vectorindex[with_ck] PASSED              [ 93%]
test_vector.py::test_vector_with_ttl FAILED                              [ 95%]
test_vector.py::test_query_vectorsearch_select_bad PASSED                [ 97%]
test_vector.py::test_query_vectorsearch_select FAILED                    [100%]

QuerthDP · 2026-03-19T15:23:23Z

There exists a known bug with timestamps not being handled correctly.

Waiting for scylladb/scylladb#15427 so that we could use writetime on vector element of :attrs. As explored there is no other possible and consistently correct workaround until then.

QuerthDP · 2026-03-20T09:42:24Z

After the recent scylladb/scylladb#29046 changes by @nyh all existing Validator and (except xfail) Python tests pass! 🎉
Working on making this PR production ready with the scylladb/scylladb#29134 (writetime) changes.

QuerthDP · 2026-03-20T10:41:43Z

Changelog:

add correct timestamp handling with writetime called on element of the :attrs

QuerthDP · 2026-03-20T11:54:06Z

Changelog:

initial rebase by CoPilot
add quoted CqlIdentifiers handling for non-alternator queries as well
add tests for quoted CqlIdentifiers

QuerthDP · 2026-03-23T09:55:15Z

Changelog:

use released CDC Rust Driver 0.6.2
use CqlIdentifier instead of quoting CQL identifiers on our own

QuerthDP · 2026-03-23T11:30:11Z

Changelog:

rebase on master

QuerthDP · 2026-03-23T11:54:19Z

Changelog:

fix clippy and fmt
change bump scylla cdc incorrect commit message

The cargo deny issue is with the Alternator API tests. I hope this will be resolved by @m-szymon in #392

m-szymon · 2026-03-23T12:00:52Z

I hope this will be resolved by @m-szymon in #392

Yes.
But actually should we merge failing tests first? Or should I somehow mark them as failing/[ignore]?

QuerthDP · 2026-03-23T12:03:42Z

I hope this will be resolved by @m-szymon in #392

Yes. But actually should we merge failing tests first? Or should I somehow mark them as failing/[ignore]?

That's the question I'm asking myself as well.
I would definitely not merge the failing tests. Better option would be to merge this and then the tests, but on the other hand we would not have this PR validated.

If you don't mind I may include your PR as an addition to this PR, just after you manage to have it ready.

nyh · 2026-03-23T12:15:06Z

Yes. But actually should we merge failing tests first? Or should I somehow mark them as failing/[ignore]?

That's the question I'm asking myself as well. I would definitely not merge the failing tests. Better option would be to merge this and then the tests, but on the other hand we would not have this PR validated.

If you already have fixes you can merge them before merging still-failing tests.

But in general, I don't see any problem with merging failing tests, properly marked "xfail". An "xfail" tests means that you believe that the test itself is correct (don't merge a test which you think is incorrect!), but Scylla has a bug.

You can find a lot of these "xfail" tests in test/alternator (over 100!), and I think they are super-important and very under-appreciated: When you write tests before writing code (test-driven development) or during writing code, you often think of cases which you didn't yet handle correctly. You have three options when you think of such cases:

Don't do anything, you'll handle them later.
Create an issue, to remember to handle them later.
Create an xfailing test showing the problem we'll need to handle later (and probably also create an issue which is cross-referenced with the xfailing test).

As you can probably see, 1 seems the most "popular" approach in Scylla, but 3 is in my opinion the best approach and what I personally always strive to do.

QuerthDP · 2026-03-23T12:58:03Z

Changelog:

move is_alternator() dead code from c135c8e to relevant commits

QuerthDP · 2026-03-23T13:55:31Z

Changelog:

update 951b07f and b0aee66 commit messages
remove unnecessary invalid column checks and comment

QuerthDP · 2026-03-23T15:52:28Z

Changelog:

remove unnecessary logging in tests
add EmbeddingSource trait to distinguish whether we read from Alternator or CQL created CDC
refactor full scan to use common branch
refactor embeddings.rs to use common functions for CQL/Alternator embeddings
refactor the comments by condensing them or removing unnecessary information
deAIize the code

QuerthDP · 2026-04-01T11:23:35Z

Failing tests from "new" @m-szymon's suite:

- alternator::update_item_vector_element_operations
- alternator::query_with_number_key
- alternator::query_with_binary_key
- alternator::ttl_expiration_verified_via_query_with_all_projected

QuerthDP · 2026-04-01T11:43:48Z

Changelog:

add KeyspaceIdentifier and TableIdentifier as wrappers around CqlIdentifier

ewienik

Now it is better solution to implement From instead of dyn trait for cdc. There is a need for rebase to master and additional small requests from me.

Bump scylla-cdc from 0.6.1 to 0.6.3 to pick up fixes on CQL identifiers quoting and Alternator's TTL which is necessary for Alternator with Vector Search integration.

Add `db_index_backend` helpers to read dimensions for CQL and Alternator indexes. For CQL indexes, dimensions are extracted from the column type via the existing regex. For Alternator indexes, dimensions are read from the index options map in `system_schema.indexes` since Alternator stores vectors inside the `:attrs` map rather than in a typed VECTOR column.

QuerthDP · 2026-04-01T15:19:33Z

Changelog:

rebased on master
renamed IndexBackend to DbIndexBackend
changed Vector type to have it's internal type private
added From<AsRef> for Keyspace- and TableIdentifiers
added From<IndexMetadata> for DbIndexBackend

Extend the backend abstraction with `vector_column_name()` and `extract_vector()`, and add `range_scan_query()` as a backend-aware helper so that each backend can customise how vectors are read during full scans and CDC consumption. For Alternator tables the vector lives inside the `:attrs` map column as a serialised blob, so range scans select `":attrs"['<name>']` and the CDC consumer extracts the vector from the `:attrs` `CqlValue`. A `Vector` type (moved out of `lib.rs` into `vector.rs`) handles conversion from both `CqlValue::Vector` and the Alternator blob format.

Wrap keyspace, table, column, and partition-key names with `CqlIdentifier` so that identifiers containing hyphens, colons, or mixed case are properly double-quoted in generated CQL. Alternator attribute names inside the `:attrs` map access are single-quoted via `CqlLiteral`. Add unit tests covering lowercase, mixed-case, uppercase, and special-character identifiers for both CQL and Alternator paths.

Move the session-wait and Statements construction above the CDC actor spawns so that a preparation failure (e.g. bad schema, unreachable node) is surfaced immediately via the returned error instead of silently orphaning background CDC tasks that would keep running with no consumer.

QuerthDP · 2026-04-01T15:23:19Z

Changelog:

added Vector::is_empty() to satisfy clippy

nyh · 2026-04-01T15:43:36Z

Congatulations on queueing! Is all the prerequisite CDC driver stuff already in, so now if I take the latest vector store it will just work with the Alternator-side patches, without changing any config files?

QuerthDP · 2026-04-01T15:45:16Z

Congatulations on queueing! Is all the prerequisite CDC driver stuff already in, so now if I take the latest vector store it will just work with the Alternator-side patches, without changing any config files?

Yes, you can use the Vector Store master from now on

This series adds support for vector search in Alternator based on the existing implementation in CQL. The series adds APIs for `CreateTable` and `UpdateTable` to add or remove vector indexes to Alternator tables, `DescribeTable` to list them and check the indexing status, and `Query` to perform a vector search - which contacts the vector store for the actual ANN (approximate nearest neighbor) search. Correct functionality of these features depend on some features of the the vector store, that were already done (see scylladb/vector-store#394). This initial implementation is fully functional, and can already be useful, but we do not yet support all the features we hope to eventually support. Here are things that we have **not** done yet, and plan to do later in follow-up pull requests: 1. Support a new optimized vector type ("V") - in addition to the "list of numbers" type supported in this version. 2. Allow choosing a different similarity function when creating an index, by SimilarityFunction in VectorIndex definition. 3. Allow choosing quantization (f32/f16/bf16/i8/b1) to ask the vector index to compress stored vectors. 4. Support oversampling and rescoring, defined per-index and per-query. 5. Support HNSW tuning parameters — maximum_node_connections, construction_beam_width, search_beam_width. 6. Support pre-filtering over key columns, which are available at the vector store, by sending the filter to the vector store (translated from DynamoDB filter syntax to the vector's store's filter syntax). A decision still need to be made if this will use KeyConditionExpression or FilterExpression. This version supports only post-filtering (with `FilterExpression`). 7. Support projecting non-key attributes into the index (Projection=INCLUDE and Projection=ALL), and then 1. pre-filtering using these attributes, and 2. efficiently return these attributes (using Select=ALL_PROJECTED_ATTRIBUTES, which today returns just the key columns). 8. Optimize the performance of `Query`, which today is inefficient for Select=ALL_ATTRIBUTES because it serially retrieves the matching items one at a time. 9. Returning the similarity scores with the items (the design proposes ReturnVectorSearchSimilarity). 10. Add more vector-search-specific metrics, beyond the metric we already have counting Query requests. For example separate latency and request-count metrics for vector-search Queries (distinct from GSI/LSI queries), and a metric accumulating the total Limit (K) across all vector search queries. 11. Consider how (and if at all) we want to run the tests in test/alternator/test_vector.py that need the vector store in the CI. Currently they are skipped in CI and only run manually (with `test/alternator/run --vs test_vector`). 12. UpdateTable 'Update' operation to modify index parameters. Only some can be modified, e.g., Oversampling. 13. Support for "local index" (separate index for each partition). 14. Make sure that vector search and Streams can be enabled concurrently on the same table - both need CDC but we need to verify that one doesn't confuse the other or disables options that the other needs. We can only do this after we have Alternator Streams running on tablets (since vector store requires tablets). Testing the new Alternator vector search end-to-end requires running both Scylla and the vector store together. We will have such end-to-end tests in the vector store repository (see scylladb/vector-store#392), but we also add in this pull request many end-to-end tests written in Python, that can be run with the command "test/alternator/run --vs test_vector.py". The "--vs" option tells the run script to run both Scylla and the vector store (currently assumed to be in `.../vector-store/target/release/vector-store`). About 65% of the tests in this pull request check supported syntax and error paths so can run without the vector store, while about 35% of the tests do perform actual Query operations and require the vector store to be running. Currently, the tests that do require the vector store will not get run by CI, but can be easily re-run manually with `test/alternator/run --vs test_vector.py`. In total, this series includes 78 functional tests in 2200 lines of Python code. This series also includes documentation for the new Alternator feature and the new APIs introduced. You can see a more detailed design document here: https://docs.google.com/document/d/1cxLI7n-AgV5hhH1DTyU_Es8_f-t8Acql-1f58eQjZLY/edit Two patches in this series split the huge alternator/executor.cc, after this series continued to grow it and it reached a whoppng 7,000 lines. These patches are just reorganization of code, no functional changes. But it's time that we finally do this (Refs #5783), we can't just continue to grow executor.cc with no end... Closes #29046 * github.com:scylladb/scylladb: test/alternator: add option to "run" script to run with vector search alternator: document vector search test/alternator: fix retries in new_dynamodb_session test/alternator: test for allowed characters in attribute names test/alternator: tests for vector index support alternator, vector: add validation of non-finite numbers in Query alternator: Query: improve error message when VectorSearch is missing alternator: add per-table metrics for vector query alternator: clean up duplicated code alternator: fix default Select of Query alternator: split executor.cc even more alternator: split alternator/executor.cc alternator: validate vector index attribute values on write alternator: DescribeTable for vector index: add IndexStatus and Backfilling alternator: implement Query with a vector index alternator: fix bug in describe_multi_item() alternator: prevent adding GSI conflicting with a vector index alternator: implement UpdateTable with a vector index alternator: implement DescribeTable with a vector index alternator: implement CreateTable with a vector index alternator: reject empty attribute names cdc: fix on_pre_create_column_families to create CDC log for vector search

github-actions Bot added P2 ai-assisted area/vector labels Mar 18, 2026

m-szymon mentioned this pull request Mar 18, 2026

validator: Alternator API test #392

Closed

nyh mentioned this pull request Mar 18, 2026

Alternator: Add vector search support scylladb/scylladb#29046

Merged

QuerthDP force-pushed the support-alternator-in-vs branch from fa37295 to fbc16e5 Compare March 19, 2026 13:28

QuerthDP force-pushed the support-alternator-in-vs branch from 7fe6ada to 2922db5 Compare March 20, 2026 11:53

QuerthDP force-pushed the support-alternator-in-vs branch from 2922db5 to aedfa96 Compare March 23, 2026 09:53

QuerthDP force-pushed the support-alternator-in-vs branch from aedfa96 to 108b3ac Compare March 23, 2026 11:30

QuerthDP force-pushed the support-alternator-in-vs branch from 108b3ac to 7cfbd87 Compare March 23, 2026 11:53

QuerthDP force-pushed the support-alternator-in-vs branch from 7cfbd87 to de480d5 Compare March 23, 2026 12:57

QuerthDP force-pushed the support-alternator-in-vs branch from de480d5 to abc12c3 Compare March 23, 2026 13:54

QuerthDP force-pushed the support-alternator-in-vs branch from abc12c3 to 39bbd63 Compare March 23, 2026 15:49

QuerthDP requested review from Copilot and removed request for Copilot March 23, 2026 15:52

QuerthDP force-pushed the support-alternator-in-vs branch from c51bdd0 to 968ca95 Compare April 1, 2026 11:43

ewienik requested changes Apr 1, 2026

View reviewed changes

Comment thread crates/vector-store/src/db_index_backend.rs Outdated

Comment thread crates/vector-store/src/vector.rs Outdated

Comment thread crates/vector-store/src/db_index_backend.rs Outdated

QuerthDP added 2 commits April 1, 2026 17:08

vector-store: bump scylla-cdc driver to 0.6.3

10984ab

Bump scylla-cdc from 0.6.1 to 0.6.3 to pick up fixes on CQL identifiers quoting and Alternator's TTL which is necessary for Alternator with Vector Search integration.

QuerthDP force-pushed the support-alternator-in-vs branch from 968ca95 to 0521d4d Compare April 1, 2026 15:17

QuerthDP added 3 commits April 1, 2026 17:22

QuerthDP force-pushed the support-alternator-in-vs branch from 0521d4d to 73bde40 Compare April 1, 2026 15:22

QuerthDP requested a review from ewienik April 1, 2026 15:23

ewienik approved these changes Apr 1, 2026

View reviewed changes

ewienik enabled auto-merge April 1, 2026 15:31

ewienik added this pull request to the merge queue Apr 1, 2026

Merged via the queue into scylladb:master with commit 599165c Apr 1, 2026
24 checks passed

QuerthDP deleted the support-alternator-in-vs branch July 16, 2026 07:45

Uh oh!

Conversation

QuerthDP commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Uh oh!

QuerthDP commented Mar 19, 2026

Uh oh!

QuerthDP commented Mar 19, 2026

Uh oh!

QuerthDP commented Mar 20, 2026

Uh oh!

QuerthDP commented Mar 20, 2026

Uh oh!

QuerthDP commented Mar 20, 2026

Uh oh!

QuerthDP commented Mar 23, 2026

Uh oh!

QuerthDP commented Mar 23, 2026

Uh oh!

QuerthDP commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

m-szymon commented Mar 23, 2026

Uh oh!

QuerthDP commented Mar 23, 2026

Uh oh!

nyh commented Mar 23, 2026

Uh oh!

QuerthDP commented Mar 23, 2026

Uh oh!

QuerthDP commented Mar 23, 2026

Uh oh!

QuerthDP commented Mar 23, 2026

Uh oh!

QuerthDP commented Apr 1, 2026

Uh oh!

QuerthDP commented Apr 1, 2026

Uh oh!

ewienik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

QuerthDP commented Apr 1, 2026

Uh oh!

QuerthDP commented Apr 1, 2026

Uh oh!

nyh commented Apr 1, 2026

Uh oh!

Uh oh!

QuerthDP commented Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

QuerthDP commented Mar 18, 2026 •

edited

Loading

QuerthDP commented Mar 23, 2026 •

edited

Loading