Add RFC for Vector Search

[gggrace14]

This RFC proposes the syntax and SPI to natively support Approximate Nearest Neighbor (ANN) vector search within Presto.

[NivinCS]

Thanks, @gggrace14 , for the write-up

The proposal currently does not describe the lifecycle management of vector indexes with respect to underlying table mutations. In a lakehouse environment such as Iceberg, operations like appends, partition backfills, deletes, file rewrites (compaction), and snapshot expiration can make the index stale or physically invalid, especially if the index stores file-level row references.

Could we clarify how index validity is expected to be maintained across such operations? Specifically:

1. How is index staleness detected?
2. Are indexes expected to be snapshot-bound?
3. How should index cleanup or rebuild be handled after file rewrites or snapshot expiry?
4. Is automatic invalidation or maintenance of index metadata in scope?

This would help ensure correctness and avoid cases where ANN search may return outdated or invalid results.

[NivinCS]

After creating the vector index metadata without building it for any partitions, the expected query behavior is not entirely clear. For example, if a user runs:

SELECT *
FROM candidates_table
ORDER BY embedding <-> query_vector
LIMIT 10;

when the index is defined but not yet populated, should the engine fall back to a full scan, ignore the index, emit a warning, or fail the query? It would be helpful to clarify the intended behavior in this scenario to ensure predictable query execution.

[NivinCS]

It would be beneficial to keep the PRIMARY KEY / ROW_ID specification optional rather than mandatory for vector index creation.

From the Iceberg V3 perspective, tables with Row Level Lineage enabled can provide a stable row-level identifier via $row_id. In such cases, the identifier column can be made optional, allowing the connector to automatically fall back to $row_id when a user-defined id is not specified.

This would avoid requiring users to explicitly provide a unique identifier for tables where lineage-based row identity is already available.

[NivinCS]

It may be beneficial to make the partition specification in UPDATE VECTOR INDEX (e.g., FOR PARTITION (...)) optional rather than mandatory.

Instead of requiring users to manually specify partition ranges for index updates, the connector could automatically detect changed partitions and perform incremental index refresh, similar to materialized view incremental refresh.

This would reduce the need for manual orchestration and improve usability for incremental index maintenance.
cc : @tdcmeehan

[NivinCS]

It will be beneficial to include a reusable toolkit/library as part of the proposal to handle internal query rewrites for vector search operations.

In many cases, executing ANN search would otherwise require users to explicitly join the base table with the corresponding vector index using a row identifier (e.g., user-defined id or $row_id). This can expose index implementation details and make query syntax more complex, requiring users to understand how embeddings and index data are related internally.

Providing a reusable component that transparently rewrites vector search predicates (e.g., ANN_SEARCH(...)) into the appropriate join between the base table and vector index would enable a simpler and more seamless user experience. This would allow users to express ANN queries without manually specifying join logic, while the underlying connector leverages the index automatically.

Such a library could be integrated by connectors (e.g., Iceberg) to perform these rewrites behind the scenes, avoiding duplication of rewrite logic across connectors and keeping index implementation details abstracted from end users.
cc : @tdcmeehan

[aditi-pandit]

Thanks @gggrace14 for this writeup. Overall I'm in favor of this proposal.

I'm also curious since FAISS functions were recently exposed in C++.... Are you planning to do any follow up work with the library ?

[aditi-pandit]

Does the optimizer know about the Index as such ? The use of the index should be abstracted by VECTOR_SEARCH function, no ?

[aditi-pandit]

Are we missing the SPI for these ? Can you elaborate ?

[aditi-pandit]

Have you considered implementing VECTOR_SEARCH with a TABLE function ? It is easier to express the input tables, partitioning and parameters with it.

[aditi-pandit]

Can you give more details about VectorSearchNode and VectorSearchOptions. Will VectorSearchNode be a PlanNode ?

[bibith4]

It would be helpful to clarify the behavior when base table partitions are dropped, since index partitions are mapped to them. Should corresponding index partitions be automatically cleaned up or require manual action?

Additionally, it may be beneficial to define lifecycle DDL such as DROP VECTOR INDEX for operational usability.

[bibith4]

It would be helpful to clarify query behavior during index build or refresh. For example, can queries read partially built partitions?

[Dilli-Babu-Godari]

what do the parameters nlist and nb represent?

1. How do these parameters affect indexing performance, recall, and query latency?
2. What guidelines should users follow to choose optimal values for different dataset sizes and vector dimensions?
3. Is there any automatic tuning mechanism available, or are users expected to benchmark and manually determine the best configuration?

[gggrace14]

Thanks, @gggrace14 , for the write-up

The proposal currently does not describe the lifecycle management of vector indexes with respect to underlying table mutations. In a lakehouse environment such as Iceberg, operations like appends, partition backfills, deletes, file rewrites (compaction), and snapshot expiration can make the index stale or physically invalid, especially if the index stores file-level row references.

Could we clarify how index validity is expected to be maintained across such operations? Specifically:

1. How is index staleness detected?
2. Are indexes expected to be snapshot-bound?
3. How should index cleanup or rebuild be handled after file rewrites or snapshot expiry?
4. Is automatic invalidation or maintenance of index metadata in scope?

This would help ensure correctness and avoid cases where ANN search may return outdated or invalid results.

Hi @NivinCS , in general it is designed to define the minimum required parts to be SPI and leave individual connectors with the max flexibility to define its own behavior.

Specifically with respect to how index validity is maintained, we could leave it to individual ConnectorPlanOptimizer to detect if the index partition is stale and return reasonable result when the ConnectorPlanOptimizer plans/rewrites the vector search query.

For SPI, thinking through your questions, I think what could be added here is a getVectorSearchIndexStatus() method in the SPI class MetadataManager and ConnectorMetadata, similar to how materialized view detects staleness. Let me know if there are something else in your mind that need to be added to SPI.

I could describe below what we will put into the HiveConnector to maintain the index partition validity at Meta, by answering your above questions. Meta has an internal HiveConnector extension. I think you can implement the Iceberg behaviors in the counterpart classes of IcebergConnector. Again I think these should belong to the connectors but not SPI.

1. How is index staleness detected?
By looking at the metadata of index partition on HiveMetastore. As index partition is just a Hive table partition, if the index partition is stale, we'll drop the partition from the HiveMetastore. We can also put a dirty mark on the metadata of the index partition.

We'll have a pub-sub service that subscribes to the metadata of the candidate table (base table). If the corresponding base table partition is updated/overwritten/deleted, it will trigger the dropping of the index partition.

Then in the search path, we have a VectorSearchRewritePlanOptimizer that implements a ConnectorPlanOptimizer. In the VectorSearchRewritePlanOptimizer, we compare the available partitions of the index vs the base table in the query range. If an index partition is available, we'll use it to get the kNN of the partition. Otherwise, we will build the index on the fly. Another option for an unavailable index partition is to fall back to the exact search. And then we'll aggregate across all partitions in the query range and compute the kNN from all kNNs from each partion.

2. Are indexes expected to be snapshot-bound?
I might not have enough context for Iceberg. However, I think you can decide the reasonable behavior in IcebergConnector.

3. How should index cleanup or rebuild be handled after file rewrites or snapshot expiry?
As mentioned above, we will have a pub-sub service. Once it detects the update or deletion event of the base table from HiveMetastore, it will trigger at least two actions in sequence, 1) drop the corresponding index partition; 2) call UPDATE VECTOR INDEX vector_index WHERE <filter>, where corresponds to the dropped partition.

4. Is automatic invalidation or maintenance of index metadata in scope?
No, the automatic invalidation is out of the scope of Presto as well as this RFC. The index validity is stored as the index metadata. And when and how to update the index metadata is out of the scope of Presto. Presto only provides the SQL as interface to create and update the index.