Skip to content

[Feature] Add scalar quantization support for vector extension#28

Open
rahul-iyer wants to merge 3 commits into
LadybugDB:mainfrom
rahul-iyer:quantized-rerank-cache
Open

[Feature] Add scalar quantization support for vector extension#28
rahul-iyer wants to merge 3 commits into
LadybugDB:mainfrom
rahul-iyer:quantized-rerank-cache

Conversation

@rahul-iyer

@rahul-iyer rahul-iyer commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Adds scalar quantization support for HNSW vector indexes behind the quantization option.

This PR supports:

  • quantization := 'sq8'
  • quantization := 'sq16'
  • quantized in-memory distance computation for ANN traversal
  • optional full-precision reranking via use_full_precision_rerank
  • SIMD-friendly SQ16 dot-product kernel with scalar fallback
  • quantized query-vector/cache alignment fixes
  • visibility checks for quantized on-disk embeddings
  • quantized cache invalidation on index mutation

Dot product with quantization is rejected for now because the quantized metric path currently supports
L2/L2SQ/Cosine-style scoring.

Behavior

Without quantization, HNSW continues to use the existing full-precision float embedding path.

With quantization enabled:

  • index traversal uses quantized embeddings
  • query vectors are quantized once per query
  • cached quantized payloads store compact integer vectors plus scale/norm metadata
  • if use_full_precision_rerank := true, final candidates are rescored using full-precision embeddings
  • if use_full_precision_rerank := false, results are returned directly from the quantized metric path

Benchmark

Ran it on OpenAI 50k small

Results:

Variant Load Index Build Ingest Query Latency DB Size Bytes/Vec Query RSS Recall@10 Precision@10
base 5.862s 51.000s 56.862s 17.812 ms/query 606.5 MiB 12718.7 746.6 MiB 0.9960 0.9960
sq8 5.765s 13.111s 18.875s 16.366 ms/query 723.4 MiB 15170.2 327.1 MiB 0.9460 0.9460
sq16 5.421s 23.337s 28.758s 16.444 ms/query 908.0 MiB 19041.5 489.4 MiB 0.9950 0.9950

Speedup vs baseline:

Variant Load Index Build Ingest Query Latency Query RSS Reduction
sq8 1.02x 3.89x 3.01x 1.09x 2.28x lower
sq16 1.08x 2.19x 1.98x 1.08x 1.53x lower

Storage overhead vs baseline:

Variant DB Size Bytes/Vec
sq8 1.19x 1.19x
sq16 1.50x 1.50x

Correctness Notes

This PR also fixes several issues found during testing:

  • quantized query buffer sizing/alignment
  • SQ8/SQ16 cached metadata layout
  • traversal using the quantized metric function instead of the full-precision metric
  • stale quantized cache after index mutation
  • visibility checks for quantized on-disk embeddings

Quantized cache invalidation is currently coarse-grained: mutations invalidate the full quantized cache.
A future improvement should make this transaction-aware and row/version based.

@rahul-iyer rahul-iyer changed the title [DRAFT][Feature] Add scalar quantization support for vector extension [Feature] Add scalar quantization support for vector extension Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant