Skip to content

[DO NOT MERGE] docs: add DiskANN disk-backed index engine SEP#499

Open
swasik wants to merge 1 commit into
scylladb:masterfrom
swasik:VECTOR-702-diskann-sep
Open

[DO NOT MERGE] docs: add DiskANN disk-backed index engine SEP#499
swasik wants to merge 1 commit into
scylladb:masterfrom
swasik:VECTOR-702-diskann-sep

Conversation

@swasik

@swasik swasik commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

This PR is used just to make the initial review of SEP. To be moved to Confluence later.

What

Adds a ScyllaDB Enhancement Proposal (SEP) for a DiskANN (SSD-backed) index
engine
in Vector Store, offered as an additional, selectable engine alongside
the existing in-memory USearch engine.

Docs-only change:

  • docs/dev/diskann/sep-diskann.md — the proposal.

Why

Vector Store builds every ANN index fully in RAM (USearch/HNSW), so RAM is the
scaling ceiling and the primary cost driver. DiskANN moves the bulk of the index
to local NVMe SSD (keeping a PQ-compressed representation + working set in RAM),
targeting:

  • Large workloads (100M–1B+): materially lower $/indexed-vector and
    $/QPS at acceptable recall, trading a modest latency increase for a large
    RAM-cost reduction.
  • Cost-sensitive smaller workloads: a lower cost floor — run on cheaper
    storage-optimized nodes when higher latency is acceptable.

USearch remains the default for latency-critical, RAM-resident workloads.

Design highlights

  • New DiskannIndexFactory behind the existing IndexFactory seam (the same
    pluggable-engine plumbing USearch uses); ingestion (CDC), routing, and the
    HTTP/CQL surface are unchanged.
  • Integrates DiskANN3 (Microsoft Research, Rust, MIT, v0.54.0) in
    disk-provider mode; node-local NVMe to avoid a network hop per beam-search
    step.
  • Real-time updates via IP-DiskANN (in-place, natively supported by
    DiskANN3) — no periodic merges or full rebuilds.
  • Engine selected per node (memory-optimized vs storage-optimized), surfaced
    in Cloud as a single latency-optimized vs cost-optimized deployment switch.
  • Durability boundary is the SSD volume; ScyllaDB remains the authoritative
    source, so index loss = rebuild-from-CDC.

Scope

  • In scope of the SEP (later milestone): filtered ANN parity with USearch.
  • Out of scope: replacing USearch, non-disk DiskANN3 providers, S3/remote
    object storage on the hot path, DistributedANN, GPU, Optane/PMem.

Notes for reviewers

  • This is a Draft SEP; several [Add link] placeholders and an Open
    Questions list (PQ defaults, cache-vs-FTS RAM split, p99 target validation,
    crash-consistency strategy) are intentionally left for delivery-time
    resolution.
  • Benchmark plan compares USearch (f32, i8, b1) vs DiskANN at 100M and 1B,
    with cost ($/indexed-vector, $/QPS) as a first-class result.

@swasik
swasik force-pushed the VECTOR-702-diskann-sep branch from e63f0dc to 85899f5 Compare July 2, 2026 10:01
@swasik
swasik requested a review from Copilot July 2, 2026 10:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a draft ScyllaDB Enhancement Proposal (SEP) describing a DiskANN (SSD-backed) index engine option for Vector Store, positioned as a capacity/cost-optimized alternative to the existing in-memory USearch engine.

Changes:

  • Introduces a new draft SEP detailing product motivation, goals, and competitive positioning for a DiskANN-based engine.
  • Documents a proposed high-level integration plan (factory selection, actor wiring, config/metrics/memory accounting).
  • Outlines QA/benchmark strategy, delivery milestones, and operational considerations for an SSD-backed tier.

Comment thread docs/dev/diskann/sep-diskann.md Outdated
Comment thread docs/dev/diskann/sep-diskann.md Outdated
Comment thread docs/dev/diskann/sep-diskann.md Outdated
Comment thread docs/dev/diskann/sep-diskann.md
Comment thread docs/dev/diskann/sep-diskann.md Outdated
Comment thread docs/dev/diskann/sep-diskann.md Outdated
Comment thread docs/dev/diskann/sep-diskann.md
Comment thread docs/dev/diskann/sep-diskann.md Outdated
@swasik
swasik force-pushed the VECTOR-702-diskann-sep branch from 85899f5 to 9728040 Compare July 2, 2026 11:43
@swasik
swasik requested review from QuerthDP, ewienik and knowack1 July 2, 2026 12:15
@swasik
swasik force-pushed the VECTOR-702-diskann-sep branch from 9728040 to b133aa0 Compare July 2, 2026 12:15
@swasik swasik changed the title docs: add DiskANN disk-backed index engine SEP [DO NOT MERGE] docs: add DiskANN disk-backed index engine SEP Jul 2, 2026
@swasik
swasik force-pushed the VECTOR-702-diskann-sep branch from b133aa0 to 1c0640b Compare July 2, 2026 14:59
Comment thread docs/dev/diskann/sep-diskann.md Outdated
Comment thread docs/dev/diskann/sep-diskann.md Outdated
Comment thread docs/dev/diskann/sep-diskann.md Outdated
Comment thread docs/dev/diskann/sep-diskann.md Outdated
Comment thread docs/dev/diskann/sep-diskann.md Outdated
Comment thread docs/dev/diskann/sep-diskann.md Outdated
Comment thread docs/dev/diskann/sep-diskann.md Outdated
Add a ScyllaDB Enhancement Proposal for integrating DiskANN (DiskANN3,
SSD-backed Vamana) as an additional, per-node-selectable Vector Store index
engine alongside the in-memory USearch engine. Covers motivation, internal
USearch-vs-DiskANN competitive analysis, 50k/100M/1B benchmark plan,
restart/durability semantics, delivery plan, and resolved design decisions.
@swasik
swasik force-pushed the VECTOR-702-diskann-sep branch from 1c0640b to e5199b8 Compare July 2, 2026 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants