Every RAG/agent stack needs a vector index, and the choice is a trade-off nobody measures cleanly: speed vs. recall vs. memory. This benchmark runs the same workload (identical vectors, queries, and exact ground truth) across FAISS (flat / IVF / IVFPQ / HNSW) and hnswlib, and reports recall@k, latency p50/p95, QPS, build time, and index memory, with a Pareto chart. Self-contained, reproducible, runs on a laptop.
Recall is measured against exact brute-force ground truth, not another approximate index, so the numbers mean what they say. The adapters share one interface, so a server-backed store (Qdrant, Milvus, Weaviate, pgvector) drops in by implementing the same three methods.
Interactive/exportable version: docs/assets/architecture.html.
$ vbench
corpus=50,000 · dim=96 · queries=1,000 · k=10
------------------------------------------------------------------------------
index recall build_s p50_ms p95_ms QPS mem_MB
faiss_ivf 1.000 0.14 0.041 0.055 23466.6 19.7
faiss_flat(exact) 1.000 0.00 0.370 0.436 2354.6 19.2
faiss_hnsw 0.996 1.11 0.078 0.103 12733.4 32.8
hnswlib 0.995 1.80 0.145 0.177 7170.4 33.0
faiss_ivfpq 0.726 6.85 0.185 0.231 5279.6 3.0
Read it as a frontier, not a winner:
faiss_ivf, recall 1.0 and ~10× the throughput of exact flat: the all-rounder here.faiss_flat, exact (recall 1.0) but ~10× slower; fine for small corpora, doesn't scale.faiss_hnsw/hnswlib, graph indexes: ~0.995 recall at high QPS, but biggest memory.faiss_ivfpq, product quantization: 3 MB vs 20 to 33 MB (~10× smaller), paying recall (0.73) for it, the pick when memory, not recall, is the constraint.
The right index depends on your recall floor, latency budget, and memory, which is
exactly the three-way trade-off the table and chart make visible. Sweep each index's knobs
(nprobe, efSearch, M) to trace its own curve.
Uses the conda
personalenv (per environment conventions, neverbase).
PY=~/miniconda3/envs/personal/bin/python
$PY -m pip install -e ".[dev]"
vbench # full benchmark + Pareto chart -> assets/pareto.png
vbench --n-base 20000 --n-query 500 # smaller/faster sweepEach run writes reports/benchmark.json and assets/pareto.png.
| Metric | Method |
|---|---|
| recall@k | overlap of returned top-k vs. exact brute-force ground truth |
| latency p50 / p95 | per-query timing, one query at a time (the serving path, not batched) |
| QPS | single-query throughput |
| build time | wall-clock to index the corpus |
| index memory | serialized index size (FAISS/hnswlib) or raw vector bytes (flat) |
| Adapter | Type | Knobs |
|---|---|---|
faiss_flat(exact) |
brute-force IP | , (recall 1.0 reference) |
faiss_ivf |
inverted file | nlist, nprobe |
faiss_ivfpq |
IVF + product quantization (compressed) | nlist, m, nbits, nprobe |
faiss_hnsw |
graph (HNSW) | M, efConstruction, efSearch |
hnswlib |
graph (HNSW) | M, ef_construction, ef |
Sweep the knobs to trace each index's own recall/QPS curve, the standard ann-benchmarks
methodology, here on embeddable libraries so it runs without standing up servers. Adding a
server-backed store (Qdrant, Milvus, Weaviate, pgvector) is just another adapter implementing
build / query / memory_mb.
vector-db-benchmark/
├── src/vbench/
│ ├── data.py synthetic clustered embeddings + exact ground truth
│ ├── adapters.py FAISS flat/IVF/IVFPQ/HNSW · hnswlib (one interface)
│ ├── metrics.py recall@k vs ground truth
│ ├── benchmark.py build/recall/latency/QPS/memory + Pareto chart (CLI: vbench)
│ └── config.py corpus size, dim, k
├── tests/ data · metric · adapter-recall · run-ordering — 8 cases
└── pyproject.toml · Dockerfile · Makefile · .github/workflows/ci.yml
Built a reproducible vector-index benchmark measuring recall@k (vs exact ground truth), latency p50/p95, QPS, build time, and memory across FAISS (flat/IVF/IVFPQ/HNSW) and hnswlib on identical data; produced the recall/throughput Pareto frontier and a pluggable adapter interface for server-backed stores, a vendor-neutral selection guide.
MIT (LICENSE).