feat(benchmarker): add bm25-benchmark command with tombstone testing by jfrancoa · Pull Request #104 · weaviate/weaviate-benchmarking

jfrancoa · 2026-07-02T07:45:59Z

Motivation

We can benchmark ANN query performance today, but we have no equivalent harness for BM25 keyword search — and in particular no way to reproduce and quantify the latency degradation that inverted-index tombstones cause. When documents are updated or deleted, their old docIDs remain in the posting lists as tombstones and must be skipped at query time until compaction reclaims them. Under update-heavy workloads this can meaningfully hurt BM25 latency/QPS, and we needed a repeatable way to measure it against real corpora.

This PR adds a bm25-benchmark command that imports a BEIR-format text corpus into a vectorless collection and measures BM25 latency/QPS through the existing worker-pool harness, with an optional tombstone-generation mode to measure performance under inverted-index churn, and an optional qrels-based quality regression gate (--measureQuality).

Approach

The guiding decision was to reuse the existing import and measurement machinery rather than fork it:

BeirDataset implements the existing Dataset interface, so the same 8-worker loadTrainData pipeline imports the corpus unchanged. Vector-specific interface methods are no-ops.
BM25 queries reuse the shared benchmark(...) worker pool and the ANN-compatible JSON result shape, so runs plug straight into existing reporting.
Tombstone density is measured phased: a 0-tombstone baseline, then delete/reinsert a fraction of docs and re-measure per iteration (or sustain churn in the background for a concurrent-churn variant).
Multi-tenancy is supported: each tenant gets a full corpus copy, and tombstone churn is tenant-aware (per-tenant Config copies, no shared mutation).

Quality gate (second commit): by default the run is a pure performance test (BM25 top-k is exact, so there is no ANN-style self-recall). With --measureQuality, the BEIR qrels become the external ground truth: the command computes NDCG@10 / Recall@100 vs qrels per phase (pytrec_eval conventions), so a change that degrades what BM25 returns — a tombstone or WAND regression — shows up as a metric drop. Rows are labeled benchmarkType: bm25-qrels so they are never conflated with ANN metrics downstream. Because a lagging multi-node cluster would otherwise be misread as a quality collapse, quality mode waits for the index to settle (judged docs findable via a competition-free probe, then the metric stops climbing) before measuring, and a retrievability probe (tombstoneRetrievability) checks that reinserted docs remain findable after churn.

The one invasive change is in ann_benchmark.go: Batch/writeChunk were generalized to carry optional text properties so the BM25 path could share them. This was done as a behavior-preserving refactor of the existing ANN write path (the loop now iterates over max(len(vectors), len(text)) and only touches vectors when present).

Hybrid (BM25 + vector) was intentionally left out but the seam is documented in bm25_query.go.

Key areas for review

benchmarker/cmd/ann_benchmark.go writeChunk — the riskiest change, the only edit to the shared ANN import path. Verify the pure-ANN branches (named vector, multivector, filter-only) are unchanged; note the inner multivector loop variable was renamed i→j to avoid shadowing the outer object index.
benchmarker/cmd/benchmark_run.go — the shared query path gained a graded-metrics branch (QueryWithNeighbors.Relevance), a quality-pass soft-fail, and a mode-gated short-result warning; the ANN branches are untouched (A/B-verified against main: identical recall/NDCG on mnist-784).
benchmarker/cmd/bm25_quality.go settleMetric — the settle logic (per-metric maxima, patience, epsilon); unit-tested with scripted poll sequences.
benchmarker/cmd/bm25_benchmark.go generateTombstones / tombstoneOnce — concurrency safety: these run alongside the query phase in --tombstoneConcurrent mode. cfg is taken by value so tenant assignment stays local; churn RPCs take a cancellable context and the goroutine join is bounded.
benchmarker/cmd/bm25_benchmark.go batchDeleteDocIDRange — the delete loop (server caps deletes at ~10k/call); terminates on Successful == 0 and propagates errors.
benchmarker/cmd/beir_dataset.go StreamTrainData — startOffset/maxRecords + absolute Offset are what make reinsert produce stable UUIDs/docIds, which makes "update" a true update rather than a duplicate insert.

Risks and mitigations

Regression in the existing ann-benchmark import path: the writeChunk rewrite is behavior-preserving; validated by inspection, the existing suite, and a live A/B run against main (identical recall/NDCG).
Race between background churn and queries (--tombstoneConcurrent): mitigated by passing per-tenant Config copies (by value) into the churn goroutine; the goroutine is stopped via stop/done channels with a cancellable context for in-flight RPCs and a bounded join, so a hung server cannot deadlock the run or discard results.
UUID/docId drift across reinsert passes would turn "update" into duplicate inserts: prevented by deriving UUIDs deterministically from the absolute corpus offset, so reinserting reuses identical UUIDs and genuinely overwrites.
False quality regressions from index lag: quality mode gates on findability + metric stability before accepting a measurement; an all-failed quality pass omits recall/ndcg (and the bm25-qrels label) instead of reporting 0.0, and rows carry qualityFailedQueries so incomplete passes are visible downstream.
Oversized corpus lines (full passages exceed bufio's 64KB default): scanner buffer raised to 16MB.

Testing

Unit tests (no Weaviate required), passing (go test ./cmd/..., incl. -race):

bm25_query_test.go — the gRPC query builder: operators, filters, tenants, defaults, and the Metadata.Uuid == true invariant.
beir_dataset_test.go / beir_qrels_test.go — corpus/query streaming, batch offsets, the range re-stream used by reinsert, qrels parsing (header detection, strict columns, dup guards).
benchmark_quality_test.go — calculateGradedNDCG / recallVsRelevant vs hand-computed pytrec_eval-convention values.
bm25_quality_test.go — settleMetric with scripted poll sequences (per-metric maxima).
benchmark_run_test.go — analyze on an all-failed pass (empty times).

Manually validated end-to-end against live Weaviate clusters (single-node and 3-node k8s) on NFCorpus, SciFact and FiQA: import → baseline → tombstone iterations → results JSON; quality values match published BEIR BM25 references (fiqa NDCG@10 = 0.236); query-only mode without --corpus; retrievability 1.0 after judged churn.

A self-review pass (2 HIGH / 6 MEDIUM findings: query-only crash, divide-by-zero on all-failed quality pass, missing Bearer auth on raw-gRPC helpers, misleading tombstoneRatio under judged churn, concurrent-churn shutdown deadlock, fatal churn errors, silent quality denominators, README drift) was addressed in the fourth commit.

🤖 Generated with Claude Code

orca-security-eu

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0 0 0 0	View in Orca
Passed	SAST	0 4 0 0	View in Orca
Passed	Secrets	0 0 0 0	View in Orca
Passed	Vulnerabilities	0 0 0 0	View in Orca

🛡️ The following SAST misconfigurations have been detected

NAME	FILE
Avoid Using `math/rand` for Cryptographic Operations in Go	...md/bm25_benchmark.go	View in code
Directory Permissions Are Set to Overly Permissive Values	...md/bm25_benchmark.go	View in code
Overly Permissive File Permissions Set in Application	...md/bm25_benchmark.go	View in code
Overly Permissive File Permissions Set in Application	...md/bm25_benchmark.go	View in code

Copilot

Pull request overview

This PR adds a new bm25-benchmark command to the benchmarker that imports a BEIR-format text corpus into a vectorless Weaviate collection and measures BM25 latency/QPS, with an optional tombstone-generation mode to quantify inverted-index tombstone impact. It reuses the existing import pipeline and query worker-pool harness, with a small generalization of the shared ANN import writer to optionally attach text properties.

Changes:

Added a BM25 benchmarking command, query builder, and BEIR dataset loader (plus unit tests) to support vectorless BM25 performance runs.
Implemented phased tombstone generation (delete/update) including an optional concurrent-churn mode during the query window.
Generalized the shared Batch/writeChunk import path to support optional text/title/docId properties for BM25 imports.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
README.md	Documents the new `bm25-benchmark` command, BEIR dataset usage, and tombstone scenario.
benchmarker/README.md	Adds CLI help + examples for BM25 benchmarking and tombstone runs.
benchmarker/cmd/root.go	Registers the new `bm25-benchmark` command in the CLI.
benchmarker/cmd/config.go	Extends config with BM25/tombstone flags and adds BM25-specific validation.
benchmarker/cmd/bm25_query.go	Implements gRPC BM25 SearchRequest builder (operator options, tenant, filter).
benchmarker/cmd/bm25_query_test.go	Unit tests for BM25 gRPC query construction invariants and options.
benchmarker/cmd/bm25_benchmark.go	Implements BM25 import + benchmark phases, tombstone generation, and results output.
benchmarker/cmd/beir_dataset.go	Adds BEIR JSONL corpus/query loader implementing the existing Dataset interface.
benchmarker/cmd/beir_dataset_test.go	Fixture-based tests for corpus streaming offsets, filters, range restream, and queries.
benchmarker/cmd/ann_benchmark.go	Extends `Batch` and generalizes `writeChunk` to support optional text properties.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add a `bm25-benchmark` CLI command that imports a BEIR-format text corpus into a vectorless collection and benchmarks BM25 keyword-search latency/QPS through the existing worker-pool harness. Reuses the shared measurement, gRPC transport, and JSON output; recall/NDCG are skipped since BM25 top-k is exact. - BEIR corpus/query loader (beir_dataset.go) implementing the Dataset interface to reuse the import pipeline. - gRPC BM25 query builder (bm25_query.go), cloned from nearVectorQueryGrpc, with AND/OR operator and an optional category filter; Hybrid seam documented. - Vectorless schema with a searchable text/title inverted index, filterable docId/category, and configurable k1/b. - Tombstone generation: delete or delete+reinsert a fraction of documents to measure BM25 latency under inverted-index tombstone load, with phased baseline-vs-tombstoned measurement and an optional concurrent-churn mode. - Multi-tenant support (per-tenant import + tenant-aware, race-free churn). Batch/writeChunk in ann_benchmark.go are extended to carry text properties (behavior-preserving for the existing ann-benchmark path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…rels) Add an opt-in --measureQuality mode to bm25-benchmark that computes NDCG@10 / Recall@100 vs BEIR qrels per phase and fills the existing recall/ndcg output fields, so the weaviate-performance-tests comparison, thresholds and Grafana pipeline work unchanged. Like ANN recall, the ground truth (qrels) ships with the dataset and each run emits an absolute number; comparison happens downstream, no baseline is stored in the tool. - BEIR qrels loader + shared scanCorpus; query/doc identity mapping (docIndex = corpus ordinal = uuidFromInt), judged-only idToIndex. - Graded NDCG@k / Recall@k (linear gain, log2 discount) behind an additive, guarded Relevance branch in processQueueGrpc; the ANN path (calculateLinearNDCG vs neighbors) is byte-for-byte unchanged. - Deterministic quality pass separate from the throughput pass; cutoffs decoupled from --limit (--ndcgCutoff/--recallCutoff). - Tombstone coverage: judged-doc-biased churn (update mode) plus a competition-free retrievability probe (docId filter, Limit 1). - Distributed-cluster settle: wait for docs to be searchable and the metric to stop climbing before measuring, so index lag isn't read as a regression. - Rows labeled benchmarkType=bm25-qrels; use negative thresholds downstream. Verified end-to-end on a 3-node k8s Weaviate against BEIR scifact and fiqa: NDCG@10 matches published BM25 values and baseline ~= tombstoned on a correct build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

orca-security-eu

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0 0 0 0	View in Orca
Passed	SAST	0 6 0 0	View in Orca
Passed	Secrets	0 0 0 0	View in Orca
Passed	Vulnerabilities	0 0 0 0	View in Orca

🛡️ The following SAST misconfigurations have been detected

NAME	FILE
Avoid Using `math/rand` for Cryptographic Operations in Go	...md/bm25_benchmark.go	View in code
Directory Permissions Are Set to Overly Permissive Values	...md/bm25_benchmark.go	View in code
Overly Permissive File Permissions Set in Application	...md/bm25_benchmark.go	View in code
Overly Permissive File Permissions Set in Application	...md/bm25_benchmark.go	View in code
Insecure Transport: TLS/SSL Certificate Verification Disabled	.../cmd/bm25_quality.go	View in code
Missing MinVersion in TLS Configuration May Allow Insecure Connections	.../cmd/bm25_quality.go	View in code

The BM25 work gated the "got N results, expected limit" gRPC warning on len(query.Neighbors) > 0, which also silenced it for non-ANN vector commands (random-vectors, dataset) that legitimately surface it when a class returns fewer than k results. Gate on cfg.Mode != "bm25-benchmark" instead, so only BM25 (where short result sets are normal) suppresses it while every vector-search command keeps main's behavior. Found via A/B exploratory testing (main vs branch) against the local k8s cluster: ann-benchmark recall/ndcg were unchanged, but random-vectors on a short-result class no longer warned. ANN recall/ndcg and import verified behavior-preserving. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

jfrancoa · 2026-07-14T07:26:24Z

+	n := len(chunk.Vectors)
+	if n == 0 {
+		// Text-only (BM25) batches carry no vectors.
+		n = len(chunk.Text)
+	}


Intentionally out of scope for this PR: no producer creates batches with both vectors and text today (ANN batches carry only vectors, BEIR batches only text), so widening the loop bound would change the most sensitive shared function for a case that cannot occur. The hybrid PR that introduces mixed batches is the right place to generalize n (the seam is documented in bm25_query.go).

jfrancoa · 2026-07-14T07:26:25Z

 		if cfg.Filter {
-			nonRefProperties, err := structpb.NewStruct(map[string]interface{}{
-				"category": strconv.Itoa(chunk.Filters[i]),
-			})
+			props["category"] = strconv.Itoa(chunk.Filters[i])
+		}


Leaving as-is deliberately: both producers construct Filters aligned with the batch by design (idx % filterCount per emitted doc), so a length mismatch can only mean a producer bug — and a panic there is the preferable fail-fast. A silent bounds-guard would import objects without their filter property and quietly skew filtered-benchmark results, which is worse than crashing.

…edium) - query-only mode no longer crashes without --corpus: corpusDocs is only counted when a corpus path is set (validateBM25 explicitly permits the corpus-less --query invocation) - analyze() guards against an empty times slice (reachable when every query in a quality pass soft-fails) instead of dividing by zero; locked by TestAnalyzeEmptyTimes - reinsertDocs/tombstoneRetrievability attach the Authorization: Bearer metadata (HTTP_AUTH) like every sibling gRPC path, and dialGrpcConn uses WithBlock so its 30s timeout actually bounds connection setup - rows record the churned fraction actually applied: judged-doc churn replaces the docId<k range churn in quality mode, so tombstoneRatio now reflects |judged|/corpusDocs instead of --tombstonePercentage (shared usesJudgedChurn helper keeps branch and label in sync) - concurrent churn is cancellable: churn RPCs take a context that is cancelled at shutdown, and the join is bounded (2m) so a hung server can no longer deadlock the run before results are written - churn errors propagate instead of log.Fatalf (batchDeleteDocIDRange, deleteUuidSlice, reinsertDocs), so a transient blip on iteration N keeps the rows collected so far; ANN callers keep fatal behavior - quality-pass completeness is surfaced: rows carry qualityFailedQueries, a partially-failed pass warns, and an all-failed pass omits recall/ndcg and the bm25-qrels label instead of reporting 0.0 as a measurement - READMEs: resolve the "qrels are ignored" contradiction with --measureQuality and document the tombstoneRatio semantics Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

- deleteUuidSlice applies cfg.Offset when deriving UUIDs, matching writeChunk/deleteChunk, so --offset runs delete the objects they actually imported (ANN update path and BM25 judged churn) - validateBM25 rejects unsupported --searchType (only "bm25" until hybrid exists) and --bm25Operator values (or/and/empty), instead of silently ignoring typos or unimplemented modes - query-only rows label dataset by the queries file instead of the unhelpful "." that filepath.Base("") produced without --corpus - writeChunk no longer allocates a properties map per object for pure-ANN unfiltered batches (restores zero-alloc import hot path) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

jfrancoa mentioned this pull request Jul 2, 2026

feat(benchmarker): add bm25-benchmark command with tombstone testing #103

Closed

orca-security-eu Bot reviewed Jul 2, 2026

View reviewed changes

jfrancoa requested a review from Copilot July 2, 2026 07:58

Copilot started reviewing on behalf of jfrancoa July 2, 2026 07:59 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread benchmarker/cmd/ann_benchmark.go Outdated

Comment thread benchmarker/cmd/config.go

Comment thread benchmarker/cmd/bm25_benchmark.go Outdated

jfrancoa force-pushed the jose/bm25-benchmark branch from 59fe7a8 to 23bcb85 Compare July 2, 2026 09:18

orca-security-eu Bot reviewed Jul 2, 2026

View reviewed changes

jfrancoa requested a review from Copilot July 13, 2026 12:52

Copilot started reviewing on behalf of jfrancoa July 13, 2026 12:53 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

jfrancoa requested a review from Copilot July 13, 2026 15:07

Copilot started reviewing on behalf of jfrancoa July 13, 2026 15:08 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread benchmarker/cmd/ann_benchmark.go

Comment thread benchmarker/cmd/config.go

jfrancoa requested a review from Copilot July 14, 2026 07:26

Copilot started reviewing on behalf of jfrancoa July 14, 2026 07:27 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(benchmarker): add bm25-benchmark command with tombstone testing#104

feat(benchmarker): add bm25-benchmark command with tombstone testing#104
jfrancoa wants to merge 5 commits into
mainfrom
jose/bm25-benchmark

jfrancoa commented Jul 2, 2026 •

edited

Loading

Uh oh!

orca-security-eu Bot left a comment •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

orca-security-eu Bot left a comment •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

jfrancoa Jul 14, 2026

Uh oh!

jfrancoa Jul 14, 2026

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jfrancoa commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Approach

Key areas for review

Risks and mitigations

Testing

Uh oh!

orca-security-eu Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Orca Security Scan Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

orca-security-eu Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Orca Security Scan Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

jfrancoa Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

jfrancoa Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jfrancoa commented Jul 2, 2026 •

edited

Loading

orca-security-eu Bot left a comment •

edited

Loading

orca-security-eu Bot left a comment •

edited

Loading