feat(benchmarker): add bm25-benchmark command with tombstone testing

[jfrancoa]

Motivation

We can benchmark ANN query performance today, but we have no equivalent harness for BM25 keyword search — and in particular no way to reproduce and quantify the latency degradation that inverted-index tombstones cause. When documents are updated or deleted, their old docIDs remain in the posting lists as tombstones and must be skipped at query time until compaction reclaims them. Under update-heavy workloads this can meaningfully hurt BM25 latency/QPS, and we needed a repeatable way to measure it against real corpora.

This PR adds a bm25-benchmark command that imports a BEIR-format text corpus into a vectorless collection and measures BM25 latency/QPS through the existing worker-pool harness, with an optional tombstone-generation mode to measure performance under inverted-index churn.

Approach

The guiding decision was to reuse the existing import and measurement machinery rather than fork it:

• BeirDataset implements the existing Dataset interface, so the same 8-worker loadTrainData pipeline imports the corpus unchanged. Vector-specific interface methods are no-ops; qrels are ignored because this is a performance test, not a relevance eval. Recall/NDCG are skipped since BM25 top-k is exact.
• BM25 queries reuse the shared benchmark(...) worker pool and the ANN-compatible JSON result shape, so runs plug straight into existing reporting.
• Tombstone density is measured phased: a 0-tombstone baseline, then delete/reinsert a fraction of docs and re-measure per iteration (or sustain churn in the background for a concurrent-churn variant).
• Multi-tenancy is supported: each tenant gets a full corpus copy, and tombstone churn is tenant-aware (per-tenant Config copies, no shared mutation).

The one invasive change is in ann_benchmark.go: Batch/writeChunk were generalized to carry optional text properties so the BM25 path could share them. This was done as a behavior-preserving refactor of the existing ANN write path (the loop now iterates over max(len(vectors), len(text)) and only touches vectors when present).

Hybrid (BM25 + vector) was intentionally left out but the seam is documented in bm25_query.go.

Key areas for review

• benchmarker/cmd/ann_benchmark.go writeChunk — the riskiest change, the only edit to the shared ANN import path. Verify the pure-ANN branches (named vector, multivector, filter-only) are unchanged; note the inner multivector loop variable was renamed i→j to avoid shadowing the outer object index.
• benchmarker/cmd/bm25_query.go bm25QueryGrpc — Metadata.Uuid must stay true; processQueueGrpc decodes each result UUID back into an int and uuid.Parse("") would panic. There's a test guarding this.
• benchmarker/cmd/bm25_benchmark.go generateTombstones / tombstoneOnce — concurrency safety: these run alongside the query phase in --tombstoneConcurrent mode. cfg is taken by value so tenant assignment stays local and the shared config is never mutated.
• benchmarker/cmd/bm25_benchmark.go batchDeleteDocIDRange — the delete loop (server caps deletes at ~10k/call); verify termination on Successful == 0.
• benchmarker/cmd/beir_dataset.go StreamTrainData — startOffset/maxRecords + absolute Offset are what make reinsert produce stable UUIDs/docIds, which makes "update" a true update rather than a duplicate insert.

Risks and mitigations

• Regression in the existing ann-benchmark import path: the writeChunk rewrite is behavior-preserving; validated by inspection and the existing suite (the ANN unit tests stay green). Worth a careful read of that function.
• Race between background churn and queries (--tombstoneConcurrent): mitigated by passing per-tenant Config copies (by value) into the churn goroutine — no shared mutable state is touched during the query window; the goroutine is stopped and joined via stop/done channels before results are written. Validation also requires --queryDuration > 0 and --tombstoneMode update for this mode.
• UUID/docId drift across reinsert passes would turn "update" into duplicate inserts: prevented by deriving UUIDs deterministically from the absolute corpus offset, so reinserting docId range [0,k) reuses identical UUIDs and genuinely overwrites.
• Oversized corpus lines (full passages exceed bufio's 64KB default): scanner buffer raised to 16MB.

Testing

Unit tests (no Weaviate required), passing (go test ./cmd/..., incl. -race):

• bm25_query_test.go — the gRPC query builder: OR operator with minimumOrTokensMatch + multi-property scoring, AND operator, category filter + tenant wiring, defaults, and the Metadata.Uuid == true invariant.
• beir_dataset_test.go — the loader end-to-end on a fixture: streaming into batches, absolute batch offsets, filter-bucket assignment (idx % filterCount), the range re-stream used by tombstone "update" reinsert, and query parsing.

Manually validated end-to-end against a live Weaviate on NFCorpus (single-tenant and 3-tenant): import → baseline → tombstone iterations → results JSON, with latency/QPS degrading as tombstone density grows, and update-mode preserving corpus size.

🤖 Generated with Claude Code

[orca-security-eu]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0   0   0   0	View in Orca
Passed	SAST	0   4   0   0	View in Orca
Passed	Secrets	0   0   0   0	View in Orca
Passed	Vulnerabilities	0   0   0   0	View in Orca

🛡️ The following SAST misconfigurations have been detected

	NAME	FILE
	Avoid Using `math/rand` for Cryptographic Operations in Go	...md/bm25_benchmark.go	View in code
	Directory Permissions Are Set to Overly Permissive Values	...md/bm25_benchmark.go	View in code
	Overly Permissive File Permissions Set in Application	...md/bm25_benchmark.go	View in code
	Overly Permissive File Permissions Set in Application	...md/bm25_benchmark.go	View in code

[jfrancoa]

Superseded by #104, which uses the branch on weaviate/weaviate-benchmarking directly instead of a fork.