[Bug]: FTS queries silently return 0 results after collection reopen (BM25 segment stats not persisted without optimize())

[Mediatros]

Environment

• zvec 0.5.1 (PyPI wheel), Python 3.14.3, macOS 15 (Darwin 25.5.0, arm64)
• FTS index declared in the schema at creation time: FieldSchema("text", DataType.STRING, index_param=FtsIndexParam())

Summary

When enough documents are upserted for segments to be sealed to disk, FTS queries
work in the writing process but silently return 0 results after the collection
is closed and reopened (new process, or same process after reopen). Vector
search on the same collection keeps working. No error or warning is raised: the
query just returns an empty result set.

Calling collection.optimize() before closing the writing process fixes it.
flush() does not. Calling optimize() after reopening does not repair an
affected collection either: the FTS index data must be rebuilt from source, or
copied to a fresh collection via fetch() + upsert() + optimize().

Reproduction

#!/usr/bin/env python3
"""Run: python repro.py write   -> FTS hits in writing process: 6
   Run: python repro.py read    -> FTS hits after reopen: 0 (expected 6)
Workaround: uncomment coll.optimize() in the write phase -> read phase returns 6."""
import shutil, os, random, sys
import zvec
from zvec import DataType, FieldSchema, FtsIndexParam, VectorSchema

PATH = "repro_fts_coll"

if sys.argv[1:] == ["write"]:
    if os.path.exists(PATH):
        shutil.rmtree(PATH)
    fields = [FieldSchema("text", DataType.STRING, index_param=FtsIndexParam())]
    vectors = VectorSchema("embedding", data_type=DataType.VECTOR_FP32, dimension=64)
    coll = zvec.create_and_open(PATH, zvec.CollectionSchema(name="mails", fields=fields, vectors=vectors))
    random.seed(1)
    # Insert enough docs, in batches, for segments to be sealed to disk.
    # 6 docs contain the marker word "needle".
    for page in range(60):
        docs = [
            zvec.Doc(
                id=f"d{page}_{i}",
                vectors={"embedding": [random.random()] * 64},
                fields={"text": "lorem ipsum dolor" + (" needle" if (page * 50 + i) % 500 == 0 else "")},
            )
            for i in range(50)
        ]
        coll.upsert(docs)
    # coll.optimize()  # <-- WORKAROUND: uncomment and the read phase works
    q = zvec.Query(field_name="text", fts=zvec.Fts(match_string="needle"))
    print("FTS hits in writing process:", len(coll.query(q, topk=10)))  # -> 6
elif sys.argv[1:] == ["read"]:
    coll = zvec.open(PATH, zvec.CollectionOption(read_only=True))
    print("doc_count:", coll.stats)
    q = zvec.Query(field_name="text", fts=zvec.Fts(match_string="needle"))
    print("FTS hits after reopen:", len(coll.query(q, topk=10)))  # -> 0 (expected 6)
    # Note: vector search still works fine after reopen; only FTS is affected.

Notes:

• With a very small dataset (a couple of docs, no sealed segment), FTS does
  survive reopen — presumably served from WAL replay. The bug only appears once
  segments are sealed, which makes it easy to miss in small tests and hit in
  production.
• The FTS RocksDB directory (0/fts.1.rocksdb) contains the expected volume of
  SST data after the write phase (hundreds of MB on our real 15k-doc corpus), so
  the postings appear to be written; they are just not usable after reopen.

Root-cause hint

While experimenting on an affected collection, internal errors point at the BM25
segment statistics:

[ERROR ... bm25_scorer.cc:44] BM25Scorer::load_segment_stats: failed to read total_docs. field[text]
[ERROR ... fts_indexer.cc:102] FtsIndexer: FtsColumnIndexer::open failed for field[text]: FtsColumnIndexer failed to load segment stats. field=text

So it looks like the per-segment stats needed by the BM25 scorer are only
persisted/registered during optimize(), and a plain close after upsert()
leaves sealed segments without loadable stats, making FTS queries match nothing.

Fallout warning (secondary, for other users hitting this)

On a collection in this state, attempting to repair by drop_index("text") +
create_index("text", FtsIndexParam()) made things much worse for us on a real
collection (15k docs, built over ~4h of embedding): the drop failed midway with

[ERROR ... rocksdb_context.cc:341] Cannot flush RocksDB[.../0/fts.1.rocksdb] in read-only mode
[ERROR ... fts_indexer.cc:148] FtsIndexer: flush failed during snapshot

and after closing, the collection could not be opened at all any more
(RuntimeError: open_fts_indexers: open failed at [.../1/fts.1.rocksdb]), i.e.
permanent loss including the vector side. We could not reduce this second
failure to a minimal repro (it seems to require a more complex segment history),
but it is worth a warning until the underlying persistence bug is fixed.

Expected behavior

FTS queries should return the same results after close/reopen as in the writing
process, without requiring an explicit optimize() — or, failing that,
upsert()/close should persist whatever the BM25 scorer needs, and reopening a
collection with unusable FTS stats should raise instead of silently returning 0
results.

[egolearner]

Thanks for the detailed repro, and sorry for the trouble this caused.

The issue is confirmed. Root cause is that BM25 segment stats (total_docs / total_tokens) were only restored on the immutable reader path. A reopened mutable writing segment could have persisted postings and doc length data, but its in-memory BM25 scorer stats stayed at zero, which made FTS scoring return no effective results after reopen unless optimize() rebuilt the immutable segment.

Opened PR #566 with the fix.

We will also do a deeper investigation into the reported drop_index("text") + create_index("text", FtsIndexParam()) failure on v0.5.1. From the logs, this looks like a separate index-mutation consistency issue: persisted segments may be opened internally in read-only mode, while the drop/create-index path still attempts to snapshot/flush their FTS RocksDB state. We will audit that path end-to-end, including partial-failure handling, CF dropping, segment metadata updates, and recovery behavior, before recommending any repair workflow.