FTS queries permanently fail with "FtsRocksdbReducer: segments not in consecutive doc_id order" after incremental upsert/delete workload (likely interrupted optimize)

[xuxiang900]

Issue draft for alibaba/zvec

Title: FTS queries permanently fail with "FtsRocksdbReducer: segments not in consecutive doc_id order" after incremental upsert/delete workload (likely interrupted optimize)

---

Environment

• @zvec/zvec 0.5.0 (Node.js binding, @zvec/bindings-darwin-arm64 0.5.0)
• macOS 15 / arm64, running inside an Electron 40 main process (Node 22 ABI)
• FTS field with jieba tokenizer + lowercase filter; one collection is FTS-only, a second collection has the same FTS field plus an HNSW VECTOR_FP32 field (dim 1536, cosine)

Summary

After weeks of incremental indexing (batched upsertSync + deleteSync followed by optimize() after each sync round), both collections reached a state where every FTS query — query() with fts.matchString as well as multiQuery() (FTS + HNSW with RRF rerank) — permanently fails with:

FtsRocksdbReducer: segments not in consecutive doc_id order. field=body

Once a collection is in this state:

• Further upsertSync / deleteSync / manifest-level syncs succeed — writes don't notice anything wrong, so the application believes the index is healthy.
• Every FTS read keeps throwing the error above.
• The only recovery we found is deleting the collection directory and rebuilding from scratch. Incremental re-indexing cannot repair it, since existing segments are untouched.

What we found on disk

Both affected collection directories contain leftover *.tmp segment directories alongside the committed ones, e.g.:

chunks/
  0.tmp/            <- full segment layout inside: fts.3.rocksdb,
  1.tmp/               embedding.index.2.proxima, scalar.index.1.rocksdb, ...
  2.tmp/
  4
  6
  7
  ...
  8058.tmp/
  8059.tmp/
  12264.tmp/
  LOCK
  del.10
  idmap.0
  manifest.23

The .tmp directories look like segment merges that were staged but never renamed/committed — consistent with optimize() being interrupted (the host application can be quit by the user while a sync+optimize round is running). Segment numbers reaching 12k+ reflect the highly incremental workload.

Where the error comes from

The check is in src/db/index/column/fts_column/fts_rocksdb_reducer.cc (~line 137): the reducer requires every non-empty segment to satisfy segment_stats.min_doc_id == segment_stats_.back().max_doc_id + 1, i.e. strictly consecutive global doc_id ranges, and returns Status::InternalError otherwise.

Our (unverified) hypothesis: an interrupted optimize() — or repeated delete/upsert rounds interleaved with partial merges — leaves a committed segment set whose doc_id ranges have gaps or overlaps. The invariant is then violated forever after, and since writes never re-validate it, the collection is silently wedged for reads.

Questions

1. Is the "strictly consecutive doc_id ranges" invariant expected to survive a process crash / kill during optimize()? If not, should open/recovery detect and quarantine stale .tmp segments and re-derive a consistent segment set?
2. Is there (or could there be) a cheaper recovery path than dropping the whole collection — e.g. a repair() / forced full re-merge that rebuilds segment stats from postings?
3. Should write paths (or optimize()) validate this invariant and surface an error early, instead of writes succeeding while every read fails?

Repro status

We don't have a minimal standalone reproduction yet — this is from a production desktop app where the collection had gone through thousands of incremental sync rounds. Happy to provide the corrupted collection's segment layout/manifest details, or test candidate fixes against the corrupted dataset (we kept a copy before rebuilding).

Possibly related to the segment-lifecycle fixes that landed recently (#566 / #569), though the reducer path itself appears unchanged since FTS was introduced in #408.

[egolearner]

Thanks for the detailed report. We have confirmed the root cause.

The strict consecutive-global-doc-id check in FtsRocksdbReducer is incorrect. A compaction that physically removes deleted documents can legitimately leave gaps between the global doc_id ranges of persisted segments. FTS postings still use dense segment-local doc IDs, and the reducer aligns deletions by feed-order scan position plus each segment doc_count, so global range adjacency is not required.

We were able to reproduce this deterministically without interrupting optimize():

1. Create multiple persisted segments.
2. Delete documents from the beginning of each segment and optimize, producing valid persisted ranges with gaps.
3. Delete more documents and optimize again.
4. The second compaction reaches the reducer and fails because the old check requires the ranges to be consecutive.

The fix is in #588. It changes the reducer contract to accept gaps while still rejecting overlapping or unordered ranges. The PR adds both a reducer-level test and a Collection-level regression test covering repeated delete/optimize rounds.

The leftover .tmp directories may indeed be artifacts of interrupted optimize operations, but they are not required to trigger this bug and are not the reason the consecutive-range check is invalid. Cleanup of stale temporary directories and a general repair API are separate lifecycle/recovery concerns and are not included in #588.

With the fix applied, an otherwise valid collection containing these gapped committed segments should be able to optimize again without a full rebuild. If the saved affected collection still fails on a build containing #588, its manifest and segment metadata would be useful for checking whether there is an additional recovery issue beyond this reducer constraint.

[egolearner]

We performed an additional verification using the code before #588, where the consecutive doc_id restriction is still present.

We reproduced the global doc_id gap and confirmed that the second optimize() fails with:

FtsRocksdbReducer: segments not in consecutive doc_id order

However, this failure did not affect FTS queries:

• Query succeeded before the failing optimize().
• Query still succeeded immediately after optimize() returned the reducer error.
• After closing and reopening the collection, Query continued to succeed.

We also checked the Node.js binding. query()/multiQuery() and optimize() use separate async workers, statuses, and Promises. The reducer error is returned by the Optimize path and should not be propagated through a Query Promise.

At this point, we cannot reproduce the reported behavior where every Query returns the reducer error. Could you provide some additional information?

1. A minimal code snippet showing the exact optimize(), query(), and multiQuery() call sequence.
2. Whether optimize() is awaited and its Promise rejection is caught.
3. Whether the failing calls use the synchronous or asynchronous APIs, for example querySync() versus query().
4. The complete JavaScript stack trace for one failed Query.
5. Whether Query still fails after restarting the process and reopening the collection without calling optimize() again.

In particular, an unhandled rejection from an asynchronous optimize() may be reported later and appear close to a subsequent Query in application logs. The full stack trace and Promise handling code should help determine whether that is happening, or whether there is another independent issue.