vector-store: interval-based Tantivy commits

[knowack1]

This PR improves FTS indexing performance by replacing per-document Tantivy commits with an interval-based commit strategy. Commits are now triggered by a time interval (3 seconds). This reduces indexing time for 10,000 small documents from roughly 2 minutes to under 1 second.
Additionally, to prevent unlimited uncommitted documents, we force a commit when the number of pending documents exceeds the 10k threshold.

This change also fixes double reader reload. In default tantivy reader configuration (that we have) reload happens automatically after commit. My must take control over reload to drop InProgress correctly after reload.

A new validator test covers correctness at scale: it inserts 10,000 documents, waits for the index to catch up, then asserts that a rare-term BM25 search returns only the expected needle documents and that a common-term search respects the query LIMIT.
Indexing and query durations are measured and logged.

Fixes: VECTOR-636
Reference: VECTOR-621

[Copilot]

Pull request overview

This PR introduces deferred/periodic commit behavior for the Tantivy-backed full-text index to avoid committing on every document update, and adds an end-to-end validator test that exercises BM25 over a large document set.

Changes:

• Add pending-commit tracking to the Tantivy FTS actor, committing on an interval and/or after enough pending bytes accumulate.
• Update Tantivy unit tests to poll for visibility (since commits are deferred) and make the commit interval configurable in tests.
• Optimize validator FTS document insertion by preparing the CQL statement once, and add a fts_large_document_set e2e test (10k docs) with basic timing instrumentation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
crates/vector-store/src/fts_index/tantivy.rs	Adds deferred commit machinery (pending bytes/dirty flag + interval tick) and updates unit tests accordingly.
crates/validator/src/fts.rs	Reuses a prepared insert statement, adds duration logging helper, and introduces a large dataset FTS e2e test.

Comments suppressed due to low confidence (1)

crates/vector-store/src/fts_index/tantivy.rs:693

• This test can become a false positive with the new deferred-commit behavior: counting immediately after add_doc can return 0 simply because the index hasn't committed yet. Waiting at least one commit interval before asserting makes it validate that the document truly was not indexed when memory allocation is denied.

        add_doc(&sender, 1, "should not be indexed").await;

        let key = make_index_key();
        let count = sender.count(key).await.unwrap();
        assert_eq!(count, 0);

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[m-szymon]

I tested that it decreases index build time dramatically.
Some concerns below to be clarified.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

crates/vector-store/src/fts_index/tantivy.rs:426

• tokio::select! cannot have an arm starting with an enum variant pattern (FtsIndex::Stats { .. } => { .. }). This will not compile, and it also suggests FtsIndex::Stats is not handled in the match msg block, making that match non-exhaustive.

                FtsIndex::Stats { index_key, tx } => {
                    let Some(state) = get_state(&states, table.as_ref(), &index_key) else {
                        _ = tx.send(Ok(FtsStats::default()));
                        continue;
                    };

[knowack1]

AI says that the merge queue failure was caused by the infra, base on this output (ANNOTATIONS section)

$ gh run view --job=89163928461 --repo scylladb/vector-store

X gh-readonly-queue/master/pr-521-40131ad2468b53707781a957f66987f49678d9ee Rust · 29994219414
Triggered via merge_group about 1 hour ago

X cargo-test in 45m0s (ID 89163928461)
  ✓ Set up job
  ✓ Run actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10
  ✓ Setup rust toolchain
  ✓ Print rustc version
  * Run cargo-test checks
  * Post Setup rust toolchain
  * Post Run actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10

ANNOTATIONS
X The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
cargo-test: .github#1


To see the logs for the failed steps, try: gh run view --log-failed --job=89163928461
View this run on GitHub: https://github.com/scylladb/vector-store/actions/runs/29994219414

But I am still looking at this.

[knowack1]

AI says that the merge queue failure was caused by the infra, base on this output (ANNOTATIONS section)

$ gh run view --job=89163928461 --repo scylladb/vector-store

X gh-readonly-queue/master/pr-521-40131ad2468b53707781a957f66987f49678d9ee Rust · 29994219414
Triggered via merge_group about 1 hour ago

X cargo-test in 45m0s (ID 89163928461)
  ✓ Set up job
  ✓ Run actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10
  ✓ Setup rust toolchain
  ✓ Print rustc version
  * Run cargo-test checks
  * Post Setup rust toolchain
  * Post Run actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10

ANNOTATIONS
X The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
cargo-test: .github#1


To see the logs for the failed steps, try: gh run view --log-failed --job=89163928461
View this run on GitHub: https://github.com/scylladb/vector-store/actions/runs/29994219414

But I am still looking at this.

I was able to reproduce locally. threshold_forces_commit_before_interval is flaky and can hang.

[knowack1]

I fixed the flaky test. The problem was a race condition: the timer was being reset before spawning the task that actually adds the document to the writer. As a result, the timer handler could run before the spawned task, committing no documents.

I changed the approach and dropped the timer reset altogether. Instead, I now commit immediately after the add/remove handlers inside the same spawned task.

This change was introduced in a dedicated commit: 515f882.

@ewienik could you please share you opinion?

[ewienik]

I fixed the flaky test. The problem was a race condition: the timer was being reset before spawning the task that actually adds the document to the writer. As a result, the timer handler could run before the spawned task, committing no documents.

I changed the approach and dropped the timer reset altogether. Instead, I now commit immediately after the add/remove handlers inside the same spawned task.

This change was introduced in a dedicated commit: 515f882.

@ewienik could you please share you opinion?

Changes seems ok. Could you combine them with the correct previous commit to stick with single implementation in the PR?

[knowack1]

I fixed the flaky test. The problem was a race condition: the timer was being reset before spawning the task that actually adds the document to the writer. As a result, the timer handler could run before the spawned task, committing no documents.
I changed the approach and dropped the timer reset altogether. Instead, I now commit immediately after the add/remove handlers inside the same spawned task.
This change was introduced in a dedicated commit: 515f882.
@ewienik could you please share you opinion?

Changes seems ok. Could you combine them with the correct previous commit to stick with single implementation in the PR?

Done.