SINDI support proximity scoring and phrase filter

[Endeavour2016]

Change Type

• Bug fix
• New feature
• Improvement/Refactor
• Documentation
• CI/Build/Infra

Linked Issue

• Resolves: [Feature] SINDI: Proximity-Aware Scoring (Slop/Phrase Support) #2017

What Changed

Adds optional position-aware scoring to the SINDI sparse index. Documents can
now carry their original token order, and queries can boost documents where
query terms appear close together, or hard-filter documents that don't contain
a given phrase.

• Position storage: SparseVector gains an optional token_sequence_ /
  token_seq_len_ field carrying the raw tokenized document. When
  store_positions is enabled, per-term positions are extracted (capped by
  max_positions_per_term, default 64) and stored as flat offset arrays + a
  shared position pool in SparseTermDataCell.
• Proximity scoring (proximity_scorer.{h,cpp}): pairwise
  1/(min_dist+1) boost over query terms — all C(n,2) pairs or adjacent-only
  pairs, ordered or unordered, multiplicative or additive. Re-ranks the top
  proximity_candidates by cosine before applying the boost.
• Phrase filter: hard filter using Lucene-style normalized-window slop
  (sloppy_phrase_match); a candidate passes only if phrase_terms occur
  within phrase_slop.
• Both runtime paths covered: proximity/phrase logic is wired into both
  search_impl and immutable_search_impl (via a templated position backend),
  so mutable and immutable indexes behave identically. Fixes a latent cursor
  misalignment in deserialize_immutable_window where the position block was
  skipped.
• New parameters: build-side store_positions, max_positions_per_term;
  search-side proximity_candidates, proximity_weight, proximity_ordered,
  proximity_boost_multiplicative, proximity_all_pairs, phrase_terms,
  phrase_slop. All default to off/no-op.
• Memory accounting: position bytes are counted in cal_memory_usage.

Test Evidence

• make fmt
• make lint
• make test
• make cov, run tests, and collect coverage
• Other (describe below)

New tests:

• proximity_scorer_test.cpp — unit coverage for all/adjacent pairwise
  proximity, ordered/unordered, sloppy phrase match, position extraction.
• sparse_term_datacell_position_test.cpp — position storage / retrieval.
• sindi_test.cpp — end-to-end proximity + phrase, plus a
  mutable-vs-immutable equality test across fp32 / sq8 / fp16 and remap.

Test details:

./build/tests/unittests -d yes  "[SINDI][Proximity],[SINDI][PhraseFilter],[SINDI][streaming],[SINDIParameter]"
Baseline+proximity search: 193.227 ms total (50 x 2 queries)
/workspace/projects/vsag-fork-zlm/src/algorithm/sindi/sindi_test.cpp:3586: warning:
  Top-1 rank changes (baseline vs proximity): 1/50
/workspace/projects/vsag-fork-zlm/src/algorithm/sindi/sindi_test.cpp:3587: warning:
  Memory estimate: 656048 bytes
5.421 s: SINDI Proximity Benchmark
[2026-07-15 12:38:09.052] SINDI Proximity Benchmark: Executed in 5.421 seconds
===============================================================================
All tests passed (954 assertions in 26 test cases)


./build/tests/unittests "[SparseTermDatacell],[Position]" -d yes
0.000 s: SparseTermDatacell Last Term Test
[2026-07-15 12:32:54.066] SparseTermDatacell Last Term Test: Executed in 0.000 seconds
===============================================================================
All tests passed (288 assertions in 14 test cases)

./build/tests/unittests "[SINDI]" -d yes
Memory estimate: 656048 bytes
5.430 s: SINDI Proximity Benchmark
[2026-07-15 12:41:38.888] SINDI Proximity Benchmark: Executed in 5.430 seconds
===============================================================================
All tests passed (185367 assertions in 34 test cases)

Compatibility Impact

• API/ABI compatibility: SparseVector gains two trailing fields
  (token_seq_len_, token_sequence_); existing callers that leave them
  zero/null are unaffected. All new index/search parameters default to
  off, so existing indexes and queries behave exactly as before.
• Behavior changes: none unless the new parameters are explicitly enabled.
  Indexes built without store_positions cannot use proximity/phrase.

Performance and Concurrency Impact

• Performance impact: none when disabled. When enabled, proximity re-ranks
  only the top proximity_candidates; per-doc cost is O(T²×P) (T = present
  query terms, P = positions/term). Position lists are read via non-owning
  PosSpan views to avoid per-candidate copies.
• Concurrency/thread-safety impact: none; position pools are read-only during
  query.

Documentation Impact

• No docs update needed
• Updated docs:
  • README.md
  • DEVELOPMENT.md
  • CONTRIBUTING.md
  • Other: user-facing SINDI docs under
    docs/docs/en/src/indexes/sindi.md and
    docs/docs/zh/src/indexes/sindi.md — added the new build params
    (store_positions, max_positions_per_term), the search params
    (proximity_*, phrase_*), and a "Proximity scoring and phrase filter"
    section covering the token_sequence_ input.

Risk and Rollback

• Risk level: low — feature is fully gated behind opt-in parameters that
  default to off.
• Rollback plan: revert this PR; no on-disk format migration is required for
  indexes built without store_positions.

Checklist

• I have linked the relevant issue (#2017)
• I have added/updated tests for new behavior or bug fixes
• I have considered API compatibility impact
• I have updated docs if behavior/workflow changed
• My commit messages follow project conventions

[vsag-bot]

/label S-waiting-on-review
/waiting-on reviewer
/request-review @jiaweizone
/request-review @wxyucs
/request-review @inabao
/request-review @LHT129

[mergify]

Merge Protections

🟢 All 3 merge protections satisfied — ready to merge.

Show 3 satisfied protections

🟢 Require kind label

• label~=^kind/

🟢 Require version label

• label~=^version/

🟢 Require linked issue for feature/bug PRs

• body~=(?im)(?:^|[\s\-\*])(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s*:?\s+(?:#\d+|[\w.\-]+/[\w.\-]+#\d+|https?://github\.com/[\w.\-]+/[\w.\-]+/issues/\d+)

[gemini-code-assist]

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

[wxyucs]

/assign @Roxanne0321
/kind feature
/version 1.0

[LHT129]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces proximity scoring and phrase filtering capabilities to the SINDI sparse retrieval index, allowing the index to store term positions and perform proximity-aware re-ranking or exact phrase filtering. The review feedback highlights several critical performance and correctness optimizations. Specifically, it suggests optimizing the ordered distance search to O(M + N) using a two-pointer approach, avoiding redundant map clearing in the search path by tracking only dirty document indices, preventing potential uint16_t overflow for long token sequences, hoisting serialization buffers out of loops to eliminate heap allocations, and improving robustness by using span.empty() instead of null pointer checks.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[Endeavour2016]

By default, position storage is disabled, proximity weight is 0, and phrase terms are empty, so the existing SINDI build/search logic is unchanged. New proximity/phrase logic only runs when explicitly enabled.