feat: add HGraph MCI clique companion index by LightWant · Pull Request #2385 · antgroup/vsag

LightWant · 2026-07-01T13:09:11Z

Summary

Add an optional HGraph MCI companion for filtered KNN search. The companion stores clique metadata inside the HGraph index, shares HGraph vector storage, and is not exposed as a standalone public index type.

Changes

Add the internal src/algorithm/mci/ module, CliqueDataCell, and MCI searcher used by HGraph filtered search.
Integrate MCI into HGraph build, incremental add maintenance, search routing, serialization, memory usage, and GetStats().
Clean up MCI parameters: index_param.mci enables build-time MCI by presence, hgraph_valid_ratio_threshold is search-only, MCI search seed parameters are mci_seed_count and mci_seed_ratio, and the default mci_seed_count is 500.
Use precise HGraph vector storage for MCI distance computation instead of depending on user-retained raw vectors.
Move MCI SIMD-specific code behind src/simd, remove the mci_v3_ file-name prefix, rename the MCI namespace to vsag::mci, and align internal naming/style with VSAG conventions.
Add an explicit read-only MemoryIO raw-buffer fast path for contiguous FP32 storage.
Update English and Chinese docs, including the note that MCI Add() is for small increments on an existing initial index and should not be used to build an MCI index from empty.
Reduce fp32_simd_test.cpp macro expansion in one large Catch2 test to lower GCC debug-info variable tracking pressure in CI.

Tests

git diff --check origin/main
clang-format-15 --dry-run --Werror on changed C++ files
cmake --build build-mkl --target vsag_static -j2
cmake --build build --target simd_test -j2

Compatibility Impact

API/ABI compatibility: no breaking public API changes expected.
Behavior changes: adds optional HGraph MCI companion support for filtered search.
HGraph MCI build config now enables MCI by the presence of index_param.mci; hgraph_valid_ratio_threshold belongs to search parameters.

Performance and Concurrency Impact

Performance impact: intended improvement for filtered HGraph search paths when MCI is enabled.
Concurrency/thread-safety impact: none expected.

Merge Protections

🟢 All 3 merge protections satisfied — ready to merge.

Show 3 satisfied protections

🟢 Require kind label

label~=^kind/

🟢 Require version label

label~=^version/

🟢 Require linked issue for feature/bug PRs

body~=(?im)(?:^|[\s\-\*])(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s*:?\s+(?:#\d+|[\w.\-]+/[\w.\-]+#\d+|https?://github\.com/[\w.\-]+/[\w.\-]+/issues/\d+)

gemini-code-assist · 2026-07-01T13:10:27Z

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

LightWant · 2026-07-02T02:26:18Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces the HGraph MCI (Maximal Clique Index) Companion feature, which optimizes filtered KNN searches by building and storing clique metadata alongside the HGraph index. The changes span documentation, examples, build configuration updates, and the core algorithms for MCI building, searching, and serialization. The code review identified several critical issues in the new implementation, including a memory corruption bug in ListLinearHeap::initRaw due to an unupdated size variable, a severe query-time performance bottleneck from allocating visited_nodes per search, a data race on num_cliques_per_node in the parallel builder, and potential undefined behavior in changeToCoreOrder upon heap pop failure. Additionally, improvements were suggested to replace raw memory allocations with RAII containers and standard printf statements with the library's logging system.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

vsag-bot · 2026-07-06T13:34:44Z

/label S-waiting-on-review
/waiting-on reviewer
/request-review @jiaweizone
/request-review @wxyucs
/request-review @inabao
/request-review @LHT129

Add an optional HGraph MCI companion that stores clique metadata with HGraph, routes filtered search through an MCI searcher, supports serialization and add maintenance, and documents the companion workflow with a self-contained example. Signed-off-by: zhuangye.yxw <2510035537@qq.com> Assisted-by: Codex:gpt-5

Signed-off-by: zhuangye.yxw <2510035537@qq.com> Assisted-by: Codex:GPT-5

Remove public build-time use_mci and hgraph_valid_ratio_threshold handling from HGraph MCI options, default the search routing threshold to 0.05, and rename MCI source files to drop the v3 file-name suffix. Signed-off-by: zhuangye.yxw <2510035537@qq.com> Assisted-by: Codex:GPT-5

Signed-off-by: zhuangye.yxw <2510035537@qq.com> Assisted-by: Codex:GPT-5

wxyucs · 2026-07-16T11:08:44Z

/review

vsag-bot · 2026-07-16T11:08:56Z

Automated pull request review completed.

Review effort: high (8275 changed lines across 45 files).

Submitted 8 inline comments.
Review: #2385 (review)

vsag-bot

Automated inline review completed.

Review effort: high (8275 changed lines across 45 files).
Submitted 8 inline comments.

Reviewed commit 5914e06.

vsag-bot · 2026-07-16T11:19:58Z

+                g.edges[g.m] = g.edges[i];
+            }
+        }
+        g.m++;


[P1] Guard the zero-edge graph before incrementing m

When edge is empty, this unconditional increment creates one default (0, 0) edge while g.n remains zero. build_csr() then writes p_idx[1] into a one-element vector; an ASan probe reproduces the heap-buffer-overflow. The optimized builder can reach this state when a seed's nearest candidate is a duplicate (distance limit zero) while the remaining candidates are distinct, so return no cliques before building the CSR when the edge list is empty.

vsag-bot · 2026-07-16T11:20:00Z

+        re_id.resize(g.n);
+    std::fill(re_id.begin(), re_id.begin() + g.n, g.n);
+
+    if (in_c.capacity() < g.max_d)


[P1] Resize reusable scratch vectors by size, not capacity

Checking capacity() does not ensure that indices below g.max_d are within the vector's logical size. This is especially immediate for vector<bool>, whose first small resize commonly leaves a capacity of 64: a later seed with a larger degree skips resize(), and this fill plus subsequent indexing run past size(). Reusing one runner for graphs of sizes 5, 8, and 9 aborts under _GLIBCXX_DEBUG; the same issue exists for re_id above, so both checks must use size() or resize unconditionally.

vsag-bot · 2026-07-16T11:20:00Z

+                mci_param.metric = this->metric_;
+                mci_param.used_precise_float_csr = &mci_used_precise_float_csr;
+            }
+            search_result = this->mci_searcher_->Search(


[P1] Do not route executor-backed attribute filters to MCI

Attribute filtering is stored in search_param.executors, but both MCI implementations only check is_inner_id_allowed and never run these executors. A request combining a selective user filter with enable_attribute_filter_ can therefore route here and return IDs that violate the attribute expression. Either incorporate the executor-produced filter into MCI's validity check or keep such requests on the HGraph path.

vsag-bot · 2026-07-16T11:20:00Z

+            if (row_stride != nullptr) {
+                *row_stride = this->code_size_ / sizeof(float);
+            }
+            return reinterpret_cast<const float*>(memory_io->GetReadOnlyRawData());


[P1] Pin the raw MemoryIO buffer for the duration of MCI search

This returns an unpinned pointer that MemoryIO::ResizeImpl or WriteImpl may invalidate through Reallocate. HGraph search's shared global_mutex_ does not protect insert_persistent_codes(), which runs before Add takes that mutex, so concurrent Add can reallocate the FP32 buffer while search_precise_float_csr() is dereferencing it. The fast path needs a lifetime/read lock spanning the search, or it must be disabled while the index is mutable.

vsag-bot · 2026-07-16T11:20:00Z

+        clique_ids.clear();
+        cliques->CollectNodeCliqueIds(candidate->inner_id, clique_ids);
+        for (auto clique_id : clique_ids) {
+            if (visited_cliques.Get(clique_id)) {


[P1] Keep the clique-count snapshot stable during traversal

The visit-mark array is sized from an unlocked TotalLogicalCliqueCount() snapshot, while each later clique lookup takes a separate lock. During a batched concurrent Add, AppendNewClique() can insert another clique between those operations, allowing CollectNodeCliqueIds() to return an ID equal to or above the marks size; visited_cliques.Get() then indexes out of bounds. Hold a stable cell snapshot across traversal, retry on a version change, or grow/check the marks before indexing.

vsag-bot · 2026-07-16T11:20:00Z

+                 max_norm_error);
+
+    auto distance_from_dot = [&](InnerIdType lhs, InnerIdType rhs, float dot) {
+        const float distance =


[P2] Honor the configured metric in optimized clique distances

The float fast path always constructs local edges from squared-L2 distance, even when HGraph is configured for cosine or inner product; it receives no metric parameter. This disagrees with the generic path's ComputePairVectors() reference behavior, and raw cosine build vectors are not necessarily normalized, so magnitude can completely change clique topology and filtered recall. Pass the metric and reproduce its distance semantics, or restrict this optimized path to L2.

vsag-bot · 2026-07-16T11:20:00Z

+    const auto candidate_limit =
+        std::min<uint64_t>({params.candidate_limit, graph.row_stride, total - 1});
+    const auto clique_threshold =
+        std::max<uint64_t>(2, std::min<uint64_t>({params.clique_max, candidate_limit + 1, total}));


[P2] Enforce clique_max as an upper bound

clique_max is used here as a minimum enumeration threshold, and no later step truncates saved cliques to that size. For example, clique_max: 4 can store all nine vertices of a complete local graph, contrary to the documented maximum and with unexpectedly larger memory and query fan-out. Apply an actual upper bound when saving cliques, or correct the public parameter contract if minimum size is intended.

vsag-bot · 2026-07-16T11:20:00Z

+        }
+    }
+    if (not seed_list_provided) {
+        for (InnerIdType seed = 0; seed < total and seeds < seed_target; ++seed) {


[P2] Avoid full-index seed scans for callback filters

A normal callback-based Filter does not implement GetValidIds(), so MCI falls back to scanning inner IDs from zero until it finds the requested number of valid seeds. With the sparse filters that trigger MCI, valid IDs near the end can make every query execute millions of filter callbacks, turning the intended sublinear search into an O(total) scan. Use bounded sampling or a reusable valid-ID source, or fall back to HGraph when seed enumeration is unavailable.

LightWant requested review from LHT129, inabao, jiaweizone and wxyucs as code owners July 1, 2026 13:09

pull-request-size Bot added the size/XXL label Jul 1, 2026

mergify Bot added module/docs module/example labels Jul 1, 2026

LightWant added kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 module/datacell module/index module/testing version/1.0 labels Jul 1, 2026

LightWant mentioned this pull request Jul 2, 2026

[feat](hgraph): add MCI companion for filtered search #2386

Open

gemini-code-assist Bot reviewed Jul 2, 2026

View reviewed changes

LightWant force-pushed the feat/hgraph-mci-companion-clean branch 3 times, most recently from fed41a2 to 78b29f3 Compare July 2, 2026 06:41

mergify Bot added the module/attr label Jul 2, 2026

wxyucs self-assigned this Jul 2, 2026

LightWant force-pushed the feat/hgraph-mci-companion-clean branch from b8eeedc to 25830d9 Compare July 3, 2026 09:37

LightWant added 6 commits July 7, 2026 07:46

fix: stabilize CI version detection

8a70a98

Signed-off-by: zhuangye.yxw <2510035537@qq.com> Assisted-by: Codex:GPT-5

fix: guard MCI SSE path on non-x86

9c21abd

Signed-off-by: zhuangye.yxw <2510035537@qq.com> Assisted-by: Codex:GPT-5

fix: keep HGraph MCI JSON parent alive in test

8f874a6

Signed-off-by: zhuangye.yxw <2510035537@qq.com> Assisted-by: Codex:GPT-5

fix: rename MCI seed search parameters

d59f3f7

Signed-off-by: zhuangye.yxw <2510035537@qq.com> Assisted-by: Codex:GPT-5

fix: address HGraph MCI companion review feedback

64c1082

Signed-off-by: zhuangye.yxw <2510035537@qq.com> Assisted-by: Codex:GPT-5

LightWant force-pushed the feat/hgraph-mci-companion-clean branch from 468cc1a to 64c1082 Compare July 7, 2026 07:55

mergify Bot added the module/simd label Jul 7, 2026

fix: reduce fp32 SIMD test debug info pressure

5914e06

Signed-off-by: zhuangye.yxw <2510035537@qq.com> Assisted-by: Codex:GPT-5

wxyucs added version/1.1 and removed version/1.0 labels Jul 13, 2026

vsag-bot self-requested a review July 16, 2026 11:08

vsag-bot reviewed Jul 16, 2026

View reviewed changes

Uh oh!

Conversation

LightWant commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Tests

Compatibility Impact

Performance and Concurrency Impact

Related

Uh oh!

mergify Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Protections

🟢 Require kind label

🟢 Require version label

🟢 Require linked issue for feature/bug PRs

Uh oh!

gemini-code-assist Bot commented Jul 1, 2026

Uh oh!

LightWant commented Jul 2, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vsag-bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wxyucs commented Jul 16, 2026

Uh oh!

vsag-bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vsag-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LightWant commented Jul 1, 2026 •

edited

Loading

mergify Bot commented Jul 1, 2026 •

edited

Loading

vsag-bot commented Jul 6, 2026 •

edited

Loading

vsag-bot commented Jul 16, 2026 •

edited

Loading