feat(pyramid): generalize build-cache feature so Pyramid is reusable by LHT129 · Pull Request #2375 · antgroup/vsag

LHT129 · 2026-06-30T08:02:51Z

Summary

Generalize VSAG's build cache (ExportCache / ImportCache) so it works for both HGraph and Pyramid.

Promote HGraphCache to a shared BuildCache under src/algorithm/ and rewire HGraph to use the renamed class without any behaviour change. This keeps HGraph's 6-phase build_with_cache pipeline intact while preparing the same cache facility for non-HGraph indexes.

Extend Pyramid with the same feature:

Read source_id from the dataset in Add() / build_by_odescent() and call LabelTable::InsertSourceId, mirroring HGraph.
Persist the label table's source_id table on Serialize() and reload it on Deserialize() behind a magic-number footer so old binary indexes still deserialize cleanly.
Add PYRAMID_PERSIST_SOURCE_ID alongside HGRAPH_PERSIST_SOURCE_ID and thread it through PyramidParameters / CheckAndMappingExternalParam.
Add BuildCache state, has_loaded_cache(), fullfill_cache(), ExportCache(), ImportCache() and build_with_cache(). fullfill_cache walks every GRAPH-status IndexNode in each hierarchy, captures each inner_id's source_id and its current bottom-graph neighbours, and stores them keyed by source_id. build_with_cache batches the dataset, rebuilds the path tree, initialises graphs, then per node translates cached neighbours to new inner_ids via a source_id_to_inner map and installs both forward and (where capacity allows) reverse edges directly into the graph; the missed set is handed to the standard Add / ODescent path.
Surface build_cache_hit_rate, build_cache_hit_nodes, build_cache_missed_nodes via GetStats(), matching HGraph.

Add three ft-cases covering smoke, miss-only, and stats reporting.

HGraph's build path is unchanged except for the rename.

Test Plan

./build-release/tests/functests Pyramid*: 22/22 pass (113092 assertions) including 3 new cache tests.
./build-release/tests/functests SIMQ*: 6/6 pass (3149 assertions).
./build-release/tests/functests *ExportCache*: 4/4 pass (68 assertions) across HGraph and Pyramid.
./build-release/tests/unittests builds and passes.
make fmt clean.

Closes #2377.

Signed-off-by: LHT129 tianlan.lht@antgroup.com
Assisted-by: OpenCode:MiniMax-M3

mergify · 2026-06-30T08:04:02Z

Merge Protections

🟢 All 3 merge protections satisfied — ready to merge.

Show 3 satisfied protections

🟢 Require kind label

label~=^kind/

🟢 Require version label

label~=^version/

🟢 Require linked issue for feature/bug PRs

body~=(?im)(?:^|[\s\-\*])(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s*:?\s+(?:#\d+|[\w.\-]+/[\w.\-]+#\d+|https?://github\.com/[\w.\-]+/[\w.\-]+/issues/\d+)

gemini-code-assist

Code Review

This pull request introduces a build cache mechanism (BuildCache, renamed from HGraphCache) to support warm-start builds for the Pyramid index, including ExportCache and ImportCache methods, and removes the unused AddMode parameter from SIMQ::Add. The code review highlights several critical bugs and optimization opportunities in the new caching implementation. Specifically, label_table_->total_count_ is not updated during build_by_odescent and build_with_cache, which leads to exporting empty caches. Additionally, the candidates heap is populated but never used to select the closest neighbors, certain variables are unused, and std::vector is used instead of the custom allocator-aware Vector. The reviewer also suggested performance improvements, such as hoisting base->GetPaths(hname) out of a loop and using resize() instead of a while loop with emplace_back() to grow the cache.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

vsag-bot · 2026-07-13T03:25:39Z

/label S-waiting-on-review
/waiting-on reviewer
/request-review @jiaweizone
/request-review @wxyucs
/request-review @inabao

vsag-bot · 2026-07-21T03:19:52Z

Automated pull request review completed.

Review effort: high (1004 changed lines across 14 files).

Submitted 2 inline comments.
Review: #2375 (review)

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

vsag-bot

Automated inline review completed.

Review effort: high (788 changed lines across 13 files).
Submitted 7 inline comments.

Reviewed commit 3943a9c.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (3)

src/algorithm/hgraph/CMakeLists.txt:1

This diff removes hgraph_cache.cpp from the hgraph target, but the cache implementation is still required (now via BuildCache/PyramidBuildCache). If build_cache.cpp (or the renamed cache .cpp) is not added to some compiled target elsewhere, this will cause unresolved symbols at link time. Fix by ensuring the cache implementation translation unit is included in the appropriate CMake target(s) (e.g., a shared algorithm object/static library) and that hgraph/pyramid link against it.

set(HGRAPH_SRCS

src/algorithm/pyramid/pyramid.cpp:1

This Pyramid parameter template still uses HGRAPH_PARAMS_TEMPLATE and the placeholder {HGRAPH_PERSIST_SOURCE_ID_KEY}. Even if the underlying string value is shared, referencing HGraph-specific symbols in Pyramid increases coupling and makes future refactors error-prone. Consider renaming the template to a Pyramid-specific name and using a Pyramid-specific placeholder key ({PYRAMID_PERSIST_SOURCE_ID_KEY}) consistently.
src/algorithm/pyramid/pyramid.cpp:1
Pyramid’s external parameter PYRAMID_PERSIST_SOURCE_ID is being mapped through HGRAPH_PERSIST_SOURCE_ID_KEY. Even though both keys currently resolve to the same string, this introduces unnecessary cross-module dependency. Prefer mapping to PYRAMID_PERSIST_SOURCE_ID_KEY so Pyramid remains self-contained.

Promote hgraph_cache.{h,cpp} to a shared BuildCache at src/algorithm/ and rewire HGraph to use the renamed class without changing any behaviour. This prepares the same ExportCache / ImportCache facility for non-HGraph indexes. Extend Pyramid with the same feature: - Read source_id from the dataset in Add() / build_by_odescent() and call LabelTable::InsertSourceId, mirroring HGraph. - Persist the label table source_id table on Serialize() and reload it on Deserialize() behind a magic-number footer so old binary indexes still deserialize cleanly. - Add PYRAMID_PERSIST_SOURCE_ID alongside HGRAPH_PERSIST_SOURCE_ID and thread it through PyramidParameters / CheckAndMappingExternalParam. - Add BuildCache state, has_loaded_cache(), fullfill_cache(), ExportCache(), ImportCache() and build_with_cache(). fullfill_cache walks every GRAPH-status IndexNode in each hierarchy, captures each inner_id source_id and its current bottom-graph neighbors, and stores them keyed by source_id. build_with_cache batches the dataset, rebuilds the path tree, initializes graphs, then per node translates cached neighbors to new inner_ids via a source_id_to_inner map and installs both forward and (where capacity allows) reverse edges directly into the graph; the missed set is handed to the standard Add/ODescent path so cold vectors still get a full graph treatment. - Surface build_cache_hit_rate, build_cache_hit_nodes, build_cache_missed_nodes via GetStats(), matching HGraph. Add three ft-cases covering smoke, miss-only, and stats reporting. Follow-ups driven by gemini-code-assist review: - Keep label_table total_count in sync after the new source_id population loops in build_by_odescent and build_with_cache, otherwise fullfill_cache() on a built index exports an empty cache. - Drop the dead variable / unused ODescent builder in the per-hierarchy init loop in build_with_cache. - Switch hit_ids/missed_ids to the allocator-aware Vector. - Hoist base->GetPaths(hname) out of the per-missed-id loop in build_with_cache. - Use the candidates heap to drop farthest entries only when the warm seed exceeds max_degree; otherwise keep the cached topology intact, so the refined graph keeps the structure captured by the cache. - Make fullfill_cache grow source_ids_ via resize() instead of an emplace_back while-loop. HGraph build path is unchanged except for the rename. Signed-off-by: LHT129 <tianlan.lht@antgroup.com> Assisted-by: OpenCode:MiniMax-M3

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

+    {
+        const uint64_t cursor_before = buffer_reader.GetCursor();
+        if (buffer_reader.Length() >= cursor_before + sizeof(uint64_t)) {
+            uint64_t magic = 0;
+            StreamReader::ReadObj(buffer_reader, magic);
+            if (magic == SOURCE_ID_TABLE_MAGIC) {
+                uint64_t sid_count = 0;
+                StreamReader::ReadObj(buffer_reader, sid_count);
+                const uint64_t label_table_size = this->label_table_->label_table_.size();
+                if (sid_count > label_table_size) {
+                    throw VsagException(
+                        ErrorType::INVALID_ARGUMENT,
+                        fmt::format("corrupted index: source_id_table sid_count ({}) "
+                                    "exceeds label_table size ({})",
+                                    sid_count,
+                                    label_table_size));
+                }
+                Vector<std::string> sid_table(sid_count, std::string{}, allocator_);
+                for (uint64_t i = 0; i < sid_count; ++i) {
+                    sid_table[i] = StreamReader::ReadString(buffer_reader);
+                }
+                this->label_table_->ReplaceSourceIdTable(std::move(sid_table));


+                for (uint64_t i = 0; i < sid_count; ++i) {
+                    sid_table[i] = StreamReader::ReadString(buffer_reader);
+                }


+MakePyramidCacheQuery(int64_t dim, const std::vector<float>& query, const std::string& path)
+    -> vsag::DatasetPtr {
+    auto dataset = vsag::Dataset::Make();
+    auto raw_query_vec = std::make_unique<float[]>(dim);
+    std::copy(query.begin(), query.end(), raw_query_vec.get());
+    auto raw_paths = std::make_unique<std::string[]>(1);
+    raw_paths[0] = path;
+    dataset->NumElements(1)
+        ->Dim(dim)
+        ->Float32Vectors(raw_query_vec.get())
+        ->Paths(raw_paths.get())
+        ->Owner(true);
+    raw_query_vec.release();
+    raw_paths.release();
+    return dataset;


        "{EF_CONSTRUCTION_KEY}": 400,
        "{NO_BUILD_LEVELS}":[],
        "{INDEX_MIN_SIZE}": 0,
-        "{SUPPORT_DUPLICATE}": false
+        "{SUPPORT_DUPLICATE}": false,
+        "{HGRAPH_PERSIST_SOURCE_ID_KEY}": false


Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

src/algorithm/hgraph/CMakeLists.txt:1

hgraph_cache.cpp was removed from HGRAPH_SRCS, but this PR still modifies src/algorithm/hgraph/hgraph_cache.cpp to provide the BuildCache/PyramidBuildCache implementations. If these symbols are still defined in that translation unit, the hgraph target (and/or pyramid) will fail to link. Fix by either (a) adding the new implementation file (wherever BuildCache now lives) to an appropriate CMake target, or (b) renaming/moving the .cpp to src/algorithm/build_cache.cpp and ensuring it is compiled into the main library.

set(HGRAPH_SRCS

src/algorithm/pyramid/pyramid.cpp:1

PYRAMID_PERSIST_SOURCE_ID is mapped to {HGRAPH_PERSIST_SOURCE_ID_KEY}. While both currently resolve to the same string (\"persist_source_id\"), this is Pyramid code and it’s easy to misread as an accidental cross-index coupling. Prefer mapping to {PYRAMID_PERSIST_SOURCE_ID_KEY} (and using the Pyramid placeholder in Pyramid param templates) to keep the parameter plumbing self-contained and avoid future divergence bugs if the keys ever change.

+        const uint64_t cursor_before = buffer_reader.GetCursor();
+        if (buffer_reader.Length() >= cursor_before + sizeof(uint64_t)) {
+            uint64_t magic = 0;
+            StreamReader::ReadObj(buffer_reader, magic);
+            if (magic == SOURCE_ID_TABLE_MAGIC) {
+                uint64_t sid_count = 0;
+                StreamReader::ReadObj(buffer_reader, sid_count);


+                    if (label_block_cursor < label_block_length) {
+                        const auto remaining = label_block_length - label_block_cursor;
+                        if (remaining < sizeof(uint64_t) * 2) {
+                            throw VsagException(ErrorType::INVALID_BINARY,
+                                                "truncated Pyramid source_id_table footer");
+                        }


+                    for (auto nb : new_neighbors) {
+                        Vector<InnerIdType> existing(allocator_);
+                        gnode->graph_->GetNeighbors(nb, existing);
+                        bool has_reverse = false;
+                        for (auto e : existing) {
+                            if (e == inner_id) {
+                                has_reverse = true;
+                                break;
+                            }
+                        }
+                        if (has_reverse) {
+                            continue;
+                        }
+                        if (existing.size() < max_deg) {
+                            existing.push_back(inner_id);
+                            gnode->graph_->InsertNeighborsById(nb, existing);
+                        }
+                    }


vsag-bot

Automated inline review completed.

Review effort: high (1004 changed lines across 14 files).
Submitted 2 inline comments.

Reviewed commit bab951c.

vsag-bot · 2026-07-21T11:05:25Z

+    if (node == nullptr) {
+        return;
+    }
+    node->Init();


[P1] Preserve NO_INDEX nodes for excluded hierarchy levels

populate_path_tree() deliberately leaves ids_ empty at every no_build_levels node, and the normal NSW build leaves those nodes as NO_INDEX so IndexNode::Search() recurses into indexed children. Calling Init() unconditionally changes each such node to an empty FLAT (or empty GRAPH when index_min_size == 0), so a prefix query such as a with no_build_levels=[0,1,2] stops there and returns no descendants. Initialize only nodes that accumulated membership IDs.

vsag-bot · 2026-07-21T11:05:25Z

+                            candidates->Pop();
+                        }
+                    }
+                    gnode->graph_->InsertNeighborsById(inner_id, new_neighbors);


[P1] Refine cached edges against the new vectors

Translated cached neighbors are installed as final adjacency, and hit points never undergo graph search or pruning using the current vectors. If stable source_ids are rebuilt with updated embeddings, every entry remains a hit while the graph still represents the old geometry, allowing recall to degrade arbitrarily; HGraph explicitly refines stale cache-hit nodes for this reason. Use cached edges only as seeds for a refinement pass against the current codes.

Copilot AI review requested due to automatic review settings June 30, 2026 08:02

LHT129 requested review from inabao, jiaweizone and wxyucs as code owners June 30, 2026 08:02

pull-request-size Bot added the size/XL label Jun 30, 2026

mergify Bot added module/api module/testing labels Jun 30, 2026

LHT129 added kind/improvement Optimizations, UX polish, or minor improvements 性能优化、体验打磨或细节改良 version/1.0 labels Jun 30, 2026

gemini-code-assist Bot reviewed Jun 30, 2026

View reviewed changes

Copilot AI reviewed Jun 30, 2026

Copilot stopped reviewing on behalf of LHT129 due to an error June 30, 2026 08:19
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

LHT129 force-pushed the P2重要-2026-06-29-通用化-build-cache-特性使-pyramid-可复用 branch from 8a566ee to a557feb Compare June 30, 2026 08:32

LHT129 mentioned this pull request Jun 30, 2026

feat: generalize build-cache (ExportCache / ImportCache) to non-HGraph indexes #2377

Open

LHT129 self-assigned this Jun 30, 2026

LHT129 added kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 and removed kind/improvement Optimizations, UX polish, or minor improvements 性能优化、体验打磨或细节改良 labels Jun 30, 2026

LHT129 force-pushed the P2重要-2026-06-29-通用化-build-cache-特性使-pyramid-可复用 branch from a557feb to 0480409 Compare June 30, 2026 09:24

wxyucs added version/1.1 and removed version/1.0 labels Jul 13, 2026

LHT129 force-pushed the P2重要-2026-06-29-通用化-build-cache-特性使-pyramid-可复用 branch from 0480409 to 3943a9c Compare July 21, 2026 03:19

Copilot AI review requested due to automatic review settings July 21, 2026 03:19

vsag-bot self-requested a review July 21, 2026 03:19

Copilot started reviewing on behalf of LHT129 July 21, 2026 03:34 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread src/algorithm/pyramid/pyramid.cpp Outdated

Comment thread src/algorithm/pyramid/pyramid.cpp Outdated

Comment thread src/algorithm/pyramid/pyramid.cpp Outdated

Comment thread src/algorithm/pyramid/pyramid.cpp Outdated

Comment thread src/algorithm/pyramid/pyramid.cpp Outdated

vsag-bot reviewed Jul 21, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 21, 2026 04:14

LHT129 force-pushed the P2重要-2026-06-29-通用化-build-cache-特性使-pyramid-可复用 branch from a1f4408 to 5756303 Compare July 21, 2026 06:36

Copilot started reviewing on behalf of LHT129 July 21, 2026 06:41 View session

LHT129 force-pushed the P2重要-2026-06-29-通用化-build-cache-特性使-pyramid-可复用 branch from 5756303 to e0bd6ef Compare July 21, 2026 06:43

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread src/algorithm/pyramid/pyramid.cpp

Copilot AI review requested due to automatic review settings July 21, 2026 06:45

Copilot started reviewing on behalf of LHT129 July 21, 2026 06:55 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread src/algorithm/pyramid/pyramid.cpp Outdated

Comment thread src/algorithm/pyramid/pyramid.cpp

Copilot AI review requested due to automatic review settings July 21, 2026 07:14

LHT129 force-pushed the P2重要-2026-06-29-通用化-build-cache-特性使-pyramid-可复用 branch 2 times, most recently from dbd925c to 5230182 Compare July 21, 2026 07:20

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread src/algorithm/pyramid/pyramid.cpp

Comment thread src/algorithm/pyramid/pyramid.cpp Outdated

Comment thread src/algorithm/pyramid/pyramid_zparameters.cpp Outdated

Comment thread tests/test_pyramid.cpp

Comment thread src/algorithm/pyramid/pyramid.cpp

Copilot AI review requested due to automatic review settings July 21, 2026 07:31

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Copilot started reviewing on behalf of LHT129 July 21, 2026 09:01 View session

Copilot started reviewing on behalf of LHT129 July 21, 2026 09:07 View session

Copilot AI review requested due to automatic review settings July 21, 2026 09:22

LHT129 force-pushed the P2重要-2026-06-29-通用化-build-cache-特性使-pyramid-可复用 branch from 5230182 to c4c8994 Compare July 21, 2026 09:22

pull-request-size Bot added size/XXL and removed size/XL labels Jul 21, 2026

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread src/algorithm/pyramid/pyramid.cpp

Comment thread src/algorithm/pyramid/pyramid.cpp

Comment thread src/algorithm/pyramid/pyramid.cpp

Copilot AI review requested due to automatic review settings July 21, 2026 09:40

Copilot started reviewing on behalf of LHT129 July 21, 2026 09:50 View session

LHT129 force-pushed the P2重要-2026-06-29-通用化-build-cache-特性使-pyramid-可复用 branch from c4c8994 to bab951c Compare July 21, 2026 09:56

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 21, 2026 09:58

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Copilot started reviewing on behalf of LHT129 July 21, 2026 10:18 View session

Copilot started reviewing on behalf of LHT129 July 21, 2026 10:33 View session

vsag-bot reviewed Jul 21, 2026

View reviewed changes

Uh oh!

Hidden character warning

Conversation

LHT129 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

mergify Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Protections

🟢 Require kind label

🟢 Require version label

🟢 Require linked issue for feature/bug PRs

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

vsag-bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vsag-bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vsag-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

LHT129 commented Jun 30, 2026 •

edited

Loading

mergify Bot commented Jun 30, 2026 •

edited

Loading

vsag-bot commented Jul 13, 2026 •

edited

Loading

vsag-bot commented Jul 21, 2026 •

edited

Loading