feat(iterator): add DocIterator for full collection traversal#597
Open
YongqiYin wants to merge 1 commit into
Open
feat(iterator): add DocIterator for full collection traversal#597YongqiYin wants to merge 1 commit into
YongqiYin wants to merge 1 commit into
Conversation
YongqiYin
requested review from
Cuiyus,
chinaux,
feihongxu0824 and
zhourrr
as code owners
July 16, 2026 12:53
There was a problem hiding this comment.
Pull request overview
This PR introduces a full-collection document iterator (“DocIterator”) across the C++ core, C API, and Python bindings to enable streaming traversal/export without relying on large topk queries (relates to #380).
Changes:
- Add C++
Collection::CreateIterator()andDocIteratorwith snapshot isolation, segment concatenation, and delete filtering. - Expose the iterator via the C API (
zvec_collection_create_iterator/next/close) and Python (Collection.iter_docs()generator). - Add C++/C/Python tests covering basic iteration, delete filtering, field selection, and concurrency/isolation.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/db/iterator_test.cc | New C++ unit/integration/concurrency/perf tests for full traversal. |
| tests/c/c_api_test.c | Adds C API iterator tests and option handling coverage. |
| src/include/zvec/db/options.h | Introduces IteratorOptions (output_fields/include_vector). |
| src/include/zvec/db/doc_iterator.h | Public C++ DocIterator interface. |
| src/include/zvec/db/collection.h | Adds Collection::CreateIterator() API. |
| src/include/zvec/c_api.h | Adds public C iterator and iterator-options API. |
| src/db/index/segment/filtering_reader.h | New Arrow reader wrapper to filter deleted docs. |
| src/db/index/segment/concatenating_reader.h | New Arrow reader to concatenate readers across segments. |
| src/db/doc_iterator.cc | Implements row-by-row Doc materialization + optional vector prefetch. |
| src/db/doc_iterator_internal.h | Internal DocIterator::Impl definition (lifetime ordering, caches). |
| src/db/collection.cc | Implements iterator creation, snapshot scan, and reader chain construction. |
| src/binding/python/model/python_collection.cc | Exposes _DocIterator + Collection.CreateIterator() to Python. |
| src/binding/python/include/python_collection.h | Declares bind_iterator() hook. |
| src/binding/c/c_api.cc | Implements iterator options + iterator handles for the C API. |
| python/zvec/model/collection.py | Adds Collection.iter_docs() streaming generator. |
| python/tests/test_iter_docs.py | New Python tests for iterator behavior and isolation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+15
to
+18
| #include <atomic> | ||
| #include <chrono> | ||
| #include <thread> | ||
| #include <gtest/gtest.h> |
|
|
||
| Result<DocIterator::Ptr> CollectionImpl::CreateIterator( | ||
| const IteratorOptions &options) { | ||
| CHECK_COLLECTION_READONLY_RETURN_STATUS_EXPECTED; |
Comment on lines
+201
to
+205
| auto uid_array = | ||
| std::dynamic_pointer_cast<arrow::StringArray>(batch.column(uid_col)); | ||
| if (uid_array) { | ||
| doc->set_pk(uid_array->GetScalar(row).ValueOrDie()->ToString()); | ||
| } |
Comment on lines
+229
to
+233
| auto str_array = std::dynamic_pointer_cast<arrow::StringArray>(array); | ||
| if (str_array) { | ||
| doc->set(field->name(), | ||
| str_array->GetScalar(row).ValueOrDie()->ToString()); | ||
| } |
Comment on lines
+6914
to
+6919
| auto *ptr = reinterpret_cast<zvec::IteratorOptions *>(options); | ||
| // NULL output_fields means "return all fields" (nullopt). | ||
| if (output_fields == nullptr || count == 0) { | ||
| ptr->output_fields_ = std::nullopt; | ||
| return ZVEC_OK; | ||
| } |
Comment on lines
+398
to
+402
| iterator = self._obj.CreateIterator(output_fields, include_vector) | ||
| for core_doc in iterator: | ||
| py_doc = convert_to_py_doc(core_doc, self.schema) | ||
| if py_doc is not None: | ||
| yield py_doc |
Comment on lines
+5897
to
+5902
| size_t ok_count = 0, err_count = 0; | ||
| zvec_collection_insert(collection, (const zvec_doc_t **)docs, 1, &ok_count, | ||
| &err_count); | ||
| zvec_doc_destroy(doc); | ||
| } | ||
| zvec_collection_flush(collection); |
| const char *fields[] = {"id"}; | ||
| err = zvec_iterator_options_set_output_fields(opts, fields, 1); | ||
| TEST_ASSERT(err == ZVEC_OK); | ||
| zvec_iterator_options_set_include_vector(opts, false); |
YongqiYin
force-pushed
the
feat/doc-iterator
branch
from
July 16, 2026 13:01
b596564 to
4aca7cb
Compare
Collaborator
Author
|
@copilot resolve the merge conflicts in this pull request |
|
|
YongqiYin
force-pushed
the
feat/doc-iterator
branch
from
July 16, 2026 16:53
4aca7cb to
feef1d7
Compare
Comment on lines
+2201
to
+2205
| // Create DocIterator, keeping segments + delete_store alive | ||
| auto impl = std::make_unique<DocIterator::Impl>(); | ||
| impl->reader = reader_result.value(); | ||
| impl->schema = schema_; | ||
| impl->segments = std::move(segments); |
Comment on lines
+129
to
+136
| void DocIterator::Close() { | ||
| if (impl_) { | ||
| impl_->closed = true; | ||
| impl_->reader.reset(); | ||
| impl_->current_batch.reset(); | ||
| impl_->vector_cache_.clear(); | ||
| } | ||
| } |
Comment on lines
+175
to
+183
| uint32_t seg_doc_id = | ||
| static_cast<uint32_t>(ga->Value(i) - min_doc_id); | ||
| auto fr = indexer->Fetch(seg_doc_id); | ||
| if (fr.has_value()) { | ||
| bufs.push_back(std::move(fr.value())); | ||
| } else { | ||
| bufs.emplace_back(); | ||
| } | ||
| } |
YongqiYin
force-pushed
the
feat/doc-iterator
branch
from
July 17, 2026 04:49
feef1d7 to
96273fc
Compare
Add streaming full-collection traversal across C++/C/Python: - C++: Collection::CreateIterator + DocIterator (isolated Flush+snapshot scan, ConcatenatingReader across segments, FilteringReader for deletes, batch-prefetched vectors) - C API: zvec_collection_create_iterator/next/close + iterator options - Python: collection.iter_docs() generator (constant memory) - Tests: C++ (unit/integration/concurrency/perf), C API, Python Relates to alibaba#380
YongqiYin
force-pushed
the
feat/doc-iterator
branch
from
July 17, 2026 06:04
96273fc to
4f542e0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add streaming full-collection traversal across C++/C/Python:
Relates to #380