feat(io): add io_uring backend by LHT129 · Pull Request #2389 · antgroup/vsag

LHT129 · 2026-07-02T05:12:38Z

What

This PR adds an optional io_uring IO backend for Linux systems with liburing available.

Changes include:

Add UringIO, UringIOContext, and UringIOParameter.
Add CMake detection for liburing via ENABLE_LIBURING, HAVE_LIBURING, and NO_LIBURING.
Wire uring_io into IO parameter parsing, IO headers, flatten/bucket/graph datacell factories, and RabitQ split IO type validation.
Fall back from uring_io to buffer_io with a one-time warning when liburing is unavailable.
Add UringIO and UringIOParameter unit tests.

Both filtered test runs passed with 4 test cases and 16805 assertions.

Closes #1533

mergify · 2026-07-02T05:13:13Z

Merge Protections

🟢 All 3 merge protections satisfied — ready to merge.

Show 3 satisfied protections

🟢 Require kind label

label~=^kind/

🟢 Require version label

label~=^version/

🟢 Require linked issue for feature/bug PRs

body~=(?im)(?:^|[\s\-\*])(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s*:?\s+(?:#\d+|[\w.\-]+/[\w.\-]+#\d+|https?://github\.com/[\w.\-]+/[\w.\-]+/issues/\d+)

gemini-code-assist

Code Review

This pull request introduces support for the io_uring asynchronous I/O framework (uring_io) as an alternative I/O backend. It adds CMake configuration options to detect and link liburing, integrates the new I/O type into various datacell factories, and implements the UringIO class along with its parameters and unit tests. The feedback highlights several critical areas for improvement: a potential file descriptor and file leak in the UringIO constructor if initialization fails, a performance bottleneck caused by mixing O_DIRECT reads with buffered writes (requiring frequent fsync calls), error-prone manual resource management in MultiReadImpl that should leverage RAII, and the use of preprocessor macros for fallback types which should be replaced with clean C++ type aliases.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Copilot

Pull request overview

Adds an optional io_uring-based I/O backend for Linux builds, integrating it into IO parameter parsing and the major datacell factories, with CMake-based detection/linking of liburing and fallback behavior when unavailable.

Changes:

Introduces UringIO, UringIOContext, and UringIOParameter (plus unit tests) under HAVE_LIBURING.
Extends IO type parsing/dispatch and datacell factories to recognize "uring_io", with fallback-to-buffer behavior when liburing is missing.
Adds a build option (ENABLE_LIBURING) and CMake detection/linking for liburing.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
src/io/uring_io.h	Declares `UringIO` and provides a no-liburing alias fallback.
src/io/uring_io.cpp	Implements `UringIO` operations including multi-read using `io_uring`.
src/io/uring_io_test.cpp	Adds unit tests for `UringIO` behavior.
src/io/uring_io_parameter.h	Defines `UringIOParameter` with fallback behavior when `liburing` is absent.
src/io/uring_io_parameter.cpp	Implements JSON (de)serialization for `UringIOParameter`.
src/io/uring_io_parameter_test.cpp	Adds a unit test for `UringIOParameter` JSON round-tripping.
src/io/uring_io_context.h	Adds an `io_uring` context wrapper and pool type.
src/io/io_parameter.cpp	Adds `"uring_io"` parsing with one-time fallback warning when `liburing` is unavailable.
src/io/io_headers.h	Wires `uring_io.h` into the common IO header set.
src/io/CMakeLists.txt	Adds new sources and links `liburing` when available.
src/inner_string_params.h	Adds the `"uring_io"` IO type string constant.
src/datacell/rabitq_split_datacell.h	Allows `"uring_io"` as a known IO type for validation.
src/datacell/graph_interface.cpp	Enables graph datacell instantiation with `UringIO`.
src/datacell/flatten_interface.cpp	Enables flatten datacell instantiation with `UringIO`.
src/datacell/bucket_interface.cpp	Enables bucket datacell instantiation with `UringIO`.
cmake/VSAGSystemDeps.cmake	Detects `liburing` and defines `HAVE_LIBURING`/`NO_LIBURING`.
cmake/VSAGOptions.cmake	Adds `ENABLE_LIBURING` option.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

vsag-bot · 2026-07-20T06:36:42Z

Automated pull request review completed.

Review effort: high (775 changed lines across 19 files).

Submitted 4 inline comments.
Review: #2389 (review)

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/io/uring_io/uring_io_test.cpp:56

The UringIO parameter test doesn’t assert the intended behavior when liburing is missing (factory should fall back to BufferIOParameter and ToJson should reflect the fallback). Adding a compile-time conditional check here (like async_io_test.cpp does) will prevent this test from silently passing while not validating the fallback path.

    auto json = JsonType::Parse(fmt::format(param_str, path));
    auto io_param = IOParameter::GetIOParameterByJson(json);
    IndexCommonParam common_param;
    common_param.allocator_ = allocator;
    auto io = std::make_unique<UringIO>(io_param, common_param);
    TestBasicReadWrite(*io);

vsag-bot

Automated inline review completed.

Review effort: high (775 changed lines across 19 files).
Submitted 4 inline comments.

Reviewed commit 9af05eb.

Add an optional UringIO backend with liburing detection, IO parameter parsing, datacell factory wiring, and unit coverage for both liburing and fallback paths. Address review feedback: - Add missing <mutex>, <cerrno>, <cstdlib>, <fmt/format.h> includes - Replace malloc/free with allocator_->Allocate/Deallocate for consistency - Fix potential integer overflow in offset bounds checks - Add uring_io.h to io_headers.h aggregation - Make ReleaseImpl non-static to access allocator Signed-off-by: LHT129 <tianlan.lht@antgroup.com> Assisted-by: OpenCode

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 5 comments.

+        int submitted = io_uring_submit(ring);
+        if (submitted < 0) {
+            for (auto& obj : objs) {
+                ReleaseUringReadObject(obj);
+            }
+            ctx_guard.Abandon();
+            throw VsagException(ErrorType::INTERNAL_ERROR,
+                                fmt::format("io_uring_submit failed: {}", strerror(-submitted)));
+        }
+
+        int remaining_to_submit = static_cast<int>(prepared) - submitted;
+        while (remaining_to_submit > 0) {
+            int ret = io_uring_submit(ring);
+            if (ret <= 0) {
+                break;
+            }
+            submitted += ret;
+            remaining_to_submit -= ret;
+        }
+
+        int first_error = 0;
+        bool has_short_read = false;
+        uint64_t completed = 0;
+        while (completed < static_cast<uint64_t>(submitted)) {


+    void
+    Abandon() {
+        static std::mutex abandon_mutex;
+        static std::vector<std::shared_ptr<UringIOContext>> abandoned;
+        std::lock_guard<std::mutex> lock(abandon_mutex);
+        abandoned.push_back(std::move(ctx_));
+        ctx_ = nullptr;
+    }


+UringIO::ResizeImpl(uint64_t size) {
+    auto ret = IOSyscall::FTruncate(this->wfd_, size);
+    if (ret == -1) {
+        throw VsagException(ErrorType::INTERNAL_ERROR, "ftruncate failed");
+    }
+    this->size_ = size;
+}


+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "io/uring_io/uring_io.h"


+UringIO::UringIO(const IOParamPtr& param, const IndexCommonParam& common_param)
+    : UringIO(std::dynamic_pointer_cast<UringIOParameter>(param), common_param) {
+}


Copilot AI review requested due to automatic review settings July 2, 2026 05:12

LHT129 requested review from inabao, jiaweizone and wxyucs as code owners July 2, 2026 05:12

LHT129 added kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 version/0.18 1. Major pyramid upgrade 2. HGraph 3. Observability 1. pyramid大升级 2. HGraph增强 3. 可观测性增强 labels Jul 2, 2026

pull-request-size Bot added the size/XL label Jul 2, 2026

Copilot started reviewing on behalf of LHT129 July 2, 2026 05:13 View session

gemini-code-assist Bot reviewed Jul 2, 2026

View reviewed changes

Comment thread src/io/uring_io/uring_io.cpp

Comment thread src/io/uring_io/uring_io.cpp

Comment thread src/io/uring_io.cpp Outdated

Comment thread src/io/uring_io/uring_io.h

Comment thread src/io/uring_io/uring_io_parameter.h

LHT129 added version/1.0 and removed version/0.18 1. Major pyramid upgrade 2. HGraph 3. Observability 1. pyramid大升级 2. HGraph增强 3. 可观测性增强 labels Jul 2, 2026

Copilot AI reviewed Jul 2, 2026

View reviewed changes

LHT129 force-pushed the feat/io-uring-support branch from 3b1b211 to da3ff4d Compare July 2, 2026 05:39

LHT129 self-assigned this Jul 2, 2026

Copilot AI review requested due to automatic review settings July 2, 2026 06:21

LHT129 force-pushed the feat/io-uring-support branch from da3ff4d to 8ca7877 Compare July 2, 2026 06:21

Copilot started reviewing on behalf of LHT129 July 2, 2026 06:25 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread src/io/uring_io.cpp Outdated

Comment thread src/io/uring_io.cpp Outdated

Comment thread src/io/uring_io.cpp Outdated

LHT129 force-pushed the feat/io-uring-support branch 2 times, most recently from 640f09c to acfe848 Compare July 2, 2026 07:07

Copilot AI review requested due to automatic review settings July 2, 2026 07:07

Copilot started reviewing on behalf of LHT129 July 2, 2026 07:19 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread src/io/uring_io.cpp Outdated

Comment thread src/io/uring_io.cpp

LHT129 force-pushed the feat/io-uring-support branch from acfe848 to c857bb3 Compare July 2, 2026 07:28

Copilot AI review requested due to automatic review settings July 2, 2026 11:29

LHT129 force-pushed the feat/io-uring-support branch from c857bb3 to 273d1d1 Compare July 2, 2026 11:29

Copilot started reviewing on behalf of LHT129 July 2, 2026 11:36 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread src/io/uring_io.cpp

Comment thread src/io/uring_io/uring_io_context.h Outdated

LHT129 force-pushed the feat/io-uring-support branch from 273d1d1 to 0481c6b Compare July 3, 2026 03:07

LHT129 force-pushed the feat/io-uring-support branch from a4e3e84 to bb6fe09 Compare July 14, 2026 11:24

Copilot started reviewing on behalf of LHT129 July 14, 2026 11:24 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 15, 2026 06:51

LHT129 force-pushed the feat/io-uring-support branch from bb6fe09 to 0c6d81c Compare July 15, 2026 06:51

Copilot started reviewing on behalf of LHT129 July 15, 2026 06:51 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread src/io/uring_io/uring_io.cpp

Comment thread src/io/uring_io/uring_io_parameter.cpp

Copilot AI review requested due to automatic review settings July 15, 2026 07:14

LHT129 force-pushed the feat/io-uring-support branch from 0c6d81c to aff8f32 Compare July 15, 2026 07:14

Copilot started reviewing on behalf of LHT129 July 15, 2026 07:15 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread src/datacell/flatten_interface.cpp Outdated

Comment thread src/datacell/bucket_interface.cpp Outdated

Comment thread src/datacell/graph_interface.cpp Outdated

Copilot AI review requested due to automatic review settings July 15, 2026 09:53

LHT129 force-pushed the feat/io-uring-support branch from aff8f32 to e793549 Compare July 15, 2026 09:53

Copilot started reviewing on behalf of LHT129 July 15, 2026 09:53 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread src/io/uring_io/uring_io.cpp

Comment thread src/datacell/graph_interface.cpp

Comment thread src/datacell/flatten_interface.cpp

Comment thread src/datacell/bucket_interface.cpp

Copilot AI review requested due to automatic review settings July 16, 2026 06:53

LHT129 force-pushed the feat/io-uring-support branch from e793549 to 0f49e96 Compare July 16, 2026 06:53

Copilot started reviewing on behalf of LHT129 July 16, 2026 06:53 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread src/io/uring_io/uring_io.cpp

Copilot AI review requested due to automatic review settings July 20, 2026 06:36

LHT129 force-pushed the feat/io-uring-support branch from 0f49e96 to 27c575e Compare July 20, 2026 06:36

vsag-bot self-requested a review July 20, 2026 06:36

Copilot started reviewing on behalf of LHT129 July 20, 2026 06:36 View session

Copilot AI reviewed Jul 20, 2026

View reviewed changes

Comment thread src/io/uring_io/uring_io.cpp

Comment thread src/io/uring_io/uring_io_test.cpp Outdated

Comment thread src/io/uring_io/uring_io_test.cpp Outdated

Comment thread src/io/uring_io/uring_io_test.cpp Outdated

Comment thread src/io/uring_io/uring_io_parameter_test.cpp Outdated

Copilot AI review requested due to automatic review settings July 20, 2026 06:58

Copilot AI reviewed Jul 20, 2026

View reviewed changes

vsag-bot reviewed Jul 20, 2026

View reviewed changes

Comment thread src/io/uring_io/uring_io.cpp

Comment thread src/io/uring_io/uring_io.cpp Outdated

Comment thread src/io/uring_io/uring_io.cpp Outdated

Comment thread src/io/uring_io/uring_io.cpp

Copilot AI reviewed Jul 20, 2026

View reviewed changes

Uh oh!

Conversation

LHT129 commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Uh oh!

mergify Bot commented Jul 2, 2026

Merge Protections

🟢 Require kind label

🟢 Require version label

🟢 Require linked issue for feature/bug PRs

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

LHT129 commented Jul 2, 2026 •

edited

Loading

vsag-bot commented Jul 20, 2026 •

edited

Loading