liburing 6.12 - Incremental buffer feature - reusing ring buffers

[MDA2AV]

Problem

IOU_PBUF_RING_INC (kernel 6.12+) lets the kernel partially consume a buffer across multiple recv CQEs. But the buffer ring was shared across all connections on a
reactor — so data from different connections would interleave in the same buffer, corrupting everything.

Solution

When IncrementalBufferConsumption = true, each connection gets its own buffer ring with a unique buffer group ID (bgid). When false, the shared reactor ring works
exactly as before (zero behavior change).

File	What
ReactorConfig.cs	Added ConnectionBufferRingEntries = 128 (per-connection ring size)
MpscUlongQueue.cs	New — MPSC queue for ulong, carries packed (fd, bid) pairs for incremental buffer returns
Connection.cs	Added per-connection ring fields: BufRing, BufRingSlab, Bgid, IncrementalMode, BufRefCounts, BufKernelDone, BufCumulativeOffset.
Updated Clear() and Dispose()
Connection.Read.LowLevelApi.cs	ReturnRing() routes to incremental queue when IncrementalMode is set
Engine.Reactor.cs	GID allocator (_freeGids stack), SetupConnectionBufRing/TeardownConnectionBufRing, ReturnConnectionBuffer (with offset reset),
EnqueueReturnQIncremental/DrainReturnQIncremental, updated InitRing() and CloseAll()
Engine.Reactor.Handle.cs	Per-connection ring setup on new connection, CQE processing with cumulative offset tracking, teardown on close
Engine.Reactor.HandleSubmitAndWaitCqe.cs	Same changes
Engine.Reactor.HandleSubmitAndWaitSingleCall.cs	Same changes

Key mechanics

• GID allocator: Stack of pre-populated GIDs 2..MaxConnections+1 (GID 1 reserved for shared ring). Pop on connect, push on close.
• Cumulative offset tracking: With IOU_PBUF_RING_INC, successive CQEs write at increasing offsets within the same buffer. BufCumulativeOffset[bid] tracks where each
  CQE's data starts. Reset to 0 when buffer is returned.
• Ref counting: BufRefCounts[bid] tracks how many RingItems the handler hasn't returned yet. BufKernelDone[bid] marks when the kernel is done with the buffer (no
  IORING_CQE_F_BUF_MORE). Buffer returns to ring only when both refcount == 0 AND kernel done.
• Return path: Handler calls ReturnRing(bid) → packs (fd, bid) into MpscUlongQueue → reactor drains, looks up connection, decrements refcount, conditionally returns
  buffer to kernel ring.
• Handler code unchanged: ReturnRing(bid) routes internally based on mode. All examples/handlers work as-is.

Validated

99.9% buffer reuse ratio under pipelined HTTP load (~675 recv completions per 32KB buffer on average).

[MDA2AV]

6.12 incremental buffers experimental feature added..