Updates to RDMA bench profiling

[cliffburdick]

No description provided.

[greptile-apps]

Greptile Summary

Adds benchmark profiling infrastructure (token-bucket pacer, rate-limiting flag, structured stdout output, DGX Spark sweep drivers) and applies several RDMA manager tuning changes.

• RDMA manager: switches to std::map for ordered WR tracking with upper_bound predecessor sweeps; adds complete_send_wrs to retire unsignaled WRs on any CQE (success or error); enlarges pools to 2047 entries; adds relaxed-ordering MR registration with fallback; raises MR alignment to max(existing, GPU_PAGE_SIZE); fixes a pool leak in get_tx_packet_burst partial-failure paths.
• RDMA bench: raises kMaxOutstanding from 5 to 64 with backpressure replacing the spin-wait; drains completions before posting; uses free_tx_metadata (not free_tx_burst) on pre-packet-allocation failure.

Confidence Score: 5/5

Safe to merge; the three previously-flagged correctness issues (double-free, missing error-path sweep, undocumented SPSC invariant) are all addressed in this commit stack.

The RDMA thread changes are substantial but well-contained: the selective-signaling predecessor sweep correctly uses std::map::upper_bound, error CQEs now call complete_send_wrs before continuing, and the SPSC ring assumption is documented. The benchmark layer changes are additive and low-risk.

The complete_send_wrs lambda in src/managers/rdma/daqiri_rdma_mgr.cpp is the highest-density changed path and worth a second human read, particularly the force-signal threshold and the two-case CRITICAL logging behavior.

Important Files Changed

Filename	Overview
src/managers/rdma/daqiri_rdma_mgr.cpp	Major RDMA thread overhaul: selective signaling, std::map for ordered WR tracking, complete_send_wrs lambda, SPSC rings, pool enlargements, relaxed-ordering MR fallback, and fixed get_tx_packet_burst error paths.
examples/rdma_bench.cpp	kMaxOutstanding raised 5→64 with backpressure instead of spin-wait; completion drain loop; TokenBucketPacer; free_tx_metadata on pre-packet-allocation failure. Free contract looks correct.
examples/socket_bench.cpp	Adds TokenBucketPacer, byte counters, structured output, and fixes a latent bug where iterations==0 would immediately exit the send/recv loop instead of running time-bounded.
examples/raw_bench_common.cpp	TokenBucketPacer implementation with chunked 10 ms sleep for stop-flag responsiveness and single-write stringstream stdout to prevent RX/TX line interleaving.
examples/run_spark_bench.sh	New DGX Spark sweep driver with per-backend payload/batch matrices, drop-source dispatch, CPU/GPU counter capture, and one CSV row per cell. RDMA and socket stdout parsing is correctly keyed by backend type.
examples/CMakeLists.txt	raw_bench_common.cpp and CUDA::cudart linked into rdma_bench and socket_bench; BUILD_RPATH added to socket_bench to fix runtime libdaqiri.so discovery.

Sequence Diagram

sequenceDiagram
    participant App as Application Thread
    participant TxRing as tx_ring (SPSC)
    participant RDMA as rdma_thread
    participant RxRing as rx_ring (SPSC)
    participant CQ as TX Completion Queue

    App->>App: create_burst_params() + rdma_set_header()
    App->>App: get_tx_packet_burst() [backpressure if pool empty]
    App->>App: set_packet_lengths() + send_tx_burst()
    App->>TxRing: "enqueue(BurstParams*)"
    RDMA->>TxRing: "dequeue(BurstParams*)"
    RDMA->>RDMA: "ibv_post_send() [signal every N or when map>=threshold]"
    RDMA->>RDMA: "outstanding_send_wr_ids[wr_id+p] = burst"
    CQ-->>RDMA: IBV_WC_SEND (signaled CQE)
    RDMA->>RDMA: complete_send_wrs(wr_id, status) upper_bound sweep
    RDMA->>RxRing: "enqueue each swept BurstParams*"
    App->>RxRing: get_rx_burst() completion
    App->>App: free_tx_burst(completion)

Loading

_{Reviews (4): Last reviewed commit: "#15 - Retire RDMA send completions on CQ..." | Re-trigger Greptile}

[cliffburdick]

@greptile review

[dleshchev]

Clarifying stack repair state as of 2026-06-03: this PR shows MERGED because it was merged into the stacked branch 15-bench-spark-tooling, not because merge commit 94a7d2efe294c8e8ccfa91ee5afd1d59841f0719 was the final mainline landing path. The main-bound RDMA profiling content has been rebuilt on top of current main in follow-up PR #118.

Cleanup note: once PR #118 lands, the stale stack branches under 15-bench-* / cburdick/* can be pruned. Keep backup/15-bench-infra-stacked-20260603 until #118 is safely landed.