Skip to content

#15 - Add Spark socket wire benchmark notes#110

Closed
cliffburdick wants to merge 2 commits into
mainfrom
cburdick/socket-wire-benchmarks
Closed

#15 - Add Spark socket wire benchmark notes#110
cliffburdick wants to merge 2 commits into
mainfrom
cburdick/socket-wire-benchmarks

Conversation

@cliffburdick
Copy link
Copy Markdown
Collaborator

This PR documents how to run Linux networking loopback tests (RDMA/Sockets) on the DGX Spark platform. Without configuring the system like this the benchmarks are not realistic since the traffic is looped through Linux rather than leaving on the wire.

Signed-off-by: Cliff Burdick <cburdick@nvidia.com>
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Jun 1, 2026

Greptile Summary

This PR adds a new SPARK_SOCKET_WIRE_README.md documenting how to configure Linux network namespaces and verify PHY-level counters for realistic DAQIRI socket benchmarks on a DGX Spark machine, alongside three small companion fixes to the socket manager.

  • Documentation: step-by-step namespace setup, PHY-counter verification with ethtool -S, verified benchmark results table, and a restore-networking script for the tested enp1s0f0np0 → enp1s0f1np1 wire pair.
  • UDP size cap: send_udp_burst now rejects any packet exceeding kMaxUdpPayloadBytes (65507) before calling sendmmsg; the burst is still freed unconditionally in send_tx_burst.
  • TCP dead-connection guards: send_tx_burst adds conn->running.load() to its null check, and socket_connect_to_server short-circuits with the existing connection only after confirming running.load() is true.

Confidence Score: 5/5

Safe to merge; the code changes are small, targeted, and all three logic paths preserve the burst-free contract.

The UDP size check is a pure validation gate that fires before any I/O and leaves the burst lifecycle untouched. The conn->running.load() additions close the window where a TX call could race against a just-exited RX thread. The connection-reuse path in socket_connect_to_server correctly checks liveness before short-circuiting. No new invariants are broken and no new data paths are introduced.

No files require special attention.

Important Files Changed

Filename Overview
SPARK_SOCKET_WIRE_README.md New documentation file describing Linux namespace setup and PHY-counter verification for Spark socket wire benchmarks; no code logic changes.
src/managers/socket/daqiri_socket_mgr.cpp Three focused fixes: UDP payload size cap at 65507 bytes, dead-connection guard added to send_tx_burst TCP path (conn->running.load()), and TCP connection reuse with live-state check in socket_connect_to_server; removal of self-erase from tcp_rx_loop is compensated by the reuse guard. Burst is freed unconditionally at end of send_tx_burst even when validation short-circuits.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[send_tx_burst] --> B{protocol?}
    B -- UDP --> C[send_udp_burst]
    C --> D{any pkt_len > 65507?}
    D -- yes --> E[return false / CONNECT_FAILURE]
    D -- no --> F[sendmmsg loop]
    F --> G[update tx_pkts / tx_bytes]
    B -- TCP --> H{conn == null OR NOT running?}
    H -- yes --> I[CONNECT_FAILURE]
    H -- no --> J[send_tcp_burst]
    J --> G
    G --> K[free_all_packets + free_tx_burst]
    E --> K
    I --> K

    subgraph socket_connect_to_server
        L[primary_conn_id set AND IP/port match?] -- yes --> M{connections_ find + running.load?}
        M -- alive --> N[return existing conn_id]
        M -- dead or missing --> O[create_tcp_client_connection]
        L -- no --> O
    end
Loading

Reviews (2): Last reviewed commit: "#15 - Guard TCP socket connection reuse" | Re-trigger Greptile

Comment thread src/managers/socket/daqiri_socket_mgr.cpp
Comment thread src/managers/socket/daqiri_socket_mgr.cpp
Signed-off-by: Cliff Burdick <cburdick@nvidia.com>
Copy link
Copy Markdown
Collaborator

@dleshchev dleshchev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like dead connections can be still an issue; also the .md file needs to be placed somewhere more appropriate.

@@ -0,0 +1,414 @@
# Spark Socket Wire Benchmark Notes
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this file live in tutorials?

CLIENT_IF=enp1s0f0np0
SERVER_IF=enp1s0f1np1

CLIENT_IP=10.250.0.1
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be needs a note that this is device-specific?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tutorial is specific to spark if that's what you mean


The socket benchmark config must use the namespace IPs. Use a large iteration
count because current `daqiri_bench_socket` treats `iterations: 0` as zero work,
not as "run until --seconds expires".
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably need an issue for that

Comment thread src/managers/socket/daqiri_socket_mgr.cpp
@cliffburdick
Copy link
Copy Markdown
Collaborator Author

Added into PR #114

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants