#15 - Add DGX Spark sweep tooling and RDMA loopback prereqs

[dleshchev]

Rebuilds the Spark tooling stack item as a clean follow-up after PR #96 landed on main.\n\nChanges:\n- Adds the Spark benchmark sweep wrapper and data-fill driver.\n- Adds the Spark RDMA loopback setup helper with current p0-to-p1 defaults.\n- Raises RDMA bench outstanding depth to match the YAML buffer depth.\n- Keeps failed/malformed bench cells from being emitted as successful zero rows.\n\nValidation:\n- bash -n examples/run_spark_bench.sh\n- bash -n scripts/spark_data_fill.sh\n- bash -n scripts/setup_spark_rdma_loopback.sh\n- python3 scripts/check_doc_refs.py\n- git diff --check origin/main...HEAD\n\nLocal hardware benchmark/build not run in this shell; the repo requires compile/run inside the project container with the Spark NIC/GPU environment.

[greptile-apps]

Greptile Summary

Adds the DGX Spark benchmark sweep stack on top of the base infra merged in PR #96: a sweep wrapper (run_spark_bench.sh), a one-shot data-fill driver (spark_data_fill.sh), and an idempotent RDMA loopback setup script (setup_spark_rdma_loopback.sh), plus a kMaxOutstanding bump in rdma_bench.cpp to match the YAML buffer depth.

• examples/run_spark_bench.sh: Sweeps payload × batch × target-gbps across DPDK, RDMA, and socket backends; emits one CSV row per successful cell and propagates failures without producing false zero rows.
• scripts/spark_data_fill.sh: Drives the DPDK/socket sweep and drop-curve modes with hugepage pre-flight, inter-run cleanup, and live log streaming via tee.
• scripts/setup_spark_rdma_loopback.sh: Sets up the p0↔p1 RDMA loopback with per-port routing tables and static ARP entries; safe to re-run.

Confidence Score: 5/5

The changes are additive tooling scripts and a one-line constant bump; they do not touch the library core, the Manager vtable, or the BurstParams contract.

All five changed files are new shell scripts or a trivial constant change in an example binary. The kMaxOutstanding bump is well-documented with the pool-drain constraint inline. The scripts handle failure isolation correctly and do not emit false zero rows. DCO sign-offs and commit format are both present.

No files require special attention; the only finding is a misleading header comment in examples/run_spark_bench.sh.

Important Files Changed

Filename	Overview
.gitignore	Adds bench-results/ output directory and pcie_schematic.png artifact to gitignore; clean and correct.
examples/rdma_bench.cpp	Raises kMaxOutstanding from 5 to 20 to match the YAML num_bufs, with an inline comment documenting the pool-drain constraint and the known structural limitation (interleaving drain/post is deferred).
examples/run_spark_bench.sh	New Spark sweep wrapper; failure cells correctly return nonzero instead of emitting false zero rows. Header comment for RX_IFACE is misleading — the variable is unused in this script and /proc/net/udp provides no per-interface filtering.
scripts/setup_spark_rdma_loopback.sh	New idempotent RDMA loopback setup script; correctly flushes routes/rules before re-adding, reads MACs from sysfs with env overrides, and uses per-port routing tables.
scripts/spark_data_fill.sh	One-shot data-fill driver with pre-flight hugepage checks, orphan-hugepage cleanup between runs, and correct PIPESTATUS capture for pipeline exit-code propagation.

Sequence Diagram

sequenceDiagram
    participant User
    participant DataFill as spark_data_fill.sh
    participant Wrapper as run_spark_bench.sh
    participant EnvCapture as bench_capture_environment.sh
    participant Bench as BenchBinary
    participant CSV

    User->>DataFill: run with backends
    DataFill->>DataFill: preflight hugepages, MAC, carrier
    loop each backend x mode
        DataFill->>DataFill: clean_orphan_hugepages
        DataFill->>Wrapper: backend mode
        Wrapper->>EnvCapture: capture env state
        loop each cell payload x batch x target_gbps
            Wrapper->>Wrapper: snapshot udp/cpu/dmon
            Wrapper->>Bench: execute with generated YAML
            Bench-->>Wrapper: stdout stats + stderr drops
            alt cell succeeded
                Wrapper->>CSV: append row
            else cell failed
                Wrapper->>Wrapper: FAILURES++
            end
        end
        Wrapper-->>DataFill: exit status
        DataFill->>DataFill: clean_orphan_hugepages
    end
    DataFill-->>User: summary and result dirs

Loading

_{Reviews (2): Last reviewed commit: "#15 - Add DGX Spark sweep tooling and RD..." | Re-trigger Greptile}

[greptile-apps]

Socket-udp BATCHES_SWEEP never varies the config

BATCHES_SWEEP=(256 32 1) is declared for socket-udp, but generate_yaml for both socket-udp and socket-tcp only substitutes message_size — the queue-level batch_size: 1 in daqiri_bench_socket_udp_tx_rx.yaml is never updated. Every element of BATCHES_SWEEP therefore produces an identical YAML, so the CSV will contain three rows per payload with different batch values but indistinguishable bench configs and throughput numbers. Either add a batch_size substitution leg to generate_yaml for socket backends (matching the DPDK -e "s|^( *batch_size: ).*|\1$batch|" pattern), or collapse BATCHES_SWEEP=(1) for socket-udp to make the sweep intent explicit.

[greptile-apps]

Doc-sync gap: examples/rdma_bench.cpp change not reflected in docs

Per the project doc-sync rule, any change to examples/*.cpp requires updating docs/tutorials/benchmarking_examples.md, docs/tutorials/configuration-walkthrough.md, and the benchmark table in AGENTS.md in the same PR. The new kMaxOutstanding value and its deadlock constraint are meaningful to anyone tuning RDMA buffer depth, and run_spark_bench.sh adds a new benchmark entry-point that both docs should mention. None of those three files are touched in this PR.

Rule Used: DAQIRI has no automated doc-sync gate beyond mkdoc... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!