feat(profilers): GPU + link profiling with directional measurements

[AlexCheema]

Summary

• New active-probe profiler subsystem (src/exo/utils/profilers/) measuring GPU FP16 TFLOPS, GPU memory bandwidth, per-edge socket bandwidth (upload + download separately), socket RTT, per-edge RDMA bandwidth (rank0→rank1 and rank1→rank0 via mx.distributed.send/recv), and RDMA RTT.
• State-driven reconciliation loop (15 s tick, replay-safe) with TTLs: GPU 1 h, socket 5 min, RDMA 6 h. GPU and RDMA probes skip while runners are active; all RDMA work serialised through a process-global lock. Subprocess isolation for jaccl init keeps the worker process clean.
• Dashboard adds: a HuggingFace-style "GPU poor → GPU rich" bar (weighted blend of TFLOPS / memory bandwidth / memory against an 8× H100 anchor; FLOPS heaviest, memory least; per-dimension cap so a single insanely high axis still pulls the thumb right), per-node TFLOPS/bandwidth labels, a directional edge label (↑X ↓Y · Z RTT), and a hover tooltip with a per-direction breakdown including inferred connection type (Wi-Fi / Ethernet / Thunderbolt 4 / Thunderbolt 5 / RDMA — generation parsed from system_profiler's linkSpeed).
• GPU-bench variance fix: long warm-up to lock the Apple GPU into peak performance state, plus best-of-N timing. Two healthy M3 Ultras now agree to ~2% (was ~15% before).

Screenshots

Topology view (no hover)

Two-node M3 Ultra cluster over Thunderbolt 5 + Tailscale. The GPU rich bar shows 47.0 TFLOPS / 1.33 TB/s / 1.00 TB; the thumb sits in the middle yellow zone — driven by the maxed-out memory dimension while compute and bandwidth are well below an 8× H100 anchor. Per-node label under each Mac shows GPU profile (e.g. 23.6 TFLOPS · 661 GB/s). Edge label reads ↑74.77 Gbps ↓74.79 Gbps · 18 µs RTT.

Hovering an edge — directional breakdown tooltip

Each row is one measured profile (one direction × one transport). Connection type is inferred per row.

Implementation notes

• Why MLX mx.sum instead of GEMV for memory bandwidth. GEMV pays for an inner reduction plus a vector broadcast and caps at ~58 % of peak DRAM bandwidth on Apple Silicon. A pure streaming mx.sum over a 2 GB FP16 buffer (well past M3 Ultra's ~96 MB SLC) hits ~80 % — the closer-to-the-ceiling number that's useful for placement decisions.
• RDMA probe is subprocess-isolated. mx.distributed.init(backend="jaccl") is process-global; running it inside the worker would clash with active inference. Probe lives in rdma_probe_main.py, spawned via /profile/rdma_probe HTTP rendezvous.
• One-way bandwidth, RTT-only latency. Socket uploads use server-side timing (POST /profile/upload returns its own recv_duration_ms) so the small response's RTT doesn't pollute the result. RDMA uses mx.distributed.send/recv per direction. Latency stays as RTT — true one-way needs sub-µs clock sync that we don't have, and synthetic half-RTT split would be misleading.
• State-driven reconciliation, not event-reactive. EXO replays the event log on master changes; reactive probe triggers would re-fire historical events. The reconciler reads current state every tick and probes anything missing or past TTL.

Test plan

• uv run basedpyright && uv run ruff check && nix fmt && uv run pytest — all green (416 pass, 16 new tests in src/exo/utils/profilers/tests/).
• Two-node end-to-end on M3 Ultra cluster (mike + james over TB5 + Tailscale): GPU profile, socket bandwidth (upload + download), socket RTT, RDMA bandwidth (upload + download), RDMA RTT all populated; values stable across hourly probes.
• Connection-type inference: TB5 RDMA / Wi-Fi / Ethernet / Tailscale all classified correctly (Tailscale shows as "Unknown" by design — we don't fake CGNAT classification).
• Inference safety: starting a generation while a GPU/RDMA probe interval elapses skips the probe cleanly (no contention, no errors).
• Reviewer eyeball: confirm dashboard rendering on a multi-node cluster including hover behaviour.

🤖 Generated with Claude Code