Swarm camera-capacity benchmark + multi-GPU render backends - #13
Open
jonasbrami wants to merge 23 commits into
Open
Swarm camera-capacity benchmark + multi-GPU render backends#13jonasbrami wants to merge 23 commits into
jonasbrami wants to merge 23 commits into
Conversation
Design for a capacity-frontier benchmark: render-backend (cpu/intel/nvidia) x camera-resolution -> max sustainable drone count, gated on delivered camera FPS + RTF + flight liveness. Includes the NVIDIA headless-render investigation that unblocks the dGPU backend. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
9 bite-sized TDD tasks: expose cam_seq on /state, pure bench package (fps/verdict, tool parsers, knee search, frontier render), host sampler, RENDER_BACKEND + CAM_* launch knobs, orchestrator, e2e smoke. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Instead of returning the first matching engine, now aggregates all matching engines with max() to properly report utilization across multi-engine GPUs (e.g. Render/3D/0, Render/3D/1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…l probes, robust intel JSON
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…il, reap sampler, close handles Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… toolkit doc; document limiting_resource limitation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ults
- bench/run_bench.py: --quick demand-ladder mode ('best you can get' per
platform, time-boxed) + --world knob + incremental REPORT.md.
- sim/launch/swarm_sim.sh: nvidia case creates the glvnd EGL vendor ICD
in-container (the --gpus all toolkit injects libEGL_nvidia but not the ICD
json) -> fixes the ogre2 CreateRenderSystem segfault; + --headless-rendering.
- docs/benchmarks/quick-20260627T201936/: live baylands results (intel & nvidia
sustain >=6 drones @ 960x540 at RTF 1.0; cpu/llvmpipe not viable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…PUSH results Adds CLI knobs to climb the demand ladder past the 4-rung quick cap so a platform is driven to its real first FAIL instead of stopping while still passing: --ladder "NxWxH,..." custom rung sequence --run-cap K max rungs per backend (was hard-coded 4) --seed-idx I where on the ladder to seed the climb PUSH run (baylands, full-stack live agents, ~16 min): both Intel iGPU and NVIDIA dGPU ceiling at 10 drones @ 1280x720; both FAIL at 12. The 12-drone wall is CPU / single-thread Gazebo physics, not GPU render — but the dGPU degrades gracefully (RTF 0.81) where the iGPU collapses (RTF 0.10). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the per-run dirs (quick, push, smoke + dozens of JSON/JSONL/CSV artifacts and several REPORT.md files) into a single authoritative docs/benchmarks/RESULTS.md. It carries everything a future runner needs to resume without redoing work: the box's spec, the pass gate, the confirmed ceiling (10 drones @ 720p on both GPUs; CPU-bound at the wall), the 6@720p headroom corner, the dead ends (cpu/llvmpipe, the nvidia ICD fix), the false-positive limiting=dgpu, untested corners, and the exact reproduce/extend commands. Raw artifacts intentionally discarded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewrites the rendering section to cover all three render backends (intel / nvidia / cpu — NVIDIA now works via the in-container glvnd ICD fix, no longer a "future toggle") and adds a "Hardware & resolution knobs" table + examples under How to run: RENDER_BACKEND, CAM_W/CAM_H/CAM_FPS, WORLD, drone count. Cross-links docs/benchmarks/RESULTS.md (measured ceilings + re-benchmark harness) and docs/nvidia-render-investigation.md. Also folds in the run_mission author-tool description and drops the now-done NVIDIA roadmap bullet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jonasbrami
force-pushed
the
bench/capacity-benchmark
branch
from
June 28, 2026 12:40
134d3c5 to
e30078f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Answers "how many camera-carrying drones at what resolution can this box sustain, per render backend?" with a reproducible harness, and makes all three render paths (Intel iGPU / NVIDIA dGPU / CPU) first-class and documented.
Headline result (this dev box: i9-12900H, Iris Xe + RTX 3070 Ti)
Both GPUs cap at the same drone count → the high-end limiter is single-thread Gazebo physics, not the renderer. The dGPU's win is graceful degradation near the limit, not a higher count. Full write-up + re-benchmark commands:
docs/benchmarks/RESULTS.md.What's in here
bench/harness (pure-logic, ROS-free, 30 unit tests): FPS/verdict metrics, tool-output probes (nvidia-smi/intel_gpu_top/docker stats/gz-RTF) + limiting-resource attribution, knee-search, frontier table/heatmap, ~1 Hz host sampler, and therun_benchorchestrator (--quickdemand-ladder with--ladder/--run-cap/--seed-idx, full sweep, smoke).RENDER_BACKEND={cpu,intel,nvidia}+CAM_W/CAM_H/CAM_FPSknobs inswarm_sim.sh/run_swarm_demo.sh(defaults preserve old behaviour).--gpus allinjectslibEGL_nvidia.so.0but not the glvnd vendor ICD, so ogre2 segfaulted; the launcher now writes the ICD in-container (idempotent). Background indocs/nvidia-render-investigation.md.cam_seqon/stateso delivered FPS is measurable.Metric / pass gate
A config PASSES iff, over a measurement window after settle: min per-drone delivered cam FPS ≥ 0.9×rate AND Gazebo RTF ≥ 0.90 AND all N drones armed. Headroom (host/container CPU%, both GPUs, RAM) sampled ~1 Hz.
Test plan
uv run --extra dev python -m pytest tests/bench/→ 30 passed.Notes
RENDER_BACKENDdefaults tointel, cam defaults unchanged (640×360@10).--ladder.🤖 Generated with Claude Code