Skip to content

docs(examples): add GB10/DGX Spark NVFP4 optimized sweep specs#21

Merged
dutifulbob merged 1 commit into
mainfrom
docs/gb10-optimized-specs
Jul 7, 2026
Merged

docs(examples): add GB10/DGX Spark NVFP4 optimized sweep specs#21
dutifulbob merged 1 commit into
mainfrom
docs/gb10-optimized-specs

Conversation

@dutifulbob

Copy link
Copy Markdown
Member

Opened on behalf of Onur Solmaz (osolmaz).

Summary

We tuned how vLLM is launched for four NVFP4 models on a GB10 (DGX Spark) and got noticeably higher throughput, but that config only lived in local run dirs.
This adds those tuned sweep specs to the repo as a versioned example, plus a README explaining what changed and why, and a record of the old vs new serve command per model.
It matters because the original hand-authored specs launched vLLM in a way that quietly hurt decode, and now the better config is reproducible instead of lost.

What Changed

Adds a new example directory with the four tuned specs and documentation. No library code changes.

  • examples/gb10-dgx-spark-nvfp4-optimized/ with qwen27b.json, qwen35b.json, gemma4-26b.json, gemma4-31b.json.
  • README.md explaining the tuning: drop --mamba-cache-mode none (the Qwen models are hybrid linear-attention and that flag disabled caching the recurrent state), drop --enforce-eager (enables CUDA graphs), drop --no-async-scheduling, and raise --max-num-batched-tokens to 16384.
  • PROFILES-old-vs-new.txt recording each model's old serve command vs the tuned one.
  • The README also notes two tested-but-not-adopted levers: MTP (helps single-stream decode ~1.6x but hurts aggregate, so off) and flashinfer autotune (no throughput gain on sm_121).

Testing

Confirmed the specs are valid and the docs pass the repo's checks. Behavior was measured on the real machine before committing.

  • All four specs round-trip localperf bench plan (39/43/39/39 planned runs).
  • npx @simpledoc/simpledoc check — clean.
  • go build ./... — ok (no code changed).
  • Live: ran the tuned sweeps end to end; qwen27b aggregate output rose ~20-30% at concurrency (4k-ref c32 157 -> 186 tok/s, decode-8k c32 61 -> 80, decode-16k c16 31 -> 40).

Risks

Low — this is example data and docs, no code paths change.

  • The specs are machine-specific: runner.vllm_command, env (CUTE_DSL_ARCH=sm_121a), and ports point at one host. The README says to edit before use elsewhere, matching the existing example convention.

Follow-ups

  • Optional: a standalone docs/ tuning note if we want the findings outside the example dir.

Throughput-tuned default-sweep specs for qwen27b, qwen35b, gemma4-26b,
gemma4-31b on GB10 (vLLM 0.24.0), plus a README explaining the tuning
and an old-vs-new serve-command record. Drops --mamba-cache-mode none,
--enforce-eager, --no-async-scheduling and raises max-num-batched-tokens
to 16384; ~20-30% higher aggregate throughput on qwen27b. Specs are
machine-specific (runtime paths/env) and meant to be edited per host.
@dutifulbob dutifulbob merged commit 817a3d8 into main Jul 7, 2026
2 checks passed
@dutifulbob dutifulbob deleted the docs/gb10-optimized-specs branch July 7, 2026 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants