docs(examples): add GB10/DGX Spark NVFP4 optimized sweep specs#21
Merged
Conversation
Throughput-tuned default-sweep specs for qwen27b, qwen35b, gemma4-26b, gemma4-31b on GB10 (vLLM 0.24.0), plus a README explaining the tuning and an old-vs-new serve-command record. Drops --mamba-cache-mode none, --enforce-eager, --no-async-scheduling and raises max-num-batched-tokens to 16384; ~20-30% higher aggregate throughput on qwen27b. Specs are machine-specific (runtime paths/env) and meant to be edited per host.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
We tuned how vLLM is launched for four NVFP4 models on a GB10 (DGX Spark) and got noticeably higher throughput, but that config only lived in local run dirs.
This adds those tuned sweep specs to the repo as a versioned example, plus a README explaining what changed and why, and a record of the old vs new serve command per model.
It matters because the original hand-authored specs launched vLLM in a way that quietly hurt decode, and now the better config is reproducible instead of lost.
What Changed
Adds a new example directory with the four tuned specs and documentation. No library code changes.
examples/gb10-dgx-spark-nvfp4-optimized/withqwen27b.json,qwen35b.json,gemma4-26b.json,gemma4-31b.json.README.mdexplaining the tuning: drop--mamba-cache-mode none(the Qwen models are hybrid linear-attention and that flag disabled caching the recurrent state), drop--enforce-eager(enables CUDA graphs), drop--no-async-scheduling, and raise--max-num-batched-tokensto 16384.PROFILES-old-vs-new.txtrecording each model's old serve command vs the tuned one.Testing
Confirmed the specs are valid and the docs pass the repo's checks. Behavior was measured on the real machine before committing.
localperf bench plan(39/43/39/39 planned runs).npx @simpledoc/simpledoc check— clean.go build ./...— ok (no code changed).Risks
Low — this is example data and docs, no code paths change.
runner.vllm_command, env (CUTE_DSL_ARCH=sm_121a), and ports point at one host. The README says to edit before use elsewhere, matching the existing example convention.Follow-ups
docs/tuning note if we want the findings outside the example dir.