perf(qwen3.6): adaptive combine warps for the hd256 flash-decode (long-context occupancy)#293
Closed
nickmopen wants to merge 1 commit into
Closed
Conversation
…ong context The hd256 split-combine ran a fixed NW=4 folding warps regardless of n_splits, so at 16k/32k (n_splits=128/256) each of 4 warps serially folds 32-64 split-partials on only num_q_heads*DG = 64 CTAs -- underfilling the GPU on the exact long-context path gittensor-ai-lab#284 just sped up. Mirror the hd128 path's adaptive dispatch (NW=16 at n_splits>=128, 8 at >=64, else 4). Byte-exact combine -- only the fold parallelism changes, same as the shipped hd128 adaptive combine -- so accuracy is unaffected; this is a pure occupancy win at long context.
jimcody1995
added a commit
to jimcody1995/sparkinfer
that referenced
this pull request
Jul 9, 2026
…anded n_splits Remove fused-router (gittensor-ai-lab#282) and hd256 combine (gittensor-ai-lab#293) to avoid copycat containment. Retain two distinct optimizations: - qknorm_rope_kv_partial_int8 fusion (new kernel, hd256 full-attn path) - banded n_splits for 8k-12k (distinct from gittensor-ai-lab#194 naive 28× threshold) Co-authored-by: Cursor <cursoragent@cursor.com>
1 task
1 task
Member
|
Closing due to inactivity (>2 days since last update). Reopen or push new commits when ready to continue. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The hd256 (Qwen3.6 full-attention) split-combine launched a fixed
NW=4folding warps regardless ofn_splits, so at long context (16k/32k →n_splits=128/256) each of 4 warps serially folds 32–64 split-partials on onlynum_q_heads × DG = 64CTAs — badly underfilling the GPU on the exact long-context path #284 just sped up. This mirrors the hd128 path's adaptive dispatch (NW=16atn_splits≥128,8at≥64, else4).Byte-exact: only the fold parallelism changes — the combine result is identical (same as the shipped hd128 adaptive combine), so accuracy is unaffected. Pure long-context occupancy win.
Proof of speedup
sm_120)Decode tok/s (end-to-end,
bench/scripts/bench.sh --download --ctx 16384, Qwen3.6-35B-A3B UD-Q4_K_M, bs=1 — long-context optimization; contexts < 8k are unchanged):At 32k the gain is larger (more splits → worse combine underfill): main 337.76 → this PR 349.86 (+3.6%).
What changed
flash_decode_split.cu: instantiatefa_combine_kernel<256,DG,8>/<256,DG,16>, addfa_launch_combine_dispatch_hd256(adaptive NW byn_splits, mirroring the hd128 dispatch), and route both hd256 combine sites (int8-MMA and tile paths) through it. Accuracy-preserving (byte-exact combine).