perf: optimize Qwen3.5 MTP decode execution on NPU.#1929
Open
pjgao wants to merge 5 commits into
Open
Conversation
- keep the target model and sharded LMHead on the original TP group. - allocate and account for the replicated draft KV cache independently.
- Use separate qkv/z/b/a projections directly in decode and speculative verification. - Preserve the packed qkvz/ba fused-split path for Qwen3Next. - Clarify the packed-token padding helper and its shape contract.
7b72e87 to
65e6750
Compare
- reuse target token IDs for greedy MTP validation - skip draft probabilities only for all-greedy sampling - preserve random, mixed, and logprob sampling paths
b722741 to
65e6750
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
修改内容与原因
1. 草稿模型主体独立使用 TP1
Qwen3.5 2B MTP草稿模型计算量较小,使用 TP2 时每层通信开销无法被计算有效掩盖。让草稿模型主题不做切分,在各
rank 本地执行避免这部分通信,同时保留体积较大的 LMHead做切分,避免在每张卡上复制完整 LMHead引起大开销。
这样草稿模型主体在每个 target rank 上各自以 TP1 执行,LMHead 仍使用target 的 TP2 group 做分片输出投影。该路径由显式运行时选项控制,并同步调整草稿KV Cache 的容量估算和张量形状。
主模型TP2、草稿模型TP2:


主模型TP2、草稿模型TP1、lm_head TP2:
2. 恢复草稿 forward 异步执行
#1823 (comment) 增加了同步,这里移除每一步草稿 forward 结束后的立即 stream 同步,恢复通过 event 表达依赖的异步执行方式。
逐步同步会把每次草稿计算和后续准备串行化,在 MTP5 下重复形成 host/device 空泡;
event 已能保证真实的数据依赖,不需要在每一步阻塞 host。
3. Qwen3.5 decode projection 直接使用独立投影结果
Qwen3Next 的qkvz/ba 合并权重布局要有split,而Qwen3.5权重本身是分离的qkv、z、a、b projection,无须先拼接再split,#1907 已在prefill/非投机的chunk-prefill阶段修改,这里放开decode/mtp veifify等其余分支限制。
4. Greedy validation 跳过无用 probability
全 greedy MTP 请求直接使用 draft token IDs 和 target token IDs 完成接受判断,不再要求草稿模型返回 probability,也不再构造、拷贝或恢复 draft probability tensor;随机采样路径保持原逻辑。
Greedy 接受判断只比较 token ID,draft probability 不参与计算。跳过这些张量操作可减少每个投机步骤的 softmax、索引和 host/device bookkeeping,同时保持 greedy validation 语义不变。
性能结果
测试条件:Qwen3.5-2B,TP2,MTP5,两张 NPU;开启 ACL Graph,关闭 graph double
buffer 和 prefix cache;
max_memory_utilization=0.84。Rank 0 绑定 CPU 240-279,Rank 1 绑定 CPU 280-319。固定请求集,输出 25 tokens,先 warmup 8 条,再执行
20 条请求 3 轮。以下指标均为 60 条成功请求的平均值。
完整优化使服务端 E2E 降低 30.160 ms(16.18%),服务端 TPOT 降低
1.115 ms(26.15%)。
各修改收益
每项同时使用按提交顺序累加和从最终代码单独移除的消融实验验证。
下表按修改组顺序累加,每一行均包含该行及其之前的所有修改:
相邻两行的差值表示对应修改组的累加收益;另外从最终代码中单独移除各组,得到以下
消融贡献:
四组修改在单项验证中均达到 2 ms 服务端 E2E 收益。
验证
origin/main@40b5ff57rebase,递归 submodule 状态一致。MAX_JOBS=16 CTEST_PARALLEL=16 python setup.py build --tilelang-jobs 16 test完成 fresh build。
spec_input_builder_test:22/22 通过。sampler_test:13 个通过,2 个仅适用于 MLU 的用例跳过。