perf: optimize qwen3.5 prefill projection and pack on npu.#1907
Merged
maojunx99 merged 3 commits intoJul 10, 2026
Conversation
1db77f3 to
3d4e3fb
Compare
3d4e3fb to
fb2afd1
Compare
Contributor
|
本优化和MTP没有关系,建议将 MTP从标题中移除 |
Collaborator
Author
|
已按建议从标题中移除 MTP。本次代码优化作用于 Qwen3.5 的 prefill/chunked prefill 路径,MTP 仅为本次性能验证配置。感谢指出。 |
fb2afd1 to
5c9cba5
Compare
yingxudeng
approved these changes
Jul 10, 2026
xiao-yu-chen
approved these changes
Jul 10, 2026
Collaborator
Author
|
本PR描述中最优tp2 e2e达到170ms,合入过程中pr1823先合入,rebase后性能劣化,e2e179ms。#1823 这里 在mtp增加了同步 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
Qwen3Next 的 projection 权重以
qkvz和ba合并形式存储,因此公共路径需要在 projection 后执行 split/reshape。Qwen3.5 的 checkpoint 已将 projection 权重拆分为独立的
qkv/z/b/a。此前为了复用 Qwen3Next 公共路径,Qwen3.5 会先把独立 projection 结果重新 merge 成qkvz + ba,随后再通过 fused kernel split 回qkv/z/b/a,产生了冗余的 merge、split、reshape 和 concat 开销。此外,单请求且无 padding 的 prefill 输入已经连续,继续执行通用 pack/unpack 会产生不必要的

cat、临时 tensor 和 copy。代码改动
1. 去除 Qwen3.5 prefill projection 的冗余 merge/split
project_prefill_split_inputs。qkv/z/b/a,不再先 merge 为qkvz + ba后再 split。2. BS=1 时跳过冗余 pack/unpack
当
batch_size == 1且有效长度等于seq_len时:processed_q/k/v/g/beta,跳过cat(...).unsqueeze(0)。zeros_like + narrow + copy_。本 PR 不修改三方依赖。TP4 功能所需的算子修复由独立 PR 处理;下述 TP4 数据来自已包含该修复的验证环境。
验证条件
num_speculative_tokens=3。max_tokens=25,ignore_eos=true。/varscounter。projection + pack均采集 TP1/TP2/TP4 稳态 msprof。性能数据
单位:ms。表格只保留与本 PR 两项改动相关的单项和组合实验。
最终组合相对 baseline:
projection split是主要 TTFT 收益来源;BS=1 pack/unpack 单独运行时收益受执行路径和波动影响,但与 projection split 组合后,TP1/TP2/TP4 的 E2E、TTFT 和 TPOT 均优于 baseline。MTP 接受率未观察到回退。Profiling
Qwen3.5-2B 共 24 层,其中 18 层为线性 attention。移除每层冗余的 projection merge/split 后,TP1 TTFT 累计降低约 15 ms。
修改前:
修改后:
本地验证
git-clang-format --diff:通过,未修改文件。python setup.py build test:通过。