Question about performance at different seqlen

[jayden222]

Great job for you guys!

My device only has 24GB VRAM, so I must set the seqlen to around 4k when running the perf test, and the performance of FSA is worse than NSA. Besides, it seems like the provided speedup ratio is also tested on default 64k seqlen, which is not very common at practice.

How's the performance of NSA _ref and FSA when the seqlen is relative shorter? It will be better that you can provide a benchmark comparing NSA_ref, FSA and full-attention along side the seqlen.

[ran-yan-hk]

Hi, great question!

What GPU are you currently using? FSA has been comprehensively tested and optimized for Hopper and Ampere architectures, which represent the most powerful GPUs available today. For older GPU generations, we haven't fine-tuned the Triton kernel design and hyperparameters, which are highly sensitive to hardware specifications including TFLOPs, HBM memory bandwidth, and shared memory capacity. Without this fine-grained optimization, the kernels may exhibit suboptimal performance, as you've observed.

We recommend using more advanced GPUs (e.g., available through Runpod) to thoroughly benchmark FSA and experience its intended performance benefits.

Regarding performance on shorter sequence lengths: please refer to our Evaluation section on GitHub, where we test sequence lengths from 8K to 64K tokens—the exact settings used in DeepSeek's NSA paper. For sequence lengths below 8K, NSA (and sparse attention methods generally) may perform worse than standard Full Attention. Since sparse attention provides limited benefits in these cases, we exclude them from our evaluations, following the same approach as the DeepSeek paper.