LETS GO AMD!!!#1229
Conversation
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Oseltamivir
requested review from
1am9trash,
billishyahao,
chunfangamd,
seungrokj and
yctseng0211
as code owners
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25232685281 |
Removed the 'seq-len-configs' section from the YAML configuration.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25232810140 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25233612426 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25239290538 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25242261206 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25242851751 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=25243060873 |
Oseltamivir
changed the title
3 participants
Refreshed as #1260
DSV4 ATOM optimizer via custom hand-prompted AI codegen AITER kernel
Builds on ROCm/ATOM#650 to bring DeepSeek-V4 (DSv4) FP4 support to ATOM on MI355X, with runtime AITER performance overlays and custom Triton kernels for sparse attention and indexer operations.
Key changes
DSv4 ATOM runtime overlay (
dsv4_fp4_mi355x_atom.sh): Patches feat(deepseek_v4): PR1 skeleton — end-to-end inference with triton MoE ROCm/ATOM#650 at launch to give each request persistent DSv4 KV/compressor/indexer cache slots, unblocking CONC>1 serving. Batches attention projections, mHC, and MoE/FFN layer-by-layer across active requests while keeping sparse attention per-sequence.AITER DSv4 perf stack: Assembles a custom AITER build from upstream
main(ROCm/aiter@bb4ea92) plus cherry-picked performance PRs:Custom Triton kernels (from Oseltamivir/aiter@0923d27):
sparse_mqa_sink— DSv4 sparse MQA sink Triton op replacing the Torch fallback in PR650'ssparse_attn_v4.pydsv4_indexer— DSv4 Indexer scorer/top-k Triton op with batched API supportATOM_DSV4_AITER_SPARSE_ATTN_*env vars) reducing repeated QK score work for D=512Sweep expansion: CONC range expanded from 1-only to 1–8 (1k1k) and 1–4 (8k1k); TP=4 comparison points added to test whether fewer ranks reduce the cross_device_reduce communication bottleneck observed in profiling (~49% of GPU kernel time at TP=8).
AITER kernel test runner (
runners/test_dsv4_aiter_kernels.sh): Standalone test harness for validating the overlaid AITER sparse MQA and indexer Triton kernels on MI355X.Benchmark infrastructure: Added
benchmark_lib.shhelpers for eval-only benchmark mode and DSv4 evaluation support inbackend_request_func.py/benchmark_serving.py.Related upstream work