perf(attn): GQA-4 shared-KV tile for Qwythos hd256 full-attn decode (+2.4% @4k)#326
perf(attn): GQA-4 shared-KV tile for Qwythos hd256 full-attn decode (+2.4% @4k)#326inference2026 wants to merge 1 commit into
Conversation
Qwythos-9B full attention is 16Q/4KV (GQA-4, head_dim=256). The hd256 flash-decode dispatch only handled GQA-8, so Qwythos fell through to the scalar fa_split_kernel<256> (one warp per q-head, no KV-tile sharing — each of the 4 q-heads in a group re-reads the same KV head). Route GQA-4 to fa_split_gqa_kernel<256,4,TILE> (shared-KV tile, one block per (kv_head,split), 4 warps sharing one smem KV tile), cutting the full-attn KV global reads ~4x. TILE=8 (smaller smem than the GQA-8 TILE=14 -> higher occupancy for the 4-warp block). Byte-identical to the scalar path (same online-softmax order). SPARKINFER_FAGQA4=0 restores scalar. Qwythos decode (RTX 5090, n=64, vs same-box main): 128: 250.5 vs 249.0 (+0.6%) 512: 248.1 vs 247.0 (+0.4%) 4k: 241.5 vs 235.8 (+2.4%) Byte-identical (score cmp 2003/2003 lines). Qwen3.6 guard (GQA-8) byte- unaffected: 16k 381.3 == baseline (never enters the GQA-4 branch).
github-actions
Bot
added
the
copycat
🐈 Flagged: copycat (real-time guard)This PR re-submits substantially the same diff (≥85% line overlap) as the earlier #195 by a different author. Duplicating another contributor's work is treated as gaming the SN74 emission mechanism. The account has been blocked and this PR closed. See |
github-actions
Bot
added
the
flagged:gaming
🚩 Flagged: eval-gamingBlocked account(s) for gaming the SN74 emission mechanism (sybil / coordinated duplicate farming): See |
|
Maintainer review requested — I believe this is a false positive of the line-overlap copycat guard, and I want to be transparent about exactly why the overlap exists. What this PR actually changes (31 lines, one file): it adds a single dispatch branch so Qwythos-9B's full-attention (16Q / 4KV = GQA-4, head_dim=256) is routed to the existing bf16 shared-KV tile kernel Why it overlaps #195: #195 (fansilas, merged) introduced the hd256 GQA-8 tile dispatch and the Why it is not a re-submission of #195's work: #195's actual contribution is the int8 tensor-core kernel I fully understand the anti-gaming stance and I'm not trying to farm anyone's work — the diff is small precisely because it reuses the correct existing kernel rather than duplicating logic. If the maintainers judge a 31-line parallel-dispatch branch too derivative to score, I completely accept closing it on those grounds. My one real request is to lift the account block on |
🐈 Flagged: copycat (real-time guard)This PR re-submits substantially the same diff (≥85% line overlap) as the earlier #195 by a different author. Duplicating another contributor's work is treated as gaming the SN74 emission mechanism. The account has been blocked and this PR closed. See |
🚩 Flagged: eval-gamingBlocked account(s) for gaming the SN74 emission mechanism (sybil / coordinated duplicate farming): See |
|
Copycat false positive cleared (maintainer). Branch restored; continued as #327 — GitHub would not reopen this PR after the head branch was recreated. |
…function block escalation. Unblock inference2026, mark gittensor-ai-lab#326 cleared in copycats.json, and cap per-function containment at WARN — template boilerplate must not trigger ≥85% block. Co-authored-by: Cursor <cursoragent@cursor.com>
skyrocket2026
added
test-on-5090
|
Copycat clearance applied: removed GitHub will not reopen this PR — branch This PR stays closed as superseded; labels updated so it no longer shows as a gaming/copycat strike. |
Co-authored-by: Cursor <cursoragent@cursor.com>
…fix (gittensor-ai-lab#334) - Credit PR gittensor-ai-lab#326 (inference2026) with gittensor-ai-lab#327 Polaris eval; void gittensor-ai-lab#327 emission in dashboard - GQA-4 Qwythos dispatch (rebased on main, cf9d521) - Copycat guard: per-function warn-only, boilerplate filter, copycat-cleared skip Closes gittensor-ai-lab#334. Refs gittensor-ai-lab#326 gittensor-ai-lab#327 gittensor-ai-lab#332.
2 participants
Summary
Route Qwythos-9B full attention (16Q / 4KV = GQA-4,
head_dim=256) to the existing shared-KV tile kernelfa_split_gqa_kernel<256, 4, …>instead of the scalarfa_split_kernel<256>fallback it was hitting. The scalar path runs one warp per q-head with no KV-tile sharing, so the 4 q-heads in a group each re-read the same KV head; the tile kernel stages KV once and shares it, cutting full-attn KV global reads ~4×. One file, +31 lines. Byte-identical output;SPARKINFER_FAGQA4=0restores the scalar path. (Rebased ontomain@ #324.)Proof of speedup
sm_120)Decode tok/s (end-to-end,
bench/scripts/bench.sh, Qwythos-9B Q4_K_M,--ctx 4096 --tokens 64, interleaved A/B median of 5 rounds, vs same-boxmain@ #324):(+2.03% @4k — the strongest scored context. 128/512 are neutral-to-positive; the KV-read sharing grows with context, so 4k benefits most.)
Correctness — byte-identical to
main:qwen3_gguf_score, greedy, tokens 100..4300 (covers the 4k scored context) → 4203 / 4203 lines identical. The GQA-4 tile does the same online-softmax over the same tokens in the same order; only the KV global-read pattern changes. So top-1 / KL vs llama.cpp equalsmainexactly.Qwen3.6 guard — unaffected: Qwen3.6 full attention is GQA-8 and never enters the new GQA-4 branch (byte-unaffected). 16k decode 381 tok/s == baseline.