Skip to content

Use unroll_count(4) for the NAX attention Q@K.T loop#3843

Open
wyanzhao wants to merge 1 commit into
ml-explore:mainfrom
wyanzhao:nax-128-qk-unroll
Open

Use unroll_count(4) for the NAX attention Q@K.T loop#3843
wyanzhao wants to merge 1 commit into
ml-explore:mainfrom
wyanzhao:nax-128-qk-unroll

Conversation

@wyanzhao

@wyanzhao wyanzhao commented Jul 13, 2026

Copy link
Copy Markdown

Proposed changes

Change the NAX attention Q @ K.T head-dimension loop from full unrolling
to #pragma clang loop unroll_count(4).

Note

This is a scheduling-only change. The accumulation order and numerical
results are unchanged.

Motivation

With full unrolling, the compiler hoists all TD K-tile loads to the start
of the block, ahead of the running MMA chain. Unrolling by four instead lets
the compiler interleave K-tile loads with matrix accumulation.

Configuration Full unroll unroll_count(4)
head_dim == 128 (TD == 8) All K-tile loads are hoisted Loads can interleave with the MMA chain
head_dim == 64 (TD == 4) Four iterations unrolled Still fully unrolled; effectively unchanged

Performance

M5 Max, 16 Q / 2 KV heads, bf16, causal, 8192 x 8192; median of 12 runs:

Head dimension Before After Change
128 5.70 ms / 48.2 TF 5.08 ms / 54.1 TF +12% throughput
64 2.45 ms 2.44 ms No material change

Scope and compatibility

  • Changes one pragma in the existing wn=1 NAX QK loop.
  • Does not change the head_dim == 256, wn=2 D-split path.
  • Preserves the exact accumulation order.
  • Composes cleanly with the head_dim == 256 PR; the two changes touch
    different regions of steel_attention_nax.h and can merge in either order.

Validation

  • Outputs remain bitwise-identical because the loop-carried accumulation
    order is unchanged.
  • Full local test_fast_sdpa.py result: 16 tests, 1 platform-dependent
    skip, otherwise passing
    .
  • A three-way merge with the head_dim == 256 PR completes without conflict.

Checklist

  • I have read the CONTRIBUTING document.
  • I have run pre-commit run --all-files.
  • Existing fast-SDPA tests pass with no numerical changes.
  • No documentation changes are required for this compiler-scheduling change.

Fully unrolling the head-dim loop makes the compiler hoist all TD K-tile
loads to the top of the block, ahead of the mma chain. Unrolling by 4
instead lets it interleave each K load with the running matmul, which
measures faster at head_dim 128 and is a no-op at head_dim 64.

M5 Max, 16 Q / 2 KV heads, bf16, causal, 8192^2 (median of 12):
  head_dim 128: 5.70 -> 5.08 ms (48.2 -> 54.1 TF, +12%)
  head_dim  64: 2.45 -> 2.44 ms (unchanged; TD == 4 is still a full
                unroll at count 4)

No numerical change (bitwise-identical accumulation order); 16/16
fast-sdpa tests pass.
@wyanzhao wyanzhao changed the title Unroll the NAX attention Q@K.T head-dim loop by 4 not fully Use unroll_count(4) for the NAX attention Q@K.T loop Jul 13, 2026
@wyanzhao
wyanzhao marked this pull request as ready for review July 13, 2026 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant