feat(qwen35): add paged-attention monolithic decode-only AR foundation

[Graffioh]

Summary

• Add an exact fixed-block paged K/V allocator (transactional allocation, stale-handle detection, immediate release).
• Add a CUDA/HIP GGML_OP_PAGED_ATTN decode op for D=256 GQA with F16, Q4_0, and Q8_0 K/V.
• Optimize decode with GQA-aware thread blocks, occupancy-aware context parallelism, adaptive long-context partitioning, and a stable split-softmax merge.
• Integrate 16-token paged autoregressive decode for dense Qwen3.5/Qwen3.6 behind --paged-attention (one CUDA or HIP device).
• Fix first-token AR bookkeeping so the first sampled token is forwarded and written at its committed K/V and RoPE position instead of advancing across an unwritten row.
• For now: keep exact chunked prefill; reject incompatible snapshots, speculative decoding, DDTree, KVFlash, layer split, and finite FA windows.
• Add allocator, nine-type numerical, block-boundary, uniform-kernel, and ragged-capacity tests.

Scope

Paged K/V is the memory-management foundation for heterogeneous 4–8-user serving. This PR establishes exact single-request Qwen integration and a multi-sequence GPU primitive first, following the block-table design in ggml-org/llama.cpp#21961.

HTTP integration stays one active sequence: Qwen3.6's Gated DeltaNet conv/SSM state is backend-global and the worker is run-to-completion. Sequence-indexed recurrent state, batched decode, and a continuous-batching scheduler are follow-up work. The benefit here is memory capacity and request-lifecycle flexibility; the benchmark below measures attention operations, not whole-model tokens/s.

Opt in with --paged-attention. Unsupported combinations fail explicitly. Requests reaching --max-ctx finish cleanly.

Benchmark results

All runs use Qwen attention dims (D=256, Hq=24, Hkv=4).

Single metric throughout: paged throughput ÷ contiguous throughput. Above 1.0x = paged faster; below 1.0x = paged slower. (Absolute times differ by GPU class, so only the within-GPU ratio is meaningful.)

Uniform batches (identical context per sequence)

Both layouts hold the same K/V payload, so this isolates paged compute/layout overhead. Q4_0 K/V, 4K context, batches n_seq = 1/2/4/8:

GPU / backend	paged ÷ contiguous throughput
RTX 3090 / CUDA	0.65–0.76x (paged slower)
Radeon 8060S / HIP	2.06–2.78x (paged faster)

Caveat: the HIP paged win largely reflects an optimization gap in GGML's native HIP flash_attn_ext_vec baseline, not an inherent advantage of paged addressing. Don't extrapolate it to whole-model serving throughput.

Ragged 8-request workload (128K + 7 × 8K, Q4_0 K/V)

What this benchmark represents: this is a standalone synthetic multi-sequence kernel benchmark, not the current HTTP/Qwen execution path. It constructs the eight sequence lengths and their interleaved block table directly. This simulates the layout that a future multi-sequence allocator would produce after concurrent requests, but it does not use PagedKvPool and does not demonstrate eight concurrent requests in the current server. The current end-to-end integration remains single-sequence (max_sequences = 1).

The case models the intended mixed-context workload. Paging stores only live, block-rounded tokens (188,416) instead of padding every sequence to 128K (1,048,576 slots).

Attention step time, same metric on each GPU:

GPU / backend	contiguous	paged	paged ÷ contiguous throughput
RTX 3090 / CUDA	3.122 ms	4.095 ms	0.76x
Radeon 8060S / HIP	11.670 ms	3.918 ms	2.98x

K/V capacity (backend-independent projection over Qwen3.6's 16 full-attention layers; excludes weights, activations, and recurrent state):

The benchmark allocates and measures one synthetic attention layer. The 16-layer values below are projections obtained by multiplying that measured K/V storage by Qwen3.6's 16 full-attention layers; they are not full-model allocations.

layout	one-layer K/V	16-layer K/V
padded per-sequence contiguous	1152 MiB	18.000 GiB
paged	207 MiB	3.234 GiB

→ 5.57x less K/V (82.03% saving). On CUDA this costs +31% attention latency (0.76x throughput); on HIP the paged path is also faster because of the weak native baseline noted above. Either way, this is primarily a memory-capacity win, not a claim that paging makes attention faster.

Roadmap

1. This PR: decode-only ggml_paged_attn, block manager, CUDA/HIP integration, benchmarks.
2. Continuous batching: sequence-indexed recurrent state, iteration-level scheduling, admission control, cancellation, dynamic decode batches.
3. Batched prefill: multiple prompts in one variable-length forward pass.
4. Chunked prefill: token budget, decode prioritization, prefill/decode interleaving.
5. Paged-aware prefill: read prefixes/preceding chunks directly from the paged K/V pool.
6. Prefix caching/CoW: share prefix blocks with reference counting and copy-on-write.

Validation

• Current head f1851f9: CUDA sm_86 and HIP gfx1151 pass paged_kv_pool, paged_attention, and paged_attention_direct; default and explicit ragged (4096,2048,1024,512) benchmarks complete on both backends. HIP also builds dflash_server.
• All nine F16/Q4_0/Q8_0 K/V combinations pass the CPU numerical oracle on CUDA and HIP.
• Correctness covers empty sequences, page boundaries, shuffled blocks, long-context partitioning, and corrupt block-table bounds guards.
• Calibrated Q4_0 4K uniform/default-ragged and exact 128K + 7 × 8K runs completed on CUDA and HIP; full HIP matrix (F16/Q4_0/Q8_0 at 4K/16K/64K/128K, batches 1/2/4/8) completed with rocprofv3 traces.
• git diff --check passes.

[cubic-dev-ai]

All reported issues were addressed across 28 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Graffioh]

the first emitted output token from prefill writes KV row only during the subsequent forward (that's right)

BUT here on the first decode step, we increment committed by 1, even though for cache we are still considering the first emitted token from prefill, leading to a gap of 1

was there a reason for this?

(in case this needs to be fixed also in other model's backend)