cuda: grouped-int4 (fmt=4) support in the expert-group kernels — opens the GPU tier to g64 and E8 containers (#334)#451
Conversation
…ugg#334) The grouped MoE kernels were per-row-only: GroupDesc had no group-size fields, row_bytes() returned 0 for fmt=4, the scale buffer was hardcoded to O floats, and a fmt=4 group that reached the generic path would have been silently decoded as int2. This closed the GPU expert tier to every grouped container — including the g64 quality line (JustVugg#225) and the E8 lattice route (JustVugg#347) whose whole point is fitting more experts in VRAM. - ColiCudaTensor gains gs/scale_count; upload allocates O*ceil(I/gs) scales for fmt=4 and applies the same offset->signed nibble conversion as fmt=2 (identical packing). New ABI entry coli_cuda_tensor_upload_g carries gs without touching the existing symbol — an old Windows DLL missing it returns 0 and the tensor simply stays CPU-side. - GroupDesc gains per-tensor group sizes; new grouped_hidden_g4_dual / grouped_down_g4 apply the per-group scale inside the accumulation (gs is required even, so a packed byte never straddles groups; gs=0 degrades to per-row, letting fmt=2 members ride the same launch). - coli_cuda_expert_group routes any group containing fmt=4 through the g4 kernels; pure-fmt=2 groups keep the existing paths byte-identical. The generic fallback now explicitly rejects fmt=4 instead of decoding garbage (JustVugg#334's prevention note, made real). tests/test_grouped_g4_cuda.cu: kernel-vs-CPU oracle over 50 trials x 3 experts — gs=64, a non-divisible tail group (200 % 64), and a per-row member in the same launch: zero mismatches on a 5090. make check 77/77; CPU, CUDA and MinGW builds clean.
|
Merge-order note, since five of my open PRs touch Independent, land anytime in any order: #429 (colibri.c only), #453 (tools only), #424 (site/docs only). Ordered, all in
#433 ↔ #451 is the sharp one (same if/else chain); the rest are near-additive. You do not need to follow this order — merge whatever you have reviewed, and I will rebase the rest the same day, same as I did for #429. Just flagging so a merge doesn't look unexpectedly conflicted. |
|
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic. While preparing the Metal twin of this work (#457) we traced #451 against Claim, hedged (traced across three files, not run): if #451 merges without #298, there is a
Bounding the trigger: it fires only if a real checkpoint carries fmt=4 on a dense We hit exactly this "two changes must land together or the intermediate state is armed" class on Great to see the CUDA grouped tier moving — the formats contract lined up cleanly in our parity |
|
@ZacharyZcR — nice work getting grouped-int4 (fmt=4) into the CUDA expert-group kernels here. Testing it on Blackwell (RTX 5080, sm_120) with the public g64 container, I hit one correctness issue you'll want to know about, and it's cleanly isolated. The expert-group path is correct, but the resident-dense fmt=4 path garbles output when dense weights are pinned to VRAM. Same binary (current
Full write-up + a fix are on #298 (which passes |
|
@monotophic — your trace is exactly right, and I can confirm it empirically: I hit precisely the silent-wrong-answer you predicted, running on Blackwell (RTX 5080, sm_120) with the public g64 container. Your two states map one-to-one to what I saw:
Clean isolation on the same current- And your read that #298 covers the dense path holds up on hardware: my rebased #298 branch, same So: your static trace + my Blackwell repro land on the same thing from opposite directions — #451 alone arms a silent-wrong-answer on the dense path under |
|
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic. @mohamedmastouri2000-boop — thank you for running this down on real hardware; a clean one-env-var @ZacharyZcR — the expert-group path holding up under that test is good news for the tier itself. |
The Metal GEMV shader (mm_gemv, backend_metal.mm) handled fmt 0-3 only. Anything else -- including dev's grouped-int4 format (fmt=4: packed int4 nibbles, same layout as fmt=2, but one f32 scale per `gs`-element group along I instead of one scale per row; CPU reference matmul_i4_grouped in quant.h, active CUDA tier in JustVugg#298/JustVugg#451) -- fell through the shader's unconditional `else` arm and was decoded AS RAW F32. This was a silent-wrong-answer defect, not just a missing- acceleration gap: bind_gemv (used by the fused decode-attention kernels, coli_metal_attn_decode/coli_metal_layer_decode) accepted a per-weight fmt from its caller with no validity gate at all, so any fmt=4 dense/attention tensor reaching that path would "succeed" (ok=1) while silently computing garbage. coli_metal_matmul/coli_metal_gemm had explicit fmt<=3 gates and so safely rejected fmt=4 (0/CPU-fallback); bind_gemv had no such safety net. COMMIT ORDERING (bisect-safety, see PR_BODY.md sec 1 for the full argument): this commit lands FIRST, before the qt_from_disk allocator fix (next commit). At this commit alone, the capability is CORRECT but INERT: qt_from_disk's fmt=4 scale buffer is still allocated with a bare falloc() (unregistered, not page-aligned), so resolve() in backend_metal.mm can never find it, and every real fmt=4 dense/ attention tensor's scale buffer permanently misses bind_gemv/ coli_metal_gemm regardless of what this commit adds to the shader -- safe CPU fallback, not a wrong answer. No commit in this branch is ever "the shader is fixed but a scale buffer resolves to the wrong/stale allocation" -- fixed-but-unreachable, then reachable-because-fixed, in that order, never the reverse. Verified empirically at this exact commit (this branch, this session): both METAL=0/METAL=1 build clean, 0 warnings, and `make metal-test` passes 27/27 -- metal-test's fmt=4 cases construct their own tensors directly via posix_memalign + coli_metal_register (never through qt_from_disk), so none of them depend on the allocator fix landing first; the capability this commit adds is exercised and proven correct here, just not yet reachable from a real loaded model. Shader: unlike fmt 1-3 (single `acc * scale[o]` after the full dot product), fmt=4's group scale must be folded into the accumulation per-group, before summing across groups -- this is matmul_i4_grouped's actual math, not a simplification. New `constant int& gsz [[buffer(9)]]` parameter carries the group size (ignored, harmlessly, for fmt!=4). Each of the 32 SIMD lanes owns one packed byte (2 elements) per 64-wide stride -- memory access stays coalesced and a group never splits a byte (gs is always even, per the loader's candidate list). Deliberately not vectorized like fmt=2's uchar4/float4 path -- see PR_BODY.md sec 7 for why that tradeoff was made on purpose. Host plumbing: gs threaded through every path that reaches mm_gemv -- coli_metal_matmul, coli_metal_gemm, and bind_gemv (-> AttnW -> coli_metal_attn_decode / coli_metal_layer_decode) -- plus the colibri.c call sites (matmul_qt_ex's Metal-GEMM gate now includes fmt==4; attention_rows/layer_forward_rows pass .gs alongside each .fmt already being passed, a no-op for every model not using fmt=4). fmt_bytes and the new fmt_scale_bytes fix coli_metal_matmul's scale-buffer wrap size, which was previously hardcoded to O floats (wrong for fmt=4's O*ceil(I/gs)). kv_b (MLA absorption core) and the batched routed-expert MoE path (moe_gemv/coli_metal_moe_block) are deliberately untouched -- both already gate fmt=4 to a safe CPU fallback today, and extending them is a structurally separate kernel family; see PR_BODY.md sec 7 for the full scope statement. Tests (tests/test_backend_metal.mm): new cpu_ref_grouped (double- precision oracle mirroring matmul_i4_grouped, same construction tests/test_i4_grouped.c already established) and run_grouped harness, using a magnitude-relative tolerance (justified in PR_BODY.md sec 5) instead of the existing ymax-relative convention, since a grouped dot product's cancellation can otherwise misreport as a kernel defect. Covers the spec's shape matrix (I mult/non-mult of 64, I<=64 degenerate, S=1/S>1, outlier-heavy rows) via coli_metal_matmul, one case via coli_metal_gemm (the large-batch path), and two via a new run_attn_grouped harness that exercises the real fused-attention bind_gemv path end-to-end (not just the standalone kernel entry point). (These two attn cases have known blind spots as first committed here -- closed two commits later, see that commit's message and PR_BODY.md sec 10; kept as originally written here for bisect fidelity.) make metal-test: 15/15 (stock e9b3614) -> 27/27 (12 new fmt=4 cases; original 15 unchanged -- the negative control). make glm METAL=0/ METAL=1: clean, 0 warnings, both before and after. make test-c / test-python: unaffected, all pass. Full gates, evidence classes, and DEVIATIONS/UNCERTAINTIES in PR_BODY.md (untracked, worktree root). (cherry picked from commit f0fa5a9) (cherry picked from commit 82984cdeef67ede71d48b32f2ddd4734f4622eba)
…#298) #298 fixed the dense-resident path's callers after they fed grouped-scale tensors to kernels that read scales per row (CUDA_DENSE=1 produced garbage). The entry points themselves still accept them: coli_cuda_matmul uploads via coli_cuda_tensor_upload (no gs), so scale_count collapses to O and the [O, ceil(I/gs)] buffer is silently truncated, then quant_matmul reads scales[o] per row. coli_cuda_expert_mlp same kernels, same assumption. Return 0 for fmt=4 at both, so a grouped tensor takes the expert-group path (#451, which carries per-group scales) or stays on the CPU — a fallback, never a wrong answer. The caller-side guard in colibri.c already excludes fmt=5 the same way; this makes the ABI safe regardless of who calls it next. make check 105/105, zero warnings.
The Metal GEMV shader (mm_gemv, backend_metal.mm) handled fmt 0-3 only. Anything else -- including dev's grouped-int4 format (fmt=4: packed int4 nibbles, same layout as fmt=2, but one f32 scale per `gs`-element group along I instead of one scale per row; CPU reference matmul_i4_grouped in quant.h, active CUDA tier in JustVugg#298/JustVugg#451) -- fell through the shader's unconditional `else` arm and was decoded AS RAW F32. This was a silent-wrong-answer defect, not just a missing- acceleration gap: bind_gemv (used by the fused decode-attention kernels, coli_metal_attn_decode/coli_metal_layer_decode) accepted a per-weight fmt from its caller with no validity gate at all, so any fmt=4 dense/attention tensor reaching that path would "succeed" (ok=1) while silently computing garbage. coli_metal_matmul/coli_metal_gemm had explicit fmt<=3 gates and so safely rejected fmt=4 (0/CPU-fallback); bind_gemv had no such safety net. COMMIT ORDERING (bisect-safety, see PR_BODY.md sec 1 for the full argument): this commit lands FIRST, before the qt_from_disk allocator fix (next commit). At this commit alone, the capability is CORRECT but INERT: qt_from_disk's fmt=4 scale buffer is still allocated with a bare falloc() (unregistered, not page-aligned), so resolve() in backend_metal.mm can never find it, and every real fmt=4 dense/ attention tensor's scale buffer permanently misses bind_gemv/ coli_metal_gemm regardless of what this commit adds to the shader -- safe CPU fallback, not a wrong answer. No commit in this branch is ever "the shader is fixed but a scale buffer resolves to the wrong/stale allocation" -- fixed-but-unreachable, then reachable-because-fixed, in that order, never the reverse. Verified empirically at this exact commit (this branch, this session): both METAL=0/METAL=1 build clean, 0 warnings, and `make metal-test` passes 27/27 -- metal-test's fmt=4 cases construct their own tensors directly via posix_memalign + coli_metal_register (never through qt_from_disk), so none of them depend on the allocator fix landing first; the capability this commit adds is exercised and proven correct here, just not yet reachable from a real loaded model. Shader: unlike fmt 1-3 (single `acc * scale[o]` after the full dot product), fmt=4's group scale must be folded into the accumulation per-group, before summing across groups -- this is matmul_i4_grouped's actual math, not a simplification. New `constant int& gsz [[buffer(9)]]` parameter carries the group size (ignored, harmlessly, for fmt!=4). Each of the 32 SIMD lanes owns one packed byte (2 elements) per 64-wide stride -- memory access stays coalesced and a group never splits a byte (gs is always even, per the loader's candidate list). Deliberately not vectorized like fmt=2's uchar4/float4 path -- see PR_BODY.md sec 7 for why that tradeoff was made on purpose. Host plumbing: gs threaded through every path that reaches mm_gemv -- coli_metal_matmul, coli_metal_gemm, and bind_gemv (-> AttnW -> coli_metal_attn_decode / coli_metal_layer_decode) -- plus the colibri.c call sites (matmul_qt_ex's Metal-GEMM gate now includes fmt==4; attention_rows/layer_forward_rows pass .gs alongside each .fmt already being passed, a no-op for every model not using fmt=4). fmt_bytes and the new fmt_scale_bytes fix coli_metal_matmul's scale-buffer wrap size, which was previously hardcoded to O floats (wrong for fmt=4's O*ceil(I/gs)). kv_b (MLA absorption core) and the batched routed-expert MoE path (moe_gemv/coli_metal_moe_block) are deliberately untouched -- both already gate fmt=4 to a safe CPU fallback today, and extending them is a structurally separate kernel family; see PR_BODY.md sec 7 for the full scope statement. Tests (tests/test_backend_metal.mm): new cpu_ref_grouped (double- precision oracle mirroring matmul_i4_grouped, same construction tests/test_i4_grouped.c already established) and run_grouped harness, using a magnitude-relative tolerance (justified in PR_BODY.md sec 5) instead of the existing ymax-relative convention, since a grouped dot product's cancellation can otherwise misreport as a kernel defect. Covers the spec's shape matrix (I mult/non-mult of 64, I<=64 degenerate, S=1/S>1, outlier-heavy rows) via coli_metal_matmul, one case via coli_metal_gemm (the large-batch path), and two via a new run_attn_grouped harness that exercises the real fused-attention bind_gemv path end-to-end (not just the standalone kernel entry point). (These two attn cases have known blind spots as first committed here -- closed two commits later, see that commit's message and PR_BODY.md sec 10; kept as originally written here for bisect fidelity.) make metal-test: 15/15 (stock e9b3614) -> 27/27 (12 new fmt=4 cases; original 15 unchanged -- the negative control). make glm METAL=0/ METAL=1: clean, 0 warnings, both before and after. make test-c / test-python: unaffected, all pass. Full gates, evidence classes, and DEVIATIONS/UNCERTAINTIES in PR_BODY.md (untracked, worktree root). (cherry picked from commit f0fa5a9) (cherry picked from commit 82984cdeef67ede71d48b32f2ddd4734f4622eba)
4 participants
Makes #334's prevention note real, and unblocks the quantization roadmap's GPU path.
Why
The grouped MoE kernels were per-row-only:
GroupDeschad no group-size fields,row_bytes()returned 0 for fmt=4, the scale buffer was hardcoded toOfloats — and a fmt=4 group that slipped to the generic path would have been silently decoded as int2. Net effect: the GPU expert tier is closed to every grouped container — the g64 quality line (#225, −63% of the int4 quality loss) and the E8 lattice route (#347, int3-e8-rot measuring better than shipped per-row int4 at 25% fewer bytes) could only run their experts on the CPU. On the bandwidth-walled decode path, bytes are speed — the grouped containers are pointless without this.What
ColiCudaTensorgainsgs/scale_count; upload allocatesO·ceil(I/gs)scales for fmt=4 and applies the same offset→signed nibble conversion as fmt=2 (identical packing). New ABI entrycoli_cuda_tensor_upload_gcarriesgswithout touching the existing symbol — an old Windows DLL missing it returns 0 and the tensor stays CPU-side, no crash.GroupDescgains per-tensor group sizes; newgrouped_hidden_g4_dual/grouped_down_g4apply the per-group scale inside the accumulation.gsmust be even (a packed byte never straddles a group);gs=0degrades to per-row, so fmt=2 members ride the same launch in mixed groups.coli_cuda_expert_grouproutes any group containing fmt=4 through the g4 kernels; pure-fmt=2 groups keep today's paths byte-identical. The generic fallback now explicitly rejects fmt=4 instead of decoding garbage.Verification
tests/test_grouped_g4_cuda.cu— kernel-vs-CPU oracle (semantics cloned frommatmul_i4_grouped): 50 trials × 3 experts covering gs=64, a non-divisible tail group (200 % 64 = 8), and a per-row member in the same launch — zero mismatches on a 5090.make check77/77; CPU, CUDA and MinGW cross builds clean.Next on this line: convert the g64/E8 containers and A/B the VRAM-resident expert tier end-to-end — this PR is the gate that had to open first.