cuda: reject fmt=4 at the per-row-only entry points (#334 follow-up to #298)

[ZacharyZcR]

Spotted while reading #298's merge notes on #108/#326.

The residue

#298 fixed the callers of the dense-resident path after they fed grouped-scale tensors to kernels that read scales per row (CUDA_DENSE=1 produced garbage). The entry points still accept them:

• coli_cuda_matmul uploads via coli_cuda_tensor_upload (the non-_g variant, no group size), so scale_count collapses to O, the [O, ceil(I/gs)] scale buffer is silently truncated, and quant_matmul then reads scales[o] per row.
• coli_cuda_expert_mlp shares those kernels and the same assumption.

Today nothing reaches them with fmt=4 (the colibri.c call site excludes fmt=5 already, and #451 routes grouped experts through the group path), so this is not a live bug — it is an unguarded door of exactly the kind #334's prevention note warns about, one call site away from reproducing #298.

The fix

Return 0 for fmt=4 at both entries. A grouped tensor then takes the expert-group path (#451, which carries per-group scales) or stays on the CPU — a fallback, never a wrong answer. Three lines, no behaviour change for anything that works today.

make check 105/105, zero warnings.

(Related: the g64/E8 line — #452, #453 merged, #458 open — will put many more fmt=4/fmt=5 tensors in front of these entries, which is why closing the door now seemed worth a small PR.)

[JustVugg]

Good instinct, but the conflict is telling you something: #298's merged form changed the entry-point semantics your patch was written against, and as-is this would break the path #298 fixed. Current dev:

• coli_cuda_matmul now takes gs as a parameter (colibri.c:534 passes w->gs), routes gs>0 uploads through coli_cuda_tensor_upload_g, and launches the gs-aware quant_matmul(..., t->gs, t->ng). So fmt=4 through this entry is now correct, and your if (fmt == 4) return 0; would knock out the dense-resident g64 GPU path that cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298 just repaired (the one hardware-verified on sm_120).
• coli_cuda_expert_mlp already rejects fmt=4: its guard reads gate->fmt!=2||up->fmt!=2||down->fmt!=2. Your second hunk is redundant there.

What survives of your point — and is worth landing: a fmt=4 tensor arriving without a group size is still the unguarded door. gs<=0 falls into the plain-upload branch, scale_count collapses to O (silent truncation), and the fmt=4 kernel branch would compute i / gs with gs=0. So the residual guard is one line in coli_cuda_matmul:

if (fmt == 4 && gs <= 0) return 0;   /* grouped scales need an explicit gs — see #298 */

(plus, if you want belt-and-braces, the same check against a cached tensor: t->fmt==4 && t->gs<=0.) Want to reshape the PR to that? Happy to merge it the moment it's green — the defensive intent is exactly right, #298 just moved the door.

[ZacharyZcR]

Rebased onto dev. Since dev's quant_matmul now handles grouped fmt=4 natively (gs/ng recorded at upload via upload_g), the blanket fmt==4 rejection is narrowed to fmt==4 with no group size — that case still truncates the scale buffer and divides by gs==0 in the kernel; the gs>0 path goes through normally.

[tonuonu]

Corroborating the gs>0 half on Blackwell: dev 68ac9ff with CUDA_DENSE=1 and a grouped gs=128 container (GLM-5.2 744B, RTX PRO 5000 72 GB, sm_120) runs coherent and fully resident — 8104 tensors / 59.65 GB VRAM, 2493 hot experts pinned, zero fallbacks. So the narrowing is right: post-#298 the gs>0 path goes through quant_matmul natively, and the only door left to shut is fmt=4 with gs==0 (scale-buffer truncation + divide-by-gs==0 in the kernel). +1 on the narrowed guard.