int3-g64 (fmt=5): per-group-scale 3-bit weights, the size/quality point your own ablation (#132) says the next container should sit at

[fabio-rovai]

This adds a fourth weight format end-to-end — int3 with per-64-group f32 scales (3.5 bits/weight) — engine, converter, and tests, so producing an int3-g64 GLM container becomes a one-flag job (--ebits 3 --xbits 3) for anyone holding the FP8 source.

Why this format, with data

Three measured inputs converge on it:

1. Quality: quant_ablation: rotation preconditioning (-rot) and int3 schemes (#81) #132's own ablation (OLMoE, fp16 A/B) found int3-g64 beats the shipped per-row int4 (−7.5pp vs −9.3pp) at ~14% fewer bits — grouped scales matter more than the 4th bit. This PR ships that exact quantization math (mirrors quant_ablation._quant_last_dim(bits=3, group=64) bit-for-bit; our test suite reproduces the effect: on outlier rows int3-g64's reconstruction RMS is 3.3× lower than per-row int4).
2. Transport: the engine is fetch-bound for most users. 0.86× bytes ≈ −14% on every cold expert read and ~50 GB off the container (357 → ~307 GB) — and in a policy simulation we ran over real routing telemetry for the Write-up: how the grammar-forced draft source (#48 → PR #70) lifts constrained-output decode — mechanism, A/B numbers, honest prior art, and why streaming-MoE is the regime where it pays most #146 work, flat int3-g64 recompression beat both plain LRU and heat-pinned int4 at every RAM budget (8/25/60 GB), which is more than we can say for the fancier tiering schemes we also simulated (they only win by serving 47–90% of calls degraded).
3. Demand: [Feature]: Support 2 bit quantization #138 asks for 2-bit; per quant_ablation: rotation preconditioning (-rot) and int3 schemes (#81) #132 (and our own int2 measurements) 2-bit steady-state is a quality cliff — int3-g64 is the honest answer to that request.

Design

• Layout (fmt=4): per row, per 64-input group: 24 bytes = 16B low plane (2 bits/val, matmul_i2's packing) + 8B high plane (1 bit/val); values in [-4,3] stored v+4; one f32 scale per group in NAME.qs.
• Format detection: fmt isn't stored on disk and row formats are inferred from weight byte counts — which int3 byte counts can alias. So the .qs scale-tensor size is the tag: O floats → row-scale formats, O*ceil(I/64) → fmt=4 (qt_fmt_from_bytes centralizes this for the dense loader and both expert_load paths, with a byte-count fallback for the degenerate I≤64 case).
• Kernel: matmul_i3, exact-f32 path with per-group scale accumulation. NEON: low plane reuses matmul_i2's unpack, high plane expanded via vtst on bit masks. Scalar reference everywhere (x86 SIMD is a follow-up — the target regime is fetch-bound). No IDOT path in v1: per-group accumulation doesn't compose with the current int8-activation kernels without restructuring, and the quality economics of IDOT are under separate scrutiny anyway (prefill: batch the attention input projections (bit-identical, -4.5% prefill) #152 and the ARM datapoint we posted there: +0.169 nats/token).
• GPU: Metal/CUDA gates already exclude unknown formats → clean CPU fallback, no shader changes needed for v1.
• Runtime-quantize path too: ./glm 64 3 3 on an unquantized checkpoint now produces fmt=4 (previously bits=3 silently became int4).

Tests (all green on Apple Silicon, make test-c + python suite)

• tests/test_int3.c — bit-exact pack/unpack vs a plain-C reference; matmul_i3 (NEON + scalar tail) vs dequant-matmul across shapes including short tail groups and the real GLM I=7168; qt_alloc/qt_fill/qt_bytes/matmul_qt/qt_addrow/qt_matvec_rows plumbing; the format tag; the outlier-rows RMS assertion (int3-g64 < per-row int4, ratio 3.34).
• tests/test_int3_load.c — writes a real single-shard .safetensors fixture (fmt=4 tensor + int4 control), indexes it with st_init, loads through qt_from_disk, asserts both formats are detected and dequantize bit-identically to the packer's output.
• tests/test_int3_convert.py — NumPy converter round-trip against an independent decoder, plus the outlier-vs-int4 property.

What this PR does NOT claim

GLM-native quality validation requires quantizing from the FP8 source (int4→int3 requantization stacks errors and is not the honest test). We don't have the 1.51 TB checkpoint; @MateoGrgic (or anyone holding it) can produce the container with --ebits 3 --xbits 3 unchanged otherwise, and #108's harness plus TF=1 are the right acceptance gates. Happy to coordinate and to run the full decode/quality ladder on Apple Silicon once a container exists.

🤖 Generated with Claude Code

[rajpratham1]

This is a substantial feature that introduces a new int3-g64 quantization format across the quantization pipeline, runtime kernels, loader logic, conversion tools, and test suite.

The implementation appears well thought out and includes comprehensive tests covering packing, loading, conversion, and numerical correctness, which is great to see.

Before merging, I'd appreciate review from maintainers on a few aspects:

• The long-term compatibility of introducing fmt=4 into the existing serialization/loading logic.
• Whether the new format-detection heuristics in qt_fmt_from_bytes() remain robust for all supported tensor shapes and future formats.
• The maintenance implications of adding a separate int3 execution path (especially the scalar/x86 fallback versus NEON implementation).

Overall the change looks promising, but given its scope and impact on core quantization/inference paths, I think it would benefit from additional review before approval.

[fabio-rovai]

Rebased onto current main (62419af, post today's merge wave — #111/#195/#199/#201/#202/#204/#205/#206/#212). One conflict, in c/Makefile TEST_BINS (union of main's new test_schema_gbnf/test_compat_direct with this branch's test_int3/test_int3_load); glm.c and the converter merged clean. Full make test-c + python suite (64 tests) green on Apple Silicon, including the int3 outlier-rows assertion (RMS ratio 3.34 vs per-row int4).

On @rajpratham1's three review questions — all three are design decisions the PR already commits to, so stating them here explicitly:

1. fmt=4 serialization compatibility — nothing is added to the on-disk header; fmt was never stored and still isn't. The tag is the .qs scale-tensor size, which is a property existing containers already have: O floats → the three existing row-scale formats, O*ceil(I/64) → fmt=4. Old containers are untouched and undetectable-as-int3 by construction (the two sizes only coincide when I≤64, which is exactly the degenerate case qt_fmt_from_bytes resolves by weight byte count instead — covered in test_int3_load).

2. Detection robustness for future formats — that's the reason detection was centralized into qt_fmt_from_bytes() rather than left inline in the dense loader and the two expert_load paths: any future format adds one arm in one function, and the existing arms are pinned by the loader test, which round-trips a real .safetensors fixture containing an fmt=4 tensor and an int4 control side by side.

3. Maintenance cost of the int3 path — the contract is scalar-reference-everywhere: every platform runs the exact-f32 scalar kernel; NEON is an optimization layered on top and tested bit-identical against it (including short tail groups and the real GLM I=7168). x86 SIMD is deliberately not in v1 because the target regime is fetch-bound — the kernel is not the bottleneck this format exists to move. And there is intentionally no IDOT path (per-group accumulation doesn't compose with the current int8-activation kernels, and prefill: batch the attention input projections (bit-identical, -4.5% prefill) #152 puts IDOT's quality economics under separate scrutiny), so the surface being maintained is one packing, one scalar kernel, one NEON overlay.

🤖 Generated with Claude Code

[JustVugg]

Heads-up: #242 (group-scaled int4) just merged to dev and it took the fmt=4 identifier. Your int3-g64 here also uses fmt=4. The on-disk detection doesn't collide (int3's weight-byte count differs from int4's, so it won't be mis-read as #242's format), but the t->fmt integer can only mean one thing in the dispatch/kernel switch. Could you renumber this to fmt=5 on the next rebase? That keeps int4-grouped (4) and int3-g64 (5) cleanly separable. Everything else about the ablation still stands.

[fabio-rovai]

New independent evidence for this PR's size/quality claim, from a per-projection ablation run last night.

I extended tools/quant_ablation.py with per-expert-projection overrides (scheme grammar gains -gate<b>/-up<b>/-down<b>, e.g. int4-g64-up2 = experts at int4-g64 but up_proj at 2 bits) and ran OLMoE-1B-7B-0924, n=200/task, quantized honestly from the bf16 source, 100% parameter coverage on every scheme:

scheme	hellaswag	arc_c	mmlu	mean acc_norm	Δ vs fp16
fp16	80.0	54.5	49.5	61.3	—
int4 (per-row, shipped)	76.5	48.5	37.0	54.0	−7.3pp
int4-g64	78.5	52.0	37.0	55.8	−5.5pp
int4-g64-up2	68.0	39.0	29.0	45.3	−16.0pp
int4-g64-up3	78.5	51.5	38.5	56.2	−5.1pp

Two takeaways relevant here:

1. 3-bit up is free. int4-g64-up3 matches (nominally beats) int4-g64 — so the most conservative deployment tranche of this PR's fmt=4 is up-only: keep gate/down at int4, put up_proj at int3-g64, and a container drops ~8% of expert bytes (disk and per-token RAM traffic) at no measured quality cost. For FP8-source container builders that's a one-flag change with the kernels this PR already provides. (FloE, arXiv 2505.05950, motivated testing up specifically.)

2. 2-bit is a cliff even confined to up (−16pp, the worst single move on the table) — consistent with quant_ablation: rotation preconditioning (-rot) and int3 schemes (#81) #132's ablation and, I'd argue, settles the [Feature]: Support 2 bit quantization #138-class "2-bit tier" question for absmax-style quantization: 3 bits with group scales is the floor, which is exactly where this PR sits.

Caveats: OLMoE as proxy model, n=200/task, 0-shot acc_norm; MMLU didn't respond to grouping in this run (recovery showed on hellaswag/arc). Happy to fold the per-projection scheme support into this PR or send it separately — it's ~15 lines in the scheme parser.

[JustVugg]

Could you rewrite this against current dev? I'm keeping it open rather than closing it, but it can't land in its present shape.

It's 121 commits behind dev. On glm.c, which changes several times a day, that isn't a rebase — it's a rewrite, and doing it for you would mean rewriting your design decisions without understanding why you made them. So I'd rather ask.

Your timing turned out to be better than anyone realised. #225 and #307 are converging on per-row int4 scales being the root cause of incoherent output — three users report a repetition loop, and @woolcoxm says gs=64 makes it disappear. Grouped scales (fmt=4) are now in the tree for CPU, and #298 is adding the CUDA half. int3-g64 sits exactly on that thread, so a rebased version would be reviewed against a live question rather than a hypothetical one.

What changed under you, which makes a rewrite cheaper than it sounds:

• CI now exists (ci: make check on ubuntu / windows (MSYS2 UCRT64) / macos — and it already caught a Windows test bug (#140) #143 + ci: GitHub Actions — engine, web, Python, CUDA syntax (free for public repos) #144, merged today): make check runs on ubuntu/windows/macos and the engine/web/python/CUDA-syntax jobs on Linux, for every PR. You'll get a verdict in ~2 minutes instead of waiting on a maintainer who can't test your platform. It caught a bug in my own test on its first Windows run.
• All PRs now target dev, not main. main is locked to admins.
• The backlog is being cleared (31 → 17 today). Your PR sat long enough to rot, and that's this repo's fault, not yours — two contributors independently rebuilt the same Windows CUDA loader this week because nobody triaged in time.

What would help it land fast: the smallest version that does one thing. A 300–700 line PR touching glm.c needs a maintainer who can run it, and for most platforms here that maintainer doesn't exist. A focused change with a test the CI can execute gets reviewed in an hour.

If you'd rather not, say so and I'll close it with thanks — no hard feelings either way. And if you think I've misjudged and it should go in as-is, push back: I've been wrong twice today already and both times a contributor caught it.

[fabio-rovai]

Rebased onto current dev and renumbered to fmt=5 as you asked. It now sits next to #242's grouped int4 (fmt=4) rather than on top of it.

One detection subtlety worth flagging: int3-g64 and grouped-int4 at gs=64 carry the same scale count (O*ceil(I/64)), so scale size alone cannot separate them. int3-g64 is tagged instead by its distinct weight-byte count (O*ceil(I/64)*24), while grouped-int4 keeps its detect_group_size path. qt_fmt_from_bytes returns fmt 1/2/3/5 from the weight bytes, then the loader upgrades fmt=2 to fmt=4 via the scale array. They coexist in qt_from_disk and both expert_load paths, and test_i4_grouped still passes alongside the new int3 tests.

The converter now carries two orthogonal axes: your per-tensor-type bits_map (--shared-bits etc.) and this PR's per-projection PROJ_BITS (--up-bits etc.). bits==3 routes to int3-g64 ahead of the grouped-int4 branch; everything else is unchanged.

Kept to the two original commits, 6 files, +503/-18. All 8 CI jobs pass. The first run earned its keep: it caught two real bugs in my own tests (an unconditional numpy import and a POSIX two-arg mkdir), now fixed to skip-if-absent and guard on _WIN32 respectively.

On the live question: if #225/#307 converge on per-row int4 scales being the repetition-loop cause, int3-g64 is the 3.5-bit point on that same grouped-scale thread. test_int3 measures 3.3x lower reconstruction error than per-row int4 on outlier rows. The Metal/CUDA gates fall back to CPU for fmt=5 today; happy to add the CUDA fmt=5 kernel to track #298, here or as a follow-up, whichever you prefer.

[JustVugg]

Heads-up: c/glm.c was just split into c/colibri.c + header modules (#391, now merged into dev). This PR touches the old glm.c, so it will need a rebase onto current dev with its changes moved into colibri.c (or the relevant extracted header: quant.h, sample.h, kv_persist.h, telemetry.h, grammar.h). The make glm target still works (alias of make colibri), so no build scripts break. Apologies for the churn — ping me if the rebase gets thorny and I can help place the hunks.

[JustVugg]

@fabio-rovai rebase heads-up: I tried to rebase this onto current dev, but it has semantic conflicts in colibri.c (8 hunks) that I won't resolve on your behalf — your int3-g64 (fmt=5) quant path overlaps code that moved a lot in dev: the #391 refactor (glm.c → colibri.c), the grouped-int4 fmt=4 work, and #413's qt_resolve_fmt which now validates weight/scale byte counts against [O,I] before use (so a new fmt=5 needs to register its expected layout there). Merging fmt=5 correctly against all that is a quant-layout judgment call I can't validate without measuring quality. Could you rebase and fold fmt=5 into the new qt_resolve_fmt gate? The converter side (convert_fp8_to_int4.py) also conflicts with #404's resume work. Happy to help with the mechanical bits once the layout call is made.

[fabio-rovai]

Rebased onto current dev (post-#426 head), layout call made. How the 8 hunks resolved:

qt_resolve_fmt registration (the judgment call you asked for): fmt=5 registers its distinct weight-byte layout O*ceil(I/64)*24 and its scale cardinality O*ceil(I/64) in #413's gate, validated against [O,I] like every other format. The ordering matters and is now documented in the code: int3-g64 and grouped-int4-at-gs=64 carry the same scale count, so the weight bytes are the int3 tag; row formats keep precedence for degenerate small-I shapes where byte counts can coincide (only possible at I<=256, where per-row is also legitimately a 1-group layout). test_int3.c now exercises the resolver directly, including the same-scale-count fmt=4/fmt=5 disambiguation pair at the real GLM I=7168.

#391 placement: matmul_i3 + pack_int3_g64 + the I3_* layout helpers went into quant.h next to their kernel family; the QT/loader/dispatch branches into colibri.c. One thing the rebase surfaced: the io_uring expert path (uring_finalize_load) still used the raw ?1:?2:3 byte inference — it never detects fmt=4 grouping and never sets gs, and would have mis-tagged int3 as int2. Converted it to qt_resolve_fmt like the other two expert paths; that closes the latent fmt-4 gap on Linux/uring as a side effect.

Backends: qt_cuda_upload returns 0 for fmt=5 (tensor stays CPU-side — same graceful pattern as your fmt=4 comment describes), the dense CUDA matmul gate excludes fmt=5, and Metal's existing gates (gemm fmt<=3, moe fmt 1/2) already reject it. Happy to add the CUDA fmt=5 kernel tracking #298's structure as a follow-up.

Converter / #404: reconciled — and one finding you'll want regardless of this PR: current dev's resume manifests record dict(PROJ_BITS) in four places (check_or_record_params + the --indir progress file) but the global is never defined, so those paths NameError at runtime today. Commit 2 supplies the definition those sites depend on, plus the --up-bits/--gate-bits/--down-bits flags.

Kept to the two original commits: 7 files, +528/-12. make check green locally on macOS (105 python tests + all C bins including the two new int3 tests).

[fabio-rovai]

@rajpratham1 your 14 Jul review asked for three things, and the version now on the branch answers each structurally rather than argumentatively:

1. Serialization/loading compatibility — the format no longer goes through ad-hoc detection. fmt=5 is registered in qt_resolve_fmt (security: reject malformed model tensors at the untrusted-mirror boundary (C half of #368) #413), the same validated gate every other format passes: weight bytes AND scale bytes are checked against [O,I] before use, and an unknown combination refuses instead of misreading.
2. Robustness of qt_fmt_from_bytes heuristics — that function no longer exists; it was superseded by qt_resolve_fmt upstream and this PR folds into it. The one genuine ambiguity (int3-g64 vs grouped-int4-at-gs=64 share scale cardinality; row vs grouped coincide at I<=256) is documented at the resolution site and covered by dedicated assertions in test_int3.c at the real GLM shape.
3. Maintenance of the separate int3 path — post-refactor: split glm.c → colibri.c + 4 header modules (−18%) #391 it lives in quant.h directly beside the int2/int4/int4-grouped kernel family it mirrors, scalar reference everywhere with NEON as an overlay, same structure as its siblings.

All 8 CI jobs are green on the rebased branch. Would you re-review when you get a chance?