tools: fmt=5 index codec — the deployable bytes for the E8/IQ3 container (#452 step 2)#458
Merged
Merged
Conversation
…ep 1) Adds `-iq3` to the ablation harness: a faithful torch model of llama.cpp's deployed 3.06-bpw IQ3_XXS format — 256-entry 4-dim magnitude grid (extracted from ggml-common.h, MIT), signs factored per 8 weights with the odd-parity constraint priced in (a violating block flips its smallest-magnitude sign), fp16 super-scale per 256 + 4-bit sub-scale per 32 searched over all 16 codes. Nearest-grid search runs as chunked matmul-argmin (|g|^2 - 2 q.g) — a full cdist materializes tens of GB on a 100M-param tensor and OOMed the first run. Measured (OLMoE-1B-7B, n=200 x hellaswag/arc/mmlu; the first four rows reproduce the published ablations exactly): fp16 58.0% int4 per-row 48.7% (-9.3pp, the shipped container's scheme) int3-g64 50.5% (-7.5pp) int3-g64-e8-rot 51.5% (-6.5pp, simulated rate-scaled ball) int3-iq3 49.3% (-8.7pp) int3-iq3-rot 51.5% (-6.5pp) The deployable IQ3 codebook plus rotation exactly ties the simulated E8 ball — that settles JustVugg#452's codebook decision toward the IQ3-style block structure, with rotation mandatory (worth 2.2pp on this codebook).
…ustVugg#452 step 2) JustVugg#453 settled the scheme; this produces the actual bytes. iq3_pack.encode/decode implement the container layout so the converter, the engine and the decode kernels can all be written against one spec: 98 bytes per 256 weights = 3.0625 bpw [ 0..63] uint8 grid index per 4-dim magnitude block [64..95] uint32 x8 — four 7-bit sign words + 4-bit sub-scale per 32 [96..97] fp16 super-scale Signs use the published odd-parity trick: 7 of every 8 are stored and the 8th is derived, so the encoder flips the smallest-magnitude weight of any block whose true signs would violate parity — the cost the JustVugg#453 ablation priced in, now actually paid. Sub-scale is searched over all 16 codes against the stored (fp16-rounded) super-scale, so encode-time and decode-time arithmetic agree. GLM-5.2 routed experts under this container: 372.7 -> 281.2 GB (-24.6%), and a 176 GB VRAM tier holds ~12,180 experts instead of ~9,190 (+33%). tests/test_iq3_pack.py: byte budget, deterministic encode, decode checked value-by-value against an independent loop-based reader written straight from the layout, sign-parity closure, and reconstruction quality in the band the chosen scheme measured (rel-RMSE 0.195 vs the torch model's 0.195).
CI keeps the runtime path dependency-free, so the Python test job runs without numpy and the new codec tests errored on import. Guard the import and skip the class instead — verified both ways: 6/6 run where numpy is present, 6/6 skip cleanly where it is not.
This was referenced Jul 20, 2026
JustVugg
added a commit
that referenced
this pull request
Jul 21, 2026
…ntainer (#452 step 4) Completes the E8/IQ3 arc: the converter can now produce a fmt=6 container and the engine loads it (after #465's decode kernel and #458's index codec). The CI failures on this PR were the stray-test-binary breakage fixed by #500, not this change — verified locally on post-#500 dev: clean merge, no binaries reintroduced, clean build, token-exact unchanged (fp 32/32, int4 21/32), full make check OK. Thanks @ZacharyZcR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 2 of #452's ladder. Stacked on #453 (which settled the scheme) — review the top commit.
What
#453 proved the scheme (IQ3-style codebook + rotation ties our simulated E8 ball, 51.5% vs 51.5%). That code quantizes to lattice points and keeps floats. This produces the actual bytes and reads them back, so the converter, the engine and the CPU/CUDA decode kernels can all be written against one spec instead of three interpretations.
Two details that matter for anyone writing a kernel against this:
Payoff arithmetic (GLM-5.2 routed experts)
On the bandwidth-walled decode path those bytes are speed, and #431's forensics put the CPU-tier expert sweep at the top of the critical path — this is the lever aimed at it.
Test
tests/test_iq3_pack.py(6 cases,make checknow 89/89): byte budget, deterministic encode, decode compared value-by-value against an independent loop-based reader written straight from the layout comment (so a shared bug in the vectorized path surfaces), sign-parity closure across every 8-weight block, and reconstruction quality in the band the chosen scheme measured (rel-RMSE 0.195 vs the torch model's 0.195).Next: fmt=5 in the container/QT plumbing + CPU decode kernel, then the CUDA sibling on #451's
GroupDescpath.