engine: fmt=6 E8/IQ3 lattice container — CPU decode kernel and dispatch (#452 step 3)#465
Merged
Merged
Conversation
main shipped v1.0.0 with prebuilt Windows binaries but no docs on main explaining what the .exe is or how to run it (JustVugg#450) — the quickstart lived only on dev. Bring docs/quickstart.md to main and add a Windows-prebuilt pointer to the README's Get started section, so users on the default branch find the steps. Docs-only. Refs JustVugg#450 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs: Quick Start guide + Windows prebuilt steps on main (JustVugg#450)
…tVugg#424) The site.yml workflow deploys on push to main, so the site only goes live once site/ is on main. Bring site/index.html + icons + the workflow so a GitHub Pages deploy (Settings -> Pages -> Source: GitHub Actions) publishes it at justvugg.github.io/colibri. Refs JustVugg#424 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
site: official website + Pages deploy on main (JustVugg#424)
Site (justvugg.github.io/colibri) is live but linked nowhere. Add website + latest-release badges and a Website header link. README-only, no code change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ch (JustVugg#452 step 3) Renumbered from fmt=5 (taken by the int3 dual-plane container, JustVugg#132) to fmt=6. quant.h gains matmul_e8 plus the 256x4 magnitude grid (ggml-common.h, MIT, header-local like the rest of the file). Decode expands one 32-weight sub-block into a stack buffer and then FMAs it against the activations: per-weight table lookups would dominate, per-sub-block expansion keeps a grid row (16 bytes) hot in L1 and lets the compiler vectorize the accumulate. Scales live inside the blocks, so `QT.s` is unused for this format and matmul_qt passes NULL. tests/test_e8_kernel.c is an oracle against tools/iq3_pack.py — the format's reference implementation. The fixture (tools/make_e8_fixture.py, regenerate if the layout moves) carries packed bytes, the codec's own dequant, an activation row and a float64 reference product, and the test checks both that expansion reproduces the codec's floats and that the matmul matches: e8 kernel oracle: O=24 I=512, dequant exact, matmul worst rel 5.59e-06 make check 111/111, zero warnings.
…JustVugg#452) fmt=6 stores W@Q for Q = D*H/sqrt(n), so inference computes y = W_rot @ (Q^T x) and activations must be transformed before the matmul. e8_fwht does that in a log-n pass (no matrix stored; the sign flip comes from the per-tensor seed) and e8_pow2_ceil gives the padded length for non-power-of-two dims (6144 -> 8192). Measured why the placement is documented in the header rather than left to the caller's judgement: the transform is 8.1 us on the hidden dim and 2.1 us on the expert-inter dim, so putting it inside the expert matmul costs ~11 ms/token (75 layers x 8 experts x 3 projections), while hoisting it to once per layer -- correct, because all routed experts of a layer consume the same input row -- costs ~1.4 ms. Against the ~47 ms/token CPU-tier sweep from JustVugg#431 that is ~3% instead of ~23%. Verified orthogonality and round-trip: worst |delta| 1.5e-07 over 1024 samples, energy preserved to 1e-9 relative. make check 111/111.
Footer said 'MIT license'; repo is Apache 2.0 (see LICENSE). Fix + link. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # README.md # site/index.html
# Conflicts: # c/Makefile
JustVugg
added a commit
that referenced
this pull request
Jul 21, 2026
…ntainer (#452 step 4) Completes the E8/IQ3 arc: the converter can now produce a fmt=6 container and the engine loads it (after #465's decode kernel and #458's index codec). The CI failures on this PR were the stray-test-binary breakage fixed by #500, not this change — verified locally on post-#500 dev: clean merge, no binaries reintroduced, clean build, token-exact unchanged (fp 32/32, int4 21/32), full make check OK. Thanks @ZacharyZcR.
This was referenced Jul 22, 2026
monotophic
added a commit
to monotophic/colibri
that referenced
this pull request
Jul 22, 2026
…nvention) Stacked on the previous two commits, closing out Branch B / PR 2's content. Adapts upstream_contribution/FORMATS_registry_draft.md (the Feature Request draft already posted, referenced verbatim on JustVugg#524) into an in-repo reference doc: renumbered fmt=100 -> fmt=7 throughout, line citations re-verified against this branch's actual current colibri.c/quant.h (substantially rewritten from the draft's source tree by the PR-1/PR-2 restructuring), and three content updates specific to this PR pair: - fmt=7's table row status is "this PR pair" (was "proposed"), since the ordinal is no longer private and the implementation is the thing under review. - A new "Scale encoding is a declared property (fmt=7)" section documents both values explicitly: f32 (implemented, what Z.ai's GLM-5.2-FP8 checkpoints ship and what this PR pair's CPU/Metal/repack code reads and writes) and UE8M0 (recognized by its distinct byte signature, refused by name, not implemented -- the DeepSeek-V4 finding the maintainer surfaced on JustVugg#524, credited here). Documents the exact refusal message, the collision check against fmt=1's own ns arithmetic, and that a stamp naming "fp8-e4m3-b128" cannot grant this build a decoder it doesn't have. - "Open questions for maintainer review" gains a second resolved question (scale-encoding-as-declared-property, decided on JustVugg#524) alongside the already-resolved JustVugg#465 sequencing question; the still-open questions (100+ convention acceptability, colibri.fmt key naming) carry forward unchanged. The PRIVATE ORDINAL BLOCK convention section is kept intact (per the task's explicit instruction), updated only to cite fmt=7 as the convention's own worked example of a private ordinal graduating to a public one. "How to add a format" gains one bullet: a format's Feature Request should declare which scale/secondary-array ENCODING(s) it implements, not just its byte layout, matching the precedent this PR pair sets. Read-only source: upstream_contribution/FORMATS_registry_draft.md (untracked, main checkout only) -- consulted for content, not modified; this commit's docs/FORMATS.md is a new, independently-committed file. No code changes in this commit; make glm METAL=1 / make test-c re-run as a sanity check (unaffected by a docs-only change) -- clean rebuild, zero warnings, full C suite green, exit 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
monotophic
added a commit
to monotophic/colibri
that referenced
this pull request
Jul 23, 2026
…nvention) Stacked on the previous two commits, closing out Branch B / PR 2's content. Adapts upstream_contribution/FORMATS_registry_draft.md (the Feature Request draft already posted, referenced verbatim on JustVugg#524) into an in-repo reference doc: renumbered fmt=100 -> fmt=7 throughout, line citations re-verified against this branch's actual current colibri.c/quant.h (substantially rewritten from the draft's source tree by the PR-1/PR-2 restructuring), and three content updates specific to this PR pair: - fmt=7's table row status is "this PR pair" (was "proposed"), since the ordinal is no longer private and the implementation is the thing under review. - A new "Scale encoding is a declared property (fmt=7)" section documents both values explicitly: f32 (implemented, what Z.ai's GLM-5.2-FP8 checkpoints ship and what this PR pair's CPU/Metal/repack code reads and writes) and UE8M0 (recognized by its distinct byte signature, refused by name, not implemented -- the DeepSeek-V4 finding the maintainer surfaced on JustVugg#524, credited here). Documents the exact refusal message, the collision check against fmt=1's own ns arithmetic, and that a stamp naming "fp8-e4m3-b128" cannot grant this build a decoder it doesn't have. - "Open questions for maintainer review" gains a second resolved question (scale-encoding-as-declared-property, decided on JustVugg#524) alongside the already-resolved JustVugg#465 sequencing question; the still-open questions (100+ convention acceptability, colibri.fmt key naming) carry forward unchanged. The PRIVATE ORDINAL BLOCK convention section is kept intact (per the task's explicit instruction), updated only to cite fmt=7 as the convention's own worked example of a private ordinal graduating to a public one. "How to add a format" gains one bullet: a format's Feature Request should declare which scale/secondary-array ENCODING(s) it implements, not just its byte layout, matching the precedent this PR pair sets. Read-only source: upstream_contribution/FORMATS_registry_draft.md (untracked, main checkout only) -- consulted for content, not modified; this commit's docs/FORMATS.md is a new, independently-committed file. No code changes in this commit; make glm METAL=1 / make test-c re-run as a sanity check (unaffected by a docs-only change) -- clean rebuild, zero warnings, full C suite green, exit 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
monotophic
added a commit
to monotophic/colibri
that referenced
this pull request
Jul 23, 2026
…nvention) Stacked on the previous two commits, closing out Branch B / PR 2's content. Adapts upstream_contribution/FORMATS_registry_draft.md (the Feature Request draft already posted, referenced verbatim on JustVugg#524) into an in-repo reference doc: renumbered fmt=100 -> fmt=7 throughout, line citations re-verified against this branch's actual current colibri.c/quant.h (substantially rewritten from the draft's source tree by the PR-1/PR-2 restructuring), and three content updates specific to this PR pair: - fmt=7's table row status is "this PR pair" (was "proposed"), since the ordinal is no longer private and the implementation is the thing under review. - A new "Scale encoding is a declared property (fmt=7)" section documents both values explicitly: f32 (implemented, what Z.ai's GLM-5.2-FP8 checkpoints ship and what this PR pair's CPU/Metal/repack code reads and writes) and UE8M0 (recognized by its distinct byte signature, refused by name, not implemented -- the DeepSeek-V4 finding the maintainer surfaced on JustVugg#524, credited here). Documents the exact refusal message, the collision check against fmt=1's own ns arithmetic, and that a stamp naming "fp8-e4m3-b128" cannot grant this build a decoder it doesn't have. - "Open questions for maintainer review" gains a second resolved question (scale-encoding-as-declared-property, decided on JustVugg#524) alongside the already-resolved JustVugg#465 sequencing question; the still-open questions (100+ convention acceptability, colibri.fmt key naming) carry forward unchanged. The PRIVATE ORDINAL BLOCK convention section is kept intact (per the task's explicit instruction), updated only to cite fmt=7 as the convention's own worked example of a private ordinal graduating to a public one. "How to add a format" gains one bullet: a format's Feature Request should declare which scale/secondary-array ENCODING(s) it implements, not just its byte layout, matching the precedent this PR pair sets. Read-only source: upstream_contribution/FORMATS_registry_draft.md (untracked, main checkout only) -- consulted for content, not modified; this commit's docs/FORMATS.md is a new, independently-committed file. No code changes in this commit; make glm METAL=1 / make test-c re-run as a sanity check (unaffected by a docs-only change) -- clean rebuild, zero warnings, full C suite green, exit 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 3 of #452's ladder. Stacked on #458 (the reference codec) — review the top commit.
Renumbered: fmt=6, not fmt=5
fmt=5is already the int3 dual-plane container (#132) ondev. The lattice container takes fmt=6; #458's codec and docs are updated to match in this branch.What
quant.hgainsmatmul_e8and the 256×4 magnitude grid (fromggml-common.h, MIT — header-local, like the rest of the file, since the engine is one translation unit).Decode strategy: expand one 32-weight sub-block into a stack buffer, then FMA it against the activations. Per-weight table lookups would dominate the loop; per-sub-block expansion keeps a grid row (16 B) hot in L1 and lets the compiler vectorize the accumulate. Scales live inside the blocks, so
QT.sis unused for this format andmatmul_qtpasses NULL — worth knowing before writing the SIMD or CUDA sibling.Verification
tests/test_e8_kernel.cis an oracle againsttools/iq3_pack.py, the format's reference implementation — the fixture comes from the codec, never from the kernel. It carries packed bytes, the codec's own dequant, an activation row and a float64 reference product, and checks two things: that sub-block expansion reproduces the codec's floats, and that the matmul matches.make check111/111, zero warnings.Not in this PR
Scalar-only decode for now (the int3 container shipped the same way, #132). The AVX2/NEON expansion, the CUDA sibling on #451's
GroupDescpath, and the converter emitting fmt=6 are the next increments — each wants its own A/B rather than a single large change.