convert: --xbits e8 — produce and load the fmt=6 E8/IQ3 container (#452 step 4)#501
Merged
Merged
Conversation
…stVugg#452 step 4) The converter rotates expert rows (W@Q, block-diagonal FWHT) and packs them with the iq3 codec; the engine now recognizes the container (.qs = 4-byte tag, scales live in-block), rotates activations once per layer for gate/up and per-expert for down, and keeps fmt=6 tensors CPU-side (no CUDA kernel yet). The rotation signs are never stored: both sides regenerate the same xorshift64* stream, and the kernel fixture pins the agreement.
Contributor
Author
|
CI note: the Engine/linux/macos jobs will stay red until #500 merges — dev currently tracks 12 stale test binaries (from #368's conflict resolution), so make skips rebuilding |
…Vugg#452 step 2) The float simulation chose the scheme (51.5% on OLMoE); this measures what a SHIPPED container gets — weights through the real codec (98B blocks, parity sign cost, fp16 super-scales, regenerated xorshift rotation signs) and back to model space via the new unrotate_rows. Tensors whose input dim is not a multiple of 256 stay float, the same rule the converter enforces.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 4 of #452, as promised there: the container becomes producible and loadable. Stacks conceptually on #465 (kernel) and #458 (codec), both merged.
What
Converter —
--xbits e8onconvert_fp8_to_int4.py: routed-expert projections are rotated (W@Q, block-diagonal FWHT) and packed with the iq3 codec into fmt=6 bytes (98B/256 weights, 3.0625 bpw). All-or-nothing across gate/up/down — gate and up share one rotated input row in the engine, so a mixed layout would buy two gather buffers for nothing. Non-expert tensors take the existing paths untouched.Engine — the container loads and runs on the CPU tier:
qt_resolve_fmt: fmt=6 keeps the uniformname + name.qsconvention;.qsis a single-float tag (real scales live inside the 98-byte blocks), andns==4is the discriminator — every other format ships at least O floats of scales.quant.hrule prescribes: once per moe() call for gate/up (all routed experts of a layer share the row, ~1.4 ms/token on GLM dims vs ~11 ms if done per expert), per-expert for down (its input is the per-expert hidden).qt_cuda_uploadrejects it, same as fmt=5) and off every whitelisted fast path (XEXP/idot/Metal gate on fmt==2) — it flows down the generic CPU path wherematmul_e8dispatches. The CUDA sibling rides cuda: grouped-int4 (fmt=4) support in the expert-group kernels — opens the GPU tier to g64 and E8 containers (#334) #451's GroupDesc in a follow-up.Rotation signs are regenerated, never stored. Both sides draw the same xorshift64* stream (seed 417+n); the block tiling is one expression (
b = rem & -rem: 6144 → 2048+4096, 1536 → 512+1024). No rotation data in the container, no way for the sides to drift without the oracle catching it.Tests
test_e8_kernelgains a rotation oracle: the fixture now carries a rotated-weights section (Python-sideencode(rotate_rows(W)), raw x, Python'sQ^T x, float64 reference). The C side reproduces the rotation to 1.2e-7 and the rotated matmul to 9.3e-7 worst-rel.test_iq3_pack.pygains the frozen-PRNG golden bits (the constants are the spec — drift decodes every rotated container to garbage), tiling cases, and the orthogonality property ((W@Q)(Q^T x) ≈ Wx).make check114/114 locally.Limits, stated plainly