Skip to content

convert: --xbits e8 — produce and load the fmt=6 E8/IQ3 container (#452 step 4)#501

Merged
JustVugg merged 2 commits into
JustVugg:devfrom
ZacharyZcR:feat/fmt6-converter
Jul 21, 2026
Merged

convert: --xbits e8 — produce and load the fmt=6 E8/IQ3 container (#452 step 4)#501
JustVugg merged 2 commits into
JustVugg:devfrom
ZacharyZcR:feat/fmt6-converter

Conversation

@ZacharyZcR

Copy link
Copy Markdown
Contributor

Step 4 of #452, as promised there: the container becomes producible and loadable. Stacks conceptually on #465 (kernel) and #458 (codec), both merged.

What

Converter--xbits e8 on convert_fp8_to_int4.py: routed-expert projections are rotated (W@Q, block-diagonal FWHT) and packed with the iq3 codec into fmt=6 bytes (98B/256 weights, 3.0625 bpw). All-or-nothing across gate/up/down — gate and up share one rotated input row in the engine, so a mixed layout would buy two gather buffers for nothing. Non-expert tensors take the existing paths untouched.

Engine — the container loads and runs on the CPU tier:

  • qt_resolve_fmt: fmt=6 keeps the uniform name + name.qs convention; .qs is a single-float tag (real scales live inside the 98-byte blocks), and ns==4 is the discriminator — every other format ships at least O floats of scales.
  • activation rotation at the placement the quant.h rule prescribes: once per moe() call for gate/up (all routed experts of a layer share the row, ~1.4 ms/token on GLM dims vs ~11 ms if done per expert), per-expert for down (its input is the per-expert hidden).
  • fmt=6 stays off CUDA (qt_cuda_upload rejects it, same as fmt=5) and off every whitelisted fast path (XEXP/idot/Metal gate on fmt==2) — it flows down the generic CPU path where matmul_e8 dispatches. The CUDA sibling rides cuda: grouped-int4 (fmt=4) support in the expert-group kernels — opens the GPU tier to g64 and E8 containers (#334) #451's GroupDesc in a follow-up.

Rotation signs are regenerated, never stored. Both sides draw the same xorshift64* stream (seed 417+n); the block tiling is one expression (b = rem & -rem: 6144 → 2048+4096, 1536 → 512+1024). No rotation data in the container, no way for the sides to drift without the oracle catching it.

Tests

  • test_e8_kernel gains a rotation oracle: the fixture now carries a rotated-weights section (Python-side encode(rotate_rows(W)), raw x, Python's Q^T x, float64 reference). The C side reproduces the rotation to 1.2e-7 and the rotated matmul to 9.3e-7 worst-rel.
  • test_iq3_pack.py gains the frozen-PRNG golden bits (the constants are the spec — drift decodes every rotated container to garbage), tiling cases, and the orthogonality property ((W@Q)(Q^T x) ≈ Wx).
  • make check 114/114 locally.

Limits, stated plainly

…stVugg#452 step 4)

The converter rotates expert rows (W@Q, block-diagonal FWHT) and packs them
with the iq3 codec; the engine now recognizes the container (.qs = 4-byte tag,
scales live in-block), rotates activations once per layer for gate/up and
per-expert for down, and keeps fmt=6 tensors CPU-side (no CUDA kernel yet).
The rotation signs are never stored: both sides regenerate the same
xorshift64* stream, and the kernel fixture pins the agreement.
@ZacharyZcR

Copy link
Copy Markdown
Contributor Author

CI note: the Engine/linux/macos jobs will stay red until #500 merges — dev currently tracks 12 stale test binaries (from #368's conflict resolution), so make skips rebuilding tests/test_e8_kernel and the runner hits EACCES before this PR's code is even exercised. Green everywhere else; re-run after #500 lands.

…Vugg#452 step 2)

The float simulation chose the scheme (51.5% on OLMoE); this measures what a
SHIPPED container gets — weights through the real codec (98B blocks, parity
sign cost, fp16 super-scales, regenerated xorshift rotation signs) and back to
model space via the new unrotate_rows. Tensors whose input dim is not a
multiple of 256 stay float, the same rule the converter enforces.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants