tools: repair int4-converted MTP heads in place; warn on --mtp --ebits <8

[NeuralNotwerk]

Summary

Adds tools/repair_mtp_int8.py — an in-place repair for colibri containers whose MTP head was quantized at per-row int4 — and a warning in convert_fp8_to_int4.py when --mtp is combined with --ebits <8 per-row scales.

The bug this repairs (found in a published container)

A pre-converted GLM-5.2 container on HF (jlnsrk/GLM-5.2-colibri-int4) shipped MTP shards made with an explicit --ebits 4, overriding --mtp's int8 default. Result: MTP acceptance 0% on that container, on every backend, while the main model works perfectly.

The mechanism is worse than ordinary quantization loss. model.layers.78.eh_proj.weight [6144, 12288] multiplies the MTP concat [embedding_norm ; hidden_norm], and its column halves differ in scale ~20-30× per row (embedding-half absmax ≈ 0.05, hidden-half ≈ 1.4-1.6). Per-row int4 has one scale per row (absmax/7), so every embedding-half weight sits at 0.24-0.34 of a quantization step — np.rint rounds the entire embedding half of all 6144 rows to exact zeros (packed bytes 0x88). The draft head literally cannot see the input token; drafts are unrelated to context (true next token ranked ~#17,000 of 154,880 in draft logits).

Verified byte-for-byte: running this repo's quant_int4 on the official zai-org/GLM-5.2-FP8 BF16 rows reproduces the published bytes exactly, scales included. This is also the mechanism behind issue #8's measurement (int4 MTP: 0-4% acceptance; int8: 39-59%) — the asymmetry explains why int4 doesn't just degrade the head but kills it.

Full elimination trail before the data was implicated (engine exonerated): CPU and Metal builds identical garbage; unpatched upstream identical; IDOT=0 exact kernels, ABSORB=0, SPEC_PIN=0, expert pinning — all 0%; stage-by-stage NumPy replay of mtp_draft matched the C engine at every step (norms 1e-8, matmuls ≤0.9%). The engine computes broken data faithfully.

What the repair script does

python3 tools/repair_mtp_int8.py --snap /path/to/glm52_i4 --dry-run   # inspect
python3 tools/repair_mtp_int8.py --snap /path/to/glm52_i4             # repair

• Finds the MTP layer's dense tensors stored as per-row int4 (eh_proj, q_a/q_b/kv_a/kv_b/o_proj, shared experts). Routed experts are left untouched.
• Re-downloads only those tensors from the FP8 source repo (~355 MB of HTTP range reads — no full 756 GB re-download, no torch, no HF token).
• Dequantizes (BF16 / FP8+128×128 block scales) and requantizes at int8 with quant_int8's exact math.
• Rewrites the affected shards atomically, keeping originals as *.bak-int4; all untouched tensors stay byte-identical.
• The engine picks up int8 automatically (qt_from_disk infers format from blob size). Cost: ~+133 MB disk/resident.

Validation (GLM-5.2 744B int4 container, macOS ARM, CPU + Metal)

	before repair	after repair
MTP acceptance (greedy)	0% (0/22)	100% (10/10)
tokens/forward	1.11	3.20
decode tok/s (Metal, 4-bit KV)	~0.18	0.30

eh_proj embedding-half cosine vs official weights: 0.0000 → 0.97.

A companion PR replacing the broken shards directly on the HF container: https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4/discussions/3

Converter change

convert_fp8_to_int4.py now prints a warning when --mtp is used with --ebits <8 and per-row scales, pointing at the default (--ebits 8) or --group-size 128 (group scales give the embedding half its own scale, making int4 viable).

🤖 Generated with Claude Code