Non-record: QAT & EMA negative results on SOTA stack (val_bpb=1.1426)

[MultiFe22]

Summary

• val_bpb: 1.1426 (baseline reproduction of PR Record: 10L Int5-MLP + BigramHash(10240) + SWA(0.4) + WD=0.04 (val_bpb=1.1428, mean 3 seeds) #180)
• Documents that adding QAT (STE fake-quantization) and EMA to the SOTA stack hurts performance due to throughput loss
• QAT costs 8% of training steps, EMA costs 32% — within the 10-minute budget, lost steps outweigh compression/averaging benefits

Results (8xH100 SXM, 600s)

Config	Steps	val_bpb	Artifact	Delta
Baseline (PR #180 repro)	6,684	1.1426	15.99 MB	—
+ QAT (warmup=500)	6,143	1.1473	15.69 MB	+0.005 (worse)
+ QAT + EMA	4,546	1.1606	16.89 MB	+0.018 (worse)

Key findings

• QAT: Better compression (15.69 vs 15.99 MB) but 8% fewer steps - net negative
• EMA: .cpu().clone() every step causes 32% throughput loss - catastrophic
• Implication: Techniques trading steps for inference quality are counterproductive under the 10-min budget

Test plan

• Baseline reproduction matches published SOTA (1.1426 vs 1.1428)
• QAT ablation on 8xH100
• QAT+EMA ablation on 8xH100
• All logs included