Non-record: 11L PartialRoPE + LNScale + EMA + SWA + TTT (1xH100 107min, val_bpb=1.2207, 15.4MB)

[nathon-lee]

Summary

Non-record submission: 11-layer 512-dim GPT with PartialRoPE, LNScale, SmearGate,
BigramHash(2048×64), U-Net skip connections, EMA + SWA, and test-time training (TTT).

Trained on 1×H100 PCIe for ~107 minutes (~equivalent to 8×H100 SXM 10 minutes).

Results

Metric	Value
val_bpb (pre-TTT)	1.2207
val_loss	2.0611
Training steps	3374 / 20000
Training time	~107 min (1×H100 PCIe)
Model params	26.7M
Artifact size	15.4 MB ✅

Key Techniques

• Partial RoPE (16/64 dims): position encoding on subset of head dims
• LN Scale: RMSNorm damped by 1/√(layer+1)
• SmearGate: per-dim gate blending current + previous token
• BigramHash(2048, dim=64): hash-based bigram context embeddings
• U-Net skip connections with learnable weights
• Muon optimizer (Newton-Schulz) + Adam for embeddings
• EMA(0.997) + SWA (last 40%)
• Uniform int5 quantization + zstd-22
• Sliding window eval (stride=64) + SGD TTT (3 epochs)

Changes from v1

• BigramHash: 4096×128 → 2048×64 (-426K params)
• Quantization: mixed int5/int6 → uniform int5
• Artifact: 17.4MB ❌ → 15.4MB ✅

Checklist

• README.md
• submission.json
• train.log
• train_gpt.py
• Artifact under 16MB ✅'