feat: cache-aware routing + co-activation expert clustering

[userFRM]

Summary

Two complementary optimizations targeting the 56% I/O bottleneck (2.41ms of 4.28ms per layer). Both are opt-in with zero impact when disabled.

1. Cache-Aware Routing (--cache-aware)

Modifies expert selection to prefer experts already in the OS page cache, with bounded quality degradation.

How it works:

1. Standard topK on raw gate logits (unchanged routing math)
2. Classify top-K experts as cached/uncached via LRU access tracking
3. For each uncached slot (weakest first), substitute the best cached expert whose score is within tolerance * score_range of the evicted expert

Why it helps: With 71% page cache hit rate, 29% of reads hit cold SSD (5.5 GB/s vs 32 GB/s). Biasing toward cached experts reduces miss rate to ~15%, cutting I/O time by ~50%.

Quality bound: Only near-tied experts are swapped. --cache-tolerance 0 = identical to baseline.

2. Co-Activation Expert Clustering

Offline tool + runtime support to reorder experts on disk so frequently co-activated ones are physically adjacent.

Measured on M1 Pro (cold SSD, F_NOCACHE):

Pattern	Time (28MB)	Throughput
4 scattered preads	8.76ms	3.2 GB/s
4 adjacent preads	6.33ms	4.4 GB/s

38% faster cold reads when co-activated experts are adjacent.

How it works:

1. Generate routing log: ./infer --collect-routing routing.bin --tokens 200
2. Cluster experts: python3 cluster_experts.py --routing routing.bin --packed-dir packed_experts
3. This produces .map files (1KB each) that translate logical expert indices to physical file positions
4. Runtime loads .map files automatically — zero config needed
5. Pread tasks are sorted by file offset for sequential I/O order

When no .map files exist: identity mapping, zero overhead, identical behavior.

Usage

# Cache-aware routing (default tolerance=0.10)
./infer --prompt "..." --tokens 100 --cache-aware

# Conservative (minimal quality impact)
./infer --prompt "..." --tokens 100 --cache-aware --cache-tolerance 0.05

# Generate clustering data then use both features
./infer --prompt "..." --tokens 200 --collect-routing routing.bin
python3 cluster_experts.py --routing routing.bin --packed-dir metal_infer/packed_experts
./infer --prompt "..." --tokens 100 --cache-aware

Safety

• K clamped to MAX_K preventing stack overflow
• K=1 uses absolute tolerance fallback
• Server mode preserves cache state across requests
• tolerance=0 produces identical behavior to baseline
• No .map files = identity mapping, zero overhead

Background: SVD analysis on real weights

Ran SVD on real Qwen3.5-397B expert weights to validate design choices:

• Expert cosine similarity: 0.0009 (orthogonal — confirms routing-level optimization is the right approach)
• Rank for 90% variance: 714 (full-rank — confirms data must be read in full, no shortcuts)

Test plan

• --cache-aware with --cache-tolerance 0 produces identical tokens to baseline
• --cache-aware shows reduced expert_io in --timing output
• cluster_experts.py produces valid .map files from routing log
• No .map files = identical behavior
• Long-context generation (>200 tokens) with --cache-aware
• Serve mode preserves cache state across requests