epic: Project Bespoke — Extract mnemonic's own LLM from Gemma 4 31B via structured pruning

[CalebisGross]

Vision

Mnemonic's "brain" is currently 99% someone else's model (Qwen 3.5 2B) with 1% of our adapters on top. Project Bespoke extracts a purpose-built model that exists because of mnemonic — smaller, faster, and genuinely ours.

The core insight from the Lottery Ticket Hypothesis (Frankle & Carlin, 2019): dense pretrained models contain sparse subnetworks that, trained in isolation, match full-model performance. Applied at scale via Sheared LLaMA (Xia et al., ICLR 2024): structured pruning with learned masks reduced LLaMA2-7B to 1.3B at 3% of from-scratch training cost, outperforming all comparably-sized models.

Target: Take a ~7-9B dense pretrained model → extract a 1.5-2B structured subnetwork optimized for mnemonic's tasks → export as a standalone GGUF → deploy as the daemon's native model.

Why This Matters

Property	Current (Qwen 2B + spokes)	Bespoke 1.5B
Identity	"Qwen with adapters"	"Mnemonic's model"
Total params	2B + 25M adapters	~1.5B standalone
Inference VRAM	~3GB	~1-1.5GB
Encoding latency	~20s	~5-8s (est.)
Daemon + training	Mutually exclusive (VRAM)	Can coexist
Architecture	Frozen base + hooks	Clean single model
Spoke overhead	Per-layer injection at inference	Baked in (or minimal)

Architecture Decision: Dense vs Hybrid Base

Critical finding: Qwen 3.5 9B uses a hybrid architecture (Gated DeltaNet + Gated Attention + Sparse MoE). All published structured pruning methods (Sheared LLaMA, SliceGPT, LLM-Pruner) assume dense attention + dense FFN. Adapting to DeltaNet/MoE is novel research.

Decision needed (Phase 0): Evaluate these candidates:

1. Qwen 3.5 9B (hybrid) — Richest representations but DeltaNet/MoE complicates pruning
2. Qwen 2.5 7B (dense) — Pure attention, Sheared LLaMA code works directly, but older generation
3. Qwen 3.5 4B (dense?) — Need to verify architecture. Smaller starting point = less aggressive pruning needed (4B → 1.5B = 2.7:1 vs 9B → 1.5B = 6:1)
4. Other dense 7-9B models — LLaMA 3 8B, Gemma 2 9B (pure attention, well-studied)

Research References

• The Lottery Ticket Hypothesis (Frankle & Carlin, 2019) — foundational theory
• Sheared LLaMA (Xia et al., ICLR 2024) — primary method. 7B → 1.3B via targeted structural pruning + continued pretraining. Code
• SliceGPT (Ashkboos et al., ICLR 2024) — post-training PCA-based width reduction, no retraining. Good for initial 20-25% reduction.
• Wanda (Sun et al., ICLR 2024) — fast pruning via weight × activation magnitude. Unstructured only, not applicable for architecture extraction.
• LLM-Pruner (Ma et al., NeurIPS 2023) — dependency-graph-aware structured pruning + LoRA recovery.

Phases

Phase 0: Feasibility & Architecture Selection (EXP-28)

Goal: Pick the base model and validate that structured pruning works on it.

• Profile Qwen 3.5 9B, 4B, and one dense alternative (LLaMA 3 8B or Gemma 2 9B) on mnemonic encoding tasks
• Verify architecture details: which are dense attention, which are hybrid/MoE
• Run SliceGPT (no retraining) on each candidate at 25% reduction — measure encoding quality retention
• Estimate MI300X compute budget for full Sheared-LLaMA-style pruning on each
• Decide: which base model to prune
• Hardware: MI300X for profiling, local 7800 XT for quality evaluation

Phase 1: Full Fine-Tune Baseline (MI300X)

Goal: Establish quality ceiling and collect importance metrics.

• Full fine-tune chosen model (all params unfrozen) on v7 encoding dataset
• Collect per-layer importance metrics during training: gradient magnitude, activation variance, attention entropy
• Evaluate: encoding quality (7 faithfulness metrics), stress test, eval loss
• This becomes the quality ceiling — the pruned model must approach this

Phase 2: Structured Pruning (MI300X)

Goal: Find the minimal architecture that maintains encoding quality.

Following Sheared LLaMA methodology:

• Define target shape: ~1.5B params (~20 layers, hidden 2048, 16 heads, FFN 5504)
• Learn pruning masks: joint optimization of task loss + pruning objective (~3K steps)
• Progressive targets: evaluate at 4B, 3B, 2B, 1.5B — find the quality cliff
• For each target size, record: which layers survived, which heads, which FFN dimensions
• Key question: Does the pruned 1.5B from 9B beat the existing full 2B on encoding? If not, pruning is not worth it.

Phase 3: Continued Pretraining (MI300X)

Goal: Recover quality lost during pruning.

• Continue pretraining the pruned model on mnemonic's encoding data
• Dynamic batch loading (per Sheared LLaMA): weight data mix by per-domain loss
• Evaluate after every 1K steps: faithfulness metrics, stress test
• Target: match or exceed Phase 1 quality ceiling on encoding tasks

Phase 4: Lottery Ticket Validation

Goal: Test whether initialization matters (the core LTH claim).

• Take the pruned architecture from Phase 2
• Variant A: Keep trained weights (standard pruning)
• Variant B: Reset to original pretrained initialization, retrain from scratch on encoding data
• Variant C: Random initialization (ablation — should fail, confirming pretrained init matters)
• Compare A vs B vs C on all metrics
• If B ≈ A: the initialization IS the value, confirming true lottery ticket
• If A >> B: the trained weights matter more, standard pruning is sufficient

Phase 5: Export & Local Deployment

Goal: Standalone bespoke model running in the daemon.

• Export pruned model as GGUF (standard format, no adapter hooks needed)
• Benchmark on RX 7800 XT: tok/s, VRAM, encoding latency
• Target: >200 tok/s, <1.5GB VRAM, <10s per encoding
• Integration test with mnemonic daemon (replace llama-server model)
• Lifecycle test: full 8-phase lifecycle with bespoke model

Phase 6: Felix-LM Integration

Goal: Add task-specific spokes to the bespoke post.

• Train spoke adapters on the pruned model (encoding, synthesis, retrieval)
• Test hot-swap capability: switch between spoke sets at inference
• This is the full Felix-LM vision: small bespoke post + swappable spoke tools
• Final benchmark: the complete system

Hardware Plan

Phase	Hardware	Estimated Time	Estimated Cost
Phase 0	MI300X droplet + local	1-2 days	~$20-30
Phase 1	MI300X droplet	4-8 hours	~$10-20
Phase 2	MI300X droplet	8-16 hours	~$20-40
Phase 3	MI300X droplet	8-24 hours	~$20-50
Phase 4	MI300X droplet	4-8 hours	~$10-20
Phase 5	Local 7800 XT	1-2 hours	$0
Phase 6	Local 7800 XT	4-8 hours	$0

Total estimated: ~$80-160 in MI300X compute, 1-2 weeks of research time.

Success Criteria

1. Quality: Pruned model matches or exceeds current Qwen 2B + spokes on all 7 faithfulness metrics + 7/7 stress test
2. Speed: >200 tok/s inference on RX 7800 XT (current: 95 tok/s)
3. Size: <1.5GB VRAM for inference (current: ~3GB)
4. Identity: Standalone GGUF with no external model dependency — this is mnemonic's model

Risks

1. Hybrid architecture complexity — If the chosen model is DeltaNet/MoE, pruning tool adaptation could take weeks
2. Quality cliff — The encoding task might not have enough signal to guide structural pruning decisions, leading to arbitrary cuts
3. Diminishing returns — If the pruned 1.5B doesn't beat the existing full 2B, the entire effort is wasted. Phase 2 has an explicit go/no-go gate for this.
4. MI300X cost — Iterative pruning experiments could exceed budget. Set hard $150 cap.

Relationship to Other Work

• EXP-26 (v7 data training) should complete first — establishes the data quality baseline
• EXP-27 (Qwen 3.5 4B) explores a larger base model, may inform Phase 0 architecture selection
• Issue EXP-25: Faithfulness probe — can Qwen 2B spokes learn to encode diverse inputs? #381 (faithfulness) — solved by EXP-25/26, provides the evaluation framework for Project Bespoke
• Felix-LM design paper (~/Projects/felixlm/docs/felix_lm_design.tex) — this is the path to realizing the post-and-spoke vision

[CalebisGross]

Architecture Decision: Gemma 4 31B

After research, the base model is Gemma 4 31B (30.7B dense, Apache 2.0).

Why Gemma 4 31B

• Dense transformer — 60 layers of standard attention. No MoE, no DeltaNet, no PLE. Structurally clean for pruning.
• Best quality in the Gemma 4 family. The E2B had faithfulness issues in our testing (EXP-19/20b).
• 60 layers — enormous pruning headroom. Even removing 40 layers leaves 20, roughly a 2B-class model.
• Apache 2.0 — fully permissive, commercially usable, no restrictions.
• Already in our ecosystem — llama.cpp Gemma 4 support exists, we have gemma_spoke_adapter.py infrastructure.
• MI300X fits it — 31B bf16 = ~62GB. MI300X has 192GB. Room for gradients + optimizer.

Why NOT alternatives

• Qwen 3.5 (all sizes) — DeltaNet hybrid architecture makes structured pruning unpredictable
• Gemma 4 E2B/E4B — E2B quality insufficient (proven in EXP-19/20b), E4B uses non-standard PLE
• Gemma 4 26B-MoE — MoE pruning is a different research problem
• LLaMA — excluded per project decision

Pruning Targets (progressive)

1. 31B → 8B (~4:1) — sanity check, should be easy
2. 31B → 4B (~8:1) — stretch goal
3. 31B → 2B (~15:1) — aspirational, find the quality cliff
4. 31B → 1.5B (~20:1) — theoretical minimum for the task

Unsloth

Training framework, not models. Useful for Phase 1/3 fine-tuning (2x faster LoRA). QLoRA broken on AMD (bitsandbytes HSA issue), but full bf16 works. Will use for MI300X fine-tuning phases.

Hardware Budget

• Pruning + continued pretraining on MI300X: 31B bf16 (62GB) + gradients (62GB) + optimizer (124GB) ≈ 188GB peak with Adam. Fits MI300X (192GB) with gradient checkpointing. Muon optimizer uses less memory than Adam.
• Resulting 2B model runs locally on RX 7800 XT.

[CalebisGross]

Research Status Update

Detailed findings posted on #387. Key takeaways:

The Good

• Gemma 4 31B is architecturally clean for pruning — dense (no MoE), no PLE, uniform FFN
• The L0 mask concept from Sheared LLaMA is model-agnostic and transferable
• 60 layers gives massive pruning headroom

The Hard

• Two structurally different layer types (sliding vs full attention) with different head dims, GQA ratios, and even different projection matrices (full layers have no v_proj — K=V). No published pruning work handles this.
• Sheared LLaMA codebase is deeply LLaMA-specific and tied to MosaicML Composer. Full port is 2-4 weeks.
• 31B + Adam exceeds MI300X VRAM (248GB > 192GB). Need alternative optimizer or approach.

Proposed Strategy

Start with Approach B (Layer-Drop + Width-Reduce) for fast validation:

1. Profile 31B layer importance on encoding tasks
2. Extract top-20 layers into standalone model
3. Apply width reduction if needed
4. LoRA fine-tune on encoding data
5. Evaluate against current system

This answers "is the vision viable?" in days, not weeks. If yes, invest in full Sheared-LLaMA-style joint optimization (Approach A). If no, pivot.

Sub-issue: #387

[CalebisGross]

Candidate Model Evaluation — Pre-Bespoke Gate

Before committing to the structured pruning path or retraining Qwen, we're evaluating the current crop of small models on mnemonic's encoding task. The AI/LLM landscape has moved fast since this issue was written — several purpose-built small models now exist that weren't available at the time.

Why

• EXP-25 (EXP-25: Faithfulness probe — can Qwen 2B spokes learn to encode diverse inputs? #381) confirmed the encoding quality problem is data, not capacity — Qwen 3.5 2B + spokes can learn faithful encoding with diverse data
• v7 dataset (5,292 train / 588 eval) is built and ready for fine-tuning
• New 3B models (SmolLM3, Ministral 3) were designed with structured output fidelity as a pretraining objective
• Pruning 31B → 2B doesn't inherently solve hallucination — the pruned model carries more latent knowledge that could bleed through
• We should know which base model benefits most from v7 fine-tuning before investing GPU time

Candidates

Model	Params	Architecture	License	Rationale
SmolLM3 3B	3B	Dense, GQA, NoPE	Apache 2.0	HuggingFace's model; strong structured output + tool calling benchmarks
Ministral 3 3B	3B	Dense	Apache 2.0	Native JSON output + function calling; designed for edge
Qwen 3.5 2B	2B	Hybrid DeltaNet	Apache 2.0	Current model, no spokes — raw baseline
Qwen 3.5 0.8B	0.8B	Hybrid DeltaNet	Apache 2.0	Minimum viable size test

Evaluation Protocol

Using the existing EXP-25 faithfulness framework (#381):

• Test set: EXP-25 probe inputs (diverse, adversarial, minimal, dense-number, production-format)
• Metrics: EPR, FR, TED, CCS, MIH, NP, SC (all 7 from EXP-25: Faithfulness probe — can Qwen 2B spokes learn to encode diverse inputs? #381)
• Conditions: Zero-shot (instruction prompt only) and few-shot (3 examples in context)
• Hardware: RX 7800 XT via llama.cpp (GGUF quantized — Q4_K_M and Q8_0)
• Baseline: Current Qwen 3.5 2B + spokes (EXP-25 rerun numbers)

Decision Gate

The evaluation answers: which base model is the best starting point for v7 fine-tuning?

• If a candidate matches or beats Qwen zero-shot → strong signal for that architecture
• If all candidates score poorly zero-shot but one shows better few-shot scaling → that model likely benefits most from fine-tuning
• If Qwen still leads → proceed with EXP-26 (Qwen + v7 data), defer Bespoke pruning

This doesn't close Project Bespoke — it informs whether we're pruning the right source model, or whether an off-the-shelf small model is a better foundation.

[CalebisGross]

EXP-29 Pre-Registered — Candidate Model Evaluation

Pre-registered in training/docs/experiment_registry.md. Full protocol documented there.

Exhaustive 2026 Model Search Results

Searched every major lab and framework for sub-4B open-weight models released in 2026:

Qualified (6 candidates):

• Qwen 3.5 0.8B, 2B, 4B (Mar 2 — Hybrid DeltaNet, Apache 2.0)
• Nemotron 3 Nano 4B (Mar 17 — Mamba-2 + Transformer, NVIDIA Open)
• Gemma 4 E2B, E4B (Apr 2 — PLE, Apache 2.0)

Searched and ruled out:

• MiniMax (no sub-4B), DeepSeek (no sub-4B), GLM-5/5.1 (744B+ MoE only)
• Cohere Tiny Aya 3.35B (CC-BY-NC non-commercial license)
• Qwen3-Coder-Next (80B total / 3B active — exceeds 16GB VRAM)
• Nemotron 30B-A3B (Dec 2025, 31.6B total)
• Falcon 3 3B (Dec 2024), Falcon-E 3B (Apr 2025), OLMo 2 1B (Apr 2025)
• SmolLM3 (Jul 2025), Ministral 3B (Dec 2025), all Phi-4 variants (2025)
• Community fine-tunes: only reasoning distillations exist, no structured extraction models

Protocol

• 25 gold-standard probe inputs (EXP-25 faithfulness framework)
• All 7 metrics: EPR, FR, TED, CCS, MIH, NP, SC
• Zero-shot + 3-shot conditions per model
• Q8_0 (quality ceiling) + Q4_K_M (deployment-realistic)
• llama-server on RX 7800 XT, one model at a time
• Secondary: tok/s, VRAM, latency

Decision Gate

Outcome	Action
Candidate SC >80%, EPR >70% zero-shot	Prioritize for v7 fine-tuning
Best scores SC 50-80%, EPR 40-70%	Fine-tune top 2-3 on v7, re-evaluate
All SC <50% zero-shot	Proceed with EXP-26 (Qwen + v7) — pretraining doesn't help much

Estimated time: ~4-6 hours on local hardware. No MI300X needed.