UNC — Universal Neural Compiler

Compiles HuggingFace transformer models into optimised native Metal inference binaries. No runtime framework, no Python — just a compiled binary that runs your model at near-hardware-limit speed on Apple Silicon, using 25% less GPU power and 1.7x better energy efficiency than mlx-lm. See the full resource & power report.

Performance

TinyLlama 1.1B on Apple M1 Pro (16GB, 200 GB/s):

UNC Q4_0  ████████████████████████████████████████████████████████████  152.0 tok/s
mlx-lm Q4 ████████████████████████████████████████████                 112.7 tok/s
UNC Q8_0  ███████████████████████████████                               76.6 tok/s
UNC F16   ███████████████████                                           47.9 tok/s

Qwen3-4B on Apple M1 Pro (Q4_0):

mlx-lm Q4 ████████████████████████████████████████████████████  49.2 tok/s
UNC Q4_0  ██████████████████████████████████████████            38.7 tok/s

Energy Efficiency (Q4_0 TinyLlama 1.1B, measured via macmon)

Metric	UNC Metal	mlx-lm Q4
Throughput	152 tok/s	113 tok/s
GPU power (decode)	11.3W	14.1W
Energy per token	74 mJ	125 mJ
Tokens per watt-hour	12,800	8,000
CPU instructions (200 tok)	5.3B	31.4B
Peak memory	4.2 GB	0.9 GB

UNC is 1.35x faster while using 25% less GPU power, resulting in 1.7x better energy efficiency. The compiled approach eliminates Python runtime and framework dispatch overhead entirely — 8.4x fewer CPU instructions means less heat, less power, and more headroom for the GPU. See unc_resource_usage.md for full methodology and traces.

Architecture

HuggingFace model
       |
  [ Frontend ]        Parse config.json + safetensors
       |
  [ IR Graph ]        Hardware-agnostic tensor graph
       |
  [ Compiler ]        Fusion, quantization, memory planning
       |
       +------------------+------------------+------------------+
       |                  |                  |                  |
  [ Metal ]          [ CUDA ]          [ ROCm ]          [ WASM ]
  Obj-C + Metal      PTX kernels       HIP kernels       WebGPU shaders
  shaders            (planned)         (planned)         (planned)
       |
  Native binary
  Mach-O (AOT) or
  .unc bundle (JIT)

IR: Hardware-agnostic typed tensor graph with BatchMatMul, QuantizedMatVec, RMSNorm, LayerNorm, QKNorm, RoPE, SDPA, SwiGLU, KVCacheAppend, Gather, etc. The IR is target-independent — the same graph can be lowered to Metal (current), CUDA, ROCm, WASM, or CPU-only backends with acceleration providers like Intel oneDNN.

Compiler passes: Weight binding, dead code elimination, QKV fusion, Gate+Up fusion, SwiGLU fusion, Add+RMSNorm fusion, RoPE+KV fusion, PSQ pipeline, dual-path (GEMM/GEMV), kernel matching, barrier analysis, memory planning with buffer aliasing.

Output modes:

Mode	Output	Use case
JIT (default)	.unc bundle — JIT-compiled via clang at first run, cached thereafter	Development, iteration
AOT (--binary)	Standalone Mach-O with embedded weights — zero dependencies	Deployment, distribution
GGUF export (quantize)	GGUF v3 file — compatible with llama.cpp, ollama, LM Studio	Interop, distribution

Setup

# Prerequisites: Rust toolchain, Xcode Command Line Tools (macOS)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Clone and build
git clone <repo-url> && cd unc
cargo build --release

Usage

Compile a model

# JIT bundle (default)
unc compile --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --quant q4-0 -o ./tinyllama
unc compile --model Qwen/Qwen3-4B --quant q4-0 -o ./qwen3

# AOT standalone binary (single Mach-O, zero dependencies)
unc compile --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --quant q4-0 --binary -o ./tinyllama

Run inference

unc run ./tinyllama.unc --prompt "The history of" --max-tokens 200

Export quantized GGUF (for llama.cpp, ollama, LM Studio)

# Quantize and export to GGUF v3
unc quantize --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --quant q4-0 -o tinyllama-q4_0.gguf
unc quantize --model Qwen/Qwen3-4B --quant q8-0 -o qwen3-q8_0.gguf

# Use with llama.cpp / ollama
llama-cli -m tinyllama-q4_0.gguf -p "Hello" -n 100
ollama create mymodel -f Modelfile  # with: FROM ./tinyllama-q4_0.gguf

GPU-accelerated quantization via Metal compute kernels with streaming double-buffered I/O. 1.8s end-to-end for 1.1B params — 9x faster than llama.cpp's full HF→GGUF pipeline (16.5s). No Python required.

Tool	HF safetensors → Q4_0	Notes
UNC	1.8s	GPU Metal, streaming, no Python
mlx-lm	2.1s	Outputs MLX format, not GGUF
llama.cpp	16.5s	Python convert + CPU quantize

See benchmark/quantization/results.md for full methodology.

Quantization options

Flag	Precision	Size (1.1B)	Inference	Export
f16	16-bit float	2.2 GB	47.9 tok/s	compile only
q8-0	8-bit	1.1 GB	76.6 tok/s	compile + GGUF
q4-0	4-bit	0.6 GB	152.0 tok/s	compile + GGUF

Supported architectures

unc list-architectures

LLaMA, Mistral, Qwen, Phi, Gemma.

Project Structure

src/
  frontend/    HuggingFace config parsing, model templates
  ir/          Typed tensor IR (ops, graph, types)
  compile/     Optimization passes, memory planner
  kernel/      Kernel registry, Metal kernel definitions
  emit/        Metal orchestrator codegen, AOT binary emission
  export/      GGUF writer, GPU quantization pipeline, tensor name mapping
  runtime/     JIT compilation, weight loading, tokenizer
  target/      Apple Silicon target detection
  unc_format/  .unc bundle serialization
kernel_sources/
  metal/
    unc_kernels/   Custom Metal shaders (fused GEMV, SDPA, RoPE, RMSNorm, etc.)
    upstream_mlx/  MLX reference kernels (QMV, sdpa_vector headers)

License

MIT — see LICENSE.