🔱 This is a fork — AMD GPU support + research

noobdev-ph/colibri — a downstream fork of JustVugg/colibri (synced through v1.0.0) that adds the first AMD GPU backend to the engine, plus generation-quality guards and a body of profiling research. Everything here stays in the upstream spirit: pure C, zero runtime dependencies, no Python in the hot path, all additions opt-in with the default CPU build untouched.

📄 Full write-up: RESEARCH.md · 🔧 GPU guide: GPU_BACKENDS.md

What this fork adds

1. AMD GPU support via HIP/ROCm — upstreamed as PR #339

The existing CUDA backend compiles unchanged for AMD through a single ~60-line mapping header (c/backend_gpu_compat.h) — the same one-shim pattern compat.h uses for the Windows port. One source, two vendors; the NVIDIA path's preprocessed source is byte-identical to upstream's.

make -C c glm HIP=1 HIP_ARCH=gfx1201     # build for an AMD GPU (ROCm)
make -C c hip-test HIP_ARCH=gfx1201      # kernel correctness on the card

HIP_VISIBLE_DEVICES=0 COLI_CUDA=1 COLI_GPU=0 CUDA_EXPERT_GB=12 CUDA_RELEASE_HOST=1 \
  ./coli chat --ram 40 --topp 0.7

It also ships a WMMA compile-gate (COLI_GPU_HAS_WMMA) that matters beyond AMD: the tensor-core kernels are guarded by __CUDA_ARCH__ >= 700, but the host dispatch checks compute_major >= 7 at runtime — and gfx GPUs report compute_major = 12. Without a compile-time gate, any non-CUDA target would dispatch into empty kernel bodies and return garbage silently.

Verified on an RX 9070 XT (gfx1201 / RDNA4, ROCm 7.2): full kernel suite passes; GLM-5.2 runs end to end.

2. GPU backend failure-path hardening — upstreamed as PR #338 (vendor-neutral)

Fixes a sticky cudaGetLastError bug (a failed allocation poisoned the next healthy launch's error check, disabling good tensors), hardens the cached-tensor upload contract, and adds a COLI_GPU_FAIL_AFTER=N fault-injection hook gating all 19 GPU compute entry points — so the engine's CPU fallback and host-rematerialization can be tested end-to-end without real hardware faults.

3. Generation-quality guards (experimental) — MIN_TOKENS, UNTIL, NOLOOP

Opt-in sampling guards for long single-artifact generations (e.g. "write this whole file"), where int4 greedy decoding degenerates. NOLOOP=1 detects periodic token loops and rewinds the KV cache to before the loop, rerolling at escalating temperature. Zero effect when unset. See RESEARCH.md §2 for the two distinct failure modes characterized and the one still open.

Key research finding: where the time actually goes

Before building AMD matrix-core (rocWMMA) kernels, we profiled where expert matmul time actually lives. The result cancelled that plan — and is the most useful thing in this fork:

decode, measured	share
expert matmul on the CPU	211.3 s (98.7%)
expert matmul on the GPU	2.66 s (1.2%)
CPU matmul throughput	4.1 GB/s
routing hitting a VRAM-resident expert	10.5%

The GPU does ~1% of the matmul because only ~10% of routing lands on a VRAM-resident expert — so matrix-core kernels would have accelerated 1.2% of the runtime. The measurement saved weeks of building the wrong thing.

It also surfaced a hardware-specific inversion worth knowing: the engine's founding rule is that streaming stays CPU-side because per-use GPU copies just trade the disk bottleneck for a PCIe one. But on a slow-CPU / fast-PCIe machine that inverts — here PCIe measured 13.4 GB/s vs CPU matmul at 4.1 GB/s, making "copy a RAM-cached expert to the GPU and compute it there" roughly 3× faster per expert than computing it on the CPU. That, not faster kernels, is the real lever on such hardware.

Measured on this fork's hardware

Ryzen 7 5700G · RX 9070 XT (16 GB) · 64 GB DDR4 · Gen3-capped NVMe — GLM-5.2 744B int4 + int8 MTP:

config	tok/s	expert hit
CPU only, cold	0.16	~30%
CPU only, warm (learned pin + --topp 0.7)	0.22	38%
HIP, VRAM tier mirrors RAM pin	0.22	33%
HIP, CUDA_RELEASE_HOST=1	0.33	61%

(Note: the 5700G is an APU — it caps all PCIe at Gen3, so a Gen5 NVMe negotiates Gen3 x4 ≈ 3.5 GB/s. Worth checking current_link_speed before trusting a drive's rated spec.)

Status

Paused pending upstream traction and performance. Both PRs are open and mergeable against upstream dev. Resume points, the full profiling data, the Rust/Go port analysis, and the ranked list of remaining levers are in RESEARCH.md.

---

English · 繁體中文

Tiny engine, immense model. Run GLM-5.2 (744B-parameter MoE) on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.

Colibrì is a lightweight, quality-preserving MoE runtime that treats VRAM, RAM, and storage as one managed memory hierarchy. Insufficient fast memory may reduce speed, but the default policy never silently changes model precision or router semantics.

$ ./coli chat
  🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
  ✓ ready in 32s · resident 9.9 GB
  › ciao!
  ◆ Ciao! 😊 Come posso aiutarti oggi?

See it running

The web dashboard (./coli web): a 744B model at 4 tok/s, TTFT 1.6 s, disk 0 — full expert residency on 6× RTX 5090, with live token metrics, the per-turn time breakdown, the VRAM/RAM/disk tier bar and the live mini-brain in the corner.

The Brain page: all 19,456 experts as a living cortex — colour is the storage tier, brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's measured topic affinity.

The Atlas page: the measured expert atlas as a 3-D galaxy — 13,260 characterised experts, 1,041 replicated specialists clustering by topic (poetry, law, Chinese, SQL…). Position is measured routing affinity, not a learned embedding. Drag to spin.

The idea

A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts):

So the model doesn't need to fit in fast memory — it needs to be placed:

• the dense part (attention, shared experts, embeddings — ~17B params) stays resident in RAM at int4 (~9.9 GB);
• the 19,456 routed experts (75 MoE layers × 256 + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier.

The engine is a single C file (c/glm.c) plus small headers. No BLAS, no Python at runtime, no GPU required.

How it works

The per-token path

Every layer of every token walks the same five steps. The design goal is that placement only ever decides speed — the router's decisions and the weights' precision are the same whether an expert answered from VRAM or from disk.

One memory hierarchy instead of one memory requirement

The same engine spans the whole range: on a 25 GB laptop everything streams from disk (slow but correct); on a large host the entire expert set becomes resident (CUDA_EXPERT_GB=auto PIN_GB=all) and disk drops out of the decode path entirely. Between the tiers sits a learning cache: the engine records which experts your workload routes to (.coli_usage, updated every turn) and pins the hottest ones automatically — colibrì literally gets faster the more you use it. On multi-socket hosts, COLI_NUMA=1 interleaves the resident weights across memory controllers (#82).

Never wait for the disk twice

Misses are expensive, so the engine spends most of its cleverness avoiding and overlapping them: each expert's three matrices are stored adjacent and read in one pread; a bounded async I/O pool (PIPE=1, default) loads missing experts while resident ones compute; batched positions read each unique expert once (batch-union); and a router-lookahead thread (PILOT=1) prefetches the next layer's experts — routing is measurably 71.6% predictable one layer ahead. On GPUs, the resident pipeline (COLI_CUDA_PIPE=2) keeps the residual stream on-device across layers so the CPU expert loop runs uninterrupted; on Apple Silicon an experimental Metal backend does the batched expert math on the unified-memory GPU.

Faithful model, compressed state

The forward pass is validated token-exact against a transformers oracle (teacher-forcing 32/32). MLA attention stores a compressed KV state — 576 floats/token instead of 32,768 (57× smaller) — and persists it across restarts (.coli_kv): conversations reopen warm with zero re-prefill, byte-identical to an uninterrupted session. DSA sparse attention (GLM-5.2's lightning indexer) is implemented faithfully and validated by forcing full-key selection to reproduce dense attention exactly.

Speculative decoding, honestly

GLM-5.2's native MTP head drafts tokens that the main model verifies in one batched forward — 2.2–2.8 tokens/forward when it pays. Two hard-won rules ship as defaults: the MTP head must be int8 (int4 heads collapse to 0–4% acceptance, #8), and draft and verify must compute the same function — SPEC_PIN=1 pins both to one kernel family (#163 is the full forensic story). Grammar-forced drafts (GRAMMAR=file.gbnf) add ~free acceptance on constrained JSON output. Whether speculation is a net win depends on your cache temperature — measure, and use DRAFT=0 when it doesn't pay.

What it achieves

Same engine, same int4 container — the hardware only changes where the experts live. Highlights from the full benchmark tables:

• 6× RTX 5090, full residency: 5.8–6.8 tok/s decode, TTFT ~13 s (experiment log);
• 128 GB CPU-only desktop: ~1.8 tok/s warm (#200);
• single RTX 5070 Ti laptop-class box: 1.07 tok/s via the GPU-resident pipeline (#273);
• 25 GB dev box: 0.05–0.1 tok/s cold — the proven floor where this project started, and still the honest baseline.

Quality is measured, not assumed: the int4 container's quantization cost and the scale-granularity/rotation ablations live in docs/benchmarks.md and #108/#81.

Get started

1. Get the model

A pre-converted GLM-5.2 int4 container is on Hugging Face — use the version with the int8 MTP heads:

https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp

⚠️ The original mirror ships int4 MTP heads → 0% draft acceptance (#8). Check yours: ls -l <model>/out-mtp-* — int8 (correct) is 3527131672 / 5366238584 / 1065950496.

Or convert from the FP8 source yourself — one resumable command that never needs the full 756 GB on disk at once:

cd c && ./setup.sh                        # checks gcc/OpenMP, builds, self-tests
./coli convert --model /nvme/glm52_i4     # download+convert shard by shard (python, one-time)

2. Run it

COLI_MODEL=/nvme/glm52_i4 ./coli chat     # RAM budget, cache and MTP auto-detected
COLI_MODEL=/nvme/glm52_i4 ./coli plan     # inspect the planned VRAM/RAM/disk placement
COLI_MODEL=/nvme/glm52_i4 ./coli doctor   # read-only readiness check
./coli web  --model /nvme/glm52_i4        # API + web dashboard on one port
./coli serve --model /nvme/glm52_i4       # OpenAI-compatible API only

The engine at runtime is pure C — python is only used by the one-time converter and the optional API gateway.

3. Go deeper

topic	doc
Benchmarks, community datapoints, quality measurements	docs/benchmarks.md
Tuning knobs, policies, the learning cache, prefetch	docs/tuning.md
Windows 11 native build (+ CUDA DLL)	docs/windows.md
CUDA backend, VRAM expert tier, full residency	docs/cuda.md
Apple Silicon Metal backend	docs/metal.md
OpenAI-compatible API, KV slots, web dashboard	docs/api.md
Grammar-forced drafts (structured output)	docs/grammar-draft.md
Environment variable inventory	docs/ENVIRONMENT.md

Supporting the project

colibrì started as a one-person project on a 12-core laptop with 25 GB of RAM; today its numbers come from a community of real machines. If it's useful to you:

• ⭐ star the repo and share it;
• 🐛 open issues with benchmark numbers from your hardware — datapoints move this project more than anything else;
• 💬 reach out via GitHub issues to sponsor development or donate hardware.

Repo layout

Makefile                  root build/check entry point
c/
├── glm.c                 single-file GLM engine
├── st.h, tok.h, json.h   runtime headers
├── backend_cuda.*        optional CUDA tier
├── Makefile              build and local checks
├── coli                  user-facing CLI
├── openai_server.py      OpenAI-compatible HTTP gateway
├── setup.sh              one-command local setup
├── tools/                offline conversion, fixtures and benchmarks
├── scripts/              long-running conversion helpers
└── tests/                dependency-free C and Python tests
web/                      browser UI (pure OpenAI-API client)
desktop/                  Tauri v2 desktop shell wrapping the web UI
docs/                     reference docs, experiments, media

The runtime path intentionally stays flat and readable: glm.c plus its small headers. From the repository root, make, make check, and make clean delegate to the engine Makefile.

Why "colibrì"

The hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. This engine keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience.

License

Apache 2.0. GLM-5.2 weights are released by Z.ai under MIT.