Merged
Conversation
…ng, adaptive cache, RMSNorm scaling, and routing EMA
…ds by 3x and accelerate prefetching
…ion to break 90% hit rate barrier
…0 EMA update to reach 90.9% hit rate and 3.25 tok/s
…loop to break 94% hit rate barrier
- Fix ENV VARS header: document PILOT=0-3, SMOOTH, CONF_LIMIT; remove stale REBAL entry - Fix per-layer EMA: apply routing momentum to all layers (not just layer 0) with correct offset - Fix in-flight slot race in expert_get: LRU eviction now skips slots with eid==-1 (being loaded) - Fix in-flight slot race in pilot_realload: same fix, prevents concurrent writes into active slot - Fix idx[] buffer overflow: clamp max_cand to 128 before E in pilot_prefetch
- Fix LRU fallback: when all evictable slots are in-flight, find oldest non-in-flight slot (pinned ok) before falling back to slot 0 - Fix pin_hot_experts: guard enqueue behind g_pilot>0, call ensure_pilot_worker_started(), and set is_queued flag to prevent duplicate in-flight loads from pilot_prefetch() - Fix token counting: increment token_count/freq_token_count by S (batch size) instead of 1 so prefill tokens are counted accurately and warmup threshold triggers at the right time
- Fix queue flush race: clear is_queued under mutex only; never move pilot_w backwards (would break r<=w ring-buffer invariant). Worker skips stale entries via new is_queued guard at start of pilot_realload - Add early-exit in pilot_realload when is_queued==0 (entry flushed between enqueue and worker pickup), preventing unnecessary loads - Fix misleading EMA struct comment: momentum_logits is used only by the PILOT prefetcher, not blended into actual MoE routing decisions - Fix Slot pinned comment: 'never evicted' was too strong; clarify that pinned slots may be displaced under extreme all-pinned cache pressure - Use st_read_f32() for scale tensor (.qs) instead of st_read_raw() to handle potential future BF16/F16 dtype changes robustly
Inkling ships an o200k-family tokenizer (case-aware Split regex, GPT-4o
lineage) rather than cl100k. tok_load now detects the family from the
pattern itself (\p{Lu} appears only in the o200k regex) so GLM behavior
is untouched, and encode dispatches to a new pretok_chunk_o200k that
replays the regex engine's backtracking order exactly: greedy optional
prefix, maximally-greedy uppercase run given back until the lowercase
run can match, contractions attached to letter runs, \p{N}{1,3}, and
the [\r\n/]* punctuation tail.
tok_unicode_o200k.h adds the two range tables the new classes need
(Lu+Lt and Lm+Lo+M), generated from Python unicodedata.
Validated against HF tokenizers on 357 adversarial strings (case
transitions, contractions, CJK, combining marks, emoji + modifiers,
zero-width chars, 300 mixed-charset fuzz cases): 357/357 identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tests/tok_o200k_tiny.json is a synthetic byte-level BPE (274 vocab, a
few KB) whose Split regex is the o200k pattern; expected ids in
tok_o200k_cases.txt were generated by HF tokenizers on that same file.
test_tok_o200k (in TEST_BINS) scores 40/40 encode + 40/40 round-trip:
case-transition splits, contractions, digit groups, the [\r\n/]* tail,
whitespace branches, CJK/Greek/Cyrillic, added-token atomicity.
The cl100k path is untouched by construction — dispatch requires
\p{Lu} in the tokenizer's own Split pattern, which cl100k lacks — and
stays covered by the GLM oracle (verified on this branch: 32/32).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MinGW's UCRT has no getline; fixed-buffer fgets with CRLF trimming reads the same case file everywhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ckend_gpu_compat.h backend_cuda.cu compiles unchanged for both vendors: under nvcc the new header passes through to cuda_runtime.h + mma.h (preprocessed source is byte-identical to before this commit); under hipcc (make HIP=1) it maps the CUDA runtime surface onto HIP 1:1. Same one-shim-header pattern compat.h uses for the Windows port. The WMMA tensor-core dispatch is compile-gated via COLI_GPU_HAS_WMMA: gfx GPUs report compute_major >= 7 (gfx1201 reports 12), so the runtime check alone would select the __CUDA_ARCH__-guarded kernels' EMPTY bodies under HIP and return garbage silently. HIP uses the portable kernels; rocWMMA matrix-core support is a natural follow-up. Build: make HIP=1 [HIP_ARCH=native|gfxXXXX], make hip-test, make gpu-compile (CI compile-only target). HIP joins the .build-config stamp so HIP<->CPU flag flips relink correctly. CI: engine-hip-syntax job mirrors engine-cuda-syntax (rocm/dev container, pinned tag). Verified on RX 9070 XT (gfx1201, ROCm 7.2.4): hip-test passes the full upstream suite incl. tensor_update; GLM-5.2 end-to-end measures 0.32 tok/s at 61% expert hit with CUDA_RELEASE_HOST=1 (CPU: 0.22-0.27). Runtime details and hardware matrix: GPU_BACKENDS.md.
Three vendor-neutral fixes to backend_cuda.cu, each with test coverage: 1. Upload check-order: a cached device tensor is now usable when the caller's host pointers are stale or NULL. CUDA_RELEASE_HOST slots null their host pointers after upload; the current engine reaches those tensors through direct handles (coli_cuda_expert_mlp etc.), but any caller going through coli_cuda_matmul/tensor_upload with a cached tensor — as matmul_qt does for QT tensors — hits the !weights check before the cached-tensor branch and fails spuriously. This hardens the API contract rather than fixing a measured regression; the contract is pinned by a 64x sustained-reuse test. 2. Sticky runtime error (real bug, test-caught): a failed allocation left the last-error state set, so the NEXT healthy launch's cudaGetLastError() check reported 'out of memory' and disabled a perfectly good tensor. cuda_ok() now consumes the error on the failure path; regression-covered. 3. COLI_GPU_FAIL_AFTER=N test hook: every GPU compute entry point (19 total: matmul, expert mlp/group, shared mlp, attention ops, pipe ops) reports failure after N successful calls, so the engine's CPU fallbacks and expert_host_ensure rematerialization can be exercised end-to-end without real hardware faults. Unset = zero effect; uploads/queries are never gated. Validated on GLM-5.2: total failure (N=0) completes coherently with every released expert rematerialized. Tests (run via make cuda-test on any CUDA GPU; vendor-neutral source): 64x sustained matmul reuse after host pointers are freed; upload from a scribbled-and-freed temporary; five graceful upload-failure cases with stats-integrity assertions; healthy-launch-after-failed-alloc (the sticky-error regression); fault-hook on/off restore. Verified on AMD RX 9070 XT via the companion HIP PR's compat header (same test source); a make cuda-test run on NVIDIA hardware would complete the matrix.
…IPE=1 pipe_wait's sched_yield spin storms the scheduler for the full 0.5-3ms of each in-flight expert read; behind COLI_PIPE_BLOCK=1 it parks on a condvar instead (~5us wake, no lost-wakeup: workers store ready with release BEFORE taking the mutex to broadcast, and the waiter re-checks under the lock). Default OFF = byte-identical spin. Pthread pool only: the URING backend has no waiter spin to replace. Setting PIPE_WORKERS>0 in the env without PIPE now implies PIPE=1 with a stderr note: sizing the pool declares the intent to use it (a full benchmark campaign ran with PIPE_WORKERS=16 and the pipe silently off). The implication fires only when the platform default left the pipe off (no-op on _WIN32 where PIPE defaults to 1), only on a positive value (PIPE_WORKERS=0/empty does not enable a clamped 1-worker pipe), and an explicit PIPE=0 always wins. The rule lives in pipe_workers_imply_pipe() so the table is unit-testable. tests/test_pipe_block.c (in TEST_BINS, all platforms): pins the implication table, and drives the pool through 200 generations under each waiter against an on-disk expert fixture — spin as control, condvar arm alternating parked and fast-path waits — verifying identical slot bytes. Measured on the spin side (2x5090 + Gen5 NVMe, GLM-5.2 744B int4, CPU decode): 1.98 -> 2.16 tok/s at 192 tokens (expert-disk service 44.6s -> 33.5s). On Metal/GPU decode an M5 Pro A/B (PR thread) measured no change, consistent with the mechanism: the win is freeing the core the spinner was stealing from the CPU matmul team. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified & applied the ColibriFixes.md audit against the current tree (adversarially confirmed; no regressions vs. legitimate GLM-5.2 loading or normal server flows). Model-file trust boundary (untrusted-mirror threat model): - glm.c: qt_check_fmt() rejects (not infers) any quantized tensor whose weight/scale byte counts disagree with the config shape, at all three format sites (qt_from_disk + both expert_load arms) — closes the matmul OOB read and the per-row scale OOB write (SEC-1); embed_row token-id bound zero-fills out-of-range ids (SEC-5). - st.h: cross-check numel==nbytes/esz for float dtypes + shape-product overflow guard + st_read_f32_cap at config-sized sinks (SEC-2/B1); st_read_slice_f32 NULL + bounds guard (B6). - tok.h: negative-id and json_get NULL/type guards (SEC-3); tk_read_file fseek/ftell/malloc/fread error handling + size cap (B3/B4). - json.h: bound \u reads against the NUL, strncmp full true/false/null keywords (SEC-4); growable string buffer replacing the silent 64KB truncation (B5). Server (openai_server.py): - fail-closed on non-loopback bind without a key (SEC-6), Host-header allowlist incl. OPTIONS preflight (SEC-7), auth-gate /health & /experts telemetry (SEC-8), scheduler head-of-line fix (B2), [DONE] under ka_lock (B9). Build / supply chain / UI: - Makefile CUDA_ARCH=portable (sm_80..120 + compute_120 PTX fallback), native stays default (D1); env-pinnable model download revisions + c/tools/requirements.txt (D3); CUDA event-leak fix (B8); flake.lock reproducibility note (D2). - web: theme-aware ErrorBoundary, API-key autocomplete, /experts error surface, tooltip viewport clamp, 401 message, responsive badge row, type=button, Probing label. Verified: test_json/test_st pass; glm.o/olmoe.o build; backend_cuda.cu compiles under nvcc 13.3; portable gencode parses; web tsc --noEmit clean; all py_compile OK. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
config.json arrives from unverified mirrors (same threat model as the qt_check_fmt / SEC-1 loader hardening). cfg_root() and the ref_glm.json oracle path both did an unbounded ftell -> malloc(n+1) -> fread with no size cap and no NULL check, so a hostile or oversized file caused a load-time OOM or, on malloc failure, a NULL-deref via b[got]=0. Add cfg_slurp(): 256 MB cap, NULL-checked alloc, full-read check (mirrors tok.h tk_read_file). Route both sites through it. config.json failure stays fatal (exit 1); the oracle path returns 1 as before. Found by an adversarial re-audit of the loader trust boundary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up hardening found by a research+audit pass (corroborated against the llama.cpp GGUF overflow CVEs and safetensors "two-authorities" TOCTOU class). The original audit hardened glm.c thoroughly but left sibling surfaces open. - olmoe.c load_expert_w (CRITICAL): the dtype-3 (U8/I8) container path did st_read_raw(name,q) + st_read_f32(qs,scale) writing header-controlled byte counts into config-sized buffers (q=malloc(O*I), scale=falloc(O)). st_init skips its numel==nbytes/esz cross-check for dtype-3, so a crafted .safetensors with an over-declared U8 weight or oversized .qs was a controlled heap OOB write — the same SEC-1 primitive glm.c's qt_check_fmt already rejects. Add the matching size check (weight bytes == O*I, scale elems == O) in the shared loader; reject, never repair. - olmoe.c load_cfg: unbounded ftell->malloc slurp of config.json (SEC-9 class, already fixed in glm.c) and json_get(...)->num deref with no NULL check (a missing key = NULL-deref). Add a 256 MB cap + NULL/short-read guards, a req_num() that exits on missing/non-numeric keys, and CKR-style range checks so bad dims can't drive a later malloc(inter*hidden) or a div-by-zero on n_heads. - tok.h (hostile tokenizer.json): type-check vocab ids and merge-pair entries as J_NUM / J_ARR-of-J_STR before dereferencing (was a NULL-deref / silent id=0), and cap maxid so n_ids=maxid+1 can't signed-overflow into a multi-GB calloc. Verified the real 151K-vocab GLM tokenizer still loads + encodes unchanged. - openai_server.py: set APIHandler.timeout=30 — a slowloris client that dribbles its request line/body could pin a worker thread (and a KV slot) forever, before the scheduler's queue bound (B2) applies. Deferred (not shipped here): the Linux io_uring finalize path (uring_finalize_load) has the same missing qt_check_fmt + no fmt=4 handling — a 5th format site, but it is #ifdef __linux__ and cannot be compiled/tested on this Windows box, so it needs its own Linux-gated PR. /profile stays public by design (a test asserts it) — maintainer's call. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Python packaging: - pyproject.toml: pip install colibri-engine (editable dev install works) - colibri/ package with __version__, CLI entry point delegating to c/coli - Optional dependency groups: [convert] (numpy, huggingface_hub), [oracle] (torch, transformers, safetensors), [bench] (tokenizers, datasets) Code style: - .editorconfig: consistent indent/charset across all file types - .clang-format: LLVM-based, 120 col, matches existing engine style Usage: pip install -e . # dev install (CLI + serve, no heavy deps) pip install -e .[convert] # adds converter dependencies pip install -e .[oracle] # adds torch/transformers for oracle validation
…(on top of #396) colibri/_version.py now reads c/version.py (#394's single source of truth -- coli --version, the release workflow, and pip metadata can no longer drift), with an importlib.metadata fallback for the installed-wheel case where c/ is not on disk. README documents that pip install -e . is the supported form: the engine lives in c/ and is not packaged into a standalone wheel. Verified in a clean venv: pip install -e . -> colibri.__version__ == 1.0.0 read from c/version.py, coli entrypoint on PATH and functional. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat(prefetch): async expert prefetcher v3.2 for the OLMoE testbed Testbed-only scope: olmoe.c + tools/oracle files; glm.c untouched (the production engine already ships the equivalent techniques: coalesced slab preads, PILOT lookahead, persistent PIN). Trivial .gitignore conflict resolved keeping both sides. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
packaging: pyproject + editorconfig + clang-format (#396) with a single version source
…dary Colibri loads model directories and safetensors from mirrors it does not control, so the file's declared shapes and byte spans are attacker-influenced input. Three memory-safety holes on that boundary, independently confirmed (incl. a from-scratch adversarial audit that re-derived the same two) and present in the shipped v1.0.0: - st.h st_read_f32: numel came from the shape, nbytes from the offsets, with no cross-check. A crafted tensor whose shape inflates numel past nbytes made the BF16/F16 loop read past the malloc'd raw buffer and the F32 memcpy write past the caller's config-sized destination (heap OOB read + write). Now enforce numel*esz == nbytes before any copy. - st.h header parse: the shape product could overflow int64 to a small/negative numel that would then pass the cross-check. Guard each multiply. - glm.c qt_resolve_fmt (new, replaces the three duplicated "?1:?2:3" fmt sites in qt_from_disk and both expert_load arms): the old inference SILENTLY fell to int2 for any unrecognized weight byte count, so a too-short weight became a valid int2 whose matmul read O*I nibbles past the buffer; and an oversized scale array overflowed the per-row t->s. Now the weight bytes must match a known int8/int4/int2 layout and the scale array must match the expected per-row (O) or grouped (O*ng) cardinality, else refuse. - glm.c config/generation_config slurp: unbounded ftell -> malloc(n+1) gave a hostile file a load-time OOM, and on malloc failure b[got]=0 was a NULL deref. Cap at 256 MB and NULL-check. Verified: TF token-exactness unchanged on every quant format (full-precision 32/32, int4 11/32, int2 1/32, mix 5/32 -- byte-identical to the pre-change binary); fmt=4 grouped path preserved (the scale check is by construction the same condition detect_group_size already imposed); a hand-crafted hostile safetensors is refused cleanly; ASan+UBSan clean on legit and hostile loads (only the pre-existing intentional startup leaks remain). These are the C trust-boundary items of #368, landed as a minimal standalone fix; the server-side and build items of that PR follow via its rebase. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
security: reject malformed model tensors at the untrusted-mirror boundary (C half of #368)
#490) #474 (eaa68f4) added an eviction guard to the two PILOT_REAL victim-selection sites (pilot_realload on the blocking path — the one Windows/sm_120 hits — and pilot_uring_batch on Linux). Its intent was to stop a speculation from evicting a just-demand-loaded, still-warm expert (the #441 thrash: +9-18% bytes for +0.5-0.7pt hit rate). The implementation shipped with the comparison inverted for a speculation: if(cs <= vs+(vs>>2)+(4u<<8)){ drop; return; } where cs is the *speculation's* score and vs the *victim's*. tier_lfru_score is (heat<<8)|recent, so one frequency count is worth 256 and recency <=255 — the test demanded the speculation beat a just-used demand victim by 25% + 4 full frequency counts. A correct speculation is, by definition, historically colder than the resident it would evict, so the condition is almost never satisfied: once the cache is full the guard drops ~100% of speculations (verified on the score math, 1e6 samples). PILOT_REAL is effectively reduced to a willneed-only hinter: no real cross-layer loads land, no LRU churn, the LRU contribution to the hit rate evaporates. This is exactly the shape change @mohamedmastouri2000-boop measured on Windows/sm_120 in #490 between 68ac9ff and 4aca059: pin share stable (47-59% -> 59%), LRU share 27-38% -> 15%, hit 84-86% -> 74.5%, 0.95 -> 0.82 tok/s — no fallbacks, coherent output (the guard is cache-placement only; a dropped speculation is demand-loaded later to the same value, so the byte-identical contract held and only the performance shape regressed). Fix: protect the victim only when it is GENUINELY warm (eheat >= 2 demand accesses) AND clearly hotter than the speculation by the same 25%+4-freq hysteresis — i.e. test the victim against speculation+margin (protect the resident), instead of the speculation against victim+margin. This matches the #474 commit message's stated intent ("never evict a WARM demand-loaded expert") rather than the over-broad formula that shipped. After the fix the score math gives: warm victim (heat>=2) -> drop 100% / allow 0%; cold victim (heat<=1) -> drop 0% / allow 100%. The #441 thrash protection is preserved; the plain-LRU churn that produces the 27-38% LRU hit share is restored. Same correction applied to pilot_uring_batch. PILOT_EVICT_GUARD=0 still restores plain-LRU for a single-binary A/B. Cache placement only -> output remains byte-identical (make check 111/111, same as the #474 baseline). Diagnostic for the field: the footer line PILOT_REAL: <N> load cross-layer completati, <M> scartati (main gia' sul layer) shows M >> N (drops dominate) on the broken build and a healthy split on the fixed one. Credit: @mohamedmastouri2000-boop — the regression report, the correct first-look at the diff (#474), the offered bisect, and the measured shape change that isolated LRU share as the collapsed component. The conservative "does this reproduce for anyone else" framing was right; the confounders flagged (.coli_usage drift, page-cache state) did not contribute. Refs #490, #474, #441.
…compat.h Adds AMD GPU support through a single-source HIP/ROCm compat layer with a WMMA dispatch gate. Both merge gates met rigorously by @noobdev-ph: (1) rebased on current dev, CI 9/9 green, zero compat-layer churn; (2) validated on real AMD hardware (RX 9070 XT / gfx1201 RDNA4, ROCm 7.2) against a real fmt=4 gs64 container — token-exact vs CPU with resident dense (CUDA_DENSE=1) AND with routed experts in VRAM (CUDA_RELEASE_HOST=1, host copies freed), in both teacher-forcing and generation, 3/3 runs. Crucially the COLI_GPU_FAIL_AFTER=0 control proves 33 live GPU calls actually executed — the token-exactness is not a silent CPU fallback. That is the stricter test for the #298 failure mode. Verified locally: clean merge, clean build, token-exact unchanged (fp 32/32, int4 21/32).
…-drop (fixes #490) Fixes the regression #474 introduced. The shipped guard tested the SPECULATION against victim+margin — but a correct speculation is by definition colder than the resident it would evict, so once the cache filled it dropped ~100% of speculations and PILOT_REAL collapsed to a willneed-only hinter (#490: LRU share 27-38%→15%, 0.95→0.82 tok/s). The fix protects the victim only when genuinely warm (eheat>=2) and clearly hotter, i.e. tests the victim against speculation+margin — matching #474's stated intent. Score math over 1e6 samples: warm victims 100% protected, cold victims 0% protected (plain-LRU churn restored). Validated by @mohamedmastouri2000-boop on the #490 box: 0.82→0.89 tok/s, LRU share 15%→18.7%. Verified locally: clean build, token-exact unchanged. Thanks @woolcoxm for the root-cause and @mohamedmastouri2000-boop for the independent validation.
st_init skips its numel*esz==nbytes cross-check for dtype-3 (U8/I8) tensors, so a crafted header from an untrusted mirror could declare nbytes far larger than the config-sized destination: st_read_raw would then write past w_block (heap overflow), and an oversized .qs would overrun s->gs. slot_ensure_allocated sizes those buffers exactly ng+ng+nd bytes and inter+inter+hidden floats, so require an exact match and refuse otherwise — same reject-never-repair contract as qt_resolve_fmt in the GLM engine. Found while reviewing #368: that PR hardened the equivalent path in the old load_expert_w, which the olmoe refactor replaced — so the hole was still open on dev in the new code. Weight/scale validation only; no behaviour change on a well-formed container. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… fmt=5 scale cap Conflict resolution (13 hunks, 6 files): - colibri.c / st.h: dev already carries an equal-or-stronger guard for the same threats (qt_resolve_fmt resolves AND validates the quant layout, refusing unknown byte counts instead of falling through to int2; the shape-product overflow guard is present). Took dev's side — no check is lost. - olmoe.c: the load_expert_w hardening targeted a function the olmoe refactor replaced; dev now validates the equivalent path in load_expert_merged. - web/*: i18n churn only, took dev's strings. This PR's unique value is preserved and is the point of the merge: the server hardening in openai_server.py (+121), plus json.h, tok.h, Makefile and the downloader/requirements pinning. Bug found and fixed while validating: st_read_f32_cap's call site bounded the scale read with the PER-ROW cardinality (O), which is correct for int8/int4/int2 but rejects grouped formats — int3-g64 (fmt=5) keeps O*i3_groups(I) scales, so a legitimate container was refused (tests/test_int3_load failed). The cap now matches the cardinality qt_resolve_fmt already validates and falloc reserved. Verified: clean build, token-exact unchanged (fp 32/32, int4 21/32), tests/test_int3_load ok, full make check OK. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rror threat model) Server-side hardening (openai_server.py +121), plus json.h/tok.h parser hardening, Makefile build flags, and downloader/requirements pinning. Rebased by maintainer onto current dev: the container-parsing hunks were resolved to dev's side, which already carries an equal-or-stronger guard for the same threats (qt_resolve_fmt validates the quant layout and refuses unknown byte counts; shape-product overflow guard; olmoe dtype-3 validation added in 9b1d87d after this PR identified the hole). A real bug in the hardening was found and fixed during validation: the bounded scale read used the per-row cardinality, which rejects grouped formats — int3-g64 (fmt=5) legitimately carries O*i3_groups(I) scales, and tests/test_int3_load failed. The cap now matches the validated cardinality. Verified: clean build, token-exact unchanged, test_int3_load ok, full make check OK. Thanks @KingIcyCreamProjects — the olmoe finding in particular was live on dev.
…ution make saw them as up-to-date targets in a fresh checkout (mode 100644, no exec bit) and skipped the build, so every CI run against dev now dies at test_e8_kernel with EACCES. The .gitignore listed test binaries one by one and had fallen behind; replaced with a pattern that ignores the build outputs and keeps the sources.
… step 4) The converter rotates expert rows (W@Q, block-diagonal FWHT) and packs them with the iq3 codec; the engine now recognizes the container (.qs = 4-byte tag, scales live in-block), rotates activations once per layer for gate/up and per-expert for down, and keeps fmt=6 tensors CPU-side (no CUDA kernel yet). The rotation signs are never stored: both sides regenerate the same xorshift64* stream, and the kernel fixture pins the agreement.
…step 2) The float simulation chose the scheme (51.5% on OLMoE); this measures what a SHIPPED container gets — weights through the real codec (98B blocks, parity sign cost, fp16 super-scales, regenerated xorshift rotation signs) and back to model space via the new unrotate_rows. Tensors whose input dim is not a multiple of 256 stay float, the same rule the converter enforces.
…ttern Compiled test binaries had been committed to c/tests/ (mode 100644), so on a fresh CI checkout make sees them as up-to-date, skips rebuilding, and run_tests.py fails to exec a non-executable file — every PR's CI was dying at test_e8_kernel (EACCES / Exec format error on macOS). Removes the 12 binaries and adds a pattern-based ignore (c/tests/test_* with negations for .c/.cu/.mm/.py) so sources stay tracked and no future binary can slip in — more robust than the per-file ignore added during #421. CI 9/9. Thanks @ZacharyZcR for spotting that it was breaking everyone, not just one PR.
…ntainer (#452 step 4) Completes the E8/IQ3 arc: the converter can now produce a fmt=6 container and the engine loads it (after #465's decode kernel and #458's index codec). The CI failures on this PR were the stray-test-binary breakage fixed by #500, not this change — verified locally on post-#500 dev: clean merge, no binaries reintroduced, clean build, token-exact unchanged (fp 32/32, int4 21/32), full make check OK. Thanks @ZacharyZcR.
Docs-only: repairs the table-of-contents anchors in docker/README.md and docker/README.IT.md (accented and punctuated headings needed proper URL-encoding). Retargeted from main to dev by the maintainer so it ships through the normal flow and lands in the v1.1.0 release — no rebase needed from the author. Thanks @DavidePapero.
…431 PR-C0) First step of the #431 plan: expert-group results stay on device instead of round-tripping to host, the dataflow a graph-captured decode needs. Validation provided as asked: sha256-identical completions with RESID=0 vs RESID=1 (6x RTX 5090, CUDA 13.3, GLM-5.2 int4, 256-token greedy, flag-only A/B on the same binary), perf at parity (6.04 vs 6.08 tok/s) — honestly presented as a structural change, not a speed claim. Rebased through the AMD/HIP single-source refactor. The red CI was the stray-test-binary breakage fixed by #500; verified locally on post-#500 dev: clean merge, clean build, token-exact unchanged, full make check OK. Thanks @ZacharyZcR.
…401) This is the root cause behind Michael's report, and it is not a tool-calling bug at all. mux_submit encoded the prompt with `tok_encode(..., maxctx-2)`. tok_encode stops dead at its cap and returns no signal that it ran out, so a prompt longer than the context was silently cut down to its FIRST maxctx-2 tokens and answered as if nothing happened. The client got HTTP 200 and a reply generated from a mutilated prompt. That is exactly what his log shows. CTX defaults to 4096, so maxctx-2 = 4094, and his engine printed `prefill 4094` -- his coding client's system prompt alone overran the context. The tail was dropped: the tool instructions and the actual user turn. The model saw a prompt cut mid-markup and emitted a bare `<`. And because a client appends to the END of a conversation while truncation keeps the HEAD, every retry produced a byte-identical prompt -- which is why his log reads `prefix 4094/4094 token, prefill 0` on every single retry, and why the reply was identical each time. Reproduced end-to-end on a tiny model with CTX=256, ~3600 tokens of prompt: before: HTTP 200 prompt_tokens=254 completion_tokens=1 content="9" [API] KV slot 0 prefix 0/254 token, prefill 254 after: HTTP 400 code=context_length_exceeded [API] prompt does not fit: >=255 token, context is 254 (CTX=256). One junk token out of a mutilated prompt, silently. Same shape as his bare `<`. Fix: encode with one token of headroom (buffer is already maxctx) purely so overflow can be DETECTED, then refuse with `ERROR <id> CONTEXT_EXCEEDED <got> <limit>`. The gateway maps that frame to a 400 with OpenAI's `context_length_exceeded` code, which is what every compatible server returns and what clients that can compact a conversation actually listen for. The engine also prints a line telling the operator to raise CTX. Prompts that fit are untouched: verified short and near-the-limit prompts still return 200 with the expected prompt_tokens. 43 unit + 4 e2e pass.
serve: refuse an over-long prompt instead of silently truncating it (#401)
The model expresses a call as <tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call> and the parser matched on BOTH tags. When the closer never came -- the model hit its token budget mid-call, or quantization mangled the tag into "</tool_cal" -- the strict regex matched nothing and the client got *zero* tool_calls, with the raw markers dumped into `content` and finish_reason=stop. The call itself was usually perfectly well-formed. That is the failure Michael reports in #401. Recover the trailing unclosed box, but only when the recovery is unambiguous, so prose that merely mentions the marker can never be turned into a call: * the last <tool_call> must not be followed by a closer (closed boxes stay the strict parser's job, unchanged); * the tail must carry a complete <arg_key>..</arg_value> pair, OR be exactly the name of a tool the client declared (the zero-argument case). A partial closing tag at end of reply ("</tool", "</tool_cal") is trimmed off the body. This is parse-side only: no change to sampling, to the prompt, or to well-formed output. It fires exactly where we used to return nothing. The recovered call is also stripped from the visible content, and the [api] log line now reports it separately from the opt-in de-mangler rather than counting it as a strict parse. Verified fail-before / pass-after through the real HTTP server against a mock engine that emits an unclosed call: before, `FAIL: the model returned no tool_calls`; after, the call arrives and the two-turn loop completes. Both response shapes covered -- non-streamed and streamed, the latter being what coding clients actually use. Tests: 8 unit cases (including two negatives that must NOT fabricate a call) + 2 e2e cases. 48 unit and 6 e2e pass. Also adds tools/try_tool_calling.py: a dependency-free two-turn probe against a running server (declare -> call -> execute -> feed back -> answer), so this can be reproduced against a real model without pulling in a coding client. It discovers the model id from /v1/models and exits non-zero on failure, so it doubles as a smoke test.
serve: recover a tool call whose closing tag never arrives (#401)
JustVugg
changed the title
27 PRs from more than 20 contributors, 216 commits since v1.0.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
release: bump to 1.1.0 and write the CHANGELOG
… runs
Download a release archive, unpack it, run `coli chat`, and it says:
engine is not built. Run: coli build
with the engine sitting right there in the same directory.
`coli` locates the engine by looking for "colibri"/"colibri.exe" (then "glm") next to itself,
or $COLI_ENGINE. The Package step copied it in as colibri-<tag>-<platform>[.exe], which
matches none of those, so it fell through to the $PREFIX/libexec layout that does not exist in
an unpacked archive. Every published archive on every platform was affected -- Windows,
Linux and macOS -- and v1.0.0 shipped this way too. Anyone who downloaded a prebuilt binary
and followed the README was told to build from source.
The version belongs in the ARCHIVE name, which is where someone downloading it looks; it does
not belong in the executable that a launcher has to find by name.
Also adds a verification step that unpacks the archive we are about to publish into a clean
directory and asserts `coli` finds the engine. A packaging mistake is invisible to every other
job here -- the build is green, the tests are green, and the artifact is still unusable -- so
this one has to assert on behaviour, not on compilation. Verified locally by reproducing the
Package step: before, "engine is not built"; after, `coli info` runs and `coli chat` reaches
the engine and generates.
The upload glob is now explicit about publishing only archives. It previously worked by
accident, because the versioned engine name did not match `colibri-*.*`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
release: ship the engine under its plain name so the archive actually runs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v1.1.0 — 27 merged PRs, 208 commits since v1.0.0.
New backend
backend_gpu_compat.hwith a WMMA dispatch gate. Validated on RX 9070 XT (RDNA4, ROCm 7.2): token-exact vs CPU on a real fmt=4 gs64 container, with resident dense and with routed experts in VRAM, plus a fail-injection control proving the GPU actually executed.Tool calling in coding clients (#401) — root cause found
CTX-2andtok_encodestops dead at its limit without reporting anything, so a prompt longer than the context was silently cut to its firstCTX-2tokens and answered anyway. With the 4096 default that is 4094 — exactly theprefill 4094in the field report. The dropped tail was the tool instructions and the user's actual turn, so the model emitted a bare<and stopped; and since clients append to the end while truncation keeps the head, every retry re-sent an identical prompt (prefill 0). Now refused with a 400context_length_exceeded. Reproduced end-to-end: beforeHTTP 200, prompt_tokens=254, content="9"; after, a clear 400.</tool_call>never arrived was dropped whole (the parser required both tags), so the client got zerotool_calls. Now recovered when unambiguous, on both the streamed and non-streamed paths.tools/try_tool_calling.py: dependency-free two-turn probe (declare → call → execute → feed back → answer) that doubles as a smoke test.Operator note:
CTXstill defaults to 4096. Coding clients send far more than that in a single system prompt — useCTX=32768.Correctness
CUDA_DENSE=1: the dense and attention kernels applied per-group scales as if they were per-row. Hardware-verified fix.OMP_PROC_BIND/OMP_PLACESwere set (~20× slowdown).CUDA_EXPERT_GB.COLI_CUDA_MTP=1andCOLI_CUDA=0are now honoured over implicit defaults.Security (untrusted-mirror threat model)
openai_server.py), JSON/tokenizer parser hardening, build flags, downloader and dependency pinning.nbytes(heap overflow).Performance — all byte-identical
qt_addrow/qt_matvec_rows)XEXP=1(one OpenMP region per expert block at S=1 full residency)Storage
COLI_MODEL_MIRROR) — read the model from two drives at once, roughly doubling streaming bandwidth on a disk-bound host.COLI_MODEL_DIRS) — capacity aggregation: a container that no single drive can hold, spread across several with no duplication.New quantization containers
Serve
response_format, per-request grammars, grammar-forced drafts.Build / CI / docs
test_e8_kernel(EACCES).convert --xbits e8: the converter now produces a fmt=6 container and the engine loads it (fmt=5: a deployable E8-lattice grouped container — int3 that beats shipped int4 quality at 25% fewer bytes (design + plan) #452 step 4).Every engine change in this release was gated on the token-exactness oracle; the performance work is byte-identical by construction.