Conversation
…check Four jobs, all fast, all free on public repos: - engine: make glm + make test-c (Linux, CPU, native arch) - engine-cuda-syntax: nvcc compile-only check (no GPU needed, catches backend_cuda.cu syntax/type errors before they reach a real machine) - web: npm ci + build + vitest (Node 22, cached deps) - python: unittest discover on c/tests/test_*.py (Python 3.12) Triggers on push/PR to dev and main. No 20-minute pipelines — the whole matrix finishes in under 3 minutes on GitHub-hosted runners.
…uption Both sentinels in c/coli (READY and END) end with \n. Under CRT text mode on Windows, printf() translates \n to \r\n, so the engine emits \x01\x01READY\x01\x01\r\n. The Python coli wrapper checks endswith(SENTINEL) which expects bare \n — the match never fires and chat hangs forever. _setmode(fileno(stdout), O_BINARY) at engine startup switches stdout to binary mode so \n passes through unchanged. This matches the belt-and-braces reasoning already in compat.h:88 (O_BINARY for file I/O). The fix is defense-in-depth: on the documented w64devkit/MinGW build path, CRT text mode may or may not apply depending on how the terminal is attached, so this ensures correctness regardless. Note: I was unable to compile-test this on the full codebase because glm.c uses POSIX-only symbols (mmap, madvise, select, fd_set, struct stat/fstat) that are unavailable on Windows without platform guards. The codebase compiles on Linux (Ubuntu 24.04, GCC 13) and macOS (Apple clang). Windows compilation requires wrapping those POSIX calls in #if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__) guards — I opened this as a separate concern for the maintainer.
…% decode on 2S, raw mbind, no libnuma (#82)
…tuning verdict) Answers 'where does it slow down on THIS machine with THIS config' so users can tune RAM_GB/PIPE/DIRECT/PIN for their hardware without folklore: - startup header: CPU/cores/RAM/backend + effective knobs (cache cap, pin, DRAFT/PIPE/DIRECT/MMAP/IDOT/DSA/PILOT/CACHE_ROUTE) — every saved log is self-describing when comparing runs across configs or machines - per-forward decode latency ring (32k) -> p50/p90/p99/max, plus a tail diagnosis when p99 >> p50 (cold-cache expert loads) - expert I/O accounting at the pread/mmap-touch sites: GB fetched, MB/token, GB/s, hit rate, loads/token, pinned/LRU tier fill - phase shares of wall time and a plain-language verdict naming the knob most likely to move tok/s (I/O-bound vs compute-bound vs attention-bound) - reports after REPLAY / PROMPT / oracle runs (stdout) and per turn in serve mode (stderr; stdout stays the framed protocol) Additive only: with PROF unset every mode's output is byte-identical. hwinfo_emit's /proc probe is factored into hw_probe() and shared. Validated end-to-end on a tiny-random unquantized fixture (REPLAY, PROF on/ off, RAM_GB squeeze flips hit 96.9%->26.6% and the verdict follows); make check clean, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TCBNCciBaHea41QmidLUMn
…path) run_serve_mux (SERVE_BATCH, used by openai_server.py / coli web) completes requests in mux_done, so the run_serve per-turn hook never fired there. Snapshot the window where hits0 is taken, record batched-forward latency around step_decode_batch, report on stderr at DONE. With KV_SLOTS>1 the window shares batched forwards across slots — same convention as the existing STAT hit%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TCBNCciBaHea41QmidLUMn
The engine already tracks where each turn's wall time goes (expert-disk service, I/O wait, expert matmul, attention, lm_head) — it just only spoke at exit or under PROF=1. Stream it instead: - glm.c: mux serve emits a per-turn "PROF" protocol line next to TIERS/HITS (window deltas per request, same convention as the STAT hit%); the phase window base is now always captured (a few loads per request). - openai_server.py: parses PROF into a 120-turn rolling window and serves it at /profile (read-only, same trust level as /health). - web: new Profiling tab — stat tiles (tok/s, wall, tokens/forward, disk service), wall-time composition bars for the last turn and the window, per-turn throughput and stacked phase columns with hover readouts, and a table of recent turns. Disk service is shown apart from the stack: it overlaps with compute, so only the I/O wait the compute thread felt counts inside wall time. Phase colours are a CVD-validated set with gaps + legend + table so identity never rides on colour alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WhTmF8yvEBgSkUKSVfZF7P
…isk stall The dashboard's wall-time stack assumed ewait = felt I/O stall and edisk = overlapped read service, but the engine measured neither: t_ewait was declared and reported yet never incremented (the blue 'I/O wait' segment always read 0.00s), while t_edisk accumulated on the COMPUTE thread — blocking OMP loads, PIPE dispatch, pipe_wait spins — i.e. exactly the felt stall. The UI excluded it from the stack as 'overlapped', so the whole disk stall landed in 'Other' (e.g. 46.9s Other vs 45.3s disk on a 54.1s turn). Now the semantics match the contract: - t_ewait accumulates the compute-thread stall (blocking loads, PIPE dispatch, pipe_wait spins) and feeds the in-stack I/O wait. - Disk service is real: expert_load is timed on whichever thread runs the read (PIPE workers, OMP loaders, pilot) into an atomic ns counter (g_edisk_ns) — thread-seconds, overlapped with compute. - prof_report's I/O share and verdict use the felt wait only, and the expert I/O line prints 'read service / felt wait' so the wait:service gap shows how well PIPE is hiding the reads. PROF wire format, /profile JSON and the web UI are unchanged — the dashboard's existing assumptions are simply true now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dd4hHD2Dsvgr7VAwEVFqWK
make_glm_oracle.py wrote glm_tiny/model.safetensors and glm_tiny/config.json
without creating the directory first. safetensors.save_file writes a temp file
inside the target dir before the atomic rename, so on a clean checkout (no
pre-existing glm_tiny/) it aborts with an opaque error:
SafetensorError: Error while serializing: I/O error:
The system cannot find the path specified. (os error 3)
at path ".../glm_tiny/.tmpXXXXXX"
The directory only ever existed because it was left over from a previous run,
so the first-ever `python tools/make_glm_oracle.py` fails for every new user
following the README's verify step.
Create glm_tiny/ with Path.mkdir(parents=True, exist_ok=True) before the save
branch — covers the fp8 path, the bf16 path, and config.json. Path is already
imported; no new dependency, no change to the CPU build.
matmul_i4_grouped is the reference the CUDA fmt=4 port (#298) is expected to reproduce, and it had no test of its own. @woolcoxm is currently debugging a CUDA backend against an oracle nobody had verified, which is two moving targets at once -- and he can't cross-check on CPU, since a 5-prompt run takes 8 hours on the 744B model. This checks matmul_i4_grouped against a plain-C reference that dequantizes nibble -> (v-8)*scale[i/gs] and accumulates in double, over 11 shapes: I a clean multiple of gs, a partial last group (the glen clamp), odd I (the scalar nibble tail), gs > I, gs=16/64/128, S>1, and the nibble extremes 0x00/0xFF -- which decode to -8/+7 because the format is offset-encoded, not two's complement. Reading that backwards turns 15 into -1 and looks like data-dependent noise rather than a bug. All 11 shapes match to ~1e-8 relative, so the CPU kernel is exact and can be trusted as the reference. One note on the tolerance, because the first draft of this test got it wrong and "found" a bug that wasn't there: the error is compared against the sum of |terms|, not against |result|. A dot product of signed terms can land near zero through cancellation, and then a 1e-6 absolute error -- ordinary f32 accumulator precision -- reads as a 1e-3 relative one. A wrong scale index or a wrong group boundary shifts the result by a fraction of the terms, so it is still caught at 1e-6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…esults move to docs/, four new figures
@woolcoxm's matmul_i4_grouped_pair reads x once instead of twice for the gate+up pair. Verified here against his branch (86e91b1 merged onto dev): correct to ~2e-8 relative vs the double reference, and BIT-EXACT against two separate matmul_i4_grouped calls on aligned shapes -- which is the shape the real g64 checkpoints have (I = 2048 / 6144, gs = 64). His kernel is good. Guarded behind COLI_HAVE_GROUPED_PAIR since the function only exists on that branch; add -DCOLI_HAVE_GROUPED_PAIR to the test's Makefile rule when #298 lands and the pair cases activate. The checks are deliberately asymmetric, and the reason is worth recording. Bit-exactness is asserted ONLY when I % gs == 0: there every group is covered by the AVX2 body, whose accumulation order matches the unfused kernel, so any difference is a real bug. With a partial last group the tail falls to scalar code and the compiler may contract/reassociate the fused body differently, producing ~1e-7 differences -- rounding, not logic. My first version demanded bit-exactness everywhere and duly "found" a bug in his kernel that did not exist; the tell was that only `up` differed and never `gate`, which is FP luck rather than a code path. Correctness is checked everywhere against the double reference; identity only where identity is actually implied. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The engine armed its stop tokens from config.json's eos_token_id and nothing
else. That trusts metadata written by third-party conversion tooling, which is
a thing we already know goes wrong: the README documents a mirror shipping
int4 MTP heads that silently give 0% draft acceptance. GLM-5.2 declares THREE
eos ids (<|endoftext|>, <|user|>, <|observation|>); a converter that rewrites
config.json with a reduced list leaves the engine stopping on fewer tokens
than the model emits, and the missed ones get detokenized and printed into the
chat as literal text while generation runs past the end of the turn.
Two independent defenses:
- eos_token_id is now unioned with generation_config.json, which is
HuggingFace's authority for generation (config.json often carries a
partial legacy copy). An extra stop is harmless; a missing one is not.
- every added-token the TOKENIZER marks "special":true is armed as a stop,
whatever the configs say. Those are control tokens (<|user|>, <|assistant|>,
<sop>, [gMASK], the image/video/audio markers) and are never legitimate
content in a reply -- GLM itself lists three of them as official eos.
<think>/<tool_call>/<arg_key> are "special":false and are deliberately NOT
swept up: they are real output. tok.h was parsing added_tokens but throwing
the "special" flag away, so the distinction wasn't available to anyone.
On the real per-row checkpoint this takes the armed set from 3 to 18:
[stop] 18 stop tokens: 154820 154827 154829 154821 ... (15 from the
tokenizer's special set)
Honesty about scope: this is hygiene for a class of bug, NOT a fix for the
trailing-junk report on #298 that prompted it. I hypothesised @woolcoxm's g64
checkpoint had lost eos ids in conversion; he checked, and it hadn't -- his
config arms all three correctly. The emit path is also innocent: is_stop() is
checked BEFORE emit() at every one of the four call sites (4215, 4256, 4908,
4987), so a correctly-armed stop cannot be printed. His trailing junk is still
unexplained and is more likely quantization noise. What this commit buys is
that a checkpoint we don't control cannot leak control tokens into a reply,
which was true before and is not now.
tests/test_stops.c covers both defenses: the union, a missing
generation_config.json, BOTH configs mutilated (the tokenizer still stops all
five control tokens while leaving <think> alone), and T=NULL (the validation
path keeps config-only behaviour).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
) GLM-5.2 MLA uses interleaved (DeepSeek-style) RoPE, which the C engine implements. transformers < 5.11.0 applied split-half (Llama-style) RoPE in GlmMoeDsa* instead; an oracle built on those versions silently drifts and the engine scores 25/32 instead of the documented 32/32 (#281). Weights come out identical across versions -- only the forward pass differs -- so a too-old transformers produces an invalid ref_glm.json with no warning. Add a version gate at the top of make_glm_oracle.py: hard sys.exit with an actionable message citing the issue and the upgrade command. Reads the version from importlib.metadata (authoritative installed-dist version) rather than the mutable transformers.__version__ attribute -- the latter gets reset by the lazy model-class import (from transformers import GlmMoeDsaForCausalLM), so reading it after that import is unreliable. The gate runs before the heavy import and falls back to the attribute only if the dist metadata lookup fails (editable/ src installs). Validated end-to-end on transformers 5.13.1: script runs, ref_glm.json and model.safetensors are byte-identical to the shipped versions, engine scores 32/32. With the floor raised to (5,14) the gate blocks with the expected message.
…t experiments) Feeds reference tokens through the normal step() decode path and reports NLL/ppl + hit rate + tok/s + RSS. Inert unless PPL=1; default path untouched (12/12 vs ref.json verified with patch applied). Cross-checked vs HF transformers bf16 on identical token ids: 12.11 vs 12.25 ppl (#108).
…), new Atlas galaxy, phone + sidebar shots
ci: GitHub Actions — engine, web, Python, CUDA syntax (free for public repos)
Two independent defects in the CUDA syntax job, both caught on its first run. 1. `cuda: '12.6.3'` — Jimver/cuda-toolkit@v0.2.19's version table stops at 12.6.2, so the install step died with "Version not available: 12.6.3" before nvcc was ever invoked. One digit. 2. `nvcc ... 2>&1 | head -40` — a pipeline exits with the status of its LAST command, so head's 0 masked every nvcc error. The job printed "CUDA syntax check passed" unconditionally: it could not fail. Defect 1 is why we found out, since it broke the step *before* the pipe. The second one is the one that matters. backend_cuda.cu is compiled by nothing else in this repo — no local build, no test — so this job is the only thing standing between a CUDA change and a user's GPU. A check that cannot fail is worse than no check: it buys false confidence in exactly the file that most needs the real thing. #298 spent a night debugging CUDA against no oracle at all; this job is supposed to be that oracle. It is also, precisely, the disease of the week in YAML form: a signal that measures its own intention rather than the thing it claims to measure. See the `~335 GB I/O saved` counter that counted dropped experts instead of bytes not read (#303), and `route_agree: 95.3%` cited as "quality preserved" when it only measures which experts coincide. The other three jobs (engine, web, python) passed on the first run. Co-Authored-By: ZacharyZcR <#144> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix: make_glm_oracle.py creates glm_tiny/ before saving (fresh-checkout failure)
docs: add winget python one-liner to Windows install block (#310)
oracle: hard-fail on transformers < 5.11.0 (interleaved-RoPE floor, #281)
docs: serve protocol reference — the engine⇄server wire format (#310)
ci: make check on ubuntu / windows (MSYS2 UCRT64) / macos — and it already caught a Windows test bug (#140)
Fix: set stdout to O_BINARY on Windows to fix READY sentinel
…a-dll-build Add build-config stamp: flag changes (e.g. CUDA_DLL=1) force a relink instead of a silent stale binary
glm: COLI_NUMA=1 — interleave resident weights across NUMA nodes (+13% decode on 2-socket, raw mbind, zero deps) (#82)
The expert lookup counted a hit identically whether the pinned hot-store or the LRU ecache served it — both Phase C branches bumped the single `m->hits`. With a warm pin profile the pin tier absorbs most hot experts, so the ecache could be serving anywhere from ~0% to most hits and the logs couldn't tell which. That made every cache-policy question unanswerable, including #223's "at what cap does the eviction policy start to win?" (a flat A/B can mean "pin absorbed everything" or "genuine floor" and the lumped counter can't distinguish them). Two counters `hit_pin`/`hit_ecache` bumped at the two lookup branches (the existing `m->hits++` stays, so all existing math is unchanged), snapshotted in ProfBase and reset alongside `hits`. Surfaced in the human-readable summaries: decode: expert hit rate 31.3% (pin 22.1% + lru 9.2%) [PROF]: hit 31.3% (X pin + Y lru / Z load) tiny: hit rate 88.1% (0 pin + 74 lru / 10 miss) The serve-mux STAT protocol line is untouched (openai_server.py parses it positionally). Invariant hit_pin + hit_ecache == hits holds by construction — exactly two sites bump hits and each also bumps one split counter, nothing else mutates hits. Verified numerically on the tiny oracle: 0 pin + 74 lru = 74 hits, 88.1%. Zero cost: two increments on paths that already increment. Reported-by: KingIcyCreamProjects <#336> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`coli serve` runs the engine with SERVE_BATCH=1 (openai_server.py), which selects run_serve_mux, which sets g_draft=0 — speculation isn't ragged-safe across the multi-slot batch. But the "[MTP] active: native speculative decoding (draft=N)" line is printed in main() BEFORE the serve path is chosen, so `coli serve DRAFT=8` announced draft=8 and then silently disabled it. @LordMZTE reported the misleading message. The load line and the [MTP] stderr line now detect the mux case (SERVE + SERVE_BATCH) and report it truthfully: "MTP DISABLED (multiplexed serve)" with draft=0, plus a one-line explanation that single-client interactive use (`coli chat`, which spawns run_serve without SERVE_BATCH) keeps MTP. No more draft=8 claim on a path that runs draft=0. This is the honesty half of #358. The feature half — MTP inside the HTTP server for a single client — is a real enhancement but needs engine work: the mux decode kernel would have to run the speculative path when exactly one KV slot is active (S=1 is not actually ragged), and it needs a server round-trip test. Tracked separately; the message no longer lies in the meantime. Reported-by: LordMZTE <#358> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pstream cuda: batch ragged attention across independent streams
…it it (#373) yurivict connected crush per the walkthrough and got 'thought for 1h8m and did nothing' — which is not a hang: agent CLIs send a 10-20k-token system preamble, and prefill on the CPU-streaming path runs at a few tok/s (attention-bound, #153). An hour of silent prefill looks exactly like a dead server. The note now spells out the arithmetic, the curl smoke-test that separates slow-but-working from broken, and honest guidance on what agent workloads are (not) viable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…the resolved plan (#382, #383) Two operator-surprise fixes: openai_server.py (#382, LordMZTE): a request without max_tokens defaulted to min(256, limit) — so `coli serve --ngen 32768` still cut every answer at 256 tokens for clients that omit the field. The operator's configured budget IS the default now; generation still ends at EOS, so it's a cap, not a target. convert_fp8_to_int4.py (#383, bokiko): the resolved conversion plan (mode, source, ebits/io/x bits, grouped-vs-per-row) prints as a [PLAN] line before any work. The two traps in #383 — --mtp defaulting ebits to 8 with the grouped branch silently gated off, and --mtp appearing to ignore --indir — are already defused by the #355 fix (the --mtp pass emits ONLY head tensors on the local path), but a 3.5-hour job must show its plan at second 1, not in a post-hoc size sanity check. bokiko's exact invocation now prints: [PLAN] mode: MTP head only | source: local ./fp8 | experts 8-bit, ... | PER-ROW (grouped branch needs bits<=4; ebits=8 disables it) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # c/Makefile # c/glm.c
cuda: keep paged ragged KV resident across decode steps
plan NUMA interleave on multi-socket Linux
…of scanning all of them (#383) When --indir contains model.safetensors.index.json, the head/indexer passes convert only the shards that actually hold the requested tensors (3 instead of scanning 141 for --mtp — every empty scan still opens a 5 GB shard). Prints the selection in the [PLAN] block. Without the index the full scan is unchanged. Output is byte-identical to the unfiltered path (sha256-verified on the GLM-5.2 FP8 head shards, with a decoy shard proving the filter excludes rather than reorders). Co-Authored-By: Claude <noreply@anthropic.com>
coli run hardcoded the nothink template (<|assistant|><think></think>), so THINK=1 had no effect in one-shot runs while serve mode honors it (glm.c serve template picks <think> vs <think></think> from the same env var). Pick the template the same way here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…interrupted passes (#383) Two gaps in the local path, both raised in #383: - The main --indir pass copied only config.json while the download path copies four files; without tokenizer.json the converted container can't run chat/serve. Copy the same four, print what was copied, and warn when the source is missing any (tokenizer.json called out explicitly). - Local passes restarted from shard 0 on every rerun. out-NNNNN names count EMITTED shards, not input indexes (shards with no relevant tensors emit nothing), so the download path's exists-check can't be mirrored directly: a sidecar manifest per prefix records input -> output (or empty) plus the conversion parameters, written atomically after each shard. Rerun skips what matches and refuses a parameter mismatch on the same outdir instead of mixing containers (the #355 failure mode). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rn on --mtp --ebits <8 Per-row int4 quantization of the MTP head's eh_proj [D,2D] zeroes its ENTIRE embedding half: the tensor's two column halves differ in scale by ~20-30x per row (embedding-half absmax ~0.05, hidden-half ~1.5 on GLM-5.2), so a single per-row scale (absmax/7) puts every embedding-half weight below half a quantization step and np.rint rounds it to exact zero (packed bytes 0x88). The draft head then cannot see the input token and MTP acceptance collapses to ~0% — the mechanism behind issue #8's measurement (int4: 0-4%; int8: 39-59%), and the reason --mtp already defaults to --ebits 8. This showed up in the wild: a published pre-converted container had its MTP shards made with an explicit --ebits 4 and drafts were pure garbage on every backend (deterministic, data-side; verified byte-for-byte by reproducing the published bytes with quant_int4 on the official BF16 rows). - tools/repair_mtp_int8.py: repairs such a container IN PLACE — finds the MTP layer's per-row-int4 dense tensors (eh_proj, q/kv/o projections, shared experts; routed experts untouched), re-downloads only those from the FP8 source repo (~355 MB of HTTP range reads, no torch, no token), requantizes at int8 with quant_int8's exact math, and rewrites the shards atomically with *.bak-int4 backups. --dry-run to inspect. The engine picks up int8 automatically (qt_from_disk detects format by blob size). Validated on GLM-5.2 744B: MTP acceptance 0% -> 100% (greedy), 1.11 -> 3.20 tokens/forward, decode 0.18 -> 0.30 tok/s (Metal, 4-bit KV). - convert_fp8_to_int4.py: print a warning when --mtp is combined with --ebits <8 and per-row scales (the exact footgun above); --group-size 128 remains a valid int4 alternative since group scales give the embedding half its own scale. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- c/version.py: single source of truth (__version__ = "1.0.0") - coli: reads version.py, banner shows dynamic version, --version flag - .github/workflows/release.yml: tag push triggers cross-platform build (Linux x86_64, macOS ARM64, Windows x86_64) and creates a GitHub Release with packaged binaries + changelog notes - CHANGELOG.md: v1.0.0 baseline documenting all shipped features To cut a release: 1. bump c/version.py 2. add a CHANGELOG section 3. git tag v1.0.0 && git push --tags
…turned False Both coli's cuda_binary() and doctor.py's cuda_linkage() detect CUDA support by running `ldd` on the engine binary and looking for a linked libcudart — but that's Linux-only (cuda_binary() checks sys.platform != "linux", and cuda_linkage() checks os.name != "posix", both short-circuiting to False on win32). Windows CUDA_DLL=1 builds never link libcudart at all: glm.exe loads coli_cuda.dll dynamically via LoadLibrary at startup (backend_loader.c), so there's no import-table entry for ldd/dumpbin to find in the first place. The practical effect: `coli doctor` always reported "NVIDIA GPU detected but the engine is CPU-only" on Windows, and `coli run/chat/serve --gpu ...` always hard-exited with "--gpu needs the CUDA build" — even on a correctly built CUDA_DLL=1 binary with coli_cuda.dll sitting right next to glm.exe. Fix: on win32, detect a COLI_CUDA build by scanning glm.exe for the marker string "[CUDA] mode: routed experts", which only exists in code compiled under #ifdef COLI_CUDA (see glm.c's cuda init block), then confirm coli_cuda.dll actually sits next to the binary — mirroring the Linux "linked but missing" distinction. Linux/macOS detection is unchanged. Verified on Windows 11 with mingw-w64 GCC 16.1.0 + CUDA 13.2 + RTX 5080: `coli doctor` now reports "CUDA engine and devices are available", and `coli run --gpu 0 --vram 8` populates VRAM (confirmed via the engine's own "[CUDA] resident set: N tensors, X GB VRAM" runtime log) instead of exiting.
# Conflicts: # .github/workflows/ci.yml
coli: honor THINK=1 in run mode
convert(fp8->int4): --indir copies the full metadata set and resumes interrupted passes (#383)
Fix CUDA detection on Windows (coli/doctor.py always report CPU-only)
tools: repair int4-converted MTP heads in place; warn on --mtp --ebits <8
fix(test): make test_stops build on Windows (mkdtemp compat shim) Conflict with #366's getenv_utf8 in compat.h resolved by keeping both blocks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nosis (#401) The gateway's tool-calling path had unit coverage (parse_tool_calls, render_chat) but nothing exercised the real subprocess wire protocol or the HTTP surface a coding client actually hits. #401 reports plain-text replies where tool_calls were expected; every documented path checks out, so pin the whole path down with a mock engine speaking SUBMIT/DATA/DONE and assert: - non-stream: tool_calls populated, finish_reason tool_calls, no raw markers - stream: markers suppressed across 20-way chunk splits, tool_calls delta - tool-result round trip: <|observation|><tool_response> rendering, text reply - no tools: plain text untouched Also emit a stderr diagnosis when tools are declared and tool-call markers are present in the reply but the strict parse matches nothing (typically quantization-mangled output) pointing at COLI_TOOL_SALVAGE=1 -- the likely field condition behind #401. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CreateProcess cannot exec a shebang script, so the gateway exits during setUpClass on the windows CI job. The gateway logic under test is platform-independent and stays covered by the POSIX jobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…point (#403) cap_for_ram's projection is an estimate: on the GB10 (#403) long generations overshot it by ~40 GB (projected 74.4, real 115.6) and the kernel killed the engine three times. Run D of the issue proves a low cap CONTAINS the growth; this guard does that automatically, keyed on MEASURED RSS instead of the projection. At the repin safe point (no moe in flight), every ~16 emitted tokens: if RSS exceeds the resolved budget (RAM_GB/auto, or an explicit RSS_GUARD_GB ceiling), free the least-used LRU expert slabs in place and lower ecap so the cache cannot regrow. Slabs are >128 KB so glibc returns the pages to the kernel immediately -- RSS actually drops. Eviction never compacts the array: with PILOT_REAL the pilot worker holds pointers into ecache[] across its preads, so the slot stays in place with eid=-1/used=0 (first candidate for reuse); reserved slots (eid<0) are never touched and victim selection happens under g_pilot_mx. resident_bytes is left alone: LRU slots are never accounted there (only pin + dense). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
serve: end-to-end tool-calling regression test + unparsed-marker diagnosis (#401)
glm: measured-RSS guard — the RAM budget enforces itself at the safe point (#403)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First tagged release of colibrì. Brings
mainup to the currentdevstate and adds the release infrastructure (#394):version.py, the cross-platform GitHub Release workflow, and the v1.0.0 CHANGELOG.Highlights since the engine started: GLM-5.2 744B MoE streaming on ~25 GB RAM in pure C; CUDA + Metal backends (ragged attention, resident pipeline, unified-memory SwiGLU); MTP speculation with kernel-pinned verification; OpenAI-compatible
coli serve+ web dashboard; cross-platform CI (Linux/macOS/Windows). Full details inCHANGELOG.md.Final pre-release batch (2026-07-19, all merged into dev, CI green)
coli runhonors THINK=1 (same convention as serve)--indirresumes interrupted passes via sidecar manifest + copies the full metadata set (tokenizer.json included)coli/doctor.py): marker-string probe for CUDA_DLL buildstools/repair_mtp_int8.py: in-place repair of int4-converted MTP heads + converter warning (eh_proj column-scale asymmetry → 0% acceptance)Deliberately not in v1.0.0: #396 (packaging) — needs a single version source (c/version.py) and honest pip-install semantics first; #399/#362/#391 and the platform PRs — post-release review queue.
CI green on
dev(8/8 jobs, all platforms). Tagv1.0.0follows this merge to trigger the release build.🤖 Generated with Claude Code