win32: auto-enable the GPU in coli chat (stacked on #298)#364
Closed
woolcoxm wants to merge 10 commits into
Closed
win32: auto-enable the GPU in coli chat (stacked on #298)#364woolcoxm wants to merge 10 commits into
woolcoxm wants to merge 10 commits into
Conversation
…ustVugg#292) The JustVugg#292 diagnostic sweep showed 'other' as the JustVugg#1 decode cost (35-70% of decode time, 28-33s) but couldn't decompose it. Added 4 fine-grained timers: t_rmsnorm: both per-layer rmsnorm calls (input norm + post-attn norm) t_router: moe() Phase A (router matmul + sigmoid + top-K selection) t_resadd: both per-layer residual adds (x += tmp) t_embed: token embedding lookup Printed as a new OTHER: line in profile_print, with 'untracked' = remaining. ALSO optimized the router top-K selection sort from O(K²×E) to O(K×E) via a taken-flag array — the old version re-scanned idx[0..kk) for every expert to check 'already taken' (O(K) per expert); a flag array makes it O(1). KEY FINDING from the smoke test: the tracked sub-buckets (rmsnorm 0.015s, router 0.2s, resadd 0.005s, embed 0s) are NEGLIGIBLE — the real 'other' is 28-33s of genuinely untracked time. The hypothesized causes (rmsnorm, router, residual adds) are NOT the bottleneck. The real cost is elsewhere — likely falloc malloc overhead, expert cache lookup loops, or pilot_prefetch/la_predict. Further instrumentation needed to localize. Refs JustVugg#292
JustVugg#292) CRITICAL FIX: profile_print's 'accounted' sum was t_ewait+t_emm+t_attn+t_head, but t_ewait is NEVER accumulated (always 0) while t_edisk (real disk-I/O wait, 30-36s) was EXCLUDED from the sum. This made 'other' appear as 28-33s — the disk I/O time that was already measured but not subtracted. Fixed: include t_edisk in the accounted sum. 'other' drops from 32.9s to 1.6s. The REAL decode breakdown is: expert-disk 36.0s (72%) — NVMe I/O is the true bottleneck expert-matmul 7.6s (15%) attention 3.9s (8%) router 0.2s (0.4%) other 1.6s (3%) Added 5 more fine-grained timers to localize the remaining 1.6s: t_cache: expert pin/LRU cache lookup loop (Phase C) t_pilot: pilot_prefetch + la_predict (I/O hints + routing prediction) t_moe_oh: moe() Phase B union + allocations t_dsa: DSA indexer key computation (reserved, inside t_attn on CPU) t_alloc: falloc allocation overhead (reserved) Verified: all sub-buckets print, 'untracked' now down to 1.3s. Refs JustVugg#292
…EAD hook (JustVugg#292) Instrumentation (committed earlier, this adds the final pieces): - Fixed profile_print accounting: t_edisk was excluded from 'accounted' while t_ewait (never accumulated) was included. This made 'other' appear as 28-33s when it was actually just disk I/O time. After fix: 1.6s. - Added 9 fine-grained sub-bucket timers for 'other' decomposition - Router top-K selection: O(K²×E) → O(K×E) via taken-flag array BATCH_READ experiment: tried coalesced sequential pre-fault of all layer- block misses. Result: WORSE (double-read — the pre-fault reads the data, then expert_load reads it again from page cache). The sequential pre-fault blocks while PIPE would have overlapped. Reverted the dispatch branch but kept the g_batch_read global + env var for future experimentation (the approach would need to replace expert_load's pread with a direct slice into the staging buffer, not just pre-fault pages). Key finding: DIRECT=1 PIPE=1 REPIN=16 already gets expert-disk from 36s (cold, no tuning) to ~15s. The <10s target requires reducing miss count (EXPERT_BUDGET/CACHE_ROUTE) not I/O request coalescing. Refs JustVugg#292
…ness The new g64 model quantizes experts with group-size 64 (fmt=4): one f32 scale per 64 elements instead of per-row. This required changes across the entire compute stack — CUDA kernel, engine dispatch, and dequant helpers — plus a new fused gate+up kernel to recover the ~40% throughput that the generic grouped path costs. CUDA backend (backend_cuda.cu): - ColiCudaTensor: add gs/ng fields for group-size metadata - row_bytes/weight_at: fmt=4 uses same packed int4 layout as fmt=2 - quant_matmul: apply per-group scales (scales[o*ng+g]) for fmt=4, matching the CPU matmul_i4_grouped accumulation exactly - coli_cuda_tensor_upload: allocate O*ng scales (not O) for fmt=4, store gs/ng on the tensor - coli_cuda_tensor_update: handle grouped scales on refresh - coli_cuda_tensor_free/tensor_bytes: VRAM accounting uses O*ng - coli_cuda_expert_group: return 0 for fmt=4 (fall back to correct per-expert path, since grouped_hidden/grouped_down kernels only handle per-row scales) - All 11 quant_matmul call sites updated to pass gs,ng Engine (glm.c): - qt_cuda_upload: pass t->gs to tensor_upload - matmul dispatch (line 816): pass w->gs to coli_cuda_matmul - kv_b shard upload: pass l->kv_b.gs - kv_b shard rb computation: add fmt=4 case ((I+1)/2) - embed_row: add fmt=4 branch with per-group scale dequant - qt_addrow/qt_matvec_rows: add fmt=4 branches (CPU MLA absorption path) - expert_gate_up: add matmul_i4_grouped_pair — fused gate+up for fmt=4 that reads x once instead of twice (+40% tok/s, 0.77 -> 1.08) - run_text: prepend [gMASK]<sop> prefix for GLM models (JustVugg#108 fix — without it, PROMPT mode generates garbage on all GLM snapshots) API (backend_cuda.h, backend_loader.c): - coli_cuda_tensor_upload and coli_cuda_matmul signatures: add gs param - Loader typedefs and wrappers thread gs through the DLL boundary Tests: - bench_tensor_core.cu, test_backend_cuda.cu, test_pipe_cuda.cu: update all calls to new API signature (append gs=0 for non-grouped) New tool: c/tools/diag_harness.py - Comprehensive model diagnostic harness (system probe, correctness smoke, deep PROFILE diagnostic, quality benchmarks via eval_glm.py, throughput with/without MTP). Outputs JSON + Markdown reports. Performance (GLM-5.2 744B g64 / RTX 5070 Ti / 32GB RAM): broken CUDA: 0.05 tok/s (every tensor fell back to CPU) fixed CUDA: 0.30 tok/s (6x — CUDA working) + full opt stack: 0.77 tok/s (CACHE_ROUTE + EXPERT_BUDGET) + fused grouped pair: 1.08 tok/s (+40% from gate+up fusion)
CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12 EXPERT_BUDGET=4 RAM_GB=28 are now the defaults whenever the harness runs. --no-optimize disables them to test the raw unoptimized path. --cuda adds COLI_CUDA_MTP=1 to the GPU stack. These are the exact settings measured at 1.08 tok/s on GLM-5.2 744B g64 / RTX 5070 Ti / 32GB RAM (commit 0430145 + EXPERT_BUDGET from d0971ff).
…R sub-buckets, routing diagnostics Audit found and fixed 8 data-capture gaps that would have caused silent information loss during the test campaign: 1. ENV NOT LOGGED: _exec now writes all custom env vars to each .log file under '--- ENV (custom) ---' for full reproducibility 2. QUALITY PHASE MISSING OPT STACK: eval_glm.py subprocess now inherits self.runner.default_env (CACHE_ROUTE/EXPERT_BUDGET/etc) so scoring runs at full speed, not 0.05 tok/s 3. ATTENTION SUB-BUCKETS NOT CAPTURED: projection/RoPE, score-softmax-value, output projection now parsed and stored as decode_attention_detail 4. OTHER SUB-BUCKETS NOT CAPTURED: rmsnorm, router, resadd, untracked, alloc, cache now parsed as decode_other_detail 5. ROUTING DIAGNOSTICS NOT CAPTURED: route_agree, route_kl, swap_pct, route_swaps, route_slots now extracted 6. STDERR FEATURE LINES NOT CAPTURED: [CACHE_ROUTE], [PIN], [USAGE], [DSA], [CUDA] mode, [MTP] activation now collected as diagnostics dict 7. ENV_STACK NOT IN REPORT: meta now includes env_stack so the report documents which optimization features were active 8. TIME ESTIMATE WRONG: quality phase now estimates at ~1 tok/s not 0.05 Also added fmt_val() helper for clean '?' formatting of missing metrics.
…Vugg review) The kv_b shard scale pointer used h0*(Q+V) which is correct for per-row scales (fmt=2: one scale per row). For fmt=4 (grouped), there are ng scales per row, so the offset must be h0*(Q+V)*ng. Without this, the shard reads from the wrong scale position on multi-GPU, producing silent corruption. Single-GPU is unaffected (no sharding). Fix: const float *scale=l->kv_b.s+(int64_t)h0*(Q+V)*(gs>0?ng:1); Refs JustVugg#298
… crash) PR JustVugg#298 added fmt=4 (grouped int4, gs=64) support to quant_matmul but the two MLA attention absorb kernels kept the per-row scale semantic (wscale[row]) from fmt=2. The g64 model's kv_b is fmt=4 (ng=8 groups/row), so COLI_CUDA_ATTN=1 / COLI_CUDA_PIPE=2 routed it through attention_absorb* which indexed the O*ng scale array with a row index -> wrong stride -> GPU memory fault -> bugcheck 0x116 VIDEO_TDR_FAILURE -> reboot. Add absorb_scale() (mirrors quant_matmul's fmt==4 branch: wscale[row*ng+k/gs] for fmt=4, wscale[row] otherwise) and apply it inside the Q- and V-projection accumulation loops of both attention_absorb_kernel and attention_absorb_batch_kernel. Thread w->gs/w->ng through all six launch sites. No extern-C signature or header changes; the tensor already carries gs/ng from tensor_upload. For fmt!=4 ng==1 so k/gs==0 and the result is bit-identical to before. Validated: COLI_CUDA_ATTN=1 and COLI_CUDA_PIPE=2 (long-prompt prefill, the exact crash config) now run clean; base path unchanged.
…elt t_ewait Rebase artifact from merging dev's edisk_s() (a cumulative atomic counter of disk-read service across all I/O threads) into the PR's profile_print 'accounted' sum. The sum should use m->t_ewait (felt wait on the compute thread, as dev does), not edisk_s() — the cumulative service number overlaps itself and exceeds elapsed, making 'other = elapsed - accounted' go negative (observed: other -258s prefill / -730s decode in the full benchmark run). dev/origin already had this correct (t_ewait); this bug was specific to the rebased PR branch. Verified: PROFILE now reads sensibly, e.g. decode 'expert-disk 57.437s service / 9.628s wait | expert-matmul 1.505s | attention 0.529s | other 0.278s' — wait+matmul+attn+other sums to wall.
This was referenced Jul 17, 2026
On Windows a bare 'coli chat' (no --gpu/--vram/--auto-tier) ALWAYS ran CPU-only, even on a CUDA build with a GPU present. Two defects: 1. cuda_binary() returned False on Windows. It detects CUDA by running 'ldd glm | grep libcudart', which is Linux-only (no ldd on win32) and meaningless anyway because the Windows engine links cudart only inside a runtime-loaded coli_cuda.dll, not as a libcudart symbol in glm.exe. So the --gpu/--vram/--auto-tier gates (which call cuda_binary()) never opened. 2. Even with detection fixed, bare 'coli chat' set no CUDA env: env_for's else-branch only enables CUDA when --gpu/--vram is passed. Nothing auto-enabled the GPU. Now: cuda_binary() on non-Linux returns True iff coli_cuda.dll exists next to glm.exe — the exact file backend_loader.c loads from the engine's own directory, so its presence is a faithful, cheap, DLL-hijack-safe proxy for a CUDA-capable build. And env_for, scoped to win32 (Linux keeps its working explicit-flag UX), auto-enables CUDA when a bare chat detects a CUDA build plus a GPU via nvidia-smi, sizing the expert-tier VRAM budget from real free VRAM via the existing build_plan/environment_for_plan machinery (same as --auto-tier, no guessed budget). If nvidia-smi is missing it falls back to CPU with a clear warning; --gpu none still forces CPU; explicit --vram/--gpu still win. CUDA_DENSE stays an explicit opt-in (matches --auto-tier). Verified on a Windows + RTX 5070 Ti box: bare 'coli chat --model <g64>' now prints '[GPU] auto-enabled CUDA ... 13.0 GB expert tier' and emits COLI_CUDA=1 / COLI_GPUS=0 / CUDA_EXPERT_GB=13.044 (was: all unset, CPU-only). Tests: 4 new cases (auto-enable, nvidia-smi-missing fallback, CPU-build silent, Linux-unchanged) plus the 4 existing default-I/O tests guarded to mock cuda_binary() so they stay host-independent. Full python suite green (env_defaults 8, resource_plan 10, doctor 8, makefile_platform 3, cli_output 3). Out of scope: doctor.cuda_linkage is also POSIX-only and mis-reports on Windows — separate follow-up.
woolcoxm
force-pushed
the
fix/win32-cuda-chat-and-pread-retry-on-g64
branch
from
July 17, 2026 16:58
921644a to
9d7d12e
Compare
woolcoxm
changed the title
This was referenced Jul 17, 2026
Owner
|
Heads-up: |
Contributor
Author
|
ok will get on this tomorrow when i wake up, out for weekend. |
Contributor
Author
Closing as superseded by #298 + #363This PR was "#298 + the win32 auto-enable launcher fix, stacked" — useful when #298 was stalled and #363 couldn't be tested against the fmt=4 engine. That's no longer the situation:
Once #298 merges to dev, #363 provides the launcher fix against a dev that already has the fmt=4 engine. The two pieces are complementary and non-overlapping (as this PR's own description notes) — there's no need for a third PR that combines them. What was hereThe single unique commit ( Verification (already done via #363)The launcher fix is tested and working: 8/8 in Closing. If #298's merge hits trouble and you want the combined stack back, the branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why stacked on #298
Without #298,
row_bytes(fmt=4)returns 0 → every expert upload fails silently → the GPU sits idle even with CUDA enabled. So the two PRs are complementary and non-overlapping: #298 = the CUDA backend works for g64; this PR =coli chatuses it by default on Windows. This branch is#298+ the single launcher commit below.The one change here
On Windows a bare
coli chat(no--gpu/--vram/--auto-tier) always ran CPU-only, even on a CUDA build with a GPU present. Two defects, both fixed:cuda_binary()returnedFalseon Windows — it detects CUDA vialdd glm | grep libcudart, Linux-only and meaningless on win32 (the engine loads cudart fromcoli_cuda.dllat runtime). Now:Trueiffcoli_cuda.dllexists next toglm.exe(the exact filebackend_loader.ctrusts).env_foronly enabled CUDA when--gpu/--vramwas passed. Now: a bare win32coli chatauto-detects the CUDA build + GPU (vianvidia-smi) and enables CUDA, sizing the expert-tier VRAM from real free VRAM via the existingbuild_planmachinery. Falls back to CPU with a warning ifnvidia-smiis missing;--gpu nonestill forces CPU; Linux unchanged.(Same change as #363, rebased onto #298. #363 stays open as the clean dev-based path in case #298 is delayed.)
What this PR no longer contains (and why)
An earlier version also included a
compat_preadtransient-read retry. That was dropped: JustVugg merged a diagnostic-only pread fix for #307 directly todev(211d448— preserves the realWinErrcode instead of collapsing toEIO). With that ondev, the retry is redundant for the #307 story; the maintainer's chosen resolution stands. This PR now touches onlyc/coli/ tests / docs and merges cleanly.Verification (this stack on a Windows + RTX 5070 Ti box)
```
[GPU] auto-enabled CUDA · NVIDIA GeForce RTX 5070 Ti · 13.2 GB expert tier
[CUDA] hot expert tier: 173/173 experts, VRAM 3.67 GB (was: 0/134, 0.00 GB on dev)
[CUDA] resident set: 423 tensors, 3.67 GB VRAM
CUDA expert tier: 173 resident experts (3.67 GB) | 482 calls served from VRAM
```
Prompt → 6-line poem, coherent, correct EOS, exit 0, no `Input/output error` (the engine now also names the real WinErr if a read does fail, thanks to
211d448). Tests: 8/8 in `test_env_defaults.py`.Relationship to the other PRs
dev#361dev's211d448+ this PR