Windows 11 nativo + tier VRAM additivi: cascata heat-driven, FFN batc…

[nuspy]

…hed, RAM dinamica

PORT WINDOWS NATIVO (validato token-exact: oracolo TF 32/32, CPU e CUDA su RTX 5090)

• compat.h: ramo _WIN32 completo — pread posizionale thread-safe (ReadFile+OVERLAPPED), O_DIRECT via FILE_FLAG_NO_BUFFERING, fadvise WILLNEED con thread di readahead dedicato, shim dirent/pthread/getline/getrusage/clock_gettime, open forzato _O_BINARY, rename con MOVEFILE_REPLACE_EXISTING, MemAvailable via GlobalMemoryStatusEx. posix_memalign accoppiato a coli_afree (_aligned_free).
• build_win.ps1: build clang (target MSVC) + nvcc dentro vcvars64 con toolset <=14.4x e senza -std (con MSVC 14.5x cudafe++ crasha, CUDA 13.1); l'obj nvcc si linka direttamente nel motore clang.
• coli: binario .exe, rilevamento CUDA dalla import table PE, build via ps1; resource_plan: statvfs -> shutil.disk_usage, MemAvailable via ctypes.

TIER GPU (fasi 1-4)

• VRAM senza backing RAM: gli slot pinnati caricati in VRAM liberano la copia host (tier ADDITIVI: 128GB RAM + 32GB VRAM = 160GB caldi); su errore CUDA demozione rumorosa con ricarica da disco, mai risultati silenziosamente sbagliati. Fix in coli_cuda_tensor_upload: il riuso di un tensore residente non richiede piu' i puntatori host.
• FFN expert fusa e batched (gate->SiLU*up->down) su stream asincrono per device, in overlap con gli expert CPU dello stesso blocco; staging host pinned; CUDA_FFN=0 per l'A/B. Fixture 313M: cuda_pin 45->61 tok/s (+35-40%, 12 run interleaved); cuda_pin_dense 13.3 tok/s vs 2.4-4.5 cpu_stream.
• Allocatore a cascata heat-driven: banda VRAM (upload a blocchi, picco RAM transitorio ~1 blocco) -> banda RAM (PIN_GB) -> LRU+disco; capienza totale RAM+VRAM; AUTOPIN parte anche con sola VRAM. Profili di dominio: COLI_PROFILE= -> storia expert separata (.coli_usage.).
• RAM dinamica a runtime: mem_watch_pass a ogni confine di turno — sotto 3.5GB liberi la LRU si restringe con evict reale, sopra 6GB cresce (MEMWATCH=0 disattiva). Margine statico auto 12%->10%, AUTOPIN 50%->85% x confidenza.
• resource_plan: rimosso il clamp min(vram, cache_bytes) — il tier VRAM e' limitato solo dalla VRAM fisica.

Test: 27 python + 3 C + cuda-test verdi; oracolo TF 32/32 in tutte le combinazioni (CPU, CUDA, pin RAM, pin VRAM-only, batched e sincrono).

Summary

Describe the problem and the smallest change that solves it.

Validation

• make -C c check
• CUDA changes were tested with make -C c cuda-test (if applicable)
• Performance claims include hardware, commands, and repeatable measurements

Compatibility

• The default CPU build remains dependency-free
• No model files, generated binaries, or benchmark artifacts are included

[JustVugg]

First — this is a lot of genuinely good engineering (the dynamic RAM mem_watch at turn boundaries, domain profiles via COLI_PROFILE, additive VRAM-without-host-backing), and thank you for it. But I have to be honest about a coordination collision before this can move, because it's not a code-quality issue — it's that several parts of this PR independently re-implement work that already landed or is in final review from other contributors:

1. The Windows port — nalepy's native Windows port merged as Windows 11 native port (MinGW-w64) — Phase 1: compiles, correct, static-linked #40 (compat.h _WIN32 branch: positional pread, O_DIRECT, fadvise readahead thread, dirent/pthread/getline shims, MemAvailable via GlobalMemoryStatusEx). Your compat.h Windows branch covers the same ground, so it now conflicts with main rather than adding to it.
2. The VRAM tier / heat-driven cascade / fused batched FFN — ZacharyZcR's consolidated runtime PR Unify continuous batching and heterogeneous runtime acceleration #68 does disjoint VRAM/RAM placement, grouped CUDA, and fused expert MLP. Same territory.

So as-is this is CONFLICTING against main and would collide with #68. I don't want to lose the parts that are genuinely yours and new — the dynamic-RAM watcher and domain profiles especially are things neither #40 nor #68 do.

The ask: could you rebase on current main and split this into just the net-new pieces — (a) MEMWATCH dynamic LRU resizing at turn boundaries, (b) COLI_PROFILE per-domain .coli_usage, (c) anything in your VRAM cascade that #68 doesn't already cover — each as a focused PR on top of the merged Windows/CUDA base? That way your original contributions land cleanly instead of fighting three other merges. Happy to review each fast. Sorry it collided — the project is moving quickly enough that two people built the same bridge, which is a good problem to have but needs untangling.

[JustVugg]

Good news from a real Windows validation: I installed a MinGW-w64 cross-compiler and built this branch's CPU-only glm.c to a Windows glm.exe, then ran it via WSL interop — it compiles clean (-D_FILE_OFFSET_BITS=64) and loads the model + runs inference on Windows. The sys/select.h / mmap / madvise / select portability path works. So the build-failure half of #123 is solid.

One blocker before it can land: this branch is based on an older dev (its make check runs 35 tests vs 58 on current dev), so it would revert the recent serve/tool-calling/gpu-flag fixes (#117/#118/#125) that just went to main. Could you rebase onto current dev? Once rebased I can integrate the CPU-portability path directly. The CUDA-on-Windows (DLL) parts I can't test here — no NVIDIA on this box — so we'd review/gate those separately with a CUDA-Windows validator. Thanks for the port work; the hard part is done and verified.

[JustVugg]

Heads up: the CPU-portability half of the native-Windows port just landed in main via #131 (086b2df) — the sys/select.h/mmap/madvise/select POSIX guards, RAM detection, the serve-mode \r\n pipe fix, and $(OS) Makefile detection (#129) are all in and validated (make check 59/59, MinGW cross-compile + run verified). So stock main now builds and runs on native Windows.

Your PR overlaps that now, so to avoid duplicate/conflicting work: if there's anything here #131 doesn't cover — a different GPU-DLL approach, additional tiers, extra fixes — a rebase onto current main focused on just that delta would be the cleanest path, and I'd review it on its own merits. Thanks for the port work; between the three Windows PRs the hard problems are now mapped and one path is validated in main.

[rajpratham1]

Thanks for the substantial contribution. This PR adds Windows native support, CUDA integration, asynchronous FFN execution, compatibility layer changes, build tooling, and extensive runtime modifications.

Before this can be merged, I think additional validation is needed:

• Please split the work into smaller, reviewable PRs where possible (Windows compatibility, CUDA runtime changes, FFN pipeline, documentation, etc.). The current PR is difficult to review and increases merge risk.
• Add CI coverage for the new Windows build path and CUDA-enabled configuration so future changes don't unintentionally break these platforms.
• The new asynchronous FFN pipeline and VRAM tier introduce complex execution and memory-management behavior. Please include tests covering failure paths, synchronization, and fallback behavior to ensure correctness.
• The README describes benchmark improvements and new runtime capabilities. Including reproducible benchmark methodology and validation results would help reviewers verify these claims.

The overall direction looks valuable, but I'd like to see the implementation validated and broken into more manageable pieces before approving.

[JustVugg]

Could you rewrite this against current dev? I'm keeping it open rather than closing it, but it can't land in its present shape.

It's 213 commits behind dev. On glm.c, which changes several times a day, that isn't a rebase — it's a rewrite, and doing it for you would mean rewriting your design decisions without understanding why you made them. So I'd rather ask.

Yours is the oldest and the furthest gone. Six Windows PRs merged this week (#275-279, #290, #302, #314) — some of what you built is probably already in, and the rest needs to be re-expressed against a tree that moved a long way. Worth checking what's left of your delta before you spend the time.

What changed under you, which makes a rewrite cheaper than it sounds:

• CI now exists (ci: make check on ubuntu / windows (MSYS2 UCRT64) / macos — and it already caught a Windows test bug (#140) #143 + ci: GitHub Actions — engine, web, Python, CUDA syntax (free for public repos) #144, merged today): make check runs on ubuntu/windows/macos and the engine/web/python/CUDA-syntax jobs on Linux, for every PR. You'll get a verdict in ~2 minutes instead of waiting on a maintainer who can't test your platform. It caught a bug in my own test on its first Windows run.
• All PRs now target dev, not main. main is locked to admins.
• The backlog is being cleared (31 → 17 today). Your PR sat long enough to rot, and that's this repo's fault, not yours — two contributors independently rebuilt the same Windows CUDA loader this week because nobody triaged in time.

What would help it land fast: the smallest version that does one thing. A 300–700 line PR touching glm.c needs a maintainer who can run it, and for most platforms here that maintainer doesn't exist. A focused change with a test the CI can execute gets reviewed in an hour.

If you'd rather not, say so and I'll close it with thanks — no hard feelings either way. And if you think I've misjudged and it should go in as-is, push back: I've been wrong twice today already and both times a contributor caught it.

[JustVugg]

Closing this one, with thanks and a clear reason rather than a silent stale-out.

It's 263 commits behind dev and the ground it covered has since been built out from other directions: the additive VRAM expert tier (CUDA_EXPERT_GB, auto-sizing, pinning) is in the engine now, and the Windows-native path landed through #131/#278/#279/#290/#314/#298. A 263-behind branch touching 13 files can't be rebased into anything meaningful — it would be a rewrite against a tree that moved a long way.

That's this repo's triage failing you (the backlog grew faster than review), not your work. If there's a specific behavior in your heat-driven cascade or FFN batching that the current VRAM tier still lacks, a small focused PR against today's dev would be reviewed fast — CI now runs on every PR, so you'd get a verdict in minutes. Thanks for pushing Windows forward early.