feat: implement I/O-aware RAM fallback router and native string-indexed memory

[dannythevibe]

This PR introduces two major performance features:

1. I/O-Aware RAM Router: Detects if the model is running on a slow HDD and actively tracks disk latency during the generation loop. If the disk starts bottlenecking, the router falls back to picking experts exclusively from the RAM-resident pool to maintain steady generation speeds.

2. Native Local Memory: Implemented a lightweight MemoryManager in the API server that logs conversation turns to a local text file and uses fast literal string matching to pull relevant historical context into the system prompt, entirely bypassing the need for heavy RAG databases.

[JustVugg]

Could you rewrite this against current dev? I'm keeping it open rather than closing it, but it can't land in its present shape.

It's 121 commits behind dev. On glm.c, which changes several times a day, that isn't a rebase — it's a rewrite, and doing it for you would mean rewriting your design decisions without understanding why you made them. So I'd rather ask.

The RAM/budget path is exactly where this week's OOM work landed (cap_for_ram now refuses to start when the projected peak exceeds available memory). A rebased version will collide with that, so it's worth checking your router still says something the new budget code doesn't.

What changed under you, which makes a rewrite cheaper than it sounds:

• CI now exists (ci: make check on ubuntu / windows (MSYS2 UCRT64) / macos — and it already caught a Windows test bug (#140) #143 + ci: GitHub Actions — engine, web, Python, CUDA syntax (free for public repos) #144, merged today): make check runs on ubuntu/windows/macos and the engine/web/python/CUDA-syntax jobs on Linux, for every PR. You'll get a verdict in ~2 minutes instead of waiting on a maintainer who can't test your platform. It caught a bug in my own test on its first Windows run.
• All PRs now target dev, not main. main is locked to admins.
• The backlog is being cleared (31 → 17 today). Your PR sat long enough to rot, and that's this repo's fault, not yours — two contributors independently rebuilt the same Windows CUDA loader this week because nobody triaged in time.

What would help it land fast: the smallest version that does one thing. A 300–700 line PR touching glm.c needs a maintainer who can run it, and for most platforms here that maintainer doesn't exist. A focused change with a test the CI can execute gets reviewed in an hour.

If you'd rather not, say so and I'll close it with thanks — no hard feelings either way. And if you think I've misjudged and it should go in as-is, push back: I've been wrong twice today already and both times a contributor caught it.

[JustVugg]

Closing this — 171 commits behind, and it lands right on top of code that was substantially rewritten in the meantime.

The RAM/fallback area specifically: cap_for_ram now auto-sizes from measured available memory and refuses to start when the projected peak would OOM (the #305/#341 work), and the expert cache sizing changed with the budget/ws_b reconciliation. An I/O-aware router touching glm.c + coli + openai_server.py from 171 commits back can't be reconciled with that without a full rewrite.

The idea — routing that accounts for I/O cost — is a live and interesting one (it's adjacent to the PILOT/prefetch and cache-policy discussions in #333/#362). If you want to bring it back against current dev, I'd suggest the smallest version that does one measurable thing, with the experts loaded/token counter showing the win — happy to review that. Thanks for the contribution.

[dannythevibe]

Thanks for the detailed breakdown and for taking the time to review it. I completely understand why it can't be merged as-is, especially with the recent memory management overhaul. No hard feelings at all

I'm glad to hear the core idea of I/O-aware routing is still of interest. I'll take a look at the current dev branch and see if I can carve out a minimal, focused PR that demonstrates the win with the new budget code in place. The new CI setup sounds fantastic and should definitely make testing this much easier.

I'll open a new PR when I have a smaller, measurable version ready for review. Thanks again for the transparent communication.