Skip to content

feat: Add NUMA-aware RAM-disk streaming#377

Draft
BColsey wants to merge 1 commit into
JustVugg:devfrom
BColsey:feature/ramdisk_streaming
Draft

feat: Add NUMA-aware RAM-disk streaming#377
BColsey wants to merge 1 commit into
JustVugg:devfrom
BColsey:feature/ramdisk_streaming

Conversation

@BColsey

@BColsey BColsey commented Jul 17, 2026

Copy link
Copy Markdown

Summary

  • add a Linux-only, Python standard-library coli ramdisk TUI and equivalent scriptable plan, prepare, status, benchmark, start, stop, and destroy commands
  • discover NUMA/kernel capabilities, calculate exact staging and runtime reserves, mount THP-enabled tmpfs safely, and stage full models or profile-guided shard closures
  • add descriptor-verified tmpfs direct mapping for complete experts while preserving the existing buffered, O_DIRECT, pipeline, and io_uring paths for SSD fallback
  • support interleaved and per-node managed engines with durable KV/usage state, crash-safe usage merging, process/mount identity checks, benchmarking, and cleanup
  • package and document the new control-plane module and add focused lifecycle, CLI, engine, integration, and packaging tests

Motivation

Large MoE models are frequently limited by repeated expert reads from SSD. This adds a managed tmpfs tier that can stage all weights, or only profile-selected shard closures, while letting eligible experts bypass slab allocation and LRU I/O through read-only mappings. Unstaged experts continue to use Colibri's existing SSD streaming path.

The canonical model is never modified. Managed state stays on durable storage, and the implementation does not change global swap settings or HugeTLB reservations.

Safety and compatibility

  • sudo is limited to the exact mount and unmount operations
  • mounts, PIDs, process groups, state paths, fingerprints, and safetensors headers are verified before destructive or signaling operations
  • preparation rechecks memory reserve before every copy and uses atomic publication
  • COLI_RAMMAP=1 and legacy COLI_MMAP=1 are mutually exclusive
  • behavior is unchanged when the new RAM-disk variables are unset

Validation

  • make check — portable build, full C suite, and 141 Python tests passed with zero compiler warnings; one privileged integration remains opt-in in the default suite
  • isolated real-tmpfs prepare/status/destroy integration passed, including no swap growth and durable KV preservation
  • git diff --check origin/dev...HEAD passed
  • rebased onto current origin/dev; upstream storage-read, sampling, filesystem-detection, Windows, Nix, and documentation changes remain covered by their tests

Hardware follow-up

Full 744B-model token parity, multi-node placement/throughput, and full/partial physical SSD-read acceptance still require a suitably provisioned NUMA host and model fixture.

@BColsey BColsey changed the title Add NUMA-aware RAM-disk streaming feat: Add NUMA-aware RAM-disk streaming Jul 18, 2026
@JustVugg

Copy link
Copy Markdown
Owner

Heads-up: c/glm.c was just split into c/colibri.c + header modules (#391, now merged into dev). This PR touches the old glm.c, so it will need a rebase onto current dev with its changes moved into colibri.c (or the relevant extracted header: quant.h, sample.h, kv_persist.h, telemetry.h, grammar.h). The make glm target still works (alias of make colibri), so no build scripts break. Apologies for the churn — ping me if the rebase gets thorny and I can help place the hunks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants