numa: per-layer pin arenas — fix the PIN_GB=all VMA explosion (#419)#429
Conversation
2-socket A/B — all acceptance criteria metMachine: 2× Xeon Silver 4510 · 251 GB · 6× RTX 5090 · GLM-5.2 int4. Both arms built from identical trees (md5-verified), 1. CPU-only,
|
| stock dev | this PR | |
|---|---|---|
| peak VMA count | 33,492 — climbs linearly at ~250/s for the whole pin phase | 1,673 — flat |
| pins loaded | 10,888 (206.0 GB, 112 s) | 10,965 (207.4 GB, 108 s) |
| decode | 0.20 tok/s · hit 63.8% | 0.20 tok/s · hit 64.5% |
numastat (node0 / node1, Private) |
79.1 / 123.0 GB | 80.5 / 124.4 GB |
The stock curve is the #419 crash in slow motion: ~3.1 VMAs per pinned expert, linear in pin count. This box pins 10.9k experts (251 GB); scale that slope to the 4 TB box's 19,456 and it crosses the default vm.max_map_count=65530 mid-load — while the arena build stays three orders of magnitude below it, independent of pin count. numastat distributions are identical between arms: the interleave behaves exactly as stock (which also confirms MPOL_MF_MOVE was doing nothing on fresh slabs).
2. Full-residency GPU stack (6× 5090, CUDA_EXPERT_GB=auto, 96-token greedy)
| stock dev | this PR | |
|---|---|---|
| peak VMA count | 28,958 | 11,771 |
| decode | 1.31 tok/s · hit 94.4% | 1.33 tok/s · hit 94.4% |
| generated text | — | byte-identical to stock (diff shows only timing lines) |
The first fix revision still peaked ~29k here: the VRAM-ranked prefix slabs (host staging, read once for upload, freed right after) were individually bound. Second commit skips binding those — transient staging earns no policy. The remaining 11.7k is the LRU ecache (its cap auto-raises when CUDA_RELEASE_HOST frees host RAM), which serves decode reads and stays correctly bound — bounded by cache cap, not model size.
Un-drafting. @snailwei your 4×A100/4TB box is the definitive test: stock vm.max_map_count, COLI_NUMA=1 PIN_GB=all, and a numastat -p after load — it should both survive and interleave this time.
|
Heads-up @ZacharyZcR — I was about to merge this (measured VMA-exhaustion root cause + 2-socket A/B is convincing, and it supersedes #425 cleanly), but merging #422 and #438 just landed in |
Follow-up finding: the arenas also fix a ±21% bandwidth lotteryWhile chasing decode noise on the 6×5090 host we found that effective CPU-expert bandwidth (P0-EXEC, identical 10,588-row workload, same env) swung 45.2–68.5 GB/s across runs (±21%) on per-slab With this PR's per-layer arenas stacked on the same tree, four consecutive runs measured 62.6 / 65.7 / 66.4 / 66.6 GB/s (±3%) — and the mean sits at the top of the historical range. Two deterministic binds per layer instead of twenty thousand lottery tickets: the placement lands in the good configuration every time. So beyond the #419 crash fix, the arenas buy run-to-run performance determinism on NUMA hosts: decode settled at 7.85–7.94 tok/s across runs where the per-slab build wandered 6.5–8.0. (Methodology note that fell out of this: on multi-socket boxes, any A/B verdict should report the P0-EXEC GB/s alongside tok/s as a machine-state fingerprint.) |
…gg#419) Root cause of JustVugg#419's 'OOM slab': every per-slab mbind carries its own memory policy, so bound regions cannot merge — measured ~2 VMAs per slab, with or without MPOL_MF_MOVE. A PIN_GB=all load (19,456 experts x slab+fslab) creates ~78k VMAs and crosses the default vm.max_map_count=65530: posix_memalign dies with terabytes free. The fix binds the pinned hot-store as ONE arena per layer. Experts of a layer share a tensor shape, so a layer's pins pack at a fixed stride into two arenas (weights + scales): 2 mbinds and a handful of VMAs per layer instead of ~500. Slices are pre-attached to the slots before the load loop — slab_cap covers expert_load's realloc check, so its alloc branch never fires and expert_load itself is untouched. aslab marks arena ownership: expert_host_release detaches instead of freeing (a REPIN gpu-swap promotion must not free() an interior pointer), and expert_host_ensure re-attaches the slice before reloading. Per-slab mbind remains for the bounded allocations (dense qalloc, LRU ecache, GPU-tier staging), now without MPOL_MF_MOVE: every bind lands before the pread that first-touches the pages, so there is nothing to migrate. numa_init also gains a capability probe done right: one page-aligned page (mbind rejects unaligned addresses with EINVAL), disabling only on errno==EPERM — so a constrained container degrades with a message instead of crashing later, and an EINVAL can never masquerade as a missing capability. The arena path activates only when interleave is actually on (g_numa_nodes>=2, Linux, non-mmap): default builds stay byte-identical.
The VRAM-ranked prefix's host slabs are upload staging — read once and freed right after — so binding them buys nothing and cost ~2 transient VMAs each: measured PEAK maps 28,958 on the six-GPU host even with the pin arenas in place. With the skip: 11,771, all of it the bounded LRU ecache, which serves decode reads and stays correctly bound.
881861a to
57e67d6
Compare
2 participants
Fixes #419. Supersedes the mechanism-level parts of #425 (and keeps its two good ideas, done so they work).
Root cause (measured, not guessed)
OOM slabon the 4 TB box is VMA count exhaustion, not memory: every per-slabmbindcarries its own memory policy, so bound regions cannot merge with neighbours — measured ~2 VMAs per slab, with or withoutMPOL_MF_MOVE(experiment). APIN_GB=allload is 19,456 experts × (slab + fslab):posix_memalign→mmap→ENOMEMat the limit, with terabytes free — and the failure lands mid-load, exactly as reported. It also explains #425's ownnumastat(362 GB on node 0): that patch's probe disables NUMA outright, so it loads by not interleaving at all.The fix
One arena per layer instead of one policy per slab. Experts of a layer share a tensor shape, so a layer's pins pack at a fixed stride into two arenas (weights + scales): 2 mbinds and a handful of VMAs per layer instead of ~500. Whole-model: ~150 policy VMAs instead of ~78,000.
slab_capcoversexpert_load's realloc check, so its alloc branch never fires —expert_loaditself is untouched.aslabmarks arena ownership:expert_host_releasedetaches instead of freeing (a REPIN gpu-swap promotion must notfree()an interior pointer),expert_host_ensurere-attaches the slice before reloading. GPU-prefix slots stay on individual allocs — their host backing really is released after upload, exactly as today.flags=0(every bind lands before the pread that first-touches the pages — nothing to migrate), and the EPERM probe — now on a page-aligned page (mbindrejects unaligned addresses with EINVAL) checkingerrno==EPERMspecifically, so a constrained container degrades with a message and an EINVAL can never masquerade as a missing capability (the fix: gracefully handle mbind EPERM and remove unnecessary MPOL_MF_MOVE #425 probe disabled NUMA unconditionally on every machine — see the review).g_numa_nodes>=2, Linux, non-mmap). Default builds stay byte-identical.Test plan
make check— 77/77, zero warnings (Linux CPU)ESlotfields inert)numastatbalanced across both nodes,wc -l /proc/<pid>/mapsflat during aPIN_GB=allload, and the GPU seems not helpful at all (0.03 TPS) - (4xA100/80G + 4TB RAM Test) #82 decode gain retainedvm.max_map_countand actually interleave; anumastat -pafter load would be the perfect confirmationMarked draft until the A/B numbers are in.