fix(plan): respect explicit COLI_CUDA_MTP=1 - skip the planner's DRAFT=0 export (opt-in was silently inert)#470
Conversation
… so the engine's auto path can engage The documented CUDA-MTP opt-in was silently inert through the wrapper: _auto_tune exported DRAFT=0 (compute-bound, or disk-bound with projected hit < 0.90), which preempts the engine's DRAFT=-1 auto path - the only place COLI_CUDA_MTP is consulted (colibri.c). Found while running the JustVugg#467 A/B: COLI_CUDA_MTP=1 alone measured pure baseline noise until DRAFT=3 was also set. Now an explicit COLI_CUDA_MTP=1 makes the planner leave DRAFT unset (engine resolves draft=3); unset keeps DRAFT=0 -> MTP off under CUDA, the measured-correct default (JustVugg#389 -42 streaming-bound). ENVIRONMENT.md documents the behaviour and the measured trade-off.
|
Merged into This closes the third-and-last wrapper gotcha from #467 — the documented opt-in now actually works, so anyone can reproduce your MTP-under-CUDA A/B (49% acceptance, −32% at 85% hit) without the With this, #467 is fully closed: findings 1/2/4 shipped (#468/#469), finding 3 measured with the default confirmed, and the opt-in now honoured (#470). Genuinely one of the best-run issues in the repo. |
What
Per the request in #467: when
COLI_CUDA_MTP=1is explicitly in the environment,_auto_tuneskips itsDRAFT=0export so the engine'sDRAFT=-1auto path (colibri.c—g_draft = (has_mtp && (!cuda || cuda_mtp)) ? 3 : 0) can actually see the opt-in. Plus the ENVIRONMENT.md line next toCOLI_CUDA_MTP.Why
The documented opt-in was silently inert on the Windows bare-run and
--auto-tierflows:resource_plan.py_auto_tuneexportedDRAFT=0(compute-bound, or disk-bound with projected hit < 0.90), which preempts the engine's auto path — the only placeCOLI_CUDA_MTPis consulted. Anyone A/B-ing speculation under CUDA got a silent no-op unless they also knew to setDRAFT=3(my first A/B pass in #467 measured exactly baseline-vs-baseline noise because of this).Behaviour
COLI_CUDA_MTPunset (default): unchanged — planner still exportsDRAFT=0on compute-bound / low-hit-disk plans; MTP stays off under CUDA. This is the measured-correct default ([Performance]: 5.42 tok/s — dual-socket Xeon Gold 6430 AVX-512 VNNI, 1TB DDR5 RAM #389 compute-bound −42%; [Performance]: RTX 5080 / 128 GB resource matrix (drive x compute x RAM) ΓÇö RAM dominates; COLI_CUDA=0 Windows quirk; PR-ready N-drive shard split #467 streaming-bound −32%).COLI_CUDA_MTP=1explicit: planner leavesDRAFTalone → engine auto path resolves draft=3. An explicit userDRAFT=ncontinues to win over everything (environment_for_planissetdefault-based).Verification
On the #467 box:
COLI_CUDA_MTP=1alone (noDRAFT) through the bare-run Windows flow now loads withMTP ACTIVE (draft=3)and speculates (2.00 tok/forwardon a 2-token smoke); without the var,draft=0as before. The doc line also records the measured trade-off so nobody reads the opt-in as free speed: acceptance holds (49% on sm_120) but at ~85% hit the widened S=4 expert union costs −32% net; the opt-in pays near full residency.Scope
c/resource_plan.py(one guard + comment in_auto_tune),docs/ENVIRONMENT.md(one row). No engine changes.