Streaming weights: coroutine-driven tensor window + 3-lane benchmark by gabewillen · Pull Request #91 · stateforward/emel.cpp

gabewillen · 2026-07-03T01:46:47Z

Streaming weights: coroutine-driven tensor window + 3-lane benchmark

Models larger than RAM: a pinned whole-file mmap source plus a sliding ring of
pre-allocated layer slots, filled ahead of decode by a bounded I/O pool, with
run-to-completion preserved end to end. Draft PR — see the handoff section for
exactly where things stand and how to resume.

What this delivers

Upstream SML external-completion scheduler (Add external-completion scheduler policy for co_sm sml.cpp#16, MERGED,
pinned here): completion_source + a plain-loop synchronous drive in co_sm
(sweep fired sources ascending → dispatch trigger → drain required completions).
Measured overhead: 2.6 ns/op fast path, 5.7 ns/op full arm+fire+require+drain
(5950X, -O3) vs 0.2 ns inline baseline. Three bot-review rounds addressed
(abort-safe dispatch state, fire idempotence, nested-dispatch depth guard,
permit burn on flag consume) — all threads resolved, CI green.
emel::model::tensor::window — tensor-owned streaming residency machine:
bind (whole-file mmap source, budget-guarded passthrough-vs-streaming), acquire
(three unconditional dispatches: resolve busy slot / settle target / publish +
ring-advance prefetch), unbind (drains in-flight loads). One staged_read actor
per slot preserves single-writer under concurrent loads; willneed/dontneed
advise flows around the ring (new io/mmap advise surface, guard-routed kinds).
Generator streamed decode route behind one engagement guard
(guard_decode_stream_window_ready, mirroring the lane-pool gate): run_layer's
streamed prologue acquires the slot view and rebases block weight records in
canonical role order. v1 streams decode only; prefill stays resident through
always-valid mmap source views. Streamed decode is token-identical to the
resident route through the public generator path (proven in tests).
weight_streaming bench (opt-in): lanes emel-window / emel-mmap /
llama.cpp-mmap; scripts/bench.sh --memory-max=<bytes|NNpct|none> wraps runs
in a systemd user scope (MemoryMax + MemorySwapMax=0, hard-fail probe).
Infra hardening shipped along the way: RAM-aware build-parallelism clamp
(scripts/build_jobs.sh, per-job memory budgets; coverage lane at 8GB/job),
bench suite-build dedup (default tree + runtime EMEL_BENCH_SUITE instead of a
per-suite llama.cpp clone), bench regressions warn-by-default with
EMEL_BENCH_STRICT_REGRESSION=1 for closeout, LFM2.5-230M fixture graduated to
current_publication with generated parity baselines.

Verified

Window machine: lifecycle + streaming content-integrity suites (races exercised
x3 runs); two real concurrency bugs found and fixed by these tests (cold-jump
ticket overwrite; prefetch aliasing the just-published slot at pass wrap).
Generator: streamed-vs-resident argmax parity, slot-byte consumption proof,
passthrough disengagement.
Upstream: 37/37 including regression tests for every review finding.
Gate lanes at f944516: bench PASS, parity PASS, lint/zig/tests PASS.

Bench numbers so far (no memory pressure, warm cache, 230M fixture)

lane	first_token	32 tokens
emel window (streamed)	409 ms	1069 ms
emel mmap (OS paging)	419 ms	564 ms
llama.cpp mmap	13.5 ms	55.5 ms

Streaming costs ~2x vs OS paging when everything fits (expected — passthrough
mode exists for that case). The EMEL-vs-llama gap is the pre-existing ~400ms
first-token path plus known decode-kernel deltas, identical in both EMEL lanes.

HANDOFF — open items to resume

The headline measurement is still blocked: capped-memory runs (the actual
model-doesn't-fit comparison) OOM on session arenas, which are sized to the
model's ADVERTISED context_length (3.57GB for the 230M fixture). Unblock:
clamp the parsed context_length in the bench fixture before prepare()
(fixture-level change in tools/bench/model/tensor/window_streaming_bench.cpp,
no src/ change), then run
scripts/bench.sh --suite=weight_streaming --memory-max=35pct.
RESUME HERE — coverage lane rerun: commit 7ef7bc1 is validated on the
zig toolchain (full suite 13/13 shards, exit codes captured directly —
beware: twice this session ctest | tail pipe status masked failures as
green). Its three fixes: (a) two UGM encode cases declare unk_id = 0
explicitly — the memset-era reset zeroed non-zero NSDMIs (unk_id = -1
etc.) and the tests silently relied on that UB state; (b) vocab resets are
in-place std::destroy_at/std::construct_at because
assignment-from-temporary materializes the multi-MB struct on the stack and
crashes g++ -O0 coverage shards; (c) pin = upstream main 8114aef. Remaining:
rerun the coverage lane (previous run failed on exactly the crash fixed by
(b)):
PATH=/tmp/emel-clang-format-venv/bin:$PATH EMEL_QUALITY_GATES_CHANGED_FILES=... scripts/quality_gates.sh
(that venv also provides gcovr; gate self-times-out at 30min unless
EMEL_QUALITY_GATES_TIMEOUT=7200s).
Fuzz lane: hard-fails on missing clang on this host (correct per
missing-tools rule). Decide: install clang, or set explicit policy
EMEL_QUALITY_GATES_FUZZ=never.
weight_streaming baselines: suite is opt-in by design (no baselines).
If it should ever be gate-enforced, snapshot with explicit consent:
scripts/bench.sh --snapshot --suite=weight_streaming --update.
Follow-on milestones (recorded in coroutine-plan.md + project memory):
wavefront x streaming multiplexing (the big >RAM throughput lever),
co_sm-throughout sweep, aarch64 repack-on-acquire, session context caps,
MoE expert-granular streaming, on-disk packed operand format.

How to run everything

Full suite: ./scripts/build_with_zig.sh && ctest --test-dir build/zig -R '^emel_tests'
Milestone tests: ./build/zig/emel_tests_bin --test-case='tensor window*,*stream*,co_sm_external_completion*'
Bench (3 lanes, unwrapped): EMEL_BENCH_ITERS=1 EMEL_BENCH_RUNS=1 EMEL_BENCH_WARMUP_ITERS=0 EMEL_BENCH_WARMUP_RUNS=0 scripts/bench.sh --suite=weight_streaming --memory-max=none
Build parallelism: bounded by scripts/build_jobs.sh; override EMEL_BUILD_JOBS /
EMEL_BUILD_JOB_MEM_GB. Never run two gates/benches/builds concurrently.

Note

High Risk
Large new async I/O + generator decode routing with pinned upstream SML; correctness depends on slot lifecycle, prefetch ring, and external-completion drain semantics under concurrency.

Overview
Adds coroutine-driven streaming weights for models larger than RAM: pins stateforward/sml to the external-completion scheduler merge, wires emel::co_sm with completion_source / external_completion_scheduler, and introduces emel::model::tensor::window — bind (mmap whole-file source, budget → passthrough vs ring slots), acquire (multi-dispatch with I/O-pool staged_read loads + mmap willneed/dontneed advise), and unbind (drain in-flight loads).

Generator gains streamed decode routes behind guard_decode_stream_window_ready (per-layer acquire + weight rebase); new LFM2.5-230M generation parity snapshots document token-identical streamed vs resident behavior.

Bench/CI infra: scripts/build_jobs.sh caps parallelism by memory; weight_streaming suite with optional systemd-run MemoryMax (--memory-max), bench build reuse, and benchmark regressions warn by default (EMEL_BENCH_STRICT_REGRESSION=1 for closeout). CMake marks SML includes SYSTEM for pedantic coverage; new tests cover mmap advise, window lifecycle/streaming, and external-completion SM.

^{Reviewed by Cursor Bugbot for commit 7ef7bc1. Bugbot is set up for automated code reviews on this repo. Configure here.}

- SML pin -> external-completion branch (stateforward/sml.cpp PR #16) - emel aliases + sm_external_completion_tests (9 cases green) - io/mmap advise_mapping surface (sequential/willneed/dontneed, guard-routed kinds, 6 cases green) - model/tensor/window machine WIP (detail/errors/events/context/guards/actions) - bench regressions warn by default (EMEL_BENCH_STRICT_REGRESSION=1 for strict) - LFM2.5-230M fixture graduated to current_publication + 4 parity baselines - SYSTEM include for sml dep + pedantic pragma guards (g++ coverage lane)

# Conflicts: # snapshots/quality_gates/timing.txt

Tensor-owned residency for models larger than RAM: whole-file mmap source, budget-guarded passthrough vs streaming decision, ring of pre-allocated layer slots filled by a 2-lane I/O pool via staged_read (one actor per slot for single-writer under concurrent loads), and coroutine suspension on slot loads through the external-completion co_sm policy. Acquire is a three-dispatch protocol (resolve/settle/publish) driven unconditionally by the typed wrapper: resolve joins a slot still mid-load for a different layer (non-sequential access), settle decides resident/loading/unscheduled and submits+requires the target, publish routes on the committed residency, publishes the slot view, and advances the prefetch ring. The prefetch guard never targets the just-published slot (the pass wrap can alias them when layer_count % slot_count != 0) and willneed/ dontneed advise flows ahead of and behind the window. Tests: lifecycle (bind validation, passthrough, rebind rejection, not_bound/ not_streaming legs, allocation-free acquire) and streaming (content integrity across ring eviction, cold jumps joining busy slots, unbind draining in-flight loads).

…sm wrappers

One engagement guard (guard_decode_stream_window_ready, mirroring the lane-pool capability gate) composed onto the scalar kernel and native_quantized flash decode routes; streamed rows sit above their resident/parallel siblings and dispatch window_mode::streamed kernel instantiations. run_layer's streamed prologue acquires the layer's slot view from the tensor window (suspending on in-flight loads via the external-completion drain) and rebases the block's matmul weight records onto the slot in canonical role order; prefill stays resident through the always-valid mmap source views (v1 streams decode only). Owner-side wiring: generator ctor overload takes the bound window actor and its bind-reported streaming_active; the initializer re-attaches both after prepare() reconstructs the backend and records the pristine per-layer weight records the rebase clones from (branch-free bookkeeping). Tests: streamed-vs-resident argmax token parity through the public generator path (2 slots over 4 layers), slot-byte consumption proof (model B's file streamed under model A's records), passthrough disengagement.

New opt-in suite (EMEL_BENCH_WEIGHT_STREAMING=1, set by scripts/bench.sh --suite=weight_streaming) comparing three lanes on the LFM2.5-230M-Q8_0 fixture at max_tokens 32 with separate first_token rows: emel window (streamed decode via the tensor window), emel mmap (passthrough whole-file mapping, OS demand paging), and a llama.cpp mmap reference at matched threads. EMEL lanes are EMEL-owned end to end; the reference lane never shares objects. Note fields record bytes_read and major_faults. scripts/bench.sh gains --memory-max=<bytes|NNpct|none> (weight_streaming only): systemd-run user-scope wrapper with MemoryMax + MemorySwapMax=0, probe with hard fail when delegation is missing, NNpct derived from the fixture size, run-only mode reusing the default bench build dir. Known limits recorded in the bench source: EMEL session arenas size to the model's advertised context_length (3.57GB RSS on this fixture), so honest capped runs need a larger fixture or a session context cap; reference rows pair by case name so the llama measurement appears under both lane names.

Every build script used bare 'cmake --build --parallel' (one job per core). The SML template TUs cost 2-4GB of compiler RSS each, so 30 jobs on a 58GB host oversubscribes memory ~2x and swaps the machine - worst under the coverage lane's g++ -O0 --coverage rebuild inside quality gates. scripts/build_jobs.sh computes jobs = min(cores, MemAvailable / 4GB) (override: EMEL_BUILD_JOBS) and exports CMAKE_BUILD_PARALLEL_LEVEL; sourced by build_with_zig, bench, test_with_coverage, and quality_gates so the gate's inner builds are bounded too. AGENTS.md records the rule plus the no-concurrent-jobs invariant for shared hosts.

Measured compiler RSS is ~3-4GB per template TU under zig -O3 but ~7-8GB under g++ -O0 --coverage, so the flat 4GB budget still let the coverage lane reach the memory ceiling at 8 jobs. The clamp now takes a per-job budget (EMEL_BUILD_JOB_MEM_GB, default 6) and the coverage build recomputes its own bound at 8GB/job (override: EMEL_COVERAGE_BUILD_JOBS).

…eaming weights

g++ -Werror=class-memaccess rejects memset on the non-trivial vocab struct (surfaced by the first coverage-lane run to get past the sm wrapper pragmas); zig/clang never flagged it.

Same -Werror=class-memaccess fix as tests/encoders: g++ coverage builds reject memset on the non-trivial vocab struct; vocab = {} value-initializes identically. 13 files, mechanical sweep.

bench_suite_build_dir() returns the unfiltered default tree whenever its bench_runner is already built: the default binary contains every suite and run_bench_runner already selects at run time via EMEL_BENCH_SUITE, so per-suite trees (each cloning and compiling a private llama.cpp reference) are only justified as a cold-start path on fresh checkouts. configure_bench_build() never applies a suite filter to the default tree, which also fixes a latent thrash where a scoped run could filter-reconfigure the shared tree.

… pin The memset-to-value-init conversion changed test semantics: memset zeroed non-zero NSDMIs (unk_id = -1, special ids, pre defaults) and two UGM encode cases silently relied on unk_id == 0. Those cases now declare their unknown token explicitly. Vocab resets use in-place destroy_at/construct_at because assignment from a temporary materializes the multi-MB struct on the stack and overflows g++ -O0 coverage builds. Pin lands on upstream main 8114aef (PR #16 merge). NOTE: full-suite validation was in flight at commit time (host restart); resume per the PR handoff.

gabewillen

I reviewed PR 91 against AGENTS.md and docs/rules/sml.rules.md, tracing the changed SML machines through the new tensor window, mmap advice surface, generator streamed decode route, and tests. I tried to submit this as REQUEST_CHANGES, but GitHub rejected that because the authenticated account owns the PR; the findings below should still be treated as merge-blocking.

The main theme is that several paths make runtime choices or enqueue/complete work outside explicit SML guards/transitions, and two error paths lose ownership of mmap resources.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 6 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7ef7bc1. Configure here.}

Copilot

Pull request overview

This PR introduces coroutine-driven streaming weight residency for models larger than RAM by adding a tensor-window actor backed by a whole-file mmap plus a bounded ring of preallocated slots, integrates it into the text generator’s decode path behind an engagement guard, and adds a 3‑lane weight_streaming benchmark plus supporting build/CI hardening.

Changes:

Add emel::model::tensor::window (bind/acquire/unbind) using external-completion co_sm semantics to overlap staged reads with run-to-completion dispatch.
Add streamed decode route selection in emel::text::generator and new tests proving streamed-vs-resident token parity and window engagement.
Add weight_streaming bench runner wiring and memory-aware scripting/build parallelism clamping; update fixture publication + parity baselines.

Reviewed changes

Copilot reviewed 64 out of 64 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tools/generation_fixture_registry.hpp	Mark LFM2.5-230M fixture as current_publication
tools/bench/dependency_manifest.txt	Add dependency records for weight_streaming suite
tools/bench/CMakeLists.txt	Enable weight_streaming bench suite plumbing
tools/bench/bench_runner_registry.cpp	Register weight_streaming runner cases
tools/bench/bench_disabled_cases.cpp	Add disabled-case stubs for weight_streaming
tools/bench/bench_dependency_manifest.cpp	Add compiled-in manifest records for weight_streaming
tools/bench/bench_cases.hpp	Declare weight_streaming case appenders
tests/text/tokenizer/tokenizer_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/tokenizer/tokenizer_action_guard_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/tokenizer/preprocessor_wpm_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/tokenizer/preprocessor_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/tokenizer/preprocessor_spm_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/tokenizer/preprocessor_rwkv_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/tokenizer/preprocessor_plamo2_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/tokenizer/preprocessor_fallback_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/tokenizer/bpe_split_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/tokenizer/bpe_regex_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/renderer/renderer_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/generator/stream_window_tests.cpp	New generator streamed-decode parity/engagement tests
tests/text/encoders/test_support.hpp	Safer vocab re-init (no memset / no stack blowup)
tests/text/encoders/common_tests.cpp	Fix UGM unk_id assumptions in tests
tests/text/detokenizer/detokenizer_tests.cpp	Replace memset resets with destroy/construct patterns
tests/text/conditioner/text_conditioner_tests.cpp	Replace memset resets with destroy/construct patterns
tests/sm/sm_external_completion_tests.cpp	New tests for external-completion co_sm behavior
tests/model/tensor/window/window_test_fixture.hpp	New shared fixture for tensor window tests
tests/model/tensor/window/streaming_tests.cpp	New tensor window streaming behavior tests
tests/model/tensor/window/lifecycle_tests.cpp	New tensor window lifecycle/validation tests
tests/io/mmap/advise_tests.cpp	New tests for mmap advise surface
src/emel/text/generator/sm.hpp	Add streamed decode decision rows + streaming ctor
src/emel/text/generator/initializer/actions.hpp	Re-attach stream window and scan pristine records
src/emel/text/generator/guards.hpp	Add guard_decode_stream_window_ready + streamed route guards
src/emel/text/generator/detail.hpp	Add streamed window binding + streamed decode kernels
src/emel/text/generator/context.hpp	Add stream_window + stream_active context fields
src/emel/text/generator/actions.hpp	Add streamed decode request actions
src/emel/sm.hpp	Import external_completion policy + pedantic suppression tweaks
src/emel/model/tensor/window/sm.hpp	New streaming weight window state machine
src/emel/model/tensor/window/guards.hpp	New guards for bind/acquire/unbind protocols
src/emel/model/tensor/window/events.hpp	New public events for window bind/acquire/unbind
src/emel/model/tensor/window/errors.hpp	New error enum for window actor
src/emel/model/tensor/window/detail.hpp	New internal types: extents/slots/tickets/runtime events
src/emel/model/tensor/window/context.hpp	New window actor context (collaborators + owned state)
src/emel/model/tensor/window/actions.hpp	New effects for mapping, streaming loads, publish, teardown
src/emel/io/mmap/sm.hpp	Add advise_mapping state machine surface + typed wrapper
src/emel/io/mmap/guards.hpp	Add guard chain for advise_mapping validation/routing
src/emel/io/mmap/events.hpp	Add advise_mapping event + done/error events
src/emel/io/mmap/errors.hpp	Add invalid_advise_range/advise_failed errors
src/emel/io/mmap/detail.hpp	Add advise runtime/status carrier structs
src/emel/io/mmap/actions.hpp	Add advise effects and publications
src/emel/io/mmap/actions.cpp	Implement platform advise helpers + advise effects
snapshots/quality_gates/timing.txt	Update recorded gate timing snapshot
snapshots/parity/generation_lfm2_5_230m_q8_0_prompt_hello_max_tokens_100.txt	Add generation parity baseline
snapshots/parity/generation_lfm2_5_230m_q8_0_prompt_hello_max_tokens_10.txt	Add generation parity baseline
snapshots/parity/generation_lfm2_5_230m_q8_0_prompt_hello_max_tokens_1.txt	Add generation parity baseline
scripts/test_with_coverage.sh	Use memory-clamped build parallelism for coverage builds
scripts/quality_gates.sh	Source build_jobs.sh for consistent build parallelism
scripts/build_with_zig.sh	Source build_jobs.sh and pass EMEL_BUILD_JOBS to builds
scripts/build_jobs.sh	New shared build-parallelism clamp helper
scripts/bench.sh	Add --memory-max wrapper + build reuse + job clamp
coroutine-plan.md	Update milestone status for streaming/external completion
CMakeLists.txt	Mark SML includes SYSTEM; add new test sources
cmake/sml_version.cmake	Pin SML tag to external-completion merge
AGENTS.md	Document build-job clamp + bench regression policy updates

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

gabewillen · 2026-07-04T14:08:40Z

Superseded by #90. worktree-arch64 contains this branch's streaming window and benchmark work plus 27 rounds of review fixes (every thread here maps to a named arch64 commit in its resolution reply), and the branch-unique commits were cherry-picked onto worktree-arch64: the post-merge SML pin (7a59a16), the bench suite build-tree reuse (9f973b7), and the madvise page-alignment issue found here was fixed there directly (6cc3afb). The memset-era vocab test hygiene already exists on arch64 in equivalent form. Closing in favor of #90.

gabewillen added 14 commits July 2, 2026 15:55

Merge remote-tracking branch 'origin/main' into worktree-dataloader

34fe029

# Conflicts: # snapshots/quality_gates/timing.txt

gate fixes: benchmark marker for window sm, pedantic pragmas for all …

cbe2883

…sm wrappers

coroutine-plan: record Phase 5 external-completion completion via str…

71075de

…eaming weights

tests/encoders: value-initialize vocab instead of memset

15a3e31

g++ -Werror=class-memaccess rejects memset on the non-trivial vocab struct (surfaced by the first coverage-lane run to get past the sm wrapper pragmas); zig/clang never flagged it.

tests/text: value-initialize vocab instead of memset across all suites

8b84c51

Same -Werror=class-memaccess fix as tests/encoders: g++ coverage builds reject memset on the non-trivial vocab struct; vocab = {} value-initializes identically. 13 files, mechanical sweep.

sml: pin to main with external-completion scheduler (PR #16 merge)

f944516

gabewillen commented Jul 3, 2026

View reviewed changes

gabewillen marked this pull request as ready for review July 4, 2026 13:40

Copilot AI review requested due to automatic review settings July 4, 2026 13:40

Copilot started reviewing on behalf of gabewillen July 4, 2026 13:40 View session

cursor Bot reviewed Jul 4, 2026

View reviewed changes

Copilot AI reviewed Jul 4, 2026

View reviewed changes

gabewillen closed this Jul 4, 2026

Uh oh!

Conversation

gabewillen commented Jul 3, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Streaming weights: coroutine-driven tensor window + 3-lane benchmark

What this delivers

Verified

Bench numbers so far (no memory pressure, warm cache, 230M fixture)

HANDOFF — open items to resume

How to run everything

Uh oh!

gabewillen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gabewillen commented Jul 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gabewillen commented Jul 3, 2026 •

edited by cursor Bot

Loading