SNDR Core Engine

Genesis vLLM Patches — runtime vLLM patches that run frontier-class open LLMs — Qwen3.6 (7B · 27B · 35B-A3B) and Gemma 4 (26B · 31B · DiffusionGemma) — on consumer NVIDIA GPUs with 24 GB (RTX 3090 / 4090 / 5090, RTX A5000 / A6000): ~1.5× faster inference, quantized tool calling that works, MTP speculative decoding, and up to 280K-token context — no fork, no rebuild.

Built on vLLM, so it serves the models the engine supports; the deep optimizations (TurboQuant KV, hybrid GDN, spec-decode, tuned kernels) are family-tuned for Qwen3.6 and Gemma 4 — the two families we validate on every pin.

🎮 Own a different card? The 24 GiB envelope is class-wide, and sndr up auto-projects VRAM for your GPU. RTX 4090 · RTX 5090 (32 GiB) · dual RTX 3090 — honest per-class gotchas (Ampere-calibrated tuning, no-NVLink P2P, idle-VRAM headroom) in the FAQ.

Contents: Get running · Who is this for · Why SNDR Core · How it compares · What it is · How it works · The platform end-to-end · Headline numbers · Fleet validation · Persistent memory · Pick your path · Install & run · FAQ · Documentation map · Repository structure · Contributing

Turn a consumer NVIDIA card into a production local-AI server. SNDR Core transforms the open-source vLLM engine in memory at boot — no fork, no rebuild — so frontier-class open models (Qwen3.6 up to 35B-A3B, Gemma 4 up to 31B) run ~1.5× faster than stock vLLM with up to a 280K-token served context, on hardware you can actually buy (A5000, RTX 4090 / 5090, A6000 — and yes, the 3090). One paste installs it; a real GUI Control Center drives it.

Two products, one engine: ⚙️ the runtime vLLM patch-overlay (faster inference) + 🧠 a persistent neural-graph memory that makes every model — local and cloud — remember and get smarter over time. Apache-2.0, self-hosted, fully auditable. 329 patches across ~23 families.

Sound familiar?

• You want 70B-class quality but only have 24 GB of VRAM.
• Tool calls break the moment you quantize the model.
• vLLM OOMs on consumer GPUs the moment you ask for long context.

That is exactly the gap this project closes — measured, reproducible, on hardware you already own.

Get running — two commands

curl -sSL https://raw.githubusercontent.com/Sandermage/sndr_core_engine/main/install.sh | bash
sndr up          # auto-picks a preset for your GPU → downloads the model → launches → opens the GUI

That's it — sndr up detects your rig, downloads the weights (skipped if present), starts the engine and the Control Center, and opens your browser at the Control Center (http://127.0.0.1:8765). Prefer the terminal? sndr run does the same and drops you straight into a chat prompt. New here? Start with docs/GETTING_STARTED.md.

The engine needs Linux + CUDA + Docker. On a Mac or Windows laptop you can't run the engine locally — but you can drive a Linux rig remotely with the same sndr CLI and GUI. See docs/RUN_ON_MAC.md (Mac), docs/RUN_ON_WINDOWS_WSL.md (Windows), docs/RUN_ON_LINUX.md (full local stack), and docs/REMOTE_ENGINE.md (client-mode reference).

Who is this for

• Homelab operators — you own (or can buy) a 24 GB-class NVIDIA card and want frontier-class local inference without datacenter hardware. Start: docs/SINGLE_CARD.md.
• On-prem / privacy deployments — data can't leave the building. Self-hosted, Apache-2.0, every applied patch logged; nothing phones home.
• Agent builders — you need an OpenAI-compatible endpoint with tool calling that survives quantization, long agentic tool-chains, and a persistent memory gateway. Start: docs/MCP.md + docs/memory/MANUAL.md.
• Researchers / performance engineers — a 329-entry patch registry with per-patch evidence, a bench suite with CV methodology, and reproducible numbers. Start: docs/BENCHMARKS.md + docs/PATCHES.md.

---

🚀 New here? → docs/GETTING_STARTED.md — who it's for, what you get, and the one install line. ❓ Quick answers → docs/FAQ.md — the questions everyone asks first. 🧠 New to local AI? → docs/LOCAL_AI_PRIMER.md — GPUs, engines, MoE, and quants in plain English. 📖 Hit an unfamiliar term (TPS · KV · MTP · TurboQuant · GDN)? → docs/GLOSSARY.md. 💸 Self-host or cloud? → docs/COMPARISONS.md — the cost-crossover trade.

Why SNDR Core — what you get

You get	How
Frontier-class models (up to 35B-A3B) with 280K served context on a card you can buy	No A100/H100 needed — TurboQuant k8v4 KV-cache quant makes 280K fit (above the model's published 256K limit); Qwen3.6 and Gemma 4 are family-tuned, and consumer Ampere / Ada / Blackwell are first-class targets, not an afterthought.
~1.5× the tokens/sec of stock vLLM — measured, not projected	MTP speculative decode + surgical kernel/scheduler patches. Same wheel, transformed at boot. The numbers below are reproduced on a 2× A5000 rig.
Tool-calling and agent workflows that don't break	The speed patches keep function-call output clean — 7/7 PASS on the dev748 promotion gate and 8/8 on the extended same-day canonical suite (thinking + non-thinking, multi-tool, error-recovery, denial; dev714, 2026-07-04), via the native qwen3_xml streaming parser.
A long-term memory for every model — local and cloud	A brain-like neural-graph memory in one CPU container: recall by meaning, self-organizing "clouds", human-like decay/reinforcement. Zero GPU on the hot path.
Nothing to memorize — one paste, then a real GUI	install.sh + sndr up gets you a running server; the Control Center drives launch, live patch summary, benches, remote hosts, and the memory graph.
Never stuck on a stale fork	It is the same upstream vLLM wheel, patched in memory — and each patch removes itself the moment upstream merges the underlying fix.
Fully yours	Apache-2.0, self-hosted, every applied patch logged and auditable. No black box; nothing phones home.

How it compares

Honest snapshot: the SNDR and stock-vLLM cells are measured on our reference rig; the rest are qualitative — we have not benched Ollama / llama.cpp / TGI here (the repo ships a llamacpp-qwen3.6-27b-q4km-1x preset if you want a measured llama.cpp row on your own rig).

	SNDR Core	Stock vLLM	Ollama	llama.cpp	TGI
35B-class single-stream TPS, 2× 24 GB	242.5 t/s (dev748 promotion gate, 2026-07-04)	~157 t/s (same rig, same model class)	not measured here	not measured here	not measured here
Long-context KV on 24 GB-class cards	280K served (TurboQuant k8v4 KV quant)	fp16 KV — context bounded by VRAM	engine defaults, GGUF KV options	GGUF KV-quant options, manual tuning	fp16 KV by default
Tool-call reliability on quantized models	7/7 promotion gate (dev748) · 8/8 extended harness (dev714, same day) — native qwen3_xml streaming parser	parser shipped, untuned for these quants	varies by model/template	varies by model/template	varies by model/template
OpenAI-compatible API	✅ (vLLM server + GUI Control Center)	✅	✅ (compat endpoint)	✅ (server mode)	partial (Messages API)
MoE + MTP speculative decode together	✅ MTP K=5 on a 35B MoE, measured	model/pin-dependent	no MTP path	draft-model spec-decode only	engine-dependent (Medusa/ngram)

The longer self-host vs cloud (and engine-alternative) discussion lives in docs/COMPARISONS.md.

What it is

A drop-in runtime patcher for vLLM. It pins to a specific vLLM nightly commit and applies 329 small, surgical changes — text edits at known anchors, class-rebind wrappers, and FastAPI middleware — that together turn an out-of-the-box vLLM into a production-grade Qwen3.6 / Gemma 4 inference server on consumer NVIDIA hardware (A5000, RTX 4090 / 5090, A6000, 3090, …) where vLLM upstream mostly targets datacenter SKUs.

It is not a fork of vLLM, a quantizer, a new inference engine, or a training framework. Patches retire automatically when upstream merges the underlying fix.

How it works

The overlay / apply model. Genesis never edits vLLM on disk. At every process start the plugin registers via vLLM's vllm.general_plugins entry point (loaded in the main process, the engine, and every worker rank) and the dispatcher walks PATCH_REGISTRY. Each patch declares an applies_to version range and an apply method — a text edit at a unique source anchor, a class-rebind wrapper, or FastAPI middleware. Patches whose anchors match and whose range covers the live pin apply; the rest print [SKIP — applies_to mismatch] and no-op. The result is an in-memory overlay: the same wheel, transformed at boot, with a structured apply summary (applied=N skipped=M failed=0) and an audit trail. Nothing is written to the vLLM package tree.

Patch families. The 329 entries group into ~23 canonical families. The largest: attention.turboquant (k8v4 KV-cache quant), spec_decode (MTP / ngram speculative decoding), attention.gdn (hybrid Gated-DeltaNet linear attention), gemma4 (Gemma-4 enablement), kv_cache, compile_safety, worker, serving, tool_parsing, and moe. The full table is docs/PATCHES.md (curated) + docs/PATCHES_AUTO.md (generated from the registry).

Pin lifecycle. Genesis pins to one canonical vLLM nightly at a time, plus an optional previous pin held for rollback during validation — at most two ("≤2-pin policy"). A bump happens only on an explicit instruction naming the target pin; there are no proactive pulls. The candidate is validated before promotion (anchor-drift resolved, the bump-preflight gate clean, boot-smoke + tokenizer-fingerprint + canonical bench), then the old 2-back pin is dropped. Current: dev748 (2dfaae752); rollback: dev714 (09663abde). See docs/PIN_BUMP_PLAYBOOK.md (canonical) + docs/ANCHOR_SOT.md.

Model catalog (current registry).

Model	Quant	KV cache	Spec-decode	Status
Qwen3.6-35B-A3B	AWQ (live PROD checkpoint; an FP8 model preset also ships)	TurboQuant k8v4	MTP K=5	✅ PROD (default)
Qwen3.6-27B-int4-AutoRound	INT4 AutoRound (hybrid GDN+Mamba)	TurboQuant k8v4	MTP K=4	✅ PROD
Gemma-4-26B-A4B	AWQ 4/8-bit	uniform fp16 / kv-auto	—	✅ boots + tool-calls (fleet-validated)
Gemma-4-31B	INT4 / kv-auto	TurboQuant or uniform fp16	MTP K=3 (separate drafter)	✅ boots + tool-calls; serving needs MM-budget config
DiffusionGemma-26B-A4B-FP8	FP8-dynamic block-diffusion MoE	TP=2	—	✅ serving at TP=2

Per-model deep-dives + the V2 layered config system: docs/MODELS.md. Hardware envelope: docs/HARDWARE.md.

Launching. Boot any model through a preset — the launcher resolves the preset, runs preflight, and renders the docker run (or podman / bare-metal / k8s) command for you with the correct pin, mounts, and env:

sndr launch prod-qwen3.6-35b-balanced            # boot a preset
sndr launch prod-qwen3.6-35b-balanced --dry-run  # inspect the rendered command, no boot

Note: prod-qwen3.6-35b-balanced is the shipped K=3 balanced default — and the right pick for a single user at a keyboard (latency-tuned, max_num_seqs=2). It is what the zero-decision sndr up / sndr quickstart auto-picks for a lone-user rig. Reach for prod-qwen3.6-35b-multiconc only when serving many concurrent requests — it is throughput-tuned (max_num_seqs=8, ~672 t/s aggregate) and trades single-stream latency for that aggregate. The headline numbers below come from the live PROD stack at MTP K=5 (re-tuned 2026-06-19, +15.8 % single-stream vs K=3) — expect the K=3 preset to land correspondingly below the K=5 figures.

Full operator manual: docs/USAGE.md.

What the platform does end-to-end

Beyond "faster tokens", SNDR Core is a full operations platform around the patched engine — every layer below is shipping today and exercised by the CI gates:

Layer	What ships
Patch engine	The 329-entry PATCH_REGISTRY with per-entry lifecycle (experimental / stable / legacy / retired / coordinator / research) walked by the dispatcher at boot. Every patch is opt-in behind a GENESIS_ENABLE_* env flag; a curated set (56 of 329 entries) is marked default_on and drives the shipped presets. Structured apply summary (applied=N skipped=M failed=0) + audit trail on every boot.
Anchor SOT + drift defense	Each pin gets a generated per-pin anchor manifest (make rebuild-pin regenerates it from the live rig). A daily drift watcher diffs anchors against upstream; a strand gate (scripts/audit_patch_targets_exist.py) fails loudly when a patch's upstream target module vanishes on a new pin — 0 unexcused stranded modules on dev748.
Pin lifecycle	Three tracked slots — current / rollback / stable — with sndr/pins.yaml as the single source of truth. make bump-pin NEW=<pin> (now with a --sha-full flag for the full commit SHA) propagates the string into every downstream artifact, and audit_pin_consistency fails loudly on a half-finished bump. Worked example — the dev714 → dev748 promotion (2026-07-04): preflight re-anchor → boot gate (fleet-wide apply failed=0) → bench gate (242.5 t/s — parity within CV vs same-day dev714, no regression) → receipts → tag rotation.
Bench suite	tools/genesis_bench_suite.py — the tool-call battery (thinking + non-thinking, multi-tool, error-recovery, denial), single-stream decode with CV methodology (n=25, CV reported with every number), an MTP accept-rate floor check (0.55), the new ctx-scaling linearity stage ([5d/8], flags --ctx-scale*) that catches long-context decode cliffs, and an agentic multi-turn depth bench (12-turn tool-chains to 39K prompt tokens).
Interfaces	GUI Control Center (docs/GUI.md) · terminal TUI (docs/TUI.md) · sndr CLI (docs/CLI_REFERENCE.md) — all driving the same product API: launch presets, live patch summary, benches, remote hosts, memory graph.
Model fleet	Qwen3.6 27B (INT4 hybrid GDN+Mamba) and 35B (AWQ / FP8 MoE), Gemma 4 26B and 31B, and DiffusionGemma 26B (block-diffusion MoE) — all seven launchable lanes validated failed=0 in the 2026-07-04 sweep, with the four digest-poisoned lanes re-validated on verified dev748 in the 2026-07-05 re-run (per-lane pin labels in the fleet table below).
Memory	The persistent neural-graph memory subsystem — one CPU container that gives any OpenAI-compatible model recall + decay/reinforcement (own section below; full manual in docs/memory/MANUAL.md).

Headline numbers (v12.1.0 current registry)

Reference rig: 2× RTX A5000 24 GB (Ampere SM 8.6), driver 580.142, CUDA 13.0.2, TurboQuant k8v4, TP=2. Live PROD stack: Qwen3.6-35B-A3B (AWQ checkpoint), MTP K=5, qwen3_xml tool parser, 280K served context.

Fresh canonical bench — pin dev748 promotion gate, 2026-07-04:

Metric	Value
Single-stream wall TPS	242.5 t/s (CV 6.9 %, n=25) — parity within CV vs the same-day dev714 run (no regression), ~1.5× the ~157 t/s stock-vLLM baseline on this rig
Decode TPOT	3.90 ms
TTFT	84.5 ms mean
Tool calls	7/7 PASS (promotion-gate battery)
MTP window accept-rate	0.653 (K=5, floor 0.55)
Context scaling 1K → 32K	LINEAR_OK — no cliff (endpoint ratio 0.84)

Same-day reference — pin dev714, 2026-07-04, extended canonical suite (kept as the labeled comparison run the parity verdict above is measured against):

Metric	Value
Single-stream wall TPS	234.2 t/s (CV 8.4 %, n=25)
Decode TPOT	4.04 ms
TTFT	88.5 ms mean (cold turn ~958 ms, warm ~200 ms — prefix cache)
Tool calls	8/8 PASS (thinking + non-thinking, multi-tool, error-recovery, denial)
MTP window accept-rate	0.660 (floor 0.55)
Agentic 12-turn tool-chain (to 39K prompt tokens)	12/12 successful, 0 silent-empty, decode p50 168 t/s, TTFT p50 1.92 s
Context scaling 1K → 32K	227 / 238 / 250 / 243 / 212 decode t/s — LINEAR_OK, no cliff (endpoint ratio 0.93)

Earlier measured records, each labeled with its pin:

Model	Stock vLLM	Genesis	Δ	Pin / date
Qwen3.6-35B-A3B (single-conc, K=5)	~157 t/s	239.7 t/s	+53 %	dev148 K-tune, 2026-06-19
Qwen3.6-35B-A3B (8-way multi-conc, K=3)	n/a	~672 t/s agg	8-way scaling	2026-05-23 cycle
Qwen3.6-27B-int4-AutoRound (single-conc, K=4)	~87 t/s	~125 t/s	+44 %	dev714, K=4 (see note below)
Tool-call clean rate (35B / 27B)	2–6 / 10	8/8 · 8/8	qualitative	35B: dev714 2026-07-04; 27B: earlier harness record

280K served context verified on the PROD preset (max_model_len: 280000), with linear decode scaling through 32K in the fresh suite. Full methodology, historical comparisons, and per-rig reproduction recipes: docs/BENCHMARKS.md.

Current pin: vLLM 0.23.1rc1.dev748+g2dfaae752 (commit 2dfaae752, image vllm/vllm-openai:nightly-2dfaae752). Per the ≤2-pin policy, dev714 (0.23.1rc1.dev714+g09663abde, commit 09663abde) is retained as the rollback pin; dev672 is dropped. A stable track also exists: the registry recognizes the tagged release v0.24.0 for operators who prefer release pins over nightlies. sndr/pins.yaml is the single source of truth for all three. dev748 was promoted 2026-07-04 through the full playbook chain — preflight re-anchor → boot gate (apply failed=0 across the whole 7-model fleet; four lanes initially booted the dev714 rollback engine via a stale image_digest and were re-run on verified dev748 on 2026-07-05 — see the fleet table below) → bench gate (242.5 t/s wall — parity within CV vs the same-day dev714 run, no regression; tool-call 7/7) → receipts → tag rotation — see docs/PIN_BUMP_PLAYBOOK.md (canonical) and docs/ANCHOR_SOT.md. The per-model table below is the historical dev148 K-tune cycle, kept for cross-model context and labeled with its pin; the fresh dev748 headline above supersedes it for the 35B PROD stack.

Validated across the fleet — 7 models (dev748; 2026-07-04 window + 2026-07-05 re-run)

The works-everywhere proof the project leans on: during the dev748 promotion window every launchable model in the catalog was booted sequentially (2× RTX A5000, TP=2), smoke-tested and mini-benched — and all seven applied their patch sets with failed=0. Post-release audit (2026-07-05): four lanes had initially booted the dev714 rollback engine (a stale hardware image_digest beat the dev748 tag at render; digest + gate since fixed) — those four were re-run on verified dev748 on 2026-07-05 (per-lane in-container version + bench fingerprint checks), and the table shows the re-run numbers. Accept rates are bench-window rates. Condensed from the full sweep table (with the labeled dev714 first-pass rows) in docs/BENCHMARKS.md:

Model	Pin	Decode	Tool-call	Note
Qwen3.6-35B-A3B AWQ (PROD)	dev748	242.5 t/s	7/7	promotion gate 2026-07-04, full canonical suite
Qwen3.6-35B-A3B FP8 (prod-qwen3.6-35b-balanced)	dev748	223.9 t/s	7/7	canonical sndr launch path; window accept 0.621; parity within CV vs dev714 (231.2)
Qwen3.6-27B INT4 TQ k8v4 (+PN520)	n/v	~130 t/s	✓	PN520 loader fix — INT4 degeneration cured (pin unattributed: fingerprint probe timed out)
Qwen3.6-27B INT4 fp8kv (+P100)	dev748	~108 t/s	—	P100 FlashInfer spec-decode runtime-validated on dev748
Gemma 4 26B-A4B AWQ (prod-gemma4-26b-default)	dev748	~141 t/s	7/7	TPOT 7.09 ms (parity vs dev714 7.12)
Gemma 4 31B AWQ (prod-gemma4-31b-kvauto-chat, +PN351)	dev748	TPOT 9.42 ms	7/7	PN351 dev748 launch variant verified in the live container; window accept 0.744; within CV of dev714 (both arms noisy — no gain claim)
DiffusionGemma 26B-A4B FP8 (prod-diffusiongemma-tp2)	dev748	n/a	7/7	diffusion lane boots + responds; AR decode metrics not applicable; tool-calls newly confirmed working on dev748

(27B thinking mode loops — a known pre-existing model trait; chat is validated with enable_thinking:false and the tool-agent workload is unaffected. Details + footnotes in docs/BENCHMARKS.md.)

Recent battle-validations. The PN520 story is the class every operator recognizes: the INT4 27B booted clean — patches applied, server healthy — and then produced garbage output. Root cause was an upstream GDN loader change silently dropping the checkpoint's split BF16 shards from the fused in_proj_ba parameter, leaving the linear-attention layers uninitialised; the PN520 loader revert routes all 96 in_proj_ba shards correctly, and the degeneration is cured (coherent chat + tool calls in the sweep above). In the same window, P100 (FlashInfer FULL-CG spec-decode) was runtime-validated on the fp8kv lane — coherent generation, 0 errors — and PN351's dev748 anchor variant was battle-validated on the head_dim=512 Gemma 4 31B in the 2026-07-05 re-run: the applied variant was read back from the live dev748 container file, and the lane served chat + 7/7 tool-calls with window accept 0.744.

Validated rig baseline — 2026-06-19 (measured on pin 0.23.1rc1.dev148+gb4c80ec0f)

Full model-cycle re-test on the reference 2× A5000 rig after the MTP K=3→K=5 re-tune, recorded on pin dev148 with the FP8 35B checkpoint of that cycle (the live PROD stack has since moved to the AWQ checkpoint — fresh dev748 numbers in the headline table above). The pin has since bumped dev148 → dev301 → dev424 → dev672 → dev714 → dev748 (current) with no decode regression (anchor regen confirmed at each bump). Each model boots the Genesis apply pipeline, applies its patch set, and is benchmarked / smoke-tested live (tools/genesis_bench_suite.py, single-stream warm sweep). The 35B and 27B single-stream rows are the dev148 K=5 re-tune record; Gemma stays K=3 (its separate drafter is optimal at K=3). Note: the live 27B config has since moved to MTP K=4 — the max coherent K for its INT4 tool-calls (K=5 emitted unparseable tool-call tokens on dev714); K=4 warm decode is ~125 t/s, within CV of the K=5 record below.

Model	Quant / KV	Patches	Decode TPS	Tool-call	Status
Qwen3.6-35B-A3B-FP8	FP8 dense · TQ k8v4 · MTP K=5	95	239.7 (CV 4.9 %)	7/7	✅ serving — +15.8 % vs K=3
Qwen3.6-27B-int4-AutoRound	INT4 AutoRound · TQ k8v4 · MTP K=5 (dev148 record; live now K=4)	93	127.4 (CV 8.3 %)	7/7	✅ serving — +8.2 % vs K=3
Gemma-4-31B	INT4 · TQ k8v4 · MTP K=3	81	—	—	⚙️ boots + patches apply; serving needs MM-budget config (multimodal-bidirectional × spec-decode)
DiffusionGemma-26B-A4B-FP8	FP8-dynamic · block-diffusion · TP=2	45	coherent	—	✅ serving at TP=2 — PN-FP8MOE-KPAD (Marlin N=352) + G4_26 (TP-vocab soft-embed); enforce-eager · max-num-seqs 2 · gpu-util 0.80

The 35B and 27B clear their historical peak band — the K=5 re-tune lifts single-stream decode to 239.7 / 127.4 t/s (+15.8 % / +8.2 % vs K=3) within CV → the v12 platform carries no decode regression. PN-FP8MOE-KPAD (backport of open vLLM PR #45703, model-agnostic Marlin-MoE intermediate-pad) plus G4_26 (backport of #45774, DiffusionGemma TP>1 vocab-sharded soft-embed all-gather) make DiffusionGemma the first block-diffusion FP8-MoE checkpoint to boot AND serve coherently at TP=2 on consumer Ampere without a kernel rebuild — validated 2026-06-17 (clears the Marlin N=352 thread-tile crash, then the probs @ embed_weight [131072,2816] TP-vocab shape mismatch; the coherent generation confirms the soft-embed all-gather yields correct TP=2 output).

Persistent memory — neural-graph (new in v12)

A brain-like persistent memory that makes every model — the internal vLLM engines and external models behind your proxy — smarter over time. Knowledge is stored as a graph whose nodes auto-form connections and cluster into "clouds" (like Obsidian), is recalled by vector similarity plus spreading activation across the graph, and decays / reinforces like human memory. It ships as one CPU-only container (Postgres + pgvector + API + GUI + gateway) — the GPU engines are untouched.

By the numbers (v12, all verified): 2 storage backends (in-memory + Postgres/ pgvector) proven identical in CI · real CPU embedder (Model2Vec) semantic match 0.85 related vs 0.01 unrelated · ~100 unit tests + a leak-soak, run on both backends (Postgres against a live pgvector in CI) · one container · zero GPU on the hot path.

Capability	What it does
Storage	Postgres + pgvector (HNSW ANN + lexical GIN); pure-stdlib in-memory reference backend (identical results, CI-verified)
Recall	vector ANN seeds → bounded, cycle-safe spreading activation over the graph, blended with decay; operator-tunable limit + expand-depth
Brain mechanics	Hebbian co-access, Ebbinghaus decay + strength reinforcement (spacing effect), communities ("clouds"), importance, bi-temporal edge invalidation
Search	vector · keyword · hybrid (catches exact terms / names / IDs)
Universal augment	OpenAI-compatible gateway: recall → inject (plain-text system block) → forward → capture, for any model. Multi-upstream — choose per request (X-Memory-Upstream)
Ingest	Obsidian vault import (notes → nodes, [[wikilinks]] → edges, #tags), path-confined; wikilinks resolve case-insensitively and by H1 title, not just exact filename
Manage	remember · forget (delete node + its edges, owner-scoped) · export (whole graph → JSON backup) · import (Obsidian vault) — all from the GUI or CLI
GUI	Obsidian-like force-directed graph (Sigma.js + ForceAtlas2): nodes colored by community, sized by importance. Toolbar shows nodes/edges/communities; List⇄Graph toggle; recall with operator limit + expand-depth; node-detail card with importance/strength/cloud badges + typed connections; Forget/Export/Import actions
Embedders	Model2Vec (real static CPU, 256-dim, no torch) · HashEmbedder (dependency-free)
CLI	sndr mem remember|recall|search|stats (+ TUI Memory panel) — same engine, no GUI required
Ops	API-key auth · owner-scoping · auto consolidate + prune (leak-bounded) · graceful Postgres-down fallback · upstream-error 502/504

GUI — Memory panel (Control Center → Engine → 🧠 Memory; served same-origin). Real screenshot of the live Control Center (dark theme):

One container, one docker run, and any OpenAI client pointed at the gateway gains memory. Deployment recipe, brain mechanics + tuned constants, every endpoint, config, security, and troubleshooting: docs/memory/MANUAL.md.

Pick your path

You have	Start here
1× consumer card (A5000 / 4090 / 5090 / 3090)	docs/SINGLE_CARD.md
2× cards (TP=2 — the reference topology)	docs/HARDWARE.md + docs/MODELS.md
A model not in the catalog	docs/MODELS.md (add-a-model + the V2 config system)
Brand-new / weighing self-host vs cloud	docs/GETTING_STARTED.md · docs/COMPARISONS.md

I want to… (by machine):

I want to…	Read
Run the full stack locally on a Linux + CUDA box	docs/RUN_ON_LINUX.md
Drive a rig from a Mac (client mode)	docs/RUN_ON_MAC.md
Run on Windows / WSL2 (GPU passthrough or client)	docs/RUN_ON_WINDOWS_WSL.md
Point the GUI / CLI at a remote engine	docs/REMOTE_ENGINE.md

Install & run

# 1. install — detects OS / Python / GPU / vLLM, installs the plugin + `sndr` CLI
curl -sSL https://raw.githubusercontent.com/Sandermage/sndr_core_engine/main/install.sh | bash

# 2. run — auto-picks a preset for your GPU, downloads the model, launches, opens the GUI
sndr up            # …or `sndr run` for a terminal chat prompt instead of the GUI

sndr up and sndr run both download the model if it isn't already present (skipped when it is), so step 2 is genuinely one command. Want to see the plan first? Add --dry-run. Pick a named preset with sndr up <preset> (browse them with sndr preset list).

Five-minute walk-through + Day-1 acceptance: docs/QUICKSTART.md. A different vLLM pin, workload, or non-interactive flag set: docs/INSTALL.md.

FAQ

The questions people actually search for. Longer answers (and ~25 more questions) live in docs/FAQ.md.

Can I run a 35B-class model on 24 GB of VRAM? On a single 24 GB card the validated recipe is the 27B INT4 preset (qa-qwen3.6-27b-tq-1x, 78K context); the 35B MoE needs 2× 24 GB at TP=2, where the PROD stack decodes at 242.5 t/s (pin dev748, 2026-07-04). sndr preflight and sndr kv-calc tell you what fits before you download weights. → docs/SINGLE_CARD.md · docs/HARDWARE.md

Why do tool calls break on quantized models? Quantization amplifies upstream parser fragility — <think> tags, multi-tool prompts, and streaming chunk splits produce malformed calls on stock configs. Genesis ships a dedicated tool-call patch family (P59 / P61 / P62 / P64 / P68 / P69) around the native qwen3_xml streaming parser: 7/7 PASS on the dev748 promotion gate and 8/8 on the extended battery (dev714, both 2026-07-04). → docs/FAQ.md · docs/TROUBLESHOOTING.md

How much faster is speculative decoding? The MTP K=3→K=5 re-tune alone lifted 35B single-stream decode +15.8 % (207 → 239.7 t/s, pin dev148, 2026-06-19); the full recommended patch set is ≈1.5× stock vLLM on the same commit (+53 % on 35B, +46 % on 27B; dev148, 2026-06-19). Current canonical figure: 242.5 wall TPS (dev748, 2026-07-04). → docs/SPEC_DECODE_GUIDE.md · docs/BENCHMARKS.md

Is this a fork of vLLM? No. It runs against an unmodified pinned vLLM wheel and applies patches in memory at boot; toggle Genesis on/off with env flags on the same binary. Each patch declares an applies_to version range and retires automatically when upstream merges the underlying fix. → docs/FAQ.md

How does 280K context fit on 24 GB cards? TurboQuant k8v4 KV-cache quantization (8-bit keys, 4-bit values) frees 2–4× more concurrent KV slots, which is what lets the PROD preset serve max_model_len: 280000 — above the model's published 256K limit — with linear decode scaling through 32K (LINEAR_OK, dev748, 2026-07-04). sndr kv-calc projects the exact KV bytes for your card. → docs/KV_PROJECTOR.md · docs/BENCHMARKS.md

Can I run it without Docker? Yes — it is a regular Python package that patches a vLLM installed in the same environment; sndr model-config render <key> --runtime bare_metal emits a venv launch script. Kubernetes and Proxmox lifecycles are wired via python3 -m sndr.cli.legacy service install <key>. → docs/FAQ.md

Is it free? Everything in this repo — sndr/**, tests, docs, bench data — is Apache 2.0. The license gate in the tree guards a commercial overlay that is absent from the public tree; it does not restrict the community tier. → docs/LICENSE_POLICY.md

Documentation map

If you want to...	Read
Two-minute orientation — who it's for, what you get, first token	docs/GETTING_STARTED.md
Learn local-AI basics first (GPUs, engines, MoE, quants — plain English)	docs/LOCAL_AI_PRIMER.md
Weigh self-host vs cloud APIs (the cost-crossover trade)	docs/COMPARISONS.md
Understand how the platform fits together (registry → byte edit, pins, configs)	docs/ARCHITECTURE.md
One-page operator manual (installer → launcher → configs → patches)	docs/USAGE.md
🧠 Persistent memory — full reference (API, gateway, embedders, Obsidian, deploy)	docs/memory/MANUAL.md
Install + first boot	docs/INSTALL.md → docs/QUICKSTART.md
Set up / fix ~/.sndr/host.yaml (paths + mounts)	docs/HOST_SETUP.md
Add your own model end-to-end (weights → YAML → bench)	docs/ADDING_MODELS.md
Operate it day-2 (health checks, swaps, rollbacks, hygiene)	docs/OPERATIONS.md
Browse sndr commands	docs/CLI_REFERENCE.md
Drive the GUI Control Center	docs/GUI.md
Stay in the terminal (TUI)	docs/TUI.md
Quick answers to common questions	docs/FAQ.md
Pick a model + hardware combo	docs/MODELS.md + docs/HARDWARE.md
Tune an env-var flag	docs/CONFIGURATION.md
Browse the patch catalogue + compatibility matrix	docs/PATCHES.md
Diagnose an OOM, cliff, or boot failure	docs/TROUBLESHOOTING.md
Roll a broken release back	docs/PIN_BUMP_PLAYBOOK.md
See current bench numbers + reproduce	docs/BENCHMARKS.md
Author a patch or community plugin	docs/CONTRIBUTING.md
Sponsorship / hardware loan / business invoicing	docs/SPONSORS.md
Disclose a security issue	SECURITY.md

Full docs index: docs/README.md.

Repository structure

The layout separates the shippable engine from the maintainer tooling and vendored third-party code, so the published wheel stays small and the apply pipeline stays auditable.

Path	What it is
sndr/	The engine. The PATCH_REGISTRY + dispatcher, the apply pipeline (text-anchor / class-rebind / middleware patchers), per-engine patch sets (sndr/engines/vllm/...), the V2 layered model-config system, the universal launcher, the CLI (sndr/genesis), and the read-only product API the GUI consumes. This is the only tree the Apache wheel ships.
gui/	The control center — a desktop/web front-end (gui/web, gui/desktop) that drives the sndr product API: launch presets, inspect the live apply summary, browse the patch catalogue, run benches, manage remote hosts, and the 🧠 Memory graph panel. Built static assets are served by the product API.
sndr/memory/	The persistent neural-graph memory engine — storage interface + in-memory & Postgres/pgvector backends, embedders, the brain mechanics (recall / Hebbian / decay / communities / prune), the ConversationMemory augment-capture middleware, the HTTP client, and the Obsidian importer. Exposed via sndr/product_api/routes/{memory,gateway}.py. See docs/memory/MANUAL.md.
deploy/memory/	The unified genesis-memory container (Postgres + pgvector + product-API + GUI + gateway in one image) — Dockerfile, entrypoint.sh, README.
tests/	The pytest suite (13k+ collected). Unit tests per subsystem under tests/unit/..., contract/bundle/proof tests, and the load-bearing CI gate. Excluded from the wheel.
docs/	All public documentation (USAGE, INSTALL, MODELS, HARDWARE, PATCHES, BENCHMARKS, the pin-bump playbook, anchor SOT, …). docs/README.md is the index.
scripts/ + tools/	Maintainer tooling — the audit gates (make gates), doc-sync / link / attribution / drift checkers, anchor-SOT regeneration, bench harnesses, and pin-bump preflight. Not shipped in the wheel.
third_party/	Vendored upstream kernel source (a curated subset of TurboMind's int4 grouped-MoE GEMM, used by the experimental G4_85 MoE kernel patch). See third_party/tm_int4_moe/README.md for provenance + license.
compose/	Reference docker-compose files for the canonical prod presets (35B / 27B, single- and multi-concurrency, long-context).
benchmarks/ + evidence/	Bench harness/data and per-patch proof artefacts (evidence/patch_proof/) plus the A/B validation evidence the registry cites for default-on/off decisions.
schemas/ + plugins/ + assets/ + release/	JSON schemas (patch-entry, config), community plugin samples, README/chart/logo assets, and release artefacts (SBOM, constraints).
pyproject.toml	Single source of truth for packaging and all tool config — [tool.pytest.ini_options], [tool.ruff], [tool.mypy], and the setuptools package layout.
Makefile	The maintainer entry point: make gates (CI gates), make test, make docs, make gui-build, pin-bump preflight, audits.

Contributing

If this saves you a GPU upgrade, a ⭐ helps others find it.

Bug reports, new patches with empirical evidence, new model recipes, and cross-rig bench reports are all welcome. The full workflow (anchor conventions, lifecycle ratchet, pin-bump playbook, PR template) is in docs/CONTRIBUTING.md. Security disclosures go through SECURITY.md.

Ecosystem / Related

• vLLM — the upstream engine SNDR Core patches. Genesis is an overlay, not a fork; each patch retires as upstream merges the underlying fix.
• Hugging Face — where the model weights the presets pull come from.

Credits + license

Apache-2.0 (see LICENSE). Per-patch attribution and upstream PR linkage in docs/CREDITS.md.

Author: Sandermage (Aleksandr Barzov), Odessa, Ukraine. Sponsorship channels (voluntary, no obligations) and hardware-loan contact: docs/SPONSORS.md.