KV8=1: fp8 e4m3 latent KV cache (CPU + CUDA) with .coli_kv v2#299
Closed
NeuralNotwerk wants to merge 1 commit into
Closed
KV8=1: fp8 e4m3 latent KV cache (CPU + CUDA) with .coli_kv v2#299NeuralNotwerk wants to merge 1 commit into
NeuralNotwerk wants to merge 1 commit into
Conversation
Store the MLA latent rows (Lc kv_lora + Rc qk_rope) as fp8 e4m3 bytes with a per-row f32 amax/448 scale instead of f32: ~3.9x less KV RAM (a 256k slot drops 47.7 -> ~13 GB), and kv_pool_bytes reports the smaller pool so cap_for_ram/expert_avail stop demoting experts to disk under multi-slot KV. Same regime as DeepSeek-V3's fp8 latent-KV practice: the latent is both key and value in the absorbed attention; per-row scaling keeps e4m3 safe. - c/kv_fp8.h: 256-entry decode LUT + RNE bit-math encoder (saturating at +/-448, signed zero preserved, NaN stored as inert 0), per-row quantizer with a subnormal-amax guard (448/amax would overflow to +inf). - attention_rows: producer quantizes the fresh normed/roped row in place of the f32 memcpy; absorb score/context loops LUT-dequant inline with the scale hoisted out of the dot; the non-absorb prefill dequants Lc into an f32 staging row for the kv_b matmul and reads Rc via the LUT. - CUDA: attention_absorb_kernel8/_batch_kernel8 (hw fp8 cvt, sm_89+) behind coli_cuda_attention_absorb8/_absorb_batch8/_project_batch8 — 1/4 the PCIe traffic of the f32 uploads; wired at all three COLI_CUDA_ATTN call sites and into the Windows DLL loader. COLI_CUDA_PIPE and Metal still read f32 rows, so KV8 forces the pipe off and auto-disables under Metal. - .coli_kv v2 (COLIKV2 magic, dtype in the header) through the persistent- handle + staging-buffer append path: fp8 rows + scales at ~46 KB/token; v1 files quantize on resume and stay intact on disk until the first append rewrites them as v2 (magic self-heal in kv_disk_open), so a crash before the first save loses nothing. - Tests: exhaustive e4m3 roundtrip + RNE ties incl. the binade-boundary mantissa carry, KV8 kv_alloc transitions, v1/v2 disk round-trip/upgrade/ reject/self-heal, CUDA fp8 kernel checks vs dequantized-host reference. Validation: tiny-oracle KV8=0 bit-identical (TF 32/32, gen 20/20, CPU and CUDA); KV8=1 flips only two positions that transformers' own f32 forward flips across versions (near-tie margins on a degenerate random-weight model); CPU and CUDA fp8 paths agree token-for-token; ASan/UBSan clean. Real-model A/B (GLM-5.2 744B int4, 3x ~450-token log-lik requests): ppl deltas +2.4% / +0.6% / -3.7% — net ~0, inside the int4 noise floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Withdrawing for now — we're running more soak testing on our own fleet first (already found a VRAM budgeting interaction between the fp8 shadow and CUDA_EXPERT_GB=auto worth fixing before this lands). Will reopen once it has more hours on it. |
Contributor
Author
|
Superseded by #399 — the VRAM-budget interaction that prompted this withdrawal (fp8 shadow vs CUDA_EXPERT_GB=auto) is fixed there, plus the stack now includes the review fixes, vectorized fp8 kernels, a 4-bit TurboQuant tier, and full Metal-backend support with parity tests. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opt-in (
KV8=1, default off) fp8 e4m3 storage for the MLA latent KV — the same latent-KV quantization regime DeepSeek-V3 uses. Per token per layer, the Lc/Rc rows become fp8 bytes plus one f32 amax/448 scale each:kv_pool_bytesreports the smaller pool socap_for_ram/expert_availstop demoting experts to disk under multi-slot KV..coli_kvv2 (~46 KB/token instead of ~182): flows through the existing persistent-handle + staging-buffer append path; v1 files quantize on resume and stay intact on disk until the first save rewrites them (magic self-heal inkv_disk_open), so a crash before the first save loses nothing.attention_absorb_kernel8/_batch_kernel8(hardware fp8 cvt, sm_89+) behind three new entry points at ¼ the PCIe upload traffic, wired at theCOLI_CUDA_ATTNcall sites and the Windows DLL loader. Metal andCOLI_CUDA_PIPEstill read f32 rows, so KV8 auto-disables under Metal and forces the pipe off.c/kv_fp8.h), exhaustively unit-tested (256-code round-trip, RNE ties incl. the binade-boundary mantissa carry, subnormal-amax guard).Validation: tiny-oracle KV8=0 is bit-identical (TF 32/32, gen 20/20, CPU and CUDA); KV8=1 flips only two positions whose logit margins are so thin that transformers' own f32 forward flips them across versions; CPU and CUDA fp8 paths agree token-for-token; ASan/UBSan clean. On the real 744B int4 model, log-likelihood A/B over three ~450-token requests: ppl deltas +2.4% / +0.6% / −3.7% — net ~0, inside the int4/routing noise floor. New tests:
tests/test_kv_fp8.c,tests/test_kv_disk.c, KV8 cases intest_kv_alloc.candtest_backend_cuda.cu;make test-candmake cuda-testgreen (sm_120 + sm_89).A follow-up PR (stacked) removes the CUDA T caps for KV8 via a device-resident fp8 shadow + split-T kernels.
🤖 Generated with Claude Code