Feature Request: TurboQuant support

[mudler]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Google recently announced TurboQuant - a new quantization method which compresses KV cache using polar coordinates, shrinking memory requirements.

https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/

Results with MLX seems to be promising as well https://x.com/i/status/2036611007523512397

Motivation

This would allow to run bigger models on smaller hardware

Possible Implementation

I'm not submitting a PR because I'm literally playing with it with claude now, but if it can help I'm experimenting at https://github.com/mudler/llama.cpp/tree/feat/turbo-quant and currently builds/starts correctly. Still evaluating it.

[am17an]

Did they release a PyTorch/JAX reference implementation?

[mudler]

Did they release a PyTorch/JAX reference implementation?

could not find it so far, from what I know only implementation in vLLM is being worked out.

[elusznik]

do you have plans for an AMD ROCm implementation or should I try taking a crack at it?

[lotfimaria331-ops]

C ok d'accord Le mer. 25 mar. 2026, 08:20, Nikodem Eluszkiewicz ***@***.***> a écrit :

…

*elusznik* left a comment (ggml-org/llama.cpp#20977) <#20977 (comment)> do you have plans for an AMD ROCm implementation or should I try taking a crack at it? — Reply to this email directly, view it on GitHub <#20977?email_source=notifications&email_token=B7VCPZTCFAAG3GFEVEMD7E34SPMKDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJSGY2TKNZRGAZ2M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-4126557103>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B7VCPZWF2VCHTS3RTPUWHS34SPMKDAVCNFSM6AAAAACW6PMVI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCMRWGU2TOMJQGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[matteoserva]

Did they release a PyTorch/JAX reference implementation?

A third party published a pytorch implementation: https://github.com/tonbistudio/turboquant-pytorch

[pdscomp]

Relevant research:
https://github.com/TheTom/turboquant_plus
https://github.com/TheTom/llama-cpp-turboquant.git

[jokedul]

this is a must! TurboQuant support

[takatodo]

Idea: optional structured rotation (FWHT/Hadamard) for speed

In the current TurboQuant WIP (dense Haar random orthogonal Q generated via Gaussian + QR), applying the rotation is O(d^2) per vector (matvec / matvec_t). For typical KV head dimensions (e.g. 64/128), this can still be a noticeable CPU cost when quantizing/dequantizing many vectors.

Would it make sense to add an optional “structured random orthogonal transform” backend for the rotation, e.g. y = (1/sqrt(d)) * H * D * x (and similarly for the inverse), where H is a Hadamard transform (FWHT) and D is a random ±1 diagonal (optionally also a permutation Π). This would reduce rotation cost to O(d log d) and may be significantly faster on CPU/AVX/NEON.

Note: this is not the exact Haar rotation from the paper, so accuracy would need benchmarking (PPL / KV error). But it could be a useful speed/accuracy trade-off (or a build/runtime option) for common power-of-two head dims.

---

I discussed the FWHT/Hadamard rotation idea with GPT-5.2 and got a few caveats worth mentioning upfront:

A structured transform like H·D is not Haar-uniform, so it won’t inherit the paper’s “Haar random rotation” assumptions/guarantees verbatim; this would be a speed/quality trade-off that needs benchmarking.
On GPUs (especially non-CUDA backends like Vulkan), a naïve FWHT can be memory-/sync-heavy (multiple stages, barriers), and may not outperform a well-optimized dense GEMV unless it’s carefully implemented and/or fused with (de)quantization to avoid extra memory traffic and dispatch overhead.
It’s likely best as an optional backend (e.g. CPU-first, power-of-two dims only, dense fallback otherwise).
I still think it’s worth exploring, but wanted to share these concerns early.