Generating GGUF from official DeepSeek releases

[MirkoCovizzi]

Hey @antirez, thanks again for your work.

I'm currently working on a port for Strix Halo.

I think it would be interesting if it was more transparent how the GGUF for ds4 was generated and being able to do so even locally using some conversion script.

Furthermore, this v4 release is a preview, so we are going to potentially see future releases, maybe with vision and such, or better intelligence. Being able to have a conversion scaffold on day one locally would be nice.

[antirez]

Hi Mirko, you have the download script download for you the GGUF with the MTP part, and there is even an MTP implementation in the current code, but the gains are very limited. About the scripts to produce the GGUF, I'm using a mix of the llama things and my private code that is not ready to be released, it's a mess because it was used just once. But actually new DeepSeek v4 Flash versions will likely be released in the future, and so the tool would be useful. Right now I have other priorities for the project around tool calling canonicalization reliability, but I'll also keep this in mind and keep this issue open.

[MirkoCovizzi]

Hi Mirko, you have the download script download for you the GGUF with the MTP part, and there is even an MTP implementation in the current code, but the gains are very limited. About the scripts to produce the GGUF, I'm using a mix of the llama things and my private code that is not ready to be released, it's a mess because it was used just once. But actually new DeepSeek v4 Flash versions will likely be released in the future, and so the tool would be useful. Right now I have other priorities for the project around tool calling canonicalization reliability, but I'll also keep this in mind and keep this issue open.

I was not expecting such a fast reply! I wrote the initial comment prematurely regarding the MTP part. I discovered soon after with surprise that there is already code covering MTP, so I quickly edited the comment.

I'm surprised to hear that MTP does not provide much speedup over baseline for DS, however I cannot test until the port to Vulkan is finished. I have had a lot of success with MTP on Qwen3.6.

[antirez]

Unfortunately your wins on Vulkan will likely not apply much to Metal, because the problem is that a N=2 specialized kernel with fast verification is needed. Since I get 65% hits and the verification is wasteful as it is implemented right now, I found no ways to accelerate it. Btw DeepSeek v4 is not particularly speculative decode friendly in the inference side.

[MirkoCovizzi]

Unfortunately your wins on Vulkan will likely not apply much to Metal, because the problem is that a N=2 specialized kernel with fast verification is needed. Since I get 65% hits and the verification is wasteful as it is implemented right now, I found no ways to accelerate it. Btw DeepSeek v4 is not particularly speculative decode friendly in the inference side.

Do you have some sources I can peruse regarding the unfriendliness of DS4 towards speculative decoding? I can only guess that it could be a side effect of the more complex attention architecture?

Edit:

Found this: https://www.lmsys.org/blog/2026-04-25-deepseek-v4/

They analyze also why MTP is not very effective for DS4. Makes sense.

[antirez]

Yep, maybe with a micro batch kernel that does everything in parallel for N=2 it will be better, but also the acceptance I see is low enough to ruin the party. Right now it is not the most terrific thing where to focus IMHO.