Pls support ComfyUI

[EidosL]

Pls support ComfyUI

[kabachuha]

Theoretically, as the cores simply give hooks to the layers, all is needed is to load DFloat11 modules from the drive, rename and substitute the model's weights + add encoding parts + add hooks, and it should work without any other modules code change (so only one new node is needed)

In real Comfy the code is more complex because of automatic model detection and parameter inference, so it will be a bit harder to implement correctly. I'm making a WIP DFloat11 support for Kijai's Wan Wrapper and although the tensors now all load and the forward passes proceed without interruption (even working with KJ's blockswap), the outputs are black image (even with sage attention disabled). What could be the reason? (see the changed files list)

kijai/ComfyUI-WanVideoWrapper#982

[EidosL]

Theoretically, as the cores simply give hooks to the layers, all is needed is to load DFloat11 modules from the drive, rename and substitute the model's weights + add encoding parts + add hooks, and it should work without any other modules code change (so only one new node is needed)

In real Comfy the code is more complex because of automatic model detection and parameter inference, so it will be a bit harder to implement correctly. I'm making a WIP DFloat11 support for Kijai's Wan Wrapper and although the tensors now all load and the forward passes proceed without interruption (even working with KJ's blockswap), the outputs are black image (even with sage attention disabled). What could be the reason? (see the changed files list)

kijai/ComfyUI-WanVideoWrapper#982

maybe we should use e5 as the weight type？

[kabachuha]

e5 is lossy, while DFloat11 is bit-by-bit identical.

As for DFloat11 representation, its main parts are based on bfloat16 (judging by the list I get with print(name, tensor_value.dtype) in loading loop), and then they undergo quantization/dequantization on the fly using the information encoded beside them in the weights

[mingyi456]

By the way, it appears that ComfyUI is supported now: https://github.com/LeanModels/ComfyUI-DFloat11

But only the flux architecture has its pattern dictionary defined, so currently only flux based models will work out of the box. It should be as trivial as defining it yourself, but I do not know how to do so.

Also, it appears the weights all need to be in a single .safetensors file for ComfyUI to load them, and currently only the DFloat11/FLUX.1-Krea-dev-DF11-ComfyUI is currently officially released in such a format.

I also tried making my own DF11 compression here: mingyi456/shuttle-jaguar-DF11. I tested that it works in the diffusers library only, but it seems like it should also work with ComfyUI because it is in a single file. If you can get the official Krea DF11 compression working, please check mine out to see if it works.