Multicontrolnet

[jloveric]

Is it possible to use a multicontrolnet with SD3.5. In the diffusers pipeline, at least, it seems to not be possible - or at least does not work. I'd like to combine canny and depth as is done in SDXL and SD1.5. Any examples (especially in diffusers!) would be appreciated if it can be done. Thanks!

[jloveric]

Anyone reading this, this seems to work. If you pass in a torch tensor the VAE preprocessor ignores it and assumes it's already been normalized. If you pass in a PIL image it does it's own normalization. So, in the diffusers pipeline, assuming control_images[0] is depth map (which is inverted compared to typical) and control_images[1] is canny, modify the images as below. The depth map you keep as a PIL image, but transform the canny to a torch tensor before passing to the pipeline. The canny normalization is the same one used on the huggingface sd3.5 controlnet example

from torchvision import transforms
pil_to_tensor = transforms.ToTensor()

control_images[0] = ImageOps.invert(control_images[0])                
control_images[1] = (pil_to_tensor(control_images[1])*255*0.5+0.5).unsqueeze(0)