Is Multiple Conditioning in Stable Diffusion XL with ControlNet Possible?

[SinaDavari]

I am encountering issues when trying to use multiple conditionings with the Stable Diffusion XL model using ControlNet. I followed a guide and successfully ran ControlNet with depth and segmentation conditionings. The guide was based on stable-diffusion-v1-5, and I wanted to adapt this setup for Stable Diffusion XL. However, I'm facing errors indicating issues with handling multiple ControlNetModels for image conditioning. The errors are as follows:

TypeError: For multiple controlnets: 'image' must be type 'list'
ValueError: For multiple controlnets: 'image' must have the same length as the number of controlnets, but got 1 images and 2 ControlNets.

I am wondering if Multiple Conditioning in Stable Diffusion XL with ControlNet is Possible.

Here is the code snippet that resulted in the aforementioned errors, intending to use depth and segmentation as two conditions with Stable Diffusion XL:

controlnet = [
    ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-depth", torch_dtype=torch.float16).to("cuda"),
    ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-seg", torch_dtype=torch.float16).to("cuda"),
]

pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    variant="fp16",
    use_safetensors=True,
    torch_dtype=torch.float16
).to("cuda")

Any insights or updates that could facilitate this use case would be greatly appreciated.

[tolgacangoz]

Could you see this documentation page for SD-XL with MultiControlNet?

[SinaDavari]

Hi Tolga, and thanks for your swift response.

I am passing an original image along with two conditions into the pipe. Here's how I do it with 1.5:

controlnet = [
    ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-depth", torch_dtype=torch.float16).to("cuda"),
    ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-seg", torch_dtype=torch.float16).to("cuda"),
]

pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    variant="fp16",
    use_safetensors=True,
    torch_dtype=torch.float16
).to("cuda")

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
generator = torch.Generator(device="cpu").manual_seed(seed_num)

for (img, image), condition_image in zip(controlnet_input_images, controlnet_conditions):
            out_image = pipe(
                prompt = prompt,
                negative_prompt = n_prompts,
                control_image = condition_image,
                image = image,
                height = output_img_size[0],
                width = output_img_size[1],
                guidance_scale = CFG,
                strength = img_strength,
                num_inference_steps=num_inference_steps,
                generator=generator,
                controlnet_conditioning_scale = controlnet_conditioning_scale,
            ).images[0]

            # Save the image with a unique filename
            out_img_path = folder_path_with_timestamp / img
            out_image.save(out_img_path)

Where img is a str, e.g., 'test001.jpg'; image is the main RGB PIL.Image input image, condition_image is a list of two RGB PIL.Image condition images (depth and mask.)

It is weird that this implementation works with SD1.5, but not with SD-XL. I tried to pass the original image as a list as well but still got the same 'image' must be type 'list' error. Any ideas on what I might be doing wrong here?

[asomoza]

you're probably using the wrong pipeline, you're using StableDiffusionXLControlNetPipeline when you should use StableDiffusionXLControlNetImg2ImgPipeline to be able to pass the image and control_image args.

[SinaDavari]

That's it. Thank you so very much, dear @asomoza and @StandardAI. You guys are the best. I really appreciate it.