How can I get correct ip adapter image embeds? I got 4D tensors and I cannnot use it.

[dai-ichiro]

Reproducible sample script

import torch
from diffusers import AutoPipelineForText2Image, DDIMScheduler
from diffusers.utils import load_image

pipeline = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
)
pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)
pipeline.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="sdxl_models",
    weight_name=[
        "ip-adapter-plus_sdxl_vit-h.safetensors",
        "ip-adapter-plus-face_sdxl_vit-h.safetensors"
    ] ,
    image_encoder_folder="models/image_encoder"
)
pipeline.set_ip_adapter_scale([0.7, 0.3])
pipeline.enable_model_cpu_offload()

face_image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/women_input.png")
style_folder = "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/style_ziggy"
style_images = [load_image(f"{style_folder}/img{i}.png") for i in range(10)]

image_embeds = pipeline.prepare_ip_adapter_image_embeds(
    ip_adapter_image=[style_images, face_image],
    ip_adapter_image_embeds=None,
    device="cuda",
    num_images_per_prompt=1,
    do_classifier_free_guidance=True
)
torch.save(image_embeds, "image_embeds.ipadpt")

print(f"type: {type(image_embeds)}")
print(f"len: {len(image_embeds)}")
for embeds in image_embeds:
    print(f"shape: {embeds.shape}")

outputs is

type: <class 'list'>
len: 2
shape: torch.Size([2, 10, 257, 1280])
shape: torch.Size([2, 1, 257, 1280])

3D tensors is preferred, but 4D can be obtained. And I cannot use it.

import torch
from diffusers import AutoPipelineForText2Image, DDIMScheduler


pipeline = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
)

pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)

pipeline.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="sdxl_models",
    weight_name=[
        "ip-adapter-plus_sdxl_vit-h.safetensors",
        "ip-adapter-plus-face_sdxl_vit-h.safetensors"
    ],
    image_encoder_folder=None
)
pipeline.set_ip_adapter_scale([0.7, 0.8])

pipeline.to("cuda")

image_embeds_fromfile =  torch.load("image_embeds.ipadpt")

generator = torch.Generator(device="cpu").manual_seed(2024)
image = pipeline(
    prompt="a woman",
    ip_adapter_image_embeds=image_embeds_fromfile,
    negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", 
    num_inference_steps=50,
    guidance_scale = 0,
    num_images_per_prompt=1,
    generator=generator,
).images[0]
image.save("result_from_image_embeds.png")

I got this error.

ValueError: `ip_adapter_image_embeds` has to be a list of 3D tensors but is 4D

[sayakpaul]

Cc: @yiyixuxu.

Are you using a particular branch of diffusers?

[dai-ichiro]

#7016 was merged, so I use main branch.

pip install git+https://github.com/huggingface/diffusers

diffusers-cli env output is

- `diffusers` version: 0.27.0.dev0
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.11.6
- PyTorch version (GPU?): 2.2.0+cu118 (True)
- Huggingface_hub version: 0.21.3
- Transformers version: 4.38.2
- Accelerate version: 0.27.2
- xFormers version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

[sayakpaul]

Probably warrants for an issue. Please tag @yiyixuxu and myself.

[dai-ichiro]

I'm sorry. I don't understand what you are saying.

Do you mean creating a new issue?

[sayakpaul]

Exactly.

[yiyixuxu]

should be fixed with #7189 now!