Another comfy-ui compatible FLUX1.dev LoRA fails to load

[AmericanPresidentJimmyCarter]

Describe the bug

https://civitai.com/models/631986/xlabs-flux-realism-lora

This LoRA fails to load. Specifically, I get a ValueError: Incompatible keys detected.

Reproduction

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16).to('cuda')

pipe.load_lora_weights('/data/models/flux_realism_lora.safetensors', adapter_name="wow_details")

prompt = "a tiny astronaut hatching from an egg on the moon"
out = pipe(
    prompt=prompt,
    guidance_scale=3.5,
    height=1024,
    width=1024,
    num_inference_steps=25,
).images[0]
out.save("image.png")

Logs

ValueError: Incompatible keys detected

System Info

Latest commit on main

Who can help?

@sayakpaul

[hlky]

Hub download of this LoRA for reference.

from huggingface_hub import hf_hub_download

file = hf_hub_download(repo_id="bigdata-pw/LoRA", filename="706528/flux_realism_lora.safetensors")

[spezialspezial]

Assuming you are talking about modelVersionId 706528, i.e. https://civitai.com/models/631986?modelVersionId=706528
It's the comfyui (merge) format. Structure for Flux mostly follows:

"comfyui": {
	"transformer_name": "diffusion_model.",
	"double_block_name": "double_blocks.",
	"single_block_name": "single_blocks.",
	"double_module_names": (
		".img_attn.proj",
		".img_attn.qkv",
		".img_mlp.0", ".img_mlp.2",
		",img_mod.lin",
		".txt_attn.proj",
		".txt_attn.qkv",
		".txt_mlp.0", ".txt_mlp.2",
		".txt_mod.lin"
	),
	"single_module_names": (
		".linear1",
		".linear2",
		".modulation.lin",
	),
	"param_names": (".lora_down.weight", ".lora_up.weight"),
	"text_encoder_names": ("text_encoders.clip_l.", "text_encoders.t5xxl."),
	"unique_meta": (),
	"comment": "getting more popular due to merging, many ill defined files in the wild"
},

Regarding modelVersionId 706529. Out of over 1000 LoRAs I have this is the only one in xlabs format. Supporting comfyui format would be more worthwhile.

Another somewhat popular kind not supported, last time I checked, would be Onetrainer style LoRAs.

[sayakpaul]

Can you open two separate issues for OneTrainer and Comfy LoRAs that you couldn’t load with the latest versions of diffusers and peft? If latest stable versions don’t work then I would also try out the from source installations.

If both fail, please do open the two separate issues and I will look into it. From memory, I think we support both but I could be wrong.

[spezialspezial]

This is the first Onetrainer LoRA I tried
https://civitai.com/models/767016?modelVersionId=857899

and it fails with:

	File "/+DEV/AI/diffusers-edge/src/diffusers/loaders/lora_pipeline.py", line 1450, in lora_state_dict
		state_dict = _convert_kohya_flux_lora_to_diffusers(state_dict)
	File "/+DEV/AI/diffusers-edge/src/diffusers/loaders/lora_conversion_utils.py", line 687, in _convert_kohya_flux_lora_to_diffusers
		return _convert_mixture_state_dict_to_diffusers(state_dict)
	File "/+DEV/AI/diffusers-edge/src/diffusers/loaders/lora_conversion_utils.py", line 659, in _convert_mixture_state_dict_to_diffusers
		if remaining_all_unet:
UnboundLocalError: local variable 'remaining_all_unet' referenced before assignment

Now I'm doing a bit of gymnastics in my current setup and I will try to verify on a simpler and cleaner setup tomorrow unless you can replicate this quickly.

Other Onetrainer LoRAs in case you are interested.
https://civitai.com/models/794095?modelVersionId=887953
https://civitai.com/models/754969?modelVersionId=884632
https://civitai.com/models/991928?modelVersionId=1111315
https://civitai.com/models/825919?modelVersionId=923640
https://civitai.com/models/1226276?modelVersionId=1381683

[spezialspezial]

These comfyui LoRAs
https://civitai.com/models/643886?modelVersionId=720252
https://civitai.com/models/960680?modelVersionId=1075587
https://civitai.com/models/995599?modelVersionId=1115596

fail with:
ValueError: Incompatible keys detected

but I remember that the listed keys in the error message are misleading

Edit: If anyone else can replicate I'll move this to a separate issue

[sayakpaul]

@spezialspezial regarding the ComfyUI LoRAs, I tried https://huggingface.co/sayakpaul/different-lora-from-civitai/blob/main/RM_AnimeNiji_v0.3.safetensors.

I could load it with the following warning:

It seems like you are using a DoRA checkpoint that is not compatible in Diffusers at the moment. So, we are going to filter out the keys associated to 'dora_scale` from the state dict. If you think this is a mistake please open an issue https://github.com/huggingface/diffusers/issues/new.

It is expected. See more here:
KohakuBlueleaf/LyCORIS#216

https://huggingface.co/sayakpaul/different-lora-from-civitai/blob/main/RM_Artistify_v1.0M.safetensors has T5 as a LoRA. We have never supported T5 LoRAs. So, will work on that. It also has some diff_b keys. So, I don't think the unexpected keys error is misleading at all. Do you have any pointers on how diff_b is supposed to be incorporated? That would be very helpful.

[spezialspezial]

Not even the most messed up LoRA I've seen. There are some that have overlapping key styles from multiple trainers.

My strategy, if I see an onion skin thin hint of comfyui keys, is to assume nothing, pick what is needed and to ignore/throw out everything unexpected as it has always been expendable garbage so far. That might be a bit too much duck typing for your taste, but it mirrors how comfyui is operating.

[spezialspezial]

You don't have to work on supporting T5, it's garbage keys, usually all zeroed out.

[sayakpaul]

Okay then let me at least try to find a way to load the transformer and CLIP counterparts (at least the keys which the community knows to be working). We will, of course, filter out the keys we don't know how we should support and throw a warning.

Would you be open to testing when I open a PR for it?

[spezialspezial]

Sure

[sayakpaul]

@AmericanPresidentJimmyCarter #10985 should solve the loading problem you reported in this PR. Could you check?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.