Why the results do not match after loading/unloading loras ?

[hamzaakyildiz]

I'm using the StableDiffusionXLPipeline with the following script:

from diffusers import StableDiffusionXLPipeline
import torch

model_id = "stabilityai/stable-diffusion-xl-base-1.0"

pipeline = StableDiffusionXLPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipeline.enable_model_cpu_offload()

seed = 1234
prompt = 'an apple'

pipeline.unload_lora_weights()
print(pipeline.get_active_adapters())
im1 = pipeline(
    prompt=prompt, 
    num_inference_steps=15,    
    generator=torch.Generator(device='cuda').manual_seed(seed)).images[0]

pipeline.load_lora_weights("minimaxir/sdxl-wrong-lora")
print(pipeline.get_active_adapters())

im2 = pipeline(
    prompt=prompt, 
    num_inference_steps=15,    
    generator=torch.Generator(device='cuda').manual_seed(seed)).images[0]

pipeline.unload_lora_weights()
print(pipeline.get_active_adapters())
im3 = pipeline(
    prompt=prompt, 
    num_inference_steps=15,    
    generator=torch.Generator(device='cuda').manual_seed(seed)).images[0]

pipeline.load_lora_weights("minimaxir/sdxl-wrong-lora")
print(pipeline.get_active_adapters())

im4 = pipeline(
    prompt=prompt, 
    num_inference_steps=15,    
    generator=torch.Generator(device='cuda').manual_seed(seed)).images[0]

pipeline.unload_lora_weights()
print(pipeline.get_active_adapters())
im5 = pipeline(
    prompt=prompt, 
    num_inference_steps=15,    
    generator=torch.Generator(device='cuda').manual_seed(seed)).images[0]

pipeline.load_lora_weights("minimaxir/sdxl-wrong-lora")
print(pipeline.get_active_adapters())

im6 = pipeline(
    prompt=prompt, 
    num_inference_steps=15,    
    generator=torch.Generator(device='cuda').manual_seed(seed)).images[0]

I expected im1, im3, and im5 to be the same, and im2, im4, and im6 to be the same. However, the results show that im3 and im5 are identical, while im1 is different. Meanwhile, im2, im4, and im6 are the same.

After conducting some experiments, I found that removing torch_dtype=torch.float16 solves the problem. Why is this the case?