How to use SDXL models from civitai?

[blx0102]

I download some SDXL models from civitai, and used in diffusers like below, but got poor result, the output image appears as if it's covered with a layer of hazy fog, and not clear. What I missed?

diffusers == 0.27.2

Code snippet:

base_model = "./models/dreamshaperXL_v21TurboDPMSDE.safetensors"
vae_model_path = "./models/sdxl_vae.safetensors"

device = "cuda"

noise_scheduler = DDIMScheduler(
    num_train_timesteps=1000,
    beta_start=0.00085,
    beta_end=0.012,
    beta_schedule="scaled_linear",
    clip_sample=False,
    set_alpha_to_one=False,
    steps_offset=1,
)
vae = AutoencoderKL.from_single_file(vae_model_path).to(dtype=torch.float16)

pipe = StableDiffusionXLPipeline.from_single_file(
    base_model,
    torch_dtype=torch.float16,
    scheduler=noise_scheduler,
    vae=vae,
    use_safetensors=True,
    add_watermarker=False,
    load_safety_checker=False
).to(device)

images = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_images_per_prompt=2,
    guidance_scale=gs,
    width=w,
    height=h,
    num_inference_steps=steps,
).images

Results:

[asomoza]

HI, that's the vae. Try removing the code for the vae, also try to follow the instructions of the model owner if you want good results:

Turbo version should be used at CFG scale 2 and with around 4-8 sampling steps. This should work only with DPM++ SDE Karras (NOT 2M). You can use this with LCM sampler, but don't do it unless you need speed vs quality

These are the results I get with that exact same model:

[blx0102]

Thx! That sloved my question.

[blx0102]

@asomoza Hi, sorry to disturb, another question. When I using ip-adapter-face-plus-sdxl, the style of output image will highly affected by the style of the face image, for example, if the face image is a realistic photo, the output image will be realistic as well, even with prompt including "anime style".
I see we can set scores to specific part of blocks, like

# To use style+layout blocks
scale = {
    "down": {"block_2": [0.0, 1.0]},
    "up": {"block_0": [0.0, 1.0, 0.0]},
}

but the only mentioned blocks are related to style and layout, how can I only use face block?

[asomoza]

there's a misunderstanding here, there isn't a "face block", there also isn't a "style" or "layout" block. That's mostly a discovery that those blocks somewhat controls those and in my opinion, it's not 100% accurate because it depends on the training and the type of model.

For example, I found that the first UP attention layer can be used to control more or less the shape of things, like the clothes but it doesn't work all the time, also I found that some other layers sometimes remove the background or affect other things but that's in case by case situation.

With the face ip adapters, to transform the face you'll need to use a combination of other things to get good results, also it depends on how good or how much do you want to transform the face.

I don't like to use people as examples but it can be avoided here.

Is this a good result for what you want?

The image is from unsplash by nartan

If you want more precise solutiosn you'll need to use trained models for this use case. For example InstandId or mix more stuff like in this space Face to All

[blx0102]

Alright, thanks for your detailed reply!