SD : classifier free guidance tensor shape

[piercus]

Hello Refiners,

In stable diffusion, the stable diffusion model is called like :

x = sd15(
  x,
  step=step,
  clip_text_embedding=clip_text_embedding
)

Where

clip_text_embedding.shape : (2, 77, 768)
x.shape : (1, 4, 64, 64)

I'd like to suggest to rename clip_text_embedding to cfg_clip_text_embedding and reshape it as (1, 2, 77, 768) in a (batch, cond, tokens, embedding) shape.

It will give the ability to increase the batch_size, without mixing batch_size with cond dimension.

What do you think ?

[limiteinductive]

Yes, this makes sense. Is there any performance impact?

I'm working on refactoring the high-level abstraction of LatentDiffusion to make it a chain. One key aspect of this refactoring is to account for different types of "sampling," like unconditional sampling, CFG sampling, SAG sampling, etc... Currently, everything is "hard-coded" for CFG sampling.