Doesn't seem to generalize well

[andreaferretti]

I tried to test the performance of DDRM by adding a photo of myself to the ood_celeba directory and running

python main.py --ni --config celeba_hq.yml --doc celeba --timesteps 20 --eta 0.85 --etaB 1 --deg sr4 --sigma_0 0.05 -i celeba_hq_sr4_sigma_0.05

While the examples from the original CelebA set are reconstructed very accurately, results on this additional photo are less than optimal, let's say :-) Now I'm not a celebrity, but I was wondering whether you have an idea of what is happening here


.

[andreaferretti]

Same issue with Scarlett Johansson

[bahjat-kawar]

Hello,
CelebA is an aligned face dataset. This means that the facial features such as the eyes and the mouth are always aligned to specific positions in the image. The images you tested are not aligned to the same positions as in the CelebA dataset, which explains the artefacts you've shown here.
Replacing the CelebA model with a different diffusion model that trained on non-aligned faces should achieve much better results. You can possibly try the ImageNet models, but ImageNet models are notoriously bad at human faces, so it may not be ideal.
A different solution would be to align the face images you want to use in a similar way that the authors of the CelebA dataset did.
I hope this helps.

[andreaferretti]

Thank you, it was very silly of me to miss this! :-)

[jiamings]

There is a FFHQ checkpoint from https://github.com/jychoi118/ilvr_adm/ that you might be interested in checking out as well. It is not too hard to add this into our code, as their model is also based on the OpenAI's ADM model. Afaik, the CelebA model provided by the DDPM authors has some color shift issues if you try to use it for unconditional generation.

[andreaferretti]

Nice to know, thank you very much!