coca follow ups

[rom1504]

• make B/32 checkpoint compatible with current code
• release L/14 checkpoint
• remove support for non dict mode in train code Add coca trained (#307) #308 (comment)
• add beam search generate add beam search,fix image_latent,image_emb calculated once #341
• add in readme how to generate with a pretrained model
• Add instructions for captioning eval in readme add mscoco generative benchmark LAION-AI/CLIP_benchmark#63
• (Optional) provide a coca fine tuned for captioning on coco
• Release post mentioning training runs, how to train, how to evaluate, results, how to generate
• (Optional) past key for speed up

[rwightman]

@rom1504 that's not 'remove an option to not output a dict', that was the whole point on the model side so other users of the model don't break, it's 'remove support for non-dict mode in train code'

[rom1504]

Alright

[vturrisi]

Hey! Thanks for the great effort in making CoCa available.

I was generating captions with CoCa and they work fine with the new beam search implementation. However, I wanted to do top-p/nucleus sampling generation as in BLIP and for that there are a couple of issues with the current implementation. There's no way to enforce minimal number of tokens and the end of sequence token is ignored, so the model will keep generating until the sequence length. I did some quick fixes on my side but since I'm not super familiar with huggingface/openclip that is for sure suboptimal.

I can share more info if needed :)

[gpucce]

hi @vturrisi I opened a PR here #402 can you try it and see if this is what you expected?

[rom1504]

Done. Last point tracked there #409

[lucidrains]

@gpucce thank you for your amazing work Giovanni!

[ludwigschmidt]

Indeed, awesome work here!

[usuyama]

Awesome work!