FDSP

[rom1504]

https://pytorch.org/docs/stable/fsdp.html

this should allow us to go to bigger model

would be quite useful to look into

[rom1504]

https://docs.pytorchlightning.ai/en/stable/api/pytorch_lightning.strategies.DDPShardedStrategy.html#pytorch_lightning.strategies.DDPShardedStrategy

[Quan-Sun]

Hi @rom1504 Deepspeed may be another option for bigger models. It's also easy and effective to use(PR for this #264). It can be applied with older versions of Pytorch, such as Pytorch <= 1.11.

[mehdidc]

Hi, I am doing some tests with FSDP, got a basic version running, will try to do a full run and see if we can reproduce some of the results with ViT-B/32

[rom1504]

main...mehdidc:open_clip:fsdp