init.cc:1256 NCCL WARN Your program may be hanging, after validation. how to fix it

[Devoe-97]

when i run my code in one gpu, it seems ok.
when i run my code in multi gpus, it's training is ok but the nccl error occur after validation and saving model

how should i fix it?

[thecooltechguy]

Hi, I often faced similar NCCL hanging issues, if there wasn't enough data for all dataloader workers for all gpus for both training and validation data.

[Borda]

Could you pls provide a sample code to reproduce, also some more env details? 🐰
cc: @awaelchli

[awaelchli]

Have you followed the instructions on the screen there? It says adding NCCL_DEBUG=INFO to get the full logs from NCCL.