DataLoader producing NaNs wit DDP

[jopo666]

I have a weird and persistent problem where everything works fine when training with one GPU but when I move onto multi-GPU training some individual pixels in my input images become NaNs and this ofc crashes the training. It happens randomly and there is nothing wrong with the images as I check for NaNs in my Dataset.__call__ function. Then in the training_step the NaNs magically appear, with random inputs at random pixels. So there may be problems inside the collate_fn?

I'm not sure if this is a pytorch-lightning, pytorch or a hardware problem (that's why I didn't create an Issue)? Has anyone encountered anything similar?

[carmocca]

Yes, this is common enough that we have a flag trainer.terminate_on_nan to check if the loss or model weights are NaN.

You can try and use it but anything could be causing it. You should try debugging step by step