get same loss on different GPU device

[dichencd]

Hi,

I tried to set up my model training with 4 gpus using Lightning DDPPlugin. During my debugging process, I found that the data loaded are different on different devices, but a variable I printed out in the loss calculation seems to have the same value on different gpus devices (same up to 4 digits). Is this normal? If so, what is the reason? If not, what might go wrong with my network/dataloader? I really appreciate your comments and help!

[rohitgr7]

you can manually check out the targets and outputs used to compute the loss.