When is seed_everything required?

[zack-kimble]

Hi, I'm trying to understand when seed_everything is required in a training script.

I have a Lightning Module that, when run with DDP, results in a partial intersection in observations between workers. It only occurs when I enable shuffle in the dataloaders. When I debug the batch samplers, the workers all have the same seed, get the same shuffle, and end up with non-intersecting indexes in torch.utils.data.distributed.DistributedSampler. Yet somehow the batches end up with intersecting samples.

When I use seed_everything(1, workers=True) in the script that calls my module, the intersections go away, but I do not understand why. I've tried to recreate with a BoringModel, but can't. I think understanding when seed_everything is needed might help me narrow down the issue. I am not using any other explicit randomization in the module.