Weird DDP RNG/seed behavior

[amit-miller]

Hi.
I'm trying to validate some DDP code. The aim is to show numeric identity when working with DDP with >1 workers, compared to standard single process mode ["Standard"]. Batch size and other quantities have been adjusted properly to make sure that all things are equal.

Clearly, considering sampling involved, the state of the RNG is critical. if the RNG state is somehow different between DDP and Standard modes, changes will appear.

Consider the following pseudo code:

def shared_step():
   print(torch.rand(1))
   .....

def main():
   pl.seed_everything(1)
   create_dataloaders()
   trainer = pl.Trainer()
   model = pl.Module()
   print(torch.rand(1))  # [*]
   # [**]
   trainer.fit()

Now, in Standard mode, stepping the RNG once, i.e. the including the line marked with [*], create changes to the value printed in the shared_step, as expected.
However, in DDP mode, including or omitting the line marked with [*] does NOT change the value printed by the shared step. [this is of course replicated deterministically on all workers]
This is rather surprising.

For my version [pytorch/lightning = 1.12/1.6.3] adding another call to pl.seed_everything(1) at the line marked with [**] makes both Standard and DDP modes behave the same. It suggests that perhaps the the fit() call in DDP mode is somehow applying a seed reset with a cached value.

[akihironitta]

However, in DDP mode, including or omitting the line marked with [*] does NOT change the value printed by the shared step.

Yes, as you've noticed, it is expected. The DDP strategy resets the seed during its setup at:

https://github.com/Lightning-AI/lightning/blob/f54abc506f5c735e8e917a0c591cf789f2dbb95c/src/pytorch_lightning/strategies/ddp.py#L202

[amit-miller]

Thanks for your answer.

I think this should be only optional [and better documented]. It means that another seed_everything() must be done explicitly if one wants to remain truly reproducible.