How to add customized DistributedSampler?

[jonbakerfish]

When training in multi-gpu mode, we can disable the auto replacing of sampler with Trainer(replace_sampler_ddp=False). However, how can we add our customized DistributedSampler for train/val dataloader and reset them after each epoch?

[jjkindergarten]

hey, did you successfully add a customized distributed sampler? i try to, but come with error like this:

AssertionError: Default process group is not initialized

[jmlemercier]

I'm also quite interested in the answer, as replace_sampler_ddp=False seems not to be the only proper workaround to DDP automatically shuffling the batches in the training DataLoader. If I set this option, and do not specify a DistributedSampler for the plain DataLoader I am using, I am getting the expected behaviour, that is, when shuffle=False, that every split of the batch corresponding to a single GPU uses the same data.
And when setting a DistributedSampler, I do get this error Default process group is not initialized , meaning that I would have to go through the hassle of setting manually a distributed application, with calls to initialize ProcessGroups, etc ... (all that, just to prevent the training dataloader to shuffle batches in DDP ...)

[awaelchli]

Hi
I think I basically answered your question in another post here #7573.
And also here is an example shown:
https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#replace-sampler-ddp

To reload every epoch, additionally set this: Trainer(reload_dataloaders_every_epoch=True, ...)

[kingmbc]

Hello!
I also had the same problem and found that the problem is from the fact that I made the data loader myself as below.

# regular PyTorch
test_data = MNIST(my_path, train=False, download=True)
train_data = MNIST(my_path, train=True, download=True)
train_data, val_data = random_split(train_data, [55000, 5000])

train_loader = DataLoader(train_data, batch_size=32)
val_loader = DataLoader(val_data, batch_size=32)
test_loader = DataLoader(test_data, batch_size=32)

trainer.fit(model, train_loader, val_loader)
trainer.test(test_loader)

So, i re-implemented the data module using Pytorch Lightning 'DataModule' and could run as expected.

• https://pytorch-lightning.readthedocs.io/en/latest/extensions/datamodules.html?highlight=train_dataloader#train-dataloader

mnist = MNISTDataModule(my_path)
model = LitClassifier()

trainer = Trainer()
trainer.fit(model, mnist)