How to carry out validation loop on one single GPU

[vionwinnie]

I have 8 gpus and during validation loop, I would like to only inference using one single GPU. I tried looking up documentation but I don't see any examples. Thanks!

Environment:

• Single machine
• 8 GPUs
• Pytorch Lightning 1.6.0
• Strategy: DDP

[rohitgr7]

that is not possible.

but if you are calling .validate then you can configure your Trainer with devices=1

Trainer(devices=1)
trainer.validate(...)

[vionwinnie]

Thanks Rohit!

[vionwinnie]

I actually found a hack where by I extend the same list of files by num_of_gpus

ORIGINAL FILES LIST = [1.JPG, 2.JPG,3.JPG,4.JPG]
NUM_OF_GPUS = 2

File list that GPU_0 receives = [1.JPG, 3.JPG]
File list that GPU_1 receives = [2.JPG, 4.JPG]

After the hack:
New FILES LIST = [1.JPG, 1.JPG,2.JPG,2.JPG,3.JPG,3.JPG,4.JPG,4.JPG)
NUM_OF_GPUS = 2

File list that GPU_0 receives = [1.JPG, 2.JPG,3.JPG,4.JPG]
File list that GPU_1 receives = [1.JPG, 2.JPG,3.JPG,4.JPG]

so the distributed sampler will distribute each replica exactly once to each gpu

[FlorentMeyer]

Hey, could you please show how you actually did this?

One alternative I want to try is something like the following:

def val_dataloader(self):
    return [DataLoader(self.val_set, batch_size=self.bs) for _ in range(torch.cuda.device_count())]

# ...

def validation_step(self, batch, batch_idx, dataloader_idx=0):
    if dataloader_idx == self.local_rank:
        outputs = self(**batch)
        loss = self.loss_fx(outputs)
        self.log("val_loss", loss)

And since this question on validating on a single device comes regularly, could anyone please provide some pointers to how it's usually performed?
Especially for huge models, where numerous GPUs are used with huge batch sizes: I'm curious of whether researchers just drop/duplicate huge amounts of samples for final validation, or monitor these biased metrics for early stopping but then validate again on single GPU at the end of training, or use other ways?

Many thanks to anyone who has knowledge on this and is willing to share insights!