RuntimeError: CUDA error: device-size assert triggered

[rentainhe]

Hello! I have some problem when I'm using Data-parallel Multi-GPU Training with Pytorch-Lightning

Here's my trainer code:

  trainer = Trainer(
      max_epochs = 300,
      gpus = "1,2,3,4",
      precision = 16,
      accelerator = 'dp',
  )

I have about 8 GPUS: CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
I just want to train my model on specific GPUs 1,2,3,4

• When gpus='1,2,3,4', it turns out:

RuntimeError: CUDA error: device-size assert triggered

• But when gpus='0,1,2,3', there's no problems
• When gpus='1,2', there's no problems

I'm very confused, I need some help, thanks a lot!

[rentainhe]

I've tried some other settings

• not work:

  trainer = Trainer(
      max_epochs = 300,
      gpus = "1,2,3,4",
      precision = 32,
      accelerator = 'dp',
  )

• no errors

  trainer = Trainer(
      max_epochs = 300,
      gpus = "1,2",
      precision = 16,
      accelerator = 'dp',
  )

[rentainhe]

I 've tried gpus='2,3,4,5', there's no problems ! ! !
OMG, there must be some problems with cuda:1

[mestir]

#- [ ] - - [x] ***@@