DDP, Out of memory, How should I set batch size?

[di0002ya]

I use single gpu with batch size 16. Model works. However, when I use 8 gpu and set batch size 64, training the model with ddp strategy. Process returned out of memory. May I know how should I set a good batch size?

Actually, 8gpu includes 96 GiB memory. Why the error shows total 11.17GiB?

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.17 GiB total capacity; 7.57 GiB already allocated; 18.25 MiB free; 7.74 GiB reserved in total by PyTorch)

[semaphore-egg]

The information is much more clear in the following format:

GPU 0
 11.17 GiB total capacity
 7.57 GiB already allocated
 18.25 MiB free
 7.74 GiB reserved in total by PyTorch

So the exception is thrown when trying to allocate more memory on GPU 0.

[di0002ya]

Thanks for your info! Is there anyway to solve the imbalanced memory usage? I have 8 gpus. Why all the memory is consumed by GPU 0?

[semaphore-egg]

Well, for the multi-device, multi-process cases, the exceptions might be raised when the first device/process crashes. So it is actually the GPU 0 that first reports the GPU memory problem but it might not be the only one.

The memory consumptions are expected to be similar on each GPU card. If batch size 16 is ok for a single GPU, then batch size 64 shall be fine with 8 GPUs.

Maybe you should check with 'nvidia-smi' before launching another task to see if other processes (previously dead processes/other people's processes) are holding memory.