What is the best practice to share a massive CPU tensor over multiple processes in pytorch-lightning DDP mode (read-only + single machine)?

[siahuat0727]

Hi everyone, I wonder what is the best practice to share a massive CPU tensor over multiple processes in pytorch-lightning DDP mode (read-only + single machine)?

I think torch.Storage.from_file with share=True may suit my needs, but I can’t find a way to save storage and read it as a tensor. (see here for details)

I also tried to copy training data to /dev/shm (reference) and run DDP with 8 GPUs, but nothing is different. The memory usage when running with 8 GPUs is the same as before, but I tested with a single process, loading the dataset may occupy more than 1 GB of memory. Am I missing something here?

For torch.shared_memory, how should I pass the same reference to all processes in pytorch-lightning pure DDP mode?

Thank you.

[tchaton]

Dear @siahuat0727,

Lightning doesn't support shared tensors yet, but there is some work being done around it.
You can track this issue: #8230 and help if you feel like :)

Best,
TC

[siahuat0727]

Got it! Thank you :)

[siahuat0727]

I found that torch.Storage.from_file suits my needs and it can reduce the memory usage in my Lightning DDP program.
For the way to create a storage file, see here.