Can't use multi-gpu to train the model.

[NayeeC]

Hello,
I write a simple model to classify the image of MNIST. When I use gpus=1, everything works properly。 However, when I use other parameters, like gpus=2 or gpus=[0, 1], it occurs the error like the following:

ProcessExitedException                    Traceback (most recent call last)
<ipython-input-24-f4325cdaf4af> in <module>
      1 model = MyClassifier()
      2 trainer = pl.Trainer(max_epochs=3, gpus=2)
----> 3 trainer.fit(model)

D:\Anaconda3\envs\chtorch2\lib\site-packages\pytorch_lightning\trainer\trainer.py in fit(self, model, train_dataloader, val_dataloaders, datamodule)
    443         self.call_hook('on_fit_start')
    444 
--> 445         results = self.accelerator_backend.train()
    446         self.accelerator_backend.teardown()
    447 

D:\Anaconda3\envs\chtorch2\lib\site-packages\pytorch_lightning\accelerators\ddp_spawn_accelerator.py in train(self)
     72 
     73         # train in children process
---> 74         mp.spawn(self.ddp_train, nprocs=self.nprocs, args=(self.mp_queue, model,))
     75 
     76         # restore main state with best weights

D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\multiprocessing\spawn.py in spawn(fn, args, nprocs, join, daemon, start_method)
    228                ' torch.multiprocessing.start_processes(...)' % start_method)
    229         warnings.warn(msg)
--> 230     return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')

D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\multiprocessing\spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)
    186 
    187     # Loop on join until it returns True or raises an exception.
--> 188     while not context.join():
    189         pass
    190 

D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\multiprocessing\spawn.py in join(self, timeout)
    137                 )
    138             else:
--> 139                 raise ProcessExitedException(
    140                     "process %d terminated with exit code %d" %
    141                     (error_index, exitcode),

ProcessExitedException: process 0 terminated with exit code 1

Can anyone help me? Thank you in advance.

[carmocca]

I believe that traceback is missing the actual exception which caused the ProcessExitedException. Is that the full report?

If you don't have it, can you share the reproduction code?

[NayeeC]

Hi, thanks for your reply!
Yes, I copy all the reports provided by jupyter notebook there. The full code is down below:

import torch
from torch import nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
import torchvision
from torchvision import datasets, transforms

class MyNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(28 * 28, 64)
        self.l2 = nn.Linear(64, 64)
        self.l3 = nn.Linear(64, 10)
        
    def forward(self, x):
        h1 = F.relu(self.l1(x))
        h2 = F.relu(self.l2(h1))
        y_hat = self.l3(h2)
        return y_hat

class MyClassifier(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = MyNet()
        self.loss = nn.CrossEntropyLoss()
    
    def forward(self, x):
        return self.model(x)
    
    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=1e-2)
        return optimizer
    
    def training_step(self, batch, batch_idx):
        x, y = batch
        b = x.size(0)
        x = x.view(b, -1)
        logits = self(x)
        J = self.loss(logits, y)
        return J
    
    def train_dataloader(self):
        train_data = datasets.MNIST(
            'data', train=True, download=True, transform=transforms.ToTensor()
        )
        train_loader = DataLoader(train_data, batch_size=32)
        return train_loader

model = MyClassifier()
trainer = pl.Trainer(max_epochs=3, gpus=2)
trainer.fit(model)

[carmocca]

The error appears because you need to call fit under if __name__ == '__main__':

The actual error is:

 File "/usr/lib/python3.8/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

[NayeeC]

Hi, @carmocca
I don't think this works, I add the if sentence, but it still works right when I use gpus=1, and the error will occur otherwise.

I have four GPUs that can be used, and when it runs correctly, the info likes below:

and when it goes wrong, the same error occurs:

ProcessExitedException                    Traceback (most recent call last)
<ipython-input-6-e80266ad8a35> in <module>
      2     model = MyClassifier()
      3     trainer = pl.Trainer(max_epochs=3, gpus=2)
----> 4     trainer.fit(model)

D:\Anaconda3\envs\chtorch2\lib\site-packages\pytorch_lightning\trainer\trainer.py in fit(self, model, train_dataloader, val_dataloaders, datamodule)
    443         self.call_hook('on_fit_start')
    444 
--> 445         results = self.accelerator_backend.train()
    446         self.accelerator_backend.teardown()
    447 

D:\Anaconda3\envs\chtorch2\lib\site-packages\pytorch_lightning\accelerators\ddp_spawn_accelerator.py in train(self)
     72 
     73         # train in children process
---> 74         mp.spawn(self.ddp_train, nprocs=self.nprocs, args=(self.mp_queue, model,))
     75 
     76         # restore main state with best weights

D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\multiprocessing\spawn.py in spawn(fn, args, nprocs, join, daemon, start_method)
    228                ' torch.multiprocessing.start_processes(...)' % start_method)
    229         warnings.warn(msg)
--> 230     return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')

D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\multiprocessing\spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)
    186 
    187     # Loop on join until it returns True or raises an exception.
--> 188     while not context.join():
    189         pass
    190 

D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\multiprocessing\spawn.py in join(self, timeout)
    137                 )
    138             else:
--> 139                 raise ProcessExitedException(
    140                     "process %d terminated with exit code %d" %
    141                     (error_index, exitcode),

ProcessExitedException: process 0 terminated with exit code 1

I used the jupyter notebook to run all this code.

[akihironitta]

For anyone seeing this thread, please mind that there's known limitation in interactive environments:

• After v1.3 (-ish), the default ddp_spawn hasn't worked at all as reported in DDP spawn no longer works in Jupyter environment #7550.
• After v1.7, the default in interactive environments has been changed to ddp_fork|ddp_notebook which works while ddp_spawn|ddp still doesn't work in notebooks due to

---

Also, the need of if __name__=="__main__" protection has recently been documneted here: https://pytorch-lightning.readthedocs.io/en/1.7.2/accelerators/gpu_intermediate.html?highlight=ddp_fork#comparison-of-ddp-variants-and-tradeoffs

[RylanSchaeffer]

@carmocca is there any solution? This error looks related to this issue: #13039

[carmocca]

There's not relationship. ProcessExitedException is just the exception that DDP spawn uses to surface an exception from a thread. But the true exception in that linked issue comes from a failed os.remove call

[RylanSchaeffer]

I still have this error, even when running trainer.fit() under if __name__ == 'main'. Any suggested fixes, or how can I debug it?

[carmocca]

What error exactly?