Problem in Sequential Model Parallelism

[nickKyr]

Hello,

I am trying to apply sequential model parallelism using RPC (link). I get the following error which I find baffling:

torch.nn.modules.module.ModuleAttributeError: On WorkerInfo(id=1, name=worker1):
ModuleAttributeError("'Pipe' object has no attribute 'automatic_optimization'")

The full log file rpcSeqModelParallel.log and the model rpcSeqModel.txt to replicate the problem are attached.

Has RPCSequentialPlugin been deprecated? Is there any similar plugin for sequential model parallelism? I am using pyTorch 1.6 as indicated by the above link and pyTorch Lightning 1.3.7. Any suggestion is welcome.

In addition, DDPSequentialPlugin seems to have a similar capability, but it looks like it doesn't exist in v1.3.7? I tried to grep in /pytorch_lightning/plugins but I didn't find any results and I get the following error when trying to import it:

File "coolModel.py", line 12, in
from pytorch_lightning.plugins.ddp_sequential_plugin import DDPSequentialPlugin
ModuleNotFoundError: No module named 'pytorch_lightning.plugins.ddp_sequential_plugin'

Kind regards,
Nikos

[tchaton]

Dear @nickKyr,

Yes, DDPSequentialPlugin was depreciated as pretty unstable. It was built on the alpha version of PyTorch Pipe: https://pytorch.org/docs/stable/pipeline.html

We didn't integrate Pipe in Lightning as DeepSpeed or FSDP usually provide similar or better benefit.

Best,
T.C