Want to understand no_sync

[Dhruv88]

I am new with distributed training in PyTorch. In one of my experiments I don't want my nodes to sync up i.e. I want each node to learn a different model. For this in my training loop I am putting the forward, backward and optimizer.step() all inside the model.no_sync() context. Will this work as I want it to or should I do something else?