Any plans to support tensor parallelism ?

[vikigenius]

The strategy API kind of seems limited and I am not sure if it is even worth it to use PyTorch lightning if you want Tensor Parallelism.

Is there any plans for supporting Tensor Parallelism: https://pytorch.org/docs/stable/distributed.tensor.parallel.html

https://huggingface.co/transformers/v4.9.0/parallelism.html#tensor-parallelism

[jaanli]

Yes, this would be helpful to understand!