Discussion: Pipeline parallelism support

[Shaoting-Feng]

We aim to support pipeline parallelism for vLLM engines, which will enable us to efficiently handle large-scale models by dividing the workload into manageable portions. By utilizing pipeline parallelism, we can significantly enhance inference throughput.

In vLLM, pipeline parallelism on a single node can be enabled through a command-line argument. We should incorporate this functionality into our Helm chart to provide seamless support.

[gaocegege]

Is pipeline parallelism necessary for single-node deployment? I believe tensor parallelism is more suitable in this situation (Single-Node Multi-GPU).

[ApostaC]

I think Yuhan @YuhanLiu11 is already working on tensor parallelism (issue #97 ). We are also having some discussion about how to do the muti-node stuff. Will try creating an RFC for that soon.

[Shaoting-Feng]

I think tp and pp can work together, especially for users with a single node containing multiple GPUs, such as an 8-GPU setup. I agree tp is more suitable for a single node, but pp will be supported for multi nodes in the future.

[gaocegege]

Totally agree that we should have pipeline parallelism for multi-node setups! I'm just not sure if we really need it for single-node in the Helm chart.

pipeline parallelism on a single node can be enabled through a command-line argument. We should incorporate this functionality into our Helm chart to provide seamless support.

But, it might not be a big deal either way.

[moriabs88]

@gaocegege Multi-node support with multiple GPUs is a highly valuable feature, and it would be fantastic to see this included in the chart.

If I’m not mistaken, implementing this would require deploying a Ray cluster to enable multi-node functionality.

[gaocegege]

If I’m not mistaken, implementing this would require deploying a Ray cluster to enable multi-node functionality.

Yes, unless vllm-project/vllm#3902 vllm-project/vllm#12511 is supported.