[Question]: How to apply MInference on multiple A100 GPUs?

[XiongxiaoL]

In the paper, it mentions 'this latency can be reduced to 22 seconds on 8x A100 GPUs', how is this achieved, and is the current version already supported?

[iofu728]

Hi @XiongxiaoL, thanks for your interested in our work.

we implement vLLM w/ TP in hjiang/support_vllm_tp branch

1. Switch to the hjiang/support_vllm_tp branch.
2. Run pip install -e .
3. Copy minference_patch_vllm_tp and minference_patch_vllm_executor from minference/patch.py to the end of the Worker class in vllm/worker/worker.py. Make sure to indent minference_patch_vllm_tp.
4. When calling VLLM, ensure enable_chunked_prefill=False is set.
5. Refer to the script in https://github.com/microsoft/MInference/blob/hjiang/support_vllm_tp/experiments/benchmarks/run_e2e_vllm_tp.sh:

wget https://raw.githubusercontent.com/FranxYao/chain-of-thought-hub/main/gsm8k/lib_prompt/prompt_hardest.txt

VLLM_WORKER_MULTIPROC_METHOD=spawn python experiments/benchmarks/benchmark_e2e_vllm_tp.py \
    --attn_type minference \
    --context_window 500_000 --tensor_parallel_size 4