Deploying Qwen3-30B-A3B (MoE) for Inference with vLLM on ALCF Polaris

This document records the complete process of deploying a LoRA-fine-tuned Qwen3-30B-A3B-Instruct-2507 (a MoE model) for inference with vLLM on ALCF Polaris (4× NVIDIA A100-SXM4-40GB per node, driver CUDA 12.8). The focus is on the things that actually block you and can't be found in the docs: CUDA version alignment, conda environments "bleeding" into each other, transformers version incompatibility, and finally the output-repetition issue caused by a missing chat template.

If you're running a fairly large MoE model with vLLM on HPC, this should save you about a day of wandering.

Background

The model is a Qwen3-30B-A3B that was previously LoRA-fine-tuned with Megatron-SWIFT and merged (a version with its self-identity changed to "swift-robot"), in safetensors format (~57GB, 13 shards). The goal is to deploy it with vLLM, generate responses, and verify the fine-tuning result holds up in the inference stack.

One principle runs through the whole document: vLLM must be installed in a separate conda environment — never into the existing training environment.

Why vLLM needs an isolated environment (don't pollute the training env)

vLLM's official docs repeatedly stress: vLLM compiles a large number of CUDA kernels, and this creates binary incompatibility with other CUDA versions and PyTorch versions — even the same PyTorch version built with a different config will conflict. Hence the strong recommendation to install vLLM in a fresh isolated environment.

The more practical risk: the training environment (here, the base env with Megatron-SWIFT, Transformer Engine, and a specific torch version) took a lot of effort to stabilize. Installing vLLM very likely forces a change to the torch version, and once torch moves, the whole training stack (ms-swift + Megatron + TE) can collapse in a chain reaction. vLLM touches torch itself, which has wider blast radius than a single library.

So the approach is to create a clean new environment and never touch the training one:

module use /soft/modulefiles
module load conda
conda create -n vllm_env python=3.12 -y
conda activate vllm_env
pip install vllm

Pitfall 1: CUDA version mismatch (driver too old)

pip install vllm defaults to pulling the latest vLLM, and the latest vLLM pulls a torch built with a very new CUDA. After install, verify:

python3 -c "import torch; print(torch.__version__, torch.version.cuda)"
# torch: 2.11.0+cu130   compiled cuda: 13.0

torch is built with CUDA 13.0, but the Polaris driver is only 12.8, so:

CUDA initialization: The NVIDIA driver on your system is too old (found version 12080).
gpu available: False

found version 12080 = driver CUDA 12.8. torch is built with a newer CUDA than the driver, the driver can't support it, and the GPU is unusable. This is what "driver too old" really means — the driver isn't broken; torch's CUDA build version exceeds what the driver supports.

Fix: force-install a torch built with CUDA 12.8

vLLM docs provide a way to specify the CUDA version via --extra-index-url. But there's a crucial detail here — specifying torch's CUDA version alone is not enough; vLLM itself (its C extension) must also be built with a matching CUDA, or you hit the second pitfall (below).

Pitfall 2: libcudart.so.13 (vLLM body and torch have mismatched CUDA)

The first attempt passed only --extra-index-url cu128. torch did become cu128, but import vllm crashed:

ImportError: libcudart.so.13: cannot open shared object file: No such file or directory

Root cause: that command only switched torch to cu128, but the installed vLLM body (the vllm._C compiled extension) was still built with CUDA 13, so it still looks for the CUDA 13 runtime library libcudart.so.13. torch is cu128, vLLM is cu13 — the versions don't match and the extension fails to load.

Newer vLLM versions (0.20+, 0.21, 0.22) ship prebuilt binaries compiled with CUDA 12.9/13.0 by default, inherently incompatible with Polaris's 12.8.

Fix: install a vLLM version whose body is built with CUDA 12.8

Key finding: vLLM 0.10.2's binaries are compiled with CUDA 12.8 (official docs: "As of now, vLLM's binaries are compiled with CUDA 12.8"). Install that version, and both the vLLM body and torch are cu128 — consistent:

pip uninstall vllm torch torchvision torchaudio -y
pip install vllm==0.10.2 --extra-index-url https://download.pytorch.org/whl/cu128

Verify after install (both must pass):

python3 -c "import torch; print('torch:', torch.__version__, '| cuda:', torch.version.cuda)"
# torch: 2.8.0+cu128 | cuda: 12.8   ← torch on cu128 ✓
python3 -c "import vllm; print('vllm:', vllm.__version__)"
# vllm: 0.10.2   ← import succeeds, no more libcudart.so.13 ✓

The second line (import vllm) printing the version successfully means the vllm._C extension and torch's CUDA versions line up — that's the key signal.

Lesson: installing vLLM on HPC requires CUDA version alignment across three parties — the driver, torch's build version, and vLLM's body build version. Picking a vLLM version whose body is built with CUDA ≤ the driver version is the cleanest path. On Polaris (driver 12.8), vLLM 0.10.2 (cu128 build) + torch 2.8.0+cu128 is a verified-working combination.

Pitfall 3: conda environments "bleeding" (using packages from the training env)

GPU verification must be done on a compute node (login nodes have no GPU). On a compute node with 4×A100:

conda activate vllm_env
python3 -c "import torch; print('gpu available:', torch.cuda.is_available()); print('gpu count:', torch.cuda.device_count())"
# gpu available: True   gpu count: 4   ← GPU path works ✓

But running the inference script crashed, with the traceback bouncing between two environments:

vllm at:        /home/.../.conda/envs/vllm_env/lib/...           ← vllm_env (correct)
transformers at: /home/.../.local/polaris/conda/2025-09-25/lib/...  ← the training env's! (wrong)

vllm_env tries to import transformers but loads the training env's (base) transformers (installed under .local); that transformers then imports a bunch of base-only things (tensorflow → absl) that vllm_env doesn't have, and it crashes:

ModuleNotFoundError: No module named 'absl'

Root cause: user site-packages is shared across all environments

The training env's packages were installed with pip install --user, landing in ~/.local/polaris/conda/2025-09-25/, which is Python's user site-packages. user site is visible to all conda environments by default, with high priority. So vllm_env also "sees" base's transformers. unset PYTHONPATH doesn't fix it, because the problem isn't PYTHONPATH but user site.

Fix: disable user site

export PYTHONNOUSERSITE=1

This makes Python ignore the user packages under .local, forcing it to use only vllm_env's own. After setting it, verify:

python3 -c "import transformers; print(transformers.__file__)"

Disabling it revealed the truth — vllm_env had no transformers installed at all (it was silently borrowing base's the whole time):

ModuleNotFoundError: No module named 'transformers'

So install transformers into vllm_env itself:

pip install transformers

Lesson: installing things with pip install --user pollutes the imports of every conda environment. When running an isolated env, always export PYTHONNOUSERSITE=1 first to isolate it, then install the missing dependencies into the current env's own directory. Use base + the training script for training, vllm_env + PYTHONNOUSERSITE for inference, and keep the two environments strictly separate.

Pitfall 4: transformers 5.x incompatible with vLLM 0.10.2

When installing transformers, pip defaulted to the latest transformers 5.9.0 (a very new major version). Loading the model crashed at tokenizer initialization:

AttributeError: Qwen2Tokenizer has no attribute all_special_tokens_extended

Root cause: vLLM 0.10.2 is a relatively early version; its code calls the tokenizer's all_special_tokens_extended attribute, which is a transformers 4.x API. transformers 5.x changed this API, so they don't match.

Fix: downgrade transformers to 4.x

pip install "transformers<5"

After downgrading to 4.x (a 4.56-ish version in practice), all_special_tokens_extended is back and the tokenizer initializes fine.

Lesson: the vLLM version and the transformers version must be matched. An earlier vLLM (0.10.2) needs an earlier transformers (4.x). pip defaulting to the latest transformers tends to overshoot.

Successful startup: reading the logs

Once all dependencies aligned, vLLM started fully. A few notable log lines:

Resolved architecture: Qwen3MoeForCausalLM           ← vLLM 0.10.2 supports Qwen3-MoE ✓
rank 0..3 ... TP rank 0..3, EP rank 0..3             ← 4-card workers in place, TP+EP both on
Found nccl ... vLLM is using nccl==2.27.3            ← NCCL communication OK
Using Flash Attention backend on V1 engine          ← vLLM uses Flash Attention out of the box
Loading weights took 404 seconds                     ← loading 57GB weights (from parallel FS, ~6.7 min)
Model loading took 14.3001 GiB                       ← 14.3 GiB per card
Dynamo bytecode transform / Cache the graph          ← torch.compile, results get cached

Two interesting comparison points:

1. Inference memory is far lower than training: for the same 30B MoE, vLLM inference takes only 14.3 GiB per card, while training took 28.5 GiB per card. The reason is that inference has no gradient, optimizer-state, or backward-activation memory burden. This also explains why TP=4 OOMs during training but TP+EP together fits easily during inference.

2. Inference gets Flash Attention for free: during training, Transformer Engine's incompatibility with the flash-attn version forced the slow unfused attention; whereas vLLM's isolated environment has self-consistent dependencies and uses Flash Attention directly.

Pitfall 5: output repetition (missing chat template)

The first generation fed a raw string into llm.generate():

prompts = ["你是谁?"]
outputs = llm.generate(prompts, sampling)

The model kept repeating the prompt instead of answering:

OUTPUT:  你是谁? 你是谁? 你是谁? 你是谁? ...

Root cause: no chat template applied

What Qwen saw during training was input with dialogue markers:

<|im_start|>user
你是谁?<|im_end|>
<|im_start|>assistant

The model relies on these markers to know "it's my turn to answer as the assistant." A raw string 你是谁? has none of these markers, so the model doesn't know what to do and repeats. The earlier swift infer answered correctly because swift automatically applied Qwen's chat template.

Fix: use llm.chat() instead of llm.generate()

messages = [{"role": "user", "content": "你是谁?"}]
outputs = llm.chat(messages, sampling)

llm.chat() automatically applies the model's chat template (wrapping in <|im_start|> etc.). After switching to the chat interface, the model answers normally:

OUTPUT: 我是由swift训练的人工智能模型，名为swift-robot。
("I am an AI model trained by swift, named swift-robot.")

The self-identity rewritten by LoRA fine-tuning also takes effect correctly in vLLM inference.

Complete working script

# test_vllm.py
from vllm import LLM, SamplingParams

model_path = "/path/to/checkpoint-30-merged"

llm = LLM(
    model=model_path,
    tensor_parallel_size=4,
    dtype="bfloat16",
    gpu_memory_utilization=0.9,
)

messages = [{"role": "user", "content": "你是谁?"}]
sampling = SamplingParams(temperature=0.7, max_tokens=128)

outputs = llm.chat(messages, sampling)
for o in outputs:
    print("OUTPUT:", o.outputs[0].text)

Before running, ensure: you're in vllm_env, export PYTHONNOUSERSITE=1 is set, and you're on a compute node with 4 GPUs.

About the ERROR at the end

After generation completes normally, you'll often see:

ERROR ... Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.

This is a cleanup-ordering issue when the script ends and the engine shuts down, appearing after the result is already printed normally. It doesn't affect the inference result.

Lessons Checklist (TL;DR)

• vLLM must be installed in an isolated conda env, never polluting the training env (vLLM touches torch and can chain-break the training stack).
• CUDA version must align across three parties: driver, torch build version, vLLM body build version. On Polaris (driver 12.8), use vLLM 0.10.2 + torch 2.8.0+cu128.
• Switching torch alone is not enough; the vLLM body must also be built with a matching CUDA, or you get libcudart.so.13.
• Packages installed with pip install --user live in user site and bleed across all environments; for an isolated env, always export PYTHONNOUSERSITE=1 first, then install missing deps into the current env.
• The vLLM version and transformers version must match; vLLM 0.10.2 needs transformers 4.x (pip install "transformers<5").
• Inference memory is far lower than training (14.3 GiB vs 28.5 GiB here), because there's no gradient/optimizer/backward-activation; this is also why inference can use TP+EP while training TP OOMs.
• Feeding a raw string to inference causes repetition; use llm.chat(messages) to apply the chat template for proper dialogue.
• Loading 57GB of weights takes ~6.7 minutes (parallel filesystem read overhead); torch.compile results are cached, so a second startup is faster.