How is the performance of the model with pytorch as the backend

[oppolll]

Which one has better performance, using pythorch as the backend or tensorrt-llm as the backend? During the actual test with qwen3, I found that the performance of using pythorch as the backend was not good, and the performance of a single gpu and multiple gpus was the same. Is this normal? Did I miss any details of the inference configuration?

[QiJune]

@oppolll Could you please share your scripts? cc @byshiue

[oppolll]

@oppolll Could you please share your scripts? cc @byshiue

Pytorch as backend reasoning code reference: https://github.com/NVIDIA/TensorRT-LLM/blob/v0.20.0rc3/examples/pytorch/quickstart.py
Lora reasoning code reference: https://github.com/NVIDIA/TensorRT-LLM/blob/v0.20.0rc3/tests/unittest/llmapi/test_llm_pytorch.py

With an input length of 8192 and an accuracy of bf16, I conducted the following several groups of experiments on the H20:

1. Using the qwen2.5-32b model and tensorrt-llm and pytorch as the backends, under the same experimental conditions, the inference performance with tensorrt-llm as the backend is superior.

2. Using the qwen3-32b model and pytorch as the backend, the performance is not much different from that of vllm inference. Usually, the inference performance of tensorrt-llm is superior to that of vllm.

3. Using the qwen3-32b model and pytorch as the backend inference, the performance was surprisingly the same on one gpu and two gpus.

4. Use qwen3-fp8 model on this link: https://huggingface.co/Qwen/Qwen3-32B-FP8, on a gpu, like bf16 reasoning performance.

5. Using the qwen3-32B model and pytorch as the backend to load lora inference did not work. Is it because qwen3 does not support lora inference?

Are the above results normal? Could it be that I overlooked some parameter configurations? May I ask if there are two types of model performance data for back-end inference to refer to?

[oppolll]

@oppolll Could you please share your scripts? cc @byshiue

Looking forward to your providing relevant guidance and suggestions. This is very important to me. Thank you.

[hypdeb]

Hello @oppolll,

More information is needed to proceed. I currently do not have the bandwidth to collect it myself, apologies.

Using the qwen2.5-32b model and tensorrt-llm and pytorch as the backends, under the same experimental conditions, the inference performance with tensorrt-llm as the backend is superior.

Does that mean comparing the TensorRT backend versus the Pytorch backend? What are you using as a benchmarking tool? Our recommendation is to use trtllm-bench. A profile of your run would allow us to understand if this is expected or not: see here to get started on profiling.

Your second experiment

Using the qwen3-32b model and pytorch as the backend, the performance is not much different from that of vllm inference. Usually, the inference performance of tensorrt-llm is superior to that of vllm.

compares the Pytorch backend of trtllm versus vllm. Same as above, a profile would allow us to proceed with an investigation.

For your third experiment:

Using the qwen3-32b model and pytorch as the backend inference, the performance was surprisingly the same on one gpu and two gpus.

How are your devices physically connected? As above, a profile would be helpful.

Fourth experiment:

Use qwen3-fp8 model on this link: https://huggingface.co/Qwen/Qwen3-32B-FP8, on a gpu, like bf16 reasoning performance.

Same as above, profiling is needed to be able to say more.

Finally:

Using the qwen3-32B model and pytorch as the backend to load lora inference did not work. Is it because qwen3 does not support lora inference?

A stack trace or error message is necessary for me to help you with that.

[oppolll]

@hypdeb Thank you for your reply! Regarding the first four questions, I will provide supplementary feedback after completing the trtllm-bench test.

Regarding the fifth question (inferring Qwen3 and LoRA files using the PyTorch backend), judging from the loading log, LoRA seems to have been loaded correctly:

...
[TensorRT-LLM] TensorRT-LLM version: 0.20.0rc3
[TensorRT-LLM][INFO] Refreshed the MPI local session
Loading weights: 100%|██████████| 1353/1353 [00:11<00:00, 114.20it/s]
Model init total -- 37.49s
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 1281 [window size=40961]
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 10.01 GiB for max tokens in paged KV cache (40992).
[TensorRT-LLM][INFO] Using 9061466112 bytes for LoRA host cache
[TensorRT-LLM][INFO] Using 9061466112 bytes for LoRA device cache
[TensorRT-LLM][INFO] Max LoRA size is 650121216
[TensorRT-LLM][INFO] LoRA host Cache can hold 7 max sized LoRAs
[TensorRT-LLM][INFO] LoRA device Cache can hold 7 max sized LoRAs
2025-05-28 01:19:59,596 - INFO - flashinfer.jit: Loading JIT ops: norm
/usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn(
/usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn(
2025-05-28 01:19:59,622 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-05-28 01:20:00,385 - INFO - flashinfer.jit: Loading JIT ops: silu_and_mul
/usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn(
/usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn(
2025-05-28 01:20:00,407 - INFO - flashinfer.jit: Finished loading JIT ops: silu_and_mul
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 1281 [window size=40961]
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 12.95 GiB for max tokens in paged KV cache (53056).
[TensorRT-LLM][INFO] Using 9061466112 bytes for LoRA host cache
[TensorRT-LLM][INFO] Using 9061466112 bytes for LoRA device cache
[TensorRT-LLM][INFO] Max LoRA size is 650121216
[TensorRT-LLM][INFO] LoRA host Cache can hold 7 max sized LoRAs
[TensorRT-LLM][INFO] LoRA device Cache can hold 7 max sized LoRAs
...

Call lora reasoning as follows, reference lora reasoning on https://github.com/NVIDIA/TensorRT-LLM/blob/main/tests/unittest/llmapi/test_llm_pytorch.py this link code.

lora_params0 = LoRARequest(
lora_name = "test0",
lora_int_id = 0,
lora_path = "lora Address"
)
outputs = llm.generate(prompts, sampling_params, True, [lora_params0])

There were no errors in the reasoning process, however, lora files did not work. May I ask what the possible reasons might be? Does Qwen3 currently not support LoRA inference?

Looking forward to your reply. Thank you very much.