trtllm-serve without any output Qwne2.5-7b

[Justin-12138]

System Info

-CPU: Intel Xeon Platinum 8352V (144) @ 3.500GHz X86
-Memory: 1031689MiB
-GPU:RTX-4090*8
-Librarys
tensorrt 10.7.0
tensorrt_cu12 10.7.0
tensorrt-cu12-bindings 10.7.0
tensorrt-cu12-libs 10.7.0
tensorrt-llm 0.16.0
nvidia driver version
Driver Version: 550.135 CUDA Version: 12.4
OS Ubuntu 22.04.5 LTS x86_64

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I ran the trtllm-serve comman like:
trtllm-serve /home/lz/tensorrt/build/Qwen2.5-7B-Instructtrt_engines/weight_only/1-gpu
--tokenizer /home/lz/tensorrt/models/Qwen2.5-7B-Instruct
--max_batch_size 128 --max_num_tokens 4096 --max_seq_len 4096
--kv_cache_free_gpu_memory_fraction 0.95

But there is no output except the:
[TensorRT-LLM] TensorRT-LLM version: 0.16.0

No errors,no warnings no Port occupation

But it ran well with the test：
python3 /home/lz/TensorRT-LLM/examples/run.py --input_text "你好，请问你叫什么？"
--max_output_len=50
--tokenizer_dir /home/lz/tensorrt/models/Qwen2.5-7B-Instruct
--engine_dir=/home/lz/tensorrt/build/Qwen2.5-7B-Instructtrt_engines/weight_only/1-gpu

What Can I do to run an OpenAI API compatible server

Expected behavior

Does it should output somemore info?

actual behavior

Nnothing but version

additional notes

Is that a problem with Qwen2.5-7b
I 'd appreciate if you guys could give me some help

[nv-guomingz]

Hi @LinPoly would u please take a look this issue?

[Justin-12138]

It works well now when using the latest nvcr.io/nvidia/tritonserver:25.02-trtllm-python-py3
but it doesn't work in my bare-metal enviroment,there is no any output or error

(trtllm) gpu8@gpu8:~/.local/bin$ ./trtllm-serve /engines/qwen2.5-32b/int4-2gpu/qwen2.5-32b \
--tokenizer /mnt/llm/models/Qwen2.5-7B-Instruct \
--max_batch_size 128 --max_num_tokens 4096 --max_seq_len 4096 \
--host "0.0.0.0" --port 4008 --tp_size 2
/usr/lib/python3/dist-packages/pytz/__init__.py:31: SyntaxWarning: invalid escape sequence '\s'
  match = re.match("^#\s*version\s*([0-9a-z]*)\s*$", line)
[TensorRT-LLM] TensorRT-LLM version: 0.17.0.post1
/home/gpu8/.local/lib/python3.12/site-packages/torch/utils/cpp_extension.py:1964: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(