LLAMA checkpoint ImportError: undefined symbol

[Pareek-Yash]

System Info

• Architecture: x86_64
• OS Ubuntu 22.04
• GPU: NVIDIA GeForce RTX 4090
• Gpu memory 2x24gb
• CPU max MHz: 5000.0000
• Driver Version: 535.183.01
• CUDA Version: 12.2
• Container: nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3
• TensorRT-LLM version: 0.10.0

Who can help?

@byshiue @nv-guomingz

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

nvidia-docker run -d -it --name trtllm     -v /home/remotessh/text-generation-webui/models/Llama-2-13b-chat-hf:/root/.cache/huggingface/llama-2-13b-chat-hf     -v /home/remotessh/TensorRT_engines:/engines     --shm-size=16G --network=host nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3 /bin/bash
11f808569af7484e003da1e5eb26729a7decd74f470b0699d983056df2ca1aef
git clone https://github.com/triton-inference-server/tensorrtllm_backend.git
cd tensorrtllm_backend/
git clone https://github.com/NVIDIA/TensorRT-LLM.git
pip install git+https://github.com/NVIDIA/TensorRT-LLM.git

mkdir /usr/local/lib/python3.10/dist-packages/tensorrt_llm/libs/

cp /opt/tritonserver/backends/tensorrtllm/* /usr/local/lib/python3.10/dist-packages/tensorrt_llm/libs/
export PYTHONPATH=/root/.cache/huggingface/llama-2-13b-chat-hf

2. This leads to the following error:

ImportError: The bindings module does not exist. Please check the package integrity. If you are attempting to use the pip development mode (editable installation), please execute build_wheels.py first, and then run pip install -e ..

python scripts/build_wheel.py
pip install -e
cd TensorRT-LLM/examples/llama
python convert_checkpoint.py --model_dir /root/.cache/huggingface/llama-2-13b-chat-hf \
                             --output_dir /workspace/tensorrt_llm/llama-2-13b-chat-hf \
                             --dtype float16 \
                             --tp_size 2

To get this error:

/opt/tritonserver/tensorrtllm_backend/TensorRT-LLM/examples/llama# python convert_checkpoint.py --model_dir /root/.cache/huggingface/llama-2-13b-chat-hf/llama-2-13b-chat-hf --output_dir /workspace/tensorrt_llm/llama-2-13b-chat-hf --dtype float16 --tp_size 2
Traceback (most recent call last):
File "/opt/tritonserver/tensorrtllm_backend/TensorRT-LLM/examples/llama/convert_checkpoint.py", line 8, in
import tensorrt_llm
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/init.py", line 32, in
import tensorrt_llm.functional as functional
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/functional.py", line 28, in
from . import graph_rewriting as gw
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/graph_rewriting.py", line 12, in
from .network import Network
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/network.py", line 27, in
from tensorrt_llm.module import Module
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/module.py", line 17, in
from ._common import default_net
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_common.py", line 31, in
from ._utils import str_dtype_to_trt
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_utils.py", line 30, in
from tensorrt_llm.bindings.BuildInfo import ENABLE_MULTI_DEVICE

ImportError: /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail14torchCheckFailEPKcS2_jRKSs

Expected behavior

convert checkpoints successfull

actual behavior

ImportError: /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail14torchCheckFailEPKcS2_jRKSs

additional notes

I'm following the official documentation and some fixes suggested by other devs

[QiJune]

Hi @Pareek-Yash , if you are using nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3, then, please use the correspoding TRT-LLM examples: https://github.com/NVIDIA/TensorRT-LLM/tree/v0.10.0/examples/llama

[Pareek-Yash]

Hi @Pareek-Yash , if you are using nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3, then, please use the correspoding TRT-LLM examples: https://github.com/NVIDIA/TensorRT-LLM/tree/v0.10.0/examples/llama

hey Thank you, changing to 0.10.0 solved the issue,

I have run into another issue which I do not understand in
python3 ../run.py --max_output_len=50 --tokenizer_dir /root/.cache/huggingface/llama-2-13b-chat-hf/ --engine_dir=./tmp/llama/13B/trt_engines/fp16/2-gpu/

[TensorRT-LLM][INFO] MPI size: 1, rank: 0
Traceback (most recent call last):
File "/opt/tritonserver/TensorRT-LLM/examples/llama/../run.py", line 571, in
main(args)
File "/opt/tritonserver/TensorRT-LLM/examples/llama/../run.py", line 420, in main
runner = runner_cls.from_dir(**runner_kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/model_runner_cpp.py", line 999, in from_dir
return ModelRunnerCppExecutor.from_dir(engine_dir, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/model_runner_cpp.py", line 98, in from_dir
world_config = WorldConfig.mpi(tensor_parallelism=tp_size,
RuntimeError: [TensorRT-LLM][ERROR] Assertion failed: mpiSize == tp * pp (/tmp/tritonbuild/tensorrtllm/tensorrt_llm/cpp/tensorrt_llm/runtime/worldConfig.cpp:99)

Steps taken:

python convert_checkpoint.py --model_dir /root/.cache/huggingface/llama-2-13b-chat-hf/                             --output_dir ./tllm_checkpoint_2gpu_tp2                             --dtype float16                             --tp_size 2
trtllm-build --checkpoint_dir ./tllm_checkpoint_2gpu_tp2             --output_dir ./tmp/llama/13B/trt_engines/fp16/2-gpu/  
           --gemm_plugin float16

[07/16/2024-07:30:55] [TRT-LLM] [I] Build TensorRT engine Unnamed Network 0
[07/16/2024-07:30:55] [TRT] [W] Unused Input: position_ids
[07/16/2024-07:30:55] [TRT] [W] Detected layernorm nodes in FP16.
[07/16/2024-07:30:55] [TRT] [W] Running layernorm after self-attention in FP16 may cause overflow. Exporting the model to the latest available ONNX opset (later than opset 17) to use the INormalizationLayer, or forcing layernorm layers to run in FP32 precision can help with preserving accuracy.
[07/16/2024-07:30:55] [TRT] [W] [RemoveDeadLayers] Input Tensor position_ids is unused or used only at compile-time, but is not being removed.
[07/16/2024-07:30:55] [TRT] [I] Global timing cache in use. Profiling results in this builder pass will be stored.
[07/16/2024-07:30:58] [TRT] [I] [GraphReduction] The approximate region cut reduction algorithm is called.
[07/16/2024-07:30:58] [TRT] [I] Detected 15 inputs and 3 output network tensors.
[07/16/2024-07:31:07] [TRT] [I] Total Host Persistent Memory: 151552
[07/16/2024-07:31:07] [TRT] [I] Total Device Persistent Memory: 0
[07/16/2024-07:31:07] [TRT] [I] Total Scratch Memory: 33559168
[07/16/2024-07:31:07] [TRT] [I] [BlockAssignment] Started assigning block shifts. This will take 778 steps to complete.
[07/16/2024-07:31:07] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 33.3864ms to assign 17 blocks to 778 nodes requiring 93334528 bytes.
[07/16/2024-07:31:07] [TRT] [I] Total Activation Memory: 93333504
[07/16/2024-07:31:07] [TRT] [I] Total Weights Memory: 13182216192
[07/16/2024-07:31:07] [TRT] [I] Engine generation completed in 12.6013 seconds.
[07/16/2024-07:31:07] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 625 MiB, GPU 12572 MiB
[07/16/2024-07:31:11] [TRT] [I] [MemUsageStats] Peak memory usage during Engine building and serialization: CPU: 28155 MiB
[07/16/2024-07:31:12] [TRT-LLM] [I] Total time of building Unnamed Network 0: 00:00:16
[07/16/2024-07:31:12] [TRT-LLM] [I] Serializing engine to ./tmp/llama/13B/trt_engines/fp16/2-gpu/rank1.engine...
[07/16/2024-07:31:17] [TRT-LLM] [I] Engine serialized. Total time: 00:00:05
[07/16/2024-07:31:17] [TRT-LLM] [I] Total time of building all engines: 00:00:48

[QiJune]

In the convert_checkpoint.py, you pass tp_size=2, so you need to use

mpirun -n 4 --allow-run-as-root \
python ../run.py xxx

[Pareek-Yash]

this has changed the MPI size, but error remains the same:

[TensorRT-LLM][INFO] MPI size: 4, rank: 3

Changed to mpi -n 2:

mpirun -n 2 --allow-run-as-root python3 ../run.py --ma
x_output_len=50                   --tokenizer_dir /root/.cache/huggingface/llama-2-13b-chat-hf/                 
  --engine_dir=/opt/tritonserver/TensorRT-LLM/examples/llama/tmp/llama/13B/trt_engines/fp16/2-gpu  --input_text 
"How do I count to nine in French?"

[TensorRT-LLM] TensorRT-LLM version: 0.10.0
[TensorRT-LLM] TensorRT-LLM version: 0.10.0
[TensorRT-LLM][INFO] Engine version 0.10.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'cross_attention' not found
[TensorRT-LLM][WARNING] Optional value for parameter cross_attention will not be set.
[TensorRT-LLM][WARNING] Parameter layer_types cannot be read from json:
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'layer_types' not found
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter kv_cache_quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'num_medusa_heads' not found
[TensorRT-LLM][WARNING] Optional value for parameter num_medusa_heads will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'max_draft_len' not found
[TensorRT-LLM][WARNING] Optional value for parameter max_draft_len will not be set.
[TensorRT-LLM][INFO] Engine version 0.10.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'cross_attention' not found
[TensorRT-LLM][WARNING] Optional value for parameter cross_attention will not be set.
[TensorRT-LLM][WARNING] Parameter layer_types cannot be read from json:
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'layer_types' not found
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter kv_cache_quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'num_medusa_heads' not found
[TensorRT-LLM][WARNING] Optional value for parameter num_medusa_heads will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'max_draft_len' not found
[TensorRT-LLM][WARNING] Optional value for parameter max_draft_len will not be set.
[TensorRT-LLM][INFO] MPI size: 2, rank: 0
[TensorRT-LLM][INFO] Engine version 0.10.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'cross_attention' not found
[TensorRT-LLM][WARNING] Optional value for parameter cross_attention will not be set.
[TensorRT-LLM][WARNING] Parameter layer_types cannot be read from json:
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'layer_types' not found
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter kv_cache_quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'num_medusa_heads' not found
[TensorRT-LLM][WARNING] Optional value for parameter num_medusa_heads will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'max_draft_len' not found
[TensorRT-LLM][WARNING] Optional value for parameter max_draft_len will not be set.
[TensorRT-LLM][INFO] Engine version 0.10.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'cross_attention' not found
[TensorRT-LLM][WARNING] Optional value for parameter cross_attention will not be set.
[TensorRT-LLM][WARNING] Parameter layer_types cannot be read from json:
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'layer_types' not found
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter kv_cache_quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'num_medusa_heads' not found
[TensorRT-LLM][WARNING] Optional value for parameter num_medusa_heads will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'max_draft_len' not found
[TensorRT-LLM][WARNING] Optional value for parameter max_draft_len will not be set.
[TensorRT-LLM][INFO] MPI size: 2, rank: 1
[TensorRT-LLM][INFO] Engine version 0.10.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'cross_attention' not found
[TensorRT-LLM][WARNING] Optional value for parameter cross_attention will not be set.
[TensorRT-LLM][INFO] Engine version 0.10.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'cross_attention' not found
[TensorRT-LLM][WARNING] Optional value for parameter cross_attention will not be set.
[TensorRT-LLM][WARNING] Parameter layer_types cannot be read from json:
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'layer_types' not found
[TensorRT-LLM][WARNING] Parameter layer_types cannot be read from json:
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'layer_types' not found
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter kv_cache_quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'num_medusa_heads' not found
[TensorRT-LLM][WARNING] Optional value for parameter num_medusa_heads will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'max_draft_len' not found
[TensorRT-LLM][WARNING] Optional value for parameter max_draft_len will not be set.
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter kv_cache_quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'num_medusa_heads' not found
[TensorRT-LLM][WARNING] Optional value for parameter num_medusa_heads will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'max_draft_len' not found
[TensorRT-LLM][WARNING] Optional value for parameter max_draft_len will not be set.
[TensorRT-LLM][INFO] MPI size: 2, rank: 0
[TensorRT-LLM][INFO] MPI size: 2, rank: 1
[TensorRT-LLM][INFO] Rank 1 is using GPU 1
[TensorRT-LLM][INFO] Rank 0 is using GPU 0
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 1
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 1
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: 2048
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 1
[TensorRT-LLM][WARNING] Device 1 peer access Device 0 is not available.
[TensorRT-LLM][INFO] Loaded engine size: 12575 MiB
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 1
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 1
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: 2048
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 1
[TensorRT-LLM][ERROR] 1: [defaultAllocator.cpp::allocate::19] Error Code 1: Cuda Runtime (out of memory)
[TensorRT-LLM][WARNING] Requested amount of GPU memory (12853706752 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[TensorRT-LLM][WARNING] Device 0 peer access Device 1 is not available.
[TensorRT-LLM][ERROR] 2: [safeDeserialize.cpp::load::269] Error Code 2: OutOfMemory (no further information)
[rezai:4750 :0:4750] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x8)
[TensorRT-LLM][INFO] Loaded engine size: 12575 MiB
[TensorRT-LLM][ERROR] 1: [defaultAllocator.cpp::allocate::19] Error Code 1: Cuda Runtime (out of memory)
[TensorRT-LLM][WARNING] Requested amount of GPU memory (12853706752 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[TensorRT-LLM][ERROR] 2: [safeDeserialize.cpp::load::269] Error Code 2: OutOfMemory (no further information)
[rezai:4749 :0:4749] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x8)
==== backtrace (tid: 4749) ====
0 0x0000000000042520 __sigaction() ???:0
1 0x0000000002401998 tensorrt_llm::runtime::TllmRuntime::TllmRuntime() ???:0
2 0x00000000026428e7 tensorrt_llm::batch_manager::TrtGptModelInflightBatching::TrtGptModelInflightBatching() ???:0
3 0x00000000025fe0b0 tensorrt_llm::batch_manager::TrtGptModelFactory::create() ???:0
4 0x0000000002664f62 tensorrt_llm::executor::Executor::Impl::createModel() ???:0
5 0x0000000002665ad2 tensorrt_llm::executor::Executor::Impl::Impl() ???:0
6 0x000000000265c4f6 tensorrt_llm::executor::Executor::Executor() ???:0
7 0x00000000000ae5b4 std::_Function_handler<void (unsigned long, at::Tensor&, std::vector<std::vector<int, std::allocator >, std::allocator<std::vector<int, std::allocator > > > const&, c10::Stream), void ()(unsigned long, at::Tensor&, std::vector<std::vector<int, std::allocator >, std::allocator<std::vector<int, std::allocator > > > const&, c10::Stream)>::_M_manager() ???:0
8 0x00000000000522b7 ???() /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so:0
9 0x000000000015a10e PyObject_CallFunctionObjArgs() ???:0
10 0x0000000000150a7b _PyObject_MakeTpCall() ???:0
11 0x0000000000168c20 PyMethod_New() ???:0
12 0x0000000000165087 _PyStack_AsDict() ???:0
13 0x0000000000150e2b _PyObject_MakeTpCall() ???:0
14 0x000000000041ecfb pybind11_meta_call() :0
15 0x0000000000150a7b _PyObject_MakeTpCall() ???:0
16 0x0000000000149629 _PyEval_EvalFrameDefault() ???:0
17 0x00000000001687f1 PyMethod_New() ???:0
18 0x0000000000169492 PyObject_Call() ???:0
19 0x00000000001455d7 _PyEval_EvalFrameDefault() ???:0
20 0x00000000001687f1 PyMethod_New() ???:0
21 0x0000000000169492 PyObject_Call() ???:0
22 0x00000000001455d7 _PyEval_EvalFrameDefault() ???:0
23 0x000000000015a9fc _PyFunction_Vectorcall() ???:0
24 0x000000000014326d _PyEval_EvalFrameDefault() ???:0
25 0x000000000013f9c6 _PyArg_ParseTuple_SizeT() ???:0
26 0x0000000000235256 PyEval_EvalCode() ???:0
27 0x0000000000260108 PyUnicode_Tailmatch() ???:0
28 0x00000000002599cb PyInit__collections() ???:0
29 0x000000000025fe55 PyUnicode_Tailmatch() ???:0
30 0x000000000025f338 _PyRun_SimpleFileObject() ???:0
31 0x000000000025ef83 _PyRun_AnyFileObject() ???:0
32 0x0000000000251a5e Py_RunMain() ???:0
33 0x000000000022802d Py_BytesMain() ???:0
34 0x0000000000029d90 __libc_init_first() ???:0
35 0x0000000000029e40 __libc_start_main() ???:0
36 0x0000000000227f25 _start() ???:0
=================================
==== backtrace (tid: 4750) ====
0 0x0000000000042520 __sigaction() ???:0
1 0x0000000002401998 tensorrt_llm::runtime::TllmRuntime::TllmRuntime() ???:0
2 0x00000000026428e7 tensorrt_llm::batch_manager::TrtGptModelInflightBatching::TrtGptModelInflightBatching() ???:0
3 0x00000000025fe0b0 tensorrt_llm::batch_manager::TrtGptModelFactory::create() ???:0
4 0x0000000002664f62 tensorrt_llm::executor::Executor::Impl::createModel() ???:0
5 0x0000000002665ad2 tensorrt_llm::executor::Executor::Impl::Impl() ???:0
6 0x000000000265c4f6 tensorrt_llm::executor::Executor::Executor() ???:0
7 0x00000000000ae5b4 std::_Function_handler<void (unsigned long, at::Tensor&, std::vector<std::vector<int, std::allocator >, std::allocator<std::vector<int, std::allocator > > > const&, c10::Stream), void ()(unsigned long, at::Tensor&, std::vector<std::vector<int, std::allocator >, std::allocator<std::vector<int, std::allocator > > > const&, c10::Stream)>::_M_manager() ???:0
8 0x00000000000522b7 ???() /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so:0
9 0x000000000015a10e PyObject_CallFunctionObjArgs() ???:0
10 0x0000000000150a7b _PyObject_MakeTpCall() ???:0
11 0x0000000000168c20 PyMethod_New() ???:0
12 0x0000000000165087 _PyStack_AsDict() ???:0
13 0x0000000000150e2b _PyObject_MakeTpCall() ???:0
14 0x000000000041ecfb pybind11_meta_call() :0
15 0x0000000000150a7b _PyObject_MakeTpCall() ???:0
16 0x0000000000149629 _PyEval_EvalFrameDefault() ???:0
17 0x00000000001687f1 PyMethod_New() ???:0
18 0x0000000000169492 PyObject_Call() ???:0
19 0x00000000001455d7 _PyEval_EvalFrameDefault() ???:0
20 0x00000000001687f1 PyMethod_New() ???:0
21 0x0000000000169492 PyObject_Call() ???:0
22 0x00000000001455d7 _PyEval_EvalFrameDefault() ???:0
23 0x000000000015a9fc _PyFunction_Vectorcall() ???:0
24 0x000000000014326d _PyEval_EvalFrameDefault() ???:0
25 0x000000000013f9c6 _PyArg_ParseTuple_SizeT() ???:0
26 0x0000000000235256 PyEval_EvalCode() ???:0
27 0x0000000000260108 PyUnicode_Tailmatch() ???:0
28 0x00000000002599cb PyInit__collections() ???:0
29 0x000000000025fe55 PyUnicode_Tailmatch() ???:0
30 0x000000000025f338 _PyRun_SimpleFileObject() ???:0
31 0x000000000025ef83 _PyRun_AnyFileObject() ???:0
32 0x0000000000251a5e Py_RunMain() ???:0
33 0x000000000022802d Py_BytesMain() ???:0
34 0x0000000000029d90 __libc_init_first() ???:0
35 0x0000000000029e40 __libc_start_main() ???:0
36 0x0000000000227f25 _start() ???:0
=================================

---

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

---

mpirun noticed that process rank 1 with PID 0 on node rezai exited on signal 11 (Segmentation fault).

[Pareek-Yash]

Update!!

The above error wasn't solved with --shm-size too

docker run -it --name trtllm --gpus all --shm-size=24g -v /home/remotessh/text-generation-webui/models/Llama-2-13b-chat-hf:/root/.cache/huggingface trtllm_final /bin/bash

mpirun -n 2 --allow-run-as-root python3 ../run.py --max_output_len=50                   --tokenizer_dir /root/.cache/huggingface/llama-2-13b-chat-hf/                   --engine_dir=/opt/tritonserver/TensorRT-LLM/examples/llama/tmp/llama/13B/trt_engines/fp16/2-gpu  --input_text "How do I count to nine in French?"

The same error as above.