Description
I have install TensorRT-LLM version: 0.15.0 in windows machine, GPU is RTX 2070 SUPER,
When run the sample:
from tensorrt_llm import LLM, SamplingParams
prompts = [
"Hello, my name is",
"The president of the United States is",
"The capital of France is",
"The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="TinyLlama-1.1B-Chat-v1.0/")
.....
a lot of runtime errors :
[TensorRT-LLM] TensorRT-LLM version: 0.15.0
Loading Model: [1/2] Loading HF model to memory
160it [00:02, 63.60it/s]
Time: 2.906s
Loading Model: [2/2] Building TRT-LLM engine
Unhandled exception caught in c10/util/AbortHandler.h
00007FFE12AE1DE400007FFE12AC2070 torch_python.dll!THPGenerator_initDefaultGenerator [ @ ]
00007FFFBA3BEE1200007FFFBA3BEDF0 ucrtbase.dll!terminate [ @ ]
00007FFF92361AAB00007FFF92361150 VCRUNTIME140_1.dll!_NLG_Return2 [ @ ]
00007FFF9236231700007FFF92361150 VCRUNTIME140_1.dll!_NLG_Return2 [ @ ]
00007FFF923640D900007FFF92364030 VCRUNTIME140_1.dll!_CxxFrameHandler4 [ @ ]
00007FFD84A3D55800007FFD8475FDF0 nvinfer_plugin_tensorrt_llm.dll!setLoggerFinder [ @ ]
00007FFFBCBD535F00007FFFBCBD5230 ntdll.dll!_chkstk [ @ ]
00007FFFBCB4E88600007FFFBCB4DDF0 ntdll.dll!RtlFindCharInUnicodeString [ @ ]
00007FFFBCB8495500007FFFBCB847C0 ntdll.dll!RtlRaiseException [ @ ]
00007FFFB9E7FE3C00007FFFB9E7FDD0 KERNELBASE.dll!RaiseException [ @ ]
00007FFF5CFE648000007FFF5CFE63F0 VCRUNTIME140.dll!CxxThrowException [ @ ]
00007FFDA1EF6BE700007FFDA1EF6B90 tensorrt_llm.dll!tensorrt_llm::common::throwRuntimeError [ @ ]
00007FFDA1F422A400007FFDA1F41EE0 tensorrt_llm.dll!tensorrt_llm::kernels::FusedMHARunnerV2::FusedMHARunnerV2 [ @ ]
00007FFD846E54D900007FFD846E52A0 nvinfer_plugin_tensorrt_llm.dll!tensorrt_llm::plugins::GPTAttentionPluginCommon::initialize [ @ ]
00007FFD846E681500007FFD846E66A0 nvinfer_plugin_tensorrt_llm.dll!tensorrt_llm::plugins::GPTAttentionPluginCommon::cloneImpl<tensorrt_llm::plugins::GPTAttentionPlugin> [ @ ]
00007FFF4E00B5B200007FFF4DFC8D10 nvinfer_10.dll!getInferLibMinorVersion [ @ ]
00007FFF4DF9441E00007FFF4DF545B0 nvinfer_10.dll!createInferBuilder_INTERNAL [ @ ]
00007FFEF606CDB9 tensorrt.cp310-win_amd64.pyd! [ @ ]
00007FFEF60170D6 tensorrt.cp310-win_amd64.pyd! [ @ ]
00007FFEF5F82EE8 tensorrt.cp310-win_amd64.pyd! [ @ ]
00007FFF5D019EEA00007FFF5D019E18 python310.dll!PyObject_IsTrue [ @ ]
00007FFF5D05FFBB00007FFF5D05FE58 python310.dll!PyObject_MakeTpCall [ @ ]
00007FFF5D19BF8F00007FFF5D17D114 python310.dll!Py_gitversion [ @ ]
00007FFF5D07729300007FFF5D0758A0 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FFF5D0749D700007FFF5D074950 python310.dll!PyFunction_Vectorcall [ @ ]
00007FFF5D05C00C00007FFF5D05BF54 python310.dll!PyVectorcall_Call [ @ ]
00007FFF5D05BE8700007FFF5D05BD44 python310.dll!PyObject_Call [ @ ]
00007FFF5D07B5E700007FFF5D0758A0 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FFF5D0749D700007FFF5D074950 python310.dll!PyFunction_Vectorcall [ @ ]
00007FFF5D05C00C00007FFF5D05BF54 python310.dll!PyVectorcall_Call [ @ ]
00007FFF5D05BE8700007FFF5D05BD44 python310.dll!PyObject_Call [ @ ]
00007FFF5D07B5E700007FFF5D0758A0 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FFF5D07361500007FFF5D0725C0 python310.dll!PyObject_GC_Malloc [ @ ]
00007FFF5D05C00C00007FFF5D05BF54 python310.dll!PyVectorcall_Call [ @ ]
00007FFF5D05BE8700007FFF5D05BD44 python310.dll!PyObject_Call [ @ ]
00007FFF5D07B5E700007FFF5D0758A0 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FFF5D0749D700007FFF5D074950 python310.dll!PyFunction_Vectorcall [ @ ]
00007FFF5D02A91700007FFF5D02A844 python310.dll!PyObject_FastCallDictTstate [ @ ]
00007FFF5D1381F400007FFF5D138178 python310.dll!PyObject_Call_Prepend [ @ ]
00007FFF5D13815000007FFF5D137064 python310.dll!PyBytesWriter_Resize [ @ ]
00007FFF5D05FFBB00007FFF5D05FE58 python310.dll!PyObject_MakeTpCall [ @ ]
00007FFF5D07C39F00007FFF5D0758A0 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FFF5D07361500007FFF5D0725C0 python310.dll!PyObject_GC_Malloc [ @ ]
00007FFF5D05C00C00007FFF5D05BF54 python310.dll!PyVectorcall_Call [ @ ]
00007FFF5D05BE8700007FFF5D05BD44 python310.dll!PyObject_Call [ @ ]
00007FFF5D07B5E700007FFF5D0758A0 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FFF5D0749D700007FFF5D074950 python310.dll!PyFunction_Vectorcall [ @ ]
00007FFF5D02A91700007FFF5D02A844 python310.dll!PyObject_FastCallDictTstate [ @ ]
00007FFF5D1381F400007FFF5D138178 python310.dll!PyObject_Call_Prepend [ @ ]
00007FFF5D13815000007FFF5D137064 python310.dll!PyBytesWriter_Resize [ @ ]
00007FFF5D05BF0300007FFF5D05BD44 python310.dll!PyObject_Call [ @ ]
00007FFF5D07B5E700007FFF5D0758A0 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FFF5D07862000007FFF5D0758A0 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FFF5D07361500007FFF5D0725C0 python310.dll!PyObject_GC_Malloc [ @ ]
00007FFF5D05C00C00007FFF5D05BF54 python310.dll!PyVectorcall_Call [ @ ]
00007FFF5D05BE8700007FFF5D05BD44 python310.dll!PyObject_Call [ @ ]
00007FFF5D07B5E700007FFF5D0758A0 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FFF5D07361500007FFF5D0725C0 python310.dll!PyObject_GC_Malloc [ @ ]
00007FFF5D05C00C00007FFF5D05BF54 python310.dll!PyVectorcall_Call [ @ ]
00007FFF5D05BE8700007FFF5D05BD44 python310.dll!PyObject_Call [ @ ]
00007FFF5D07B5E700007FFF5D0758A0 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FFF5D0749D700007FFF5D074950 python310.dll!PyFunction_Vectorcall [ @ ]
00007FFF5D02A91700007FFF5D02A844 python310.dll!PyObject_FastCallDictTstate [ @ ]
00007FFF5D1381F400007FFF5D138178 python310.dll!PyObject_Call_Prepend [ @ ]
Process finished with exit code -1073740791 (0xC0000409)