ModuleNotFoundError: No module named 'exllamav2'

[hushuitian]

The test environment: the server OS is Ubuntu 22.04 and there is a T4 GPU available.
(1)did below install and infer steps according to https://huggingface.co/TigerResearch/tigerbot-13b-chat-4bit-exl2:
conda create --name tigerbot python=3.8
conda activate tigerbot
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
git clone https://github.com/TigerResearch/TigerBot
cd TigerBot
pip install -r requirements.txt

git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
cd ..

CUDA_VISIBLE_DEVICES=0 python other_infer/exllamav2_hf_infer.py --model_path TigerResearch/tigerbot-13b-chat-4bit-exl2

(2)Got the error:
Traceback (most recent call last):
File "other_infer/exllamav2_hf_infer.py", line 9, in
from exllamav2 import ExLlamaV2, ExLlamaV2Cache, ExLlamaV2Config
ModuleNotFoundError: No module named 'exllamav2'

(3)Please advise how to fix this issue.

[Vivicai1005]

try "pip install exllamav2"? and "pip show exllamav2" to check if installed successfully

[hushuitian]

(1)Did:
pip install exllamav2
CUDA_VISIBLE_DEVICES=0 python other_infer/exllamav2_hf_infer.py --model_path TigerResearch/tigerbot-13b-chat-4bit-exl2
(2)Got:
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-12.1'
Traceback (most recent call last):
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/ext.py", line 14, in
import exllamav2_ext
ModuleNotFoundError: No module named 'exllamav2_ext'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "other_infer/exllamav2_hf_infer.py", line 9, in
from exllamav2 import ExLlamaV2, ExLlamaV2Cache, ExLlamaV2Config
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/init.py", line 3, in
from exllamav2.model import ExLlamaV2
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/model.py", line 12, in
from exllamav2.linear import ExLlamaV2Linear
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/linear.py", line 5, in
from exllamav2 import ext
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/ext.py", line 121, in
exllamav2_ext = load
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1308, in load
return _jit_compile(
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1710, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1810, in _write_ninja_file_and_build_library
_write_ninja_file_to_build_library(
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2199, in _write_ninja_file_to_build_library
cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1980, in _get_cuda_arch_flags
arch_list[-1] += '+PTX'
IndexError: list index out of range

[wangyu1997]

Hello, you can upgrade your Python version to 3.9 first and try again. We work normally with python=3.10 and CUDA version=11.6. The recommended Python version is >=3.9. We will also investigate the solution as soon as possible.

[wangyu1997]

I think I found the reason. When switched to Python 3.8, I encountered the same problem as you. I realized that it might be due to a mismatch between the torch version and the CUDA version. Therefore, I reinstalled torch under Python 3.8 and ensured that it was consistent with the current CUDA version. The model loaded smoothly. Good Luck !

[hushuitian]

conda create --name tigerbot python=3.8
conda activate tigerbot
#pytorch official installation
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
git clone https://github.com/TigerResearch/TigerBot
cd TigerBot
pip install -r requirements.txt
git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
pip install exllamav2
cd ..
CUDA_VISIBLE_DEVICES=0 python other_infer/exllamav2_hf_infer.py --model_path TigerResearch/tigerbot-13b-chat-4bit-exl2
loading model: TigerResearch/tigerbot-13b-chat-4bit-exl2...
Traceback (most recent call last):
File "other_infer/exllamav2_hf_infer.py", line 300, in
fire.Fire(main)
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "other_infer/exllamav2_hf_infer.py", line 190, in main
model = get_model(model_path)
File "other_infer/exllamav2_hf_infer.py", line 177, in get_model
model = Exllamav2HF.from_pretrained(model)
File "other_infer/exllamav2_hf_infer.py", line 161, in from_pretrained
config.prepare()
File "/home/test/anaconda3/envs/tigerbot/lib/python3.8/site-packages/exllamav2/config.py", line 52, in prepare
assert os.path.exists(self.model_dir), "Can't find " + self.model_dir
AssertionError: Can't find TigerResearch/tigerbot-13b-chat-4bit-exl2

[wangyu1997]

pip install torch==2.0.1+cu117 torchvision torchaudio --extra-index-url https://mirror.sjtu.edu.cn/pytorch-wheels/cu117

[hushuitian]

yes. installed torch==2.0.1+cu117.

[wangyu1997]

yes, it's a bug. exllamav2 will first check local path. You can download model weights from HF manually. We will repair it soon.

[hushuitian]

Thank you! Another option is to do 4 bits quantization with AutoGPTQ only.