QLoRA 4bit works fine but 8 bit raise AttributeError: 'MatmulLtState' object has no attribute 'memory_efficient_backward'

[randydl]

Reminder

• I have read the README and searched the existing issues.

System Info

Package                                 Version        Editable project location
--------------------------------------- -------------- -----------------------------------------------
absl-py                                 2.1.0
accelerate                              1.0.1
aiofiles                                23.2.1
aiohappyeyeballs                        2.4.4
aiohttp                                 3.11.11
aiohttp-cors                            0.7.0
aiosignal                               1.3.2
airportsdata                            20241001
albucore                                0.0.23
albumentations                          1.4.24
altair                                  5.5.0
annotated-types                         0.7.0
antlr4-python3-runtime                  4.9.3
anyio                                   4.7.0
argon2-cffi                             23.1.0
argon2-cffi-bindings                    21.2.0
arrow                                   1.3.0
astor                                   0.8.1
asttokens                               3.0.0
async-lru                               2.0.4
async-timeout                           4.0.3
attrs                                   24.3.0
autotiktokenizer                        0.2.2
av                                      14.0.1
babel                                   2.16.0
beautifulsoup4                          4.12.3
bitsandbytes                            0.45.0
blake3                                  1.0.0
bleach                                  6.2.0
blinker                                 1.9.0
cachetools                              5.5.0
certifi                                 2024.12.14
cffi                                    1.17.1
charset-normalizer                      3.4.1
chonkie                                 0.4.0
click                                   8.1.8
cloudpickle                             3.1.0
colorful                                0.5.6
comm                                    0.2.2
compressed-tensors                      0.8.1
contourpy                               1.3.1
cupy-cuda12x                            13.3.0
cut-cross-entropy                       24.12.3
cycler                                  0.12.1
dataclasses-json                        0.6.7
datasets                                3.1.0
debugpy                                 1.8.11
decorator                               5.1.1
decord                                  0.6.0
deepspeed                               0.15.4
defusedxml                              0.7.1
Deprecated                              1.2.15
depyf                                   0.18.0
diffusers                               0.32.1
dill                                    0.3.8
dirtyjson                               1.0.8
diskcache                               5.6.3
distlib                                 0.3.9
distro                                  1.9.0
docstring_parser                        0.16
easydict                                1.13
einops                                  0.8.0
einx                                    0.3.0
et_xmlfile                              2.0.0
exceptiongroup                          1.2.2
executing                               2.1.0
fastapi                                 0.115.6
fastjsonschema                          2.21.1
fastrlock                               0.8.3
fasttext                                0.9.3
ffmpy                                   0.5.0
filelock                                3.16.1
filetype                                1.2.0
fire                                    0.7.0
flash-attn                              2.7.2.post1
Flask                                   3.1.0
fonttools                               4.55.3
fqdn                                    1.5.1
frozendict                              2.4.6
frozenlist                              1.5.0
fsspec                                  2024.9.0
ftfy                                    6.3.1
gguf                                    0.10.0
gitdb                                   4.0.12
GitPython                               3.1.44
google-api-core                         2.24.0
google-auth                             2.37.0
googleapis-common-protos                1.66.0
grad-cam                                1.5.4
gradio                                  4.44.1
gradio_client                           1.3.0
greenlet                                3.1.1
grpcio                                  1.68.1
h11                                     0.14.0
hf_transfer                             0.1.8
hjson                                   3.1.0
httpcore                                1.0.7
httptools                               0.6.4
httpx                                   0.28.1
httpx-sse                               0.4.0
huggingface-hub                         0.27.0
idna                                    3.10
imageio                                 2.36.1
importlib_metadata                      8.5.0
importlib_resources                     6.4.5
iniconfig                               2.0.0
interegular                             0.3.3
ipykernel                               6.29.5
ipython                                 8.31.0
ipython-genutils                        0.2.0
ipywidgets                              8.1.5
isoduration                             20.11.0
itsdangerous                            2.2.0
jedi                                    0.19.2
jieba                                   0.42.1
Jinja2                                  3.1.5
jiter                                   0.8.2
joblib                                  1.4.2
json5                                   0.10.0
jsonpatch                               1.33
jsonpointer                             3.0.0
jsonschema                              4.23.0
jsonschema-specifications               2024.10.1
jupyter_client                          8.6.3
jupyter_core                            5.7.2
jupyter-events                          0.11.0
jupyter-lsp                             2.2.5
jupyter_server                          2.15.0
jupyter_server_terminals                0.5.3
jupyterlab                              4.3.4
jupyterlab_pygments                     0.3.0
jupyterlab_server                       2.27.3
jupyterlab_widgets                      3.0.13
kiwisolver                              1.4.8
kornia                                  0.7.4
kornia_rs                               0.1.8
langchain                               0.3.13
langchain-community                     0.3.13
langchain-core                          0.3.28
langchain-openai                        0.2.14
langchain-text-splitters                0.3.4
langsmith                               0.2.7
lark                                    1.2.2
lazy_loader                             0.4
liger_kernel                            0.5.2
lightgbm                                4.5.0
lightning-utilities                     0.11.9
linkify-it-py                           2.0.3
litserve                                0.2.5
llama-cloud                             0.1.7
llama-index                             0.12.9
llama-index-agent-openai                0.4.1
llama-index-cli                         0.4.0
llama-index-core                        0.12.9
llama-index-embeddings-openai           0.3.1
llama-index-indices-managed-llama-cloud 0.6.3
llama-index-llms-openai                 0.3.12
llama-index-multi-modal-llms-openai     0.4.2
llama-index-program-openai              0.3.1
llama-index-question-gen-openai         0.3.0
llama-index-readers-file                0.4.1
llama-index-readers-llama-parse         0.4.0
llama-parse                             0.5.19
llamafactory                            0.9.2.dev0     /nas_data/userdata/randy/projects/LLaMA-Factory
lm-format-enforcer                      0.10.9
loguru                                  0.7.3
lxml                                    5.3.0
Markdown                                3.7
markdown-it-py                          3.0.0
MarkupSafe                              2.1.5
marshmallow                             3.23.2
matplotlib                              3.10.0
matplotlib-inline                       0.1.7
mdit-py-plugins                         0.4.2
mdurl                                   0.1.2
memray                                  1.15.0
mistral_common                          1.5.1
mistune                                 3.1.0
model2vec                               0.3.3
modelscope                              1.21.1
mpmath                                  1.3.0
msgpack                                 1.1.0
msgspec                                 0.19.0
multidict                               6.1.0
multiprocess                            0.70.16
mypy-extensions                         1.0.0
narwhals                                1.20.1
nbclassic                               1.1.0
nbclient                                0.10.2
nbconvert                               7.16.4
nbformat                                5.10.4
nest-asyncio                            1.6.0
networkx                                3.4.2
ninja                                   1.11.1.3
nltk                                    3.9.1
notebook                                7.3.2
notebook_shim                           0.2.4
numpy                                   1.26.4
nvidia-cublas-cu12                      12.4.5.8
nvidia-cuda-cupti-cu12                  12.4.127
nvidia-cuda-nvrtc-cu12                  12.4.127
nvidia-cuda-runtime-cu12                12.4.127
nvidia-cudnn-cu12                       9.1.0.70
nvidia-cufft-cu12                       11.2.1.3
nvidia-curand-cu12                      10.3.5.147
nvidia-cusolver-cu12                    11.6.1.9
nvidia-cusparse-cu12                    12.3.1.170
nvidia-ml-py                            12.560.30
nvidia-nccl-cu12                        2.21.5
nvidia-nvjitlink-cu12                   12.4.127
nvidia-nvtx-cu12                        12.4.127
nvitop                                  1.4.0
omegaconf                               2.3.0
open_clip_torch                         2.29.0
openai                                  1.58.1
OpenCC                                  1.1.9
opencensus                              0.11.4
opencensus-context                      0.1.3
opencv-python                           4.10.0.84
opencv-python-headless                  4.10.0.84
openpyxl                                3.1.5
orjson                                  3.10.13
outlines                                0.1.11
outlines_core                           0.1.26
overrides                               7.7.0
packaging                               24.2
pandas                                  2.2.3
pandocfilters                           1.5.1
parse                                   1.20.2
parso                                   0.8.4
partial-json-parser                     0.2.1.1.post4
peft                                    0.12.0
pexpect                                 4.9.0
pillow                                  10.4.0
pip                                     24.3.1
platformdirs                            4.3.6
plotly                                  5.24.1
pluggy                                  1.5.0
prometheus_client                       0.21.1
prometheus-fastapi-instrumentator       7.0.0
prompt_toolkit                          3.0.48
propcache                               0.2.1
proto-plus                              1.25.0
protobuf                                3.20.3
psutil                                  6.1.1
ptyprocess                              0.7.0
pure_eval                               0.2.3
py-cpuinfo                              9.0.0
py-spy                                  0.4.0
pyarrow                                 18.1.0
pyasn1                                  0.6.1
pyasn1_modules                          0.4.1
pybind11                                2.13.6
pycountry                               24.6.1
pycparser                               2.22
pydantic                                2.10.4
pydantic_core                           2.27.2
pydantic-settings                       2.7.1
pydeck                                  0.9.1
pydub                                   0.25.1
Pygments                                2.18.0
PyMuPDF                                 1.25.1
pyparsing                               3.2.1
pypdf                                   5.1.0
pytest                                  8.3.4
python-dateutil                         2.9.0.post0
python-dotenv                           1.0.1
python-json-logger                      3.2.1
python-multipart                        0.0.20
pytorch-metric-learning                 2.8.1
pytz                                    2024.2
PyYAML                                  6.0.2
pyzmq                                   26.2.0
RapidFuzz                               3.11.0
ray                                     2.40.0
referencing                             0.35.1
regex                                   2024.11.6
requests                                2.32.3
requests-toolbelt                       1.0.0
rfc3339-validator                       0.1.4
rfc3986-validator                       0.1.1
rich                                    13.9.4
rouge-chinese                           1.0.3
rpds-py                                 0.22.3
rsa                                     4.9
ruff                                    0.8.4
safetensors                             0.4.5
scikit-image                            0.25.0
scikit-learn                            1.6.0
scipy                                   1.14.1
seaborn                                 0.13.2
semantic-version                        2.10.0
Send2Trash                              1.8.3
sentence-transformers                   3.3.1
sentencepiece                           0.2.0
setuptools                              75.6.0
shellingham                             1.5.4
shtab                                   1.7.1
SimpleITK                               2.4.0
simsimd                                 6.2.1
six                                     1.17.0
smart-open                              7.1.0
smmap                                   5.0.2
sniffio                                 1.3.1
soupsieve                               2.6
SQLAlchemy                              2.0.36
sse-starlette                           2.2.1
stack-data                              0.6.3
starlette                               0.41.3
streamlit                               1.41.1
stringzilla                             3.11.3
striprtf                                0.0.26
supervision                             0.25.1
sympy                                   1.13.1
tenacity                                9.0.0
tensorboard                             2.18.0
tensorboard-data-server                 0.7.2
termcolor                               2.5.0
terminado                               0.18.1
textual                                 1.0.0
threadpoolctl                           3.5.0
tifffile                                2024.12.12
tiktoken                                0.7.0
timm                                    1.0.12
tinycss2                                1.4.0
tokenizers                              0.20.3
toml                                    0.10.2
tomli                                   2.2.1
tomlkit                                 0.12.0
torch                                   2.5.1
torchinfo                               1.8.0
torchmetrics                            1.6.1
torchshow                               0.5.1
torchvision                             0.20.1
tornado                                 6.4.2
tqdm                                    4.67.1
traitlets                               5.14.3
transformers                            4.46.1
triton                                  3.1.0
trl                                     0.9.6
ttach                                   0.0.3
typer                                   0.15.1
types-python-dateutil                   2.9.0.20241206
typing_extensions                       4.12.2
typing-inspect                          0.9.0
tyro                                    0.8.14
tzdata                                  2024.2
uc-micro-py                             1.0.3
ultralytics                             8.3.56
ultralytics-thop                        2.0.13
unsloth                                 2024.12.12
unsloth_zoo                             2024.12.7
uri-template                            1.3.0
urllib3                                 2.3.0
uvicorn                                 0.34.0
uvloop                                  0.21.0
virtualenv                              20.28.0
vllm                                    0.6.6.post1
watchdog                                6.0.0
watchfiles                              1.0.3
wcwidth                                 0.2.13
webcolors                               24.11.1
webencodings                            0.5.1
websocket-client                        1.8.0
websockets                              12.0
Werkzeug                                3.1.3
wheel                                   0.45.1
widgetsnbextension                      4.0.13
wrapt                                   1.17.0
x-transformers                          1.43.5
xformers                                0.0.28.post3
xgboost                                 2.1.3
xgrammar                                0.1.8
xxhash                                  3.5.0
yarl                                    1.18.3
zipp                                    3.21.0

Reproduction

### model
model_name_or_path: /nas_data/userdata/randy/models/Qwen2.5-32B-Instruct

### method
stage: pt
do_train: true
finetuning_type: lora
lora_target: all
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.05
deepspeed: examples/deepspeed/ds_z2_config.json
flash_attn: fa2
# use_dora: true
quantization_bit: 8
quantization_method: bitsandbytes
trust_remote_code: true

### dataset
dataset: jedec-ddr5
template: default
cutoff_len: 2048
overwrite_cache: true
preprocessing_num_workers: 64

### output
output_dir: saves/qwen
overwrite_output_dir: true
logging_steps: 10
logging_first_step: true
save_steps: 100
save_total_limit: 10
load_best_model_at_end: true
plot_loss: true
report_to: tensorboard

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
warmup_ratio: 0.1
learning_rate: 1e-4
num_train_epochs: 10.0
lr_scheduler_type: cosine
neftune_noise_alpha: 5
ddp_timeout: 180000000
bf16: true

### eval
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 100
val_size: 0.01

Others

10.252.32.13: [rank10]:   File "/nas_data/userdata/randy/projects/LLaMA-Factory/randy/train.py", line 21, in <module>
10.252.32.13: [rank10]:     run_exp()
10.252.32.13: [rank10]:   File "/nas_data/userdata/randy/projects/LLaMA-Factory/src/llamafactory/train/tuner.py", line 57, in run_exp
10.252.32.13: [rank10]:     run_pt(model_args, data_args, training_args, finetuning_args, callbacks)
10.252.32.13: [rank10]:   File "/nas_data/userdata/randy/projects/LLaMA-Factory/src/llamafactory/train/pt/workflow.py", line 47, in run_pt
10.252.32.13: [rank10]:     model = load_model(tokenizer, model_args, finetuning_args, training_args.do_train)
10.252.32.13: [rank10]:   File "/nas_data/userdata/randy/projects/LLaMA-Factory/src/llamafactory/model/loader.py", line 169, in load_model
10.252.32.13: [rank10]:     model = init_adapter(config, model, model_args, finetuning_args, is_trainable)
10.252.32.13: [rank10]:   File "/nas_data/userdata/randy/projects/LLaMA-Factory/src/llamafactory/model/adapter.py", line 299, in init_adapter
10.252.32.13: [rank10]:     model = _setup_lora_tuning(
10.252.32.13: [rank10]:   File "/nas_data/userdata/randy/projects/LLaMA-Factory/src/llamafactory/model/adapter.py", line 250, in _setup_lora_tuning
10.252.32.13: [rank10]:     model = get_peft_model(model, lora_config)
10.252.32.13: [rank10]:   File "/home/app.e0016372/miniconda3/envs/llmtuner/lib/python3.10/site-packages/peft/mapping.py", line 183, in get_peft_model
10.252.32.13: [rank10]:     return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](
10.252.32.13: [rank10]:   File "/home/app.e0016372/miniconda3/envs/llmtuner/lib/python3.10/site-packages/peft/peft_model.py", line 1542, in __init__
10.252.32.13: [rank10]:     super().__init__(model, peft_config, adapter_name, **kwargs)
10.252.32.13: [rank10]:   File "/home/app.e0016372/miniconda3/envs/llmtuner/lib/python3.10/site-packages/peft/peft_model.py", line 155, in __init__
10.252.32.13: [rank10]:     self.base_model = cls(model, {adapter_name: peft_config}, adapter_name)
10.252.32.13: [rank10]:   File "/home/app.e0016372/miniconda3/envs/llmtuner/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 139, in __init__
10.252.32.13: [rank10]:     super().__init__(model, config, adapter_name)
10.252.32.13: [rank10]:   File "/home/app.e0016372/miniconda3/envs/llmtuner/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 175, in __init__
10.252.32.13: [rank10]:     self.inject_adapter(self.model, adapter_name)
10.252.32.13: [rank10]:   File "/home/app.e0016372/miniconda3/envs/llmtuner/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 431, in inject_adapter
10.252.32.13: [rank10]:     self._create_and_replace(peft_config, adapter_name, target, target_name, parent, current_key=key)
10.252.32.13: [rank10]:   File "/home/app.e0016372/miniconda3/envs/llmtuner/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 224, in _create_and_replace
10.252.32.13: [rank10]:     new_module = self._create_new_module(lora_config, adapter_name, target, **kwargs)
10.252.32.13: [rank10]:   File "/home/app.e0016372/miniconda3/envs/llmtuner/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 340, in _create_new_module
10.252.32.13: [rank10]:     new_module = dispatcher(target, adapter_name, lora_config=lora_config, **kwargs)
10.252.32.13: [rank10]:   File "/home/app.e0016372/miniconda3/envs/llmtuner/lib/python3.10/site-packages/peft/tuners/lora/bnb.py", line 273, in dispatch_bnb_8bit
10.252.32.13: [rank10]:     "memory_efficient_backward": target.state.memory_efficient_backward,
10.252.32.13: [rank10]: AttributeError: 'MatmulLtState' object has no attribute 'memory_efficient_backward'

[hiyouga]

see huggingface/peft#2245, try install bitsandbytes==0.43.0