Broken Qwen2 7B Vision SFT Colab?: 'Qwen2VLCausalLMOutputWithPast' has no attribute 'forward'

[Any-Winter-4079]

Installing latest version of unsloth:

!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git

breaks the Qwen2 7B Vision Colab it seems(?)

from unsloth import FastVisionModel # FastLanguageModel for LLMs
import torch

# 4bit pre quantized models we support for 4x faster downloading + no OOMs.
fourbit_models = [
    "unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit", # Llama 3.2 vision support
    "unsloth/Llama-3.2-11B-Vision-bnb-4bit",
    "unsloth/Llama-3.2-90B-Vision-Instruct-bnb-4bit", # Can fit in a 80GB card!
    "unsloth/Llama-3.2-90B-Vision-bnb-4bit",

    "unsloth/Pixtral-12B-2409-bnb-4bit",              # Pixtral fits in 16GB!
    "unsloth/Pixtral-12B-Base-2409-bnb-4bit",         # Pixtral base model

    "unsloth/Qwen2-VL-2B-Instruct-bnb-4bit",          # Qwen2 VL support
    "unsloth/Qwen2-VL-7B-Instruct-bnb-4bit",
    "unsloth/Qwen2-VL-72B-Instruct-bnb-4bit",

    "unsloth/llava-v1.6-mistral-7b-hf-bnb-4bit",      # Any Llava variant works!
    "unsloth/llava-1.5-7b-hf-bnb-4bit",
] # More models at https://huggingface.co/unsloth

model, tokenizer = FastVisionModel.from_pretrained(
    "unsloth/Qwen2-VL-7B-Instruct",
    load_in_4bit = True, # Use 4bit to reduce memory use. False for 16bit LoRA.
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for long context
)

Leading to:

🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
🦥 Unsloth Zoo will now patch everything to make training faster!
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-2-dd19b029c094>](https://localhost:8080/#) in <cell line: 22>()
     20 ] # More models at https://huggingface.co/unsloth
     21 
---> 22 model, tokenizer = FastVisionModel.from_pretrained(
     23     "unsloth/Qwen2-VL-7B-Instruct",
     24     load_in_4bit = True, # Use 4bit to reduce memory use. False for 16bit LoRA.

3 frames
[/usr/local/lib/python3.10/dist-packages/unsloth_zoo/compiler.py](https://localhost:8080/#) in patch_gradient_accumulation(modeling_file, module)
    789     functions = dir(modeling_file)
    790     module = eval(f"modeling_file.{module}")
--> 791     forward = module.forward
    792     source = inspect.getsource(forward)
    793     has_kwargs = tuple(inspect.signature(forward).parameters.values())[-1].kind == inspect._VAR_KEYWORD

AttributeError: type object 'Qwen2VLCausalLMOutputWithPast' has no attribute 'forward'

[Any-Winter-4079]

Never mind. It seems it works with the latest commit, but for some reason, the first time the cell is run fails, and after re-running it, works. Very weird, but it still works.

[Any-Winter-4079]

I'm reopening the issue because I've tried to run it on Runpod, and I get the same issue:

...
  File "/usr/local/lib/python3.12/dist-packages/unsloth_zoo/compiler.py", line 791, in patch_gradient_accumulation
    forward = module.forward
              ^^^^^^^^^^^^^^
AttributeError: type object 'Qwen2VLCausalLMOutputWithPast' has no attribute 'forward'

with the way to fix it being

try:
    model, tokenizer = FastVisionModel.from_pretrained(
        f"{username}/{model_name}",
        load_in_4bit = False, # Use 4bit to reduce memory use. False for 16bit LoRA.
        use_gradient_checkpointing = "unsloth", # True or "unsloth" for long context
    )
except:
    try:
        model, tokenizer = FastVisionModel.from_pretrained(
            f"{username}/{model_name}",
            load_in_4bit = False, # Use 4bit to reduce memory use. False for 16bit LoRA.
            use_gradient_checkpointing = "unsloth", # True or "unsloth" for long context
    )
    except:
        print('error')

i.e. have the model loading part run twice, which is a bit of a hack (and while it's more okay on Colab, on scripts running on the cloud it's more of a pain)...
So maybe someone has come across this error or knows the issue?
Am I doing something wrong? Is there a small bug somewhere?

[anindyamitra2002]

same issue here

[kaykyr]

I'm facing same issue here! It wasn't breaking things like that... Maybe because latest commits/updates

[kaykyr]

In my case:

🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
🦥 Unsloth Zoo will now patch everything to make training faster!
Traceback (most recent call last):
  File "/ors/workdir/aura-omini/train.py", line 12, in <module>
    model, tokenizer = FastVisionModel.from_pretrained(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/unsloth_env2/lib/python3.11/site-packages/unsloth/models/loader.py", line 459, in from_pretrained
    model_types = unsloth_compile_transformers(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/unsloth_env2/lib/python3.11/site-packages/unsloth/models/_utils.py", line 1216, in unsloth_compile_transformers
    _unsloth_compile_transformers(
  File "/root/anaconda3/envs/unsloth_env2/lib/python3.11/site-packages/unsloth_zoo/compiler.py", line 1418, in unsloth_compile_transformers
    new_source = patch_gradient_accumulation(modeling_file, module)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/unsloth_env2/lib/python3.11/site-packages/unsloth_zoo/compiler.py", line 791, in patch_gradient_accumulation
    forward = module.forward
              ^^^^^^^^^^^^^^
AttributeError: type object 'Qwen2VLCausalLMOutputWithPast' has no attribute 'forward'

[kaykyr]

My previous enviroment unsloth_env still working...

[kaykyr]

I was able to rollback previous commits and find last working version of Unsloth:

python -m pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git@44185c473ba1b20c4f4a7bde2cd8abd7d30e2514" -U --force
python -m pip install "unsloth_zoo @ git+https://github.com/unslothai/unsloth-zoo.git@e9950f5c9895dc2cb1d6e7810713b810e6d94285" -U --force

After installation, go to:
/root/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/vision.py

Remove import "merge_and_overwrite_lora" and save.

Your algorithm will work again.

[ariefwijaya]

use pip install "unsloth==2024.12.11" until the problem solved

[developer0hye]

The latest version(=2024.12.7) of 'usloth-zoo' has an issue.

pip install "unsloth==2024.12.11"
pip install "unsloth-zoo==2024.12.6"