Qwen3-VL-4B-IT lora_vision sft

[nx3782]

Thanks so much for sharing this amazing fine-tuning scripts on the github. Just curious if I can also fine-tune "MODEL_NAME="Qwen/Qwen3-VL-4B-Instruct"" with this "finetune_lora_vision.sh" script.

Also is there a way to monitor the training and validation loss by chance?

Thanks again.

[2U1]

Hi, thanks for the kind words!
Yes, Qwen/Qwen3-VL-4B-Instruct is supported. In scripts/finetune_lora_vision.sh, change:

MODEL_NAME="Qwen/Qwen3-VL-4B-Instruct"

Since Qwen3-VL uses a 32×32 pixel unit, please also update:

--image_min_pixels $((256 * 32 * 32)) \
--image_max_pixels $((1280 * 32 * 32)) \

The script already logs the training loss through TensorBoard using:

--logging_steps 1 \
--report_to tensorboard \

To monitor validation loss as well, prepare a validation JSON file in the same format as the training data and add:

--eval_path /path/to/your/validation/data.json \
--eval_strategy "steps" \
--eval_steps 200 \
--per_device_eval_batch_size 1 \
--logging_dir output/lora_vision_test/tensorboard \

Then launch TensorBoard:

tensorboard --logdir output/lora_vision_test/tensorboard

You should then be able to monitor both loss and eval_loss. Please note that validation adds some training overhead.
Hope this helps!

[nx3782]

Thanks a lot. Just a quick question here. When setting True for "lora_enable" and "vision_lora", you are freezing the base weights for the vision tower and llm backbone and perform lora update on weights for the following, right?

--freeze_vision_tower True
--freeze_llm True \