Skip to content

feat: add Qwen2.5 0.5B Compatibility #282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kjh2159
Copy link

@kjh2159 kjh2159 commented May 20, 2025

1. Description

The current mllm cannot handle Qwen2.5 0.5B models.
For example, the existing mllm throws dummy words even though the model is FP32.
However, now QwenConfig can handle Qwen2.5 0.5B models.

Basically, this modified QwenConfig class and its member functions do not harm the compatibility of the existing functionalities.

Compatible Quantized Models: fp32, q4_0

2. Usage

The default Qwen model version of QwenConfig is 1.5.
To use Qwen2.5 0.5B models, you can add only "Qwen2.5" string to the arguments when constructing "QwenConfig" object.

// demo_qwen.cpp

// model_version can be "qwen2.5" or "Qwen2.5"
// default model_version is "Qwen1.5"
QWenConfig config(tokens_limit, model_billion, RoPEType::HFHUBROPE, model_version);

// ex)
// QWenConfig config(512, "0.5B", RoPEType::HFHUBROPE, "Qwen2.5");
# execution command example
./bin/demo_qwen \
    -m qwen2.5-0.5b-instruct-q4_0.mllm \
    -v vocab/qwen2.5_vocab.mllm \
    -e vocab/qwen2.5_merges.txt \
    -b 0.5B \
    -r Qwen2.5

3. Model Files

I utilized the quantization and converters that mllm team provides.
And, the files for deployment of qwen2.5-0.5b models are uploaded on my huggingface repository and you can download the file with curl command.

Click here for the huggingface repository.

Execute the following command to download it.

curl -L https://huggingface.co/kjh2159/Qwen2.5-0.5B-Instruct-MLLM/resolve/main/qwen2.5-0.5b-instruct_q4_0.mllm --output qwen2.5-0.5b-instruct_q4_0.mllm

Now QwenConfig can handle Qwen2.5 0.5B models
Compatibility: fp32, q4_0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant