[Feature] Support llava onevision #2783

deepindeed2022 · 2024-11-21T03:06:45Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Support llava-onevision with LMDeploy

Modification

support llava-onevision series
refactor the llava related code for device_map
refactor llava_next.py with transformers new features

Use cases (Optional)

git clone https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov
lmdeploy serve api_server llava-onevision-qwen2-7b-ov

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
The documentation has been modified accordingly, like docstring or example tutorials.

docs/zh_cn/multi_modal/llava.md

lmdeploy/vl/model/llava.py

lvhan028 · 2024-11-25T14:10:53Z

Hi, @deepindeed2022 thanks for your contribution to LMDeploy
We are currently refactoring VLM modules in PR #2810, which conflicts to this PR.
I'd like to push the review of #2810 at first. After it is merged, I'll come back to this PR and resolve the conflicts.
Does it work for you?

deepindeed2022 · 2024-11-26T01:53:49Z

Hi, @deepindeed2022 thanks for your contribution to LMDeploy We are currently refactoring VLM modules in PR #2810, which conflicts to this PR. I'd like to push the review of #2810 at first. After it is merged, I'll come back to this PR and resolve the conflicts. Does it work for you?

Yes, I 'll follow up after this PR #2810 merge.

lmdeploy/vl/model/llava_next.py

lvhan028 · 2024-12-13T11:12:32Z

Hi, @deepindeed2022
It has been a while.
How are you doing?
We finally merged the VLM refactoring PR #2810
Would it be possible for you to keep working on this PR?

- update llava doc - fix multi-gpus load issue - support llava onevision qwen

deepindeed2022 · 2024-12-25T10:18:42Z

@lvhan028 the pr_ete_test failed with GPU resources. How to confirm CUDA_VISIBLE_DEVICES=5 is free or how to retry?

RunningLeon · 2024-12-26T02:48:38Z

@lvhan028 the pr_ete_test failed with GPU resources. How to confirm CUDA_VISIBLE_DEVICES=5 is free or how to retry?

@deepindeed2022 hi, there's an issue with pr_ete_test and the fixing pr will come soon. Pls ignore at the moment.

docs/en/multi_modal/llava.md

docs/zh_cn/multi_modal/llava.md

lvhan028 · 2025-01-06T10:23:19Z

lmdeploy/vl/model/llava.py

 from transformers import AutoConfig, AutoModelForCausalLM

 from lmdeploy.utils import get_logger
 from lmdeploy.vl.model.llava_hf import VISION_MODELS, LlavaHfVisionModel
-from lmdeploy.vl.model.utils import disable_logging, rewrite_ctx
+from lmdeploy.vl.model.utils import (disable_logging,
+                                     get_vision_encoder_device_map,


get_vision_encoder_device_map is shared by LlavaHfVisionModel, LlavaVisionModel and LlavaNextVisionModel
I think it's better to put this function in the base class LlavaHfVisionModel

This reverts commit c2fef11.

…mdeploy into support_llava_onevision

README.md

irexyc · 2025-01-08T05:58:32Z

lmdeploy/vl/model/llava_hf.py

+            return device_map
+
+        for keys in same_device_keys:
+            fuzzy_keys = [kk for kk in device_map for k in keys if kk.find(k)]


fuzzy_keys = [kk for kk in device_map for k in keys if kk.find(k) != -1] ?

after changing it to

fuzzy_keys = [kk for kk in device_map for k in keys if k in kk]

it suffers RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:2!

I will fix it later on

RunningLeon

tested ok on llavaonvision, llava, llava-hf

RunningLeon

LGTM

deepindeed2022 · 2025-01-15T02:12:42Z

Is there any problem with this PR ? When is it expected to be used directly in the official code repository ?

lvhan028 · 2025-01-15T04:22:40Z

Hi, @deepindeed2022
It hasn't passed our test yet.

deepindeed2022 changed the title ~~Support llava onevision~~ [Feature] Support llava onevision Nov 21, 2024

lvhan028 added the enhancement New feature or request label Nov 21, 2024

lvhan028 requested review from RunningLeon and irexyc November 21, 2024 11:03

RunningLeon reviewed Nov 22, 2024

View reviewed changes

docs/zh_cn/multi_modal/llava.md Outdated Show resolved Hide resolved

RunningLeon reviewed Nov 22, 2024

View reviewed changes

lmdeploy/vl/model/llava.py Show resolved Hide resolved

deepindeed2022 force-pushed the support_llava_onevision branch from c15538a to 948fffe Compare November 25, 2024 07:20

lvhan028 reviewed Nov 25, 2024

View reviewed changes

lmdeploy/vl/model/llava.py Show resolved Hide resolved

RunningLeon reviewed Nov 26, 2024

View reviewed changes

lmdeploy/vl/model/llava_next.py Outdated Show resolved Hide resolved

support llava-onevision-qwen

7b5a83d

- update llava doc - fix multi-gpus load issue - support llava onevision qwen

deepindeed2022 force-pushed the support_llava_onevision branch from 948fffe to 7b5a83d Compare December 24, 2024 11:51

update doc

468a750

lvhan028 reviewed Jan 6, 2025

View reviewed changes

docs/en/multi_modal/llava.md Show resolved Hide resolved

lvhan028 reviewed Jan 6, 2025

View reviewed changes

docs/zh_cn/multi_modal/llava.md Show resolved Hide resolved

lvhan028 reviewed Jan 6, 2025

View reviewed changes

lvhan028 and others added 7 commits January 6, 2025 21:20

Merge branch 'main' into PR-2783

523d09b

update

bedc917

Merge branch 'PR-2783' into support_llava_onevision

69843f4

test commit

c2fef11

Revert "test commit"

cecb103

This reverts commit c2fef11.

Merge branch 'support_llava_onevision' of github.com:deepindeed2022/l…

613ee66

…mdeploy into support_llava_onevision

remove get_vision_encoder_device_map from utils.py

ba82f2d

RunningLeon reviewed Jan 7, 2025

View reviewed changes

README.md Show resolved Hide resolved

update readme

1a8ea2a

irexyc reviewed Jan 8, 2025

View reviewed changes

RunningLeon reviewed Jan 8, 2025

View reviewed changes

RunningLeon approved these changes Jan 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Support llava onevision #2783

[Feature] Support llava onevision #2783

deepindeed2022 commented Nov 21, 2024 •

edited

Loading

lvhan028 commented Nov 25, 2024

deepindeed2022 commented Nov 26, 2024

lvhan028 commented Dec 13, 2024 •

edited

Loading

deepindeed2022 commented Dec 25, 2024

RunningLeon commented Dec 26, 2024

lvhan028 Jan 6, 2025

irexyc Jan 8, 2025

lvhan028 Jan 9, 2025

RunningLeon left a comment

RunningLeon left a comment

deepindeed2022 commented Jan 15, 2025

lvhan028 commented Jan 15, 2025

[Feature] Support llava onevision #2783

Are you sure you want to change the base?

[Feature] Support llava onevision #2783

Conversation

deepindeed2022 commented Nov 21, 2024 • edited Loading

Motivation

Modification

Use cases (Optional)

Checklist

lvhan028 commented Nov 25, 2024

deepindeed2022 commented Nov 26, 2024

lvhan028 commented Dec 13, 2024 • edited Loading

deepindeed2022 commented Dec 25, 2024

RunningLeon commented Dec 26, 2024

lvhan028 Jan 6, 2025

Choose a reason for hiding this comment

irexyc Jan 8, 2025

Choose a reason for hiding this comment

lvhan028 Jan 9, 2025

Choose a reason for hiding this comment

RunningLeon left a comment

Choose a reason for hiding this comment

RunningLeon left a comment

Choose a reason for hiding this comment

deepindeed2022 commented Jan 15, 2025

lvhan028 commented Jan 15, 2025

deepindeed2022 commented Nov 21, 2024 •

edited

Loading

lvhan028 commented Dec 13, 2024 •

edited

Loading