[wwb] Add support of visual embeddings model#4135
Draft
sbalandi wants to merge 1 commit into
Draft
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the Who-What Benchmark (WWB) tooling to evaluate visual embedding pipelines by adding support for image-embedding and video-embedding model types, including special handling for Qwen3-VL-style inputs.
Changes:
- Added
image-embeddingandvideo-embeddingas WWB--model-typeoptions and routed them through the embeddings evaluator path. - Updated embeddings evaluation to generate default datasets for text/image/video embedding and added a Qwen3-VL embedding wrapper/preprocessing path.
- Updated embedding model loading to use the newer GenAI embedding pipeline when available and added a wrapper hook for Qwen3-VL.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/who_what_benchmark/whowhatbench/wwb.py | Adds image/video embedding model types and a GenAI embedding generation path for visual inputs. |
| tools/who_what_benchmark/whowhatbench/whowhat_metrics.py | Adds shape checking for embedding comparisons (but currently includes debug prints). |
| tools/who_what_benchmark/whowhatbench/utils.py | Extends video dataset items to include videos_metadata. |
| tools/who_what_benchmark/whowhatbench/model_loaders.py | Introduces embedding wrapper hook and switches to GenAI embedding pipeline API when available. |
| tools/who_what_benchmark/whowhatbench/embeddings_evaluator.py | Extends embeddings evaluator to handle text/image/video inputs and adds Qwen3-VL preprocessing/pooling support. |
Comment on lines
139
to
142
| "text-embedding", | ||
| "image-embedding", | ||
| "video-embedding", | ||
| "text-reranking", |
Comment on lines
139
to
142
| "text-embedding", | ||
| "image-embedding", | ||
| "video-embedding", | ||
| "text-reranking", |
Comment on lines
+937
to
+938
| if prompt: | ||
| media_inputs["embedding_prompt"] = prompt[0] |
Comment on lines
+945
to
+956
| videos, videos_metadata = videos_info.values() | ||
| if videos is not None: | ||
| media_inputs["videos"] = [] | ||
| for i, video in enumerate(videos): | ||
| import openvino_genai | ||
|
|
||
| media_inputs["videos"].append(ov.Tensor(np.stack(video, axis=0))) | ||
| video_metadata = openvino_genai.VideoMetadata() | ||
| video_metadata.fps = videos_metadata[i]["fps"] | ||
| video_metadata.frames_indices = range(10) | ||
| media_inputs["videos_metadata"] = [video_metadata] | ||
|
|
Comment on lines
+273
to
+275
| print("gold_data shape:", gold_data.shape, "prediction_data shape:", prediction_data.shape) | ||
| print("gold_data shape:", gold_data, "prediction_data shape:", prediction_data) | ||
|
|
Comment on lines
+51
to
+53
| def prepare_default_data_video(num_samples=None, num_frames=10): | ||
| items = prepare_video_dataset(num_samples=None, num_frames=10) | ||
| data = [] |
Comment on lines
307
to
316
| if self.test_data: | ||
| if isinstance(self.test_data, str): | ||
| data = pd.read_csv(self.test_data) | ||
| else: | ||
| if isinstance(self.test_data, dict): | ||
| assert "prompts" in self.test_data | ||
| if not ("passages" in self.test_data or "images" in self.test_data or "videos" in self.test_data): | ||
| raise RuntimeError("Test data must contain 'passages' or 'images' or 'videos' keys") | ||
| data = dict(self.test_data) | ||
| else: | ||
| data = {"prompts": list(self.test_data)} | ||
| data = pd.DataFrame.from_dict(data) | ||
| else: |
| if self.num_samples is None | ||
| else data.values[: self.num_samples] | ||
| ) | ||
| prompt = (Qwen3VLEmbeddingWrapper.system_prompt if Qwen3VLEmbeddingWrapper.is_qwen3_vl_model(model) else None,) |
Comment on lines
+354
to
+370
| kwargs = {} | ||
| text_data_input = None | ||
| if texts_input: | ||
| kwargs = { | ||
| "padding_side": self.padding_side, | ||
| "pooling_type": self.pooling_type, | ||
| "normalize": self.normalize, | ||
| } | ||
| batch_size = self.batch_size or len(texts_input) | ||
| data_len = len(texts_input) | ||
|
|
||
| if batch_size <= data_len: | ||
| text_data_input = texts_input[:batch_size] | ||
| else: | ||
| # Duplicate data to reach batch_size | ||
| text_data_input = list(itertools.islice(itertools.cycle(texts_input), batch_size)) | ||
| passages.append(text_data_input) |
Comment on lines
958
to
962
| elif model_type == "text-embedding": | ||
| return load_embedding_model(model_id, device, ov_options, use_hf, use_genai, **sanitized_kwargs) | ||
| elif model_type == "text-embedding" or model_type == "image-embedding" or model_type == "video-embedding": | ||
| return load_embedding_model(model_id, device, ov_options, use_hf, use_genai, **kwargs) | ||
| elif model_type == "text-reranking": |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
CVS-###
Fixes #(issue)
Checklist: