import BasicGenerationConfiguration from '@site/docs/use-cases/_shared/_basic_generation_configuration.mdx'; import ChatScenario from '@site/docs/use-cases/_shared/_chat_scenario.mdx'; import GenerationConfigurationWorkflow from '@site/docs/use-cases/_shared/_generation_configuration_workflow.mdx'; import Streaming from '@site/docs/use-cases/_shared/_streaming.mdx';
:::tip Check out Python, C++ and JavaScript visual language chat samples. :::
The prompt can contain <ov_genai_image_i> with i replaced with an actual zero based index to refer to an image. Reference to images used in previous prompts isn't implemented. A model's native image tag can be used instead of <ov_genai_image_i>. These tags are:
- InternVL2:
<image>\n - llava-1.5-7b-hf:
<image> - LLaVA-NeXT:
<image> - LLaVA-NeXT-Video:
<image> - nanoLLaVA:
<image>\n - nanoLLaVA-1.5:
<image>\n - MiniCPM-o-2_6:
<image>./</image>\n - MiniCPM-V-2_6:
<image>./</image>\n - Phi-3-vision:
<|image_i|>\n- the index starts with one - Phi-4-multimodal-instruct:
<|image_i|>\n- the index starts with one - Qwen2-VL:
<|vision_start|><|image_pad|><|vision_end|> - Qwen2.5-VL:
<|vision_start|><|image_pad|><|vision_end|> - Qwen3-VL:
<|vision_start|><|image_pad|><|vision_end|> - gemma-3-4b-it:
<start_of_image> - gemma4:
<|image|>
Model's native video tag can be used to refer to a video. These tags are:
- LLaVA-NeXT-Video:
<video> - Qwen2-VL:
<|vision_start|><|video_pad|><|vision_end|> - Qwen2.5-VL:
<|vision_start|><|video_pad|><|vision_end|> - Qwen3-VL:
<|vision_start|><|video_pad|><|vision_end|> - VideoChat_Flash_Qwen:
<|image_i|>\n- the index starts with one - gemma4:
<|video|>
If the prompt doesn't contain image or video tags, but images or videos are provided, the tags are prepended to the prompt.
Similar to text generation, VLM pipelines support various generation parameters to control the text output.
```python import openvino_genai as ov_genai pipe = ov_genai.VLMPipeline(model_path, "CPU")
# Get default configuration
config = pipe.get_generation_config()
# Modify parameters
config.max_new_tokens = 100
config.temperature = 0.7
config.top_k = 50
config.top_p = 0.9
config.repetition_penalty = 1.2
# Generate text with custom configuration
output = pipe.generate(prompt, images, config)
```
</TabItemPython>
<TabItemCpp>
```cpp
int main() {
ov::genai::VLMPipeline pipe(model_path, "CPU");
// Get default configuration
auto config = pipe.get_generation_config();
// Modify parameters
config.max_new_tokens = 100;
config.temperature = 0.7f;
config.top_k = 50;
config.top_p = 0.9f;
config.repetition_penalty = 1.2f;
// Generate text with custom configuration
auto output = pipe.generate(prompt, images, config);
}
```
</TabItemCpp>
<TabItemJS>
```javascript
import { VLMPipeline } from 'openvino-genai-node';
const pipe = await VLMPipeline(modelPath, "CPU", {});
// Create custom generation configuration
const config = {
max_new_tokens: 100,
temperature: 0.7,
top_k: 50,
top_p: 0.9,
repetition_penalty: 1.2
};
// Generate text with custom configuration
const output = await pipe.generate(prompt, {
images: images,
generationConfig: config
});
```
</TabItemJS>
When a video is provided as an input, video metadata can be passed along with the video to control the sampling of video frames.
Passing VideoMetadata with frames_indices defined enables sampling based on provided indices within the pipeline, and any model-specific sampling logic will be skipped (if defined).
Leave frames_indices empty to apply model-specific sampling (e.g. for Qwen3-VL).
Refer to the video-to-text Python and C++ samples for more details.
For Visual Language Models (VLMs), LoRA adapters can customize the generated text by applying adapters to the language-model (LLM) part. LoRA adapters that target the vision encoder or other multimodal components are not supported.
Refer to the LoRA Adapters guide for more details on working with LoRA adapters.