Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions site/docs/getting-started/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ This library is friendly to PC and laptop execution, and optimized for resource

## Key Features and Benefits

- **📦 Pre-built Generative AI Pipelines:** Ready-to-use pipelines for text generation (LLMs), image generation (Diffuser-based), speech recognition (Whisper), and visual language models (VLMs). See all [supported use cases](/docs/category/use-cases).
- **📦 Pre-built Generative AI Pipelines:** Ready-to-use pipelines for text generation (LLMs), image generation (Diffuser-based), automatic speech recognition, and visual language models (VLMs). See all [supported use cases](/docs/category/use-cases).
- **👣 Minimal Footprint:** Smaller binary size and reduced memory footprint compared to other frameworks.
- **🚀 Performance Optimization:** Hardware-specific optimizations for CPU, GPU, and NPU devices.
- **👨‍💻 Programming Language Support:** Comprehensive APIs in Python, C++, and Node.js.
Expand All @@ -37,7 +37,7 @@ Using OpenVINO GenAI typically involves three main steps:
Refer to [Model Preparation](/docs/category/model-preparation) for more details.

:::
2. **Pipeline Setup:** Initialize the appropriate pipeline for your task (`LLMPipeline`, `Text2ImagePipeline`, `WhisperPipeline`, `VLMPipeline`, etc.) with the converted model.
2. **Pipeline Setup:** Initialize the appropriate pipeline for your task (`LLMPipeline`, `Text2ImagePipeline`, `ASRPipeline`, `VLMPipeline`, etc.) with the converted model.
3. **Inference:** Run the model with your inputs using the pipeline's simple API.

![OpenVINO GenAI Workflow](/img/openvino-genai-workflow.svg)
Expand Down
2 changes: 1 addition & 1 deletion site/docs/guides/streaming.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 3
For more interactive UIs during generation, you can stream output tokens.

:::info
Streaming is supported for `LLMPipeline`, `VLMPipeline` and `WhisperPipeline`.
Streaming is supported for `LLMPipeline`, `VLMPipeline` and `ASRPipeline`.
:::

## Streaming Function
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,18 @@ type WhisperModelType = {
};

export const WHISPER_MODELS: WhisperModelType[] = [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also rename this?

Suggested change
export const WHISPER_MODELS: WhisperModelType[] = [
export const SPEECH_RECOGNITION_MODELS: SpeechRecognitionModelType[] = [

or

Suggested change
export const WHISPER_MODELS: WhisperModelType[] = [
export const ASR_MODELS: ASRModelType[] = [

{
architecture: 'Qwen3ASRForConditionalGeneration',
models: [
{
name: 'Qwen3-ASR',
links: [
'https://huggingface.co/Qwen/Qwen3-ASR-0.6B',
'https://huggingface.co/Qwen/Qwen3-ASR-1.7B',
],
},
],
},
Comment on lines 9 to +21
{
architecture: 'WhisperForConditionalGeneration',
models: [
Expand Down
10 changes: 9 additions & 1 deletion site/docs/supported-models/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -107,14 +107,22 @@ The model requires `transformers==5.2` for the export with `optimum-cli`.
The model requires `transformers>=4.48` for the export with `optimum-cli`.
:::

## Speech Recognition Models (Whisper-based)
## Speech Recognition Models

:::info LoRA Support
Speech recognition pipeline does **not** support LoRA adapters.
:::

<WhisperModelsTable />

:::warning Speech Recognition Models Notes
Qwen3-ASR models require the `qwen-asr` package for export with `optimum-cli`:

```bash
pip install qwen-asr
```
:::

## Speech Generation Models

:::info LoRA Support
Expand Down
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
import CodeBlock from '@theme/CodeBlock';

<CodeBlock language="cpp" showLineNumbers>
{`#include "openvino/genai/whisper_pipeline.hpp"
{`#include "openvino/genai/automatic_speech_recognition/pipeline.hpp"
#include "audio_utils.hpp"
#include <iostream>

int main(int argc, char* argv[]) {
std::filesystem::path models_path = argv[1];
std::string wav_file_path = argv[2];

ov::genai::RawSpeechInput raw_speech = utils::audio::read_wav(wav_file_path);
std::vector<float> raw_speech = utils::audio::read_wav(wav_file_path);

ov::genai::WhisperPipeline pipe(models_path, "${props.device || 'CPU'}");
ov::genai::ASRPipeline pipe(models_path, "${props.device || 'CPU'}");
auto result = pipe.generate(raw_speech, ov::genai::max_new_tokens(100));
std::cout << result << std::endl;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ def read_wav(filepath):

raw_speech = read_wav('sample.wav')

pipe = ov_genai.WhisperPipeline(model_path, "${props.device || 'CPU'}")
pipe = ov_genai.ASRPipeline(model_path, "${props.device || 'CPU'}")
result = pipe.generate(raw_speech, max_new_tokens=100)
print(result)
`}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@ import CodeExampleJS from './_code_example_js.mdx';

## Run Model Using OpenVINO GenAI

OpenVINO GenAI introduces the [`WhisperPipeline`](https://docs.openvino.ai/2026/api/genai_api/_autosummary/openvino_genai.WhisperPipeline.html) pipeline for inference of speech recognition Whisper models.
OpenVINO GenAI introduces the [`ASRPipeline`](https://docs.openvino.ai/2026/api/genai_api/_autosummary/openvino_genai.ASRPipeline.html) pipeline for automatic speech recognition models.
You can construct it straight away from the folder with the converted model.
It will automatically load the model, tokenizer, detokenizer and default generation configuration.

:::info
`WhisperPipeline` expects normalized audio files in WAV format at sampling rate of 16 kHz as input.
`ASRPipeline` expects normalized audio files in WAV format at sampling rate of 16 kHz as input.
:::

<LanguageTabs>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import Streaming from '@site/docs/use-cases/_shared/_streaming.mdx';
## Additional Usage Options

:::tip
Check out [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/whisper_speech_recognition), [C++](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/whisper_speech_recognition), and [JavaScript](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/js/whisper_speech_recognition) Whisper speech recognition samples.
Check out [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/automatic_speech_recognition), [C++](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/automatic_speech_recognition), and [JavaScript](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/js/whisper_speech_recognition) speech recognition samples.
:::

### Use Different Generation Parameters
Expand All @@ -19,7 +19,7 @@ Check out [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master
```python
import openvino_genai as ov_genai

pipe = ov_genai.WhisperPipeline(model_path, "CPU")
pipe = ov_genai.ASRPipeline(model_path, "CPU")

# Get default configuration
config = pipe.get_generation_config()
Expand All @@ -38,7 +38,7 @@ Check out [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master
<TabItemCpp>
```cpp
int main() {
ov::genai::WhisperPipeline pipe(model_path, "CPU");
ov::genai::ASRPipeline pipe(model_path, "CPU");

// Get default configuration
auto config = pipe.get_generation_config();
Expand Down Expand Up @@ -87,7 +87,7 @@ Check out [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master
```python
import openvino_genai as ov_genai

pipe = ov_genai.WhisperPipeline(model_path, "CPU")
pipe = ov_genai.ASRPipeline(model_path, "CPU")

# Get default generation config
config = pipe.get_generation_config()
Expand All @@ -105,10 +105,10 @@ Check out [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master
<TabItemCpp>
```cpp
int main() {
ov::genai::WhisperPipeline pipe(model_path, "CPU");
ov::genai::ASRPipeline pipe(model_path, "CPU");

// Get default generation config
ov::genai::GenerationConfig config = pipe.get_generation_config();
ov::genai::ASRGenerationConfig config = pipe.get_generation_config();

// Modify parameters
config.max_new_tokens = 256;
Expand Down Expand Up @@ -144,47 +144,51 @@ Check out [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master
</LanguageTabs>
</BeamSearchGeneration>

### Transcription
:::note
Beam search support depends on the selected model architecture. Qwen3-ASR models do not support beam search decoding.
:::

#### Transcription

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing header L3 to L4 might exclude it from right side page navigation, please double check.


Whisper models can automatically detect the language of the input audio, or you can specify the language to improve accuracy. The detected (or specified) language is always available in the result via the `language` field:
ASR models can automatically detect the language of the input audio, or you can specify the language to improve accuracy. The detected (or specified) language is available in the result via the `languages` field:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is no abbreviation description in the page above (before this section).

Suggested change
ASR models can automatically detect the language of the input audio, or you can specify the language to improve accuracy. The detected (or specified) language is available in the result via the `languages` field:
Automatic speech recognition (ASR) models can automatically detect the language of the input audio, or you can specify the language to improve accuracy. The detected (or specified) language is available in the result via the `languages` field:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it is better to have "Automatic speech recognition (ASR)" in the beginning of the page (or even in page header) so we can refer just "ASR models" further on the page.


<LanguageTabs>
<TabItemPython>
```python
pipe = ov_genai.WhisperPipeline(model_path, "CPU")
pipe = ov_genai.ASRPipeline(model_path, "CPU")

# Automatic language detection
raw_speech = read_wav("speech_sample.wav")
result = pipe.generate(raw_speech)
print(result.language) # e.g. "en", "fr", "de"
print(result.languages[0]) # e.g. "en", "fr", "de"

# Explicitly specify language (English)
# Explicitly specify language (English). Use "English" for Qwen3-ASR models.
result = pipe.generate(raw_speech, language="<|en|>")
print(result.language) # "en"
print(result.languages[0]) # "en"

# French speech sample
raw_speech = read_wav("french_sample.wav")
result = pipe.generate(raw_speech, language="<|fr|>")
print(result.language) # "fr"
print(result.languages[0]) # "fr"
```
</TabItemPython>
<TabItemCpp>
```cpp
int main() {
ov::genai::WhisperPipeline pipe(model_path, "CPU");
ov::genai::ASRPipeline pipe(model_path, "CPU");

// Automatic language detection
auto result = pipe.generate(raw_speech);
std::cout << result.language << "\n"; // e.g. "en", "fr", "de"
std::cout << result.languages[0] << "\n"; // e.g. "en", "fr", "de"

// Explicitly specify language (English)
// Explicitly specify language (English). Use "English" for Qwen3-ASR models.
result = pipe.generate(raw_speech, ov::genai::language("<|en|>"));
std::cout << result.language << "\n"; // "en"
std::cout << result.languages[0] << "\n"; // "en"

// French speech sample
raw_speech = utils::audio::read_wav("french_sample.wav");
result = pipe.generate(raw_speech, ov::genai::language("<|fr|>"));
std::cout << result.language << "\n"; // "fr"
std::cout << result.languages[0] << "\n"; // "fr"
}
```
</TabItemCpp>
Expand All @@ -206,15 +210,21 @@ Whisper models can automatically detect the language of the input audio, or you
</TabItemJS>
</LanguageTabs>

### Translation
### Whisper-specific Features

:::info
These features are specific to Whisper models used with `ASRPipeline`. They may not apply to other ASR model architectures.
:::

#### Translation

By default, Whisper performs transcription, keeping the output in the same language as the input.
To translate non-English speech to English, use the `translate` task:

<LanguageTabs>
<TabItemPython>
```python
pipe = ov_genai.WhisperPipeline(model_path, "CPU")
pipe = ov_genai.ASRPipeline(model_path, "CPU")

# Translate French audio to English
raw_speech = read_wav("french_sample.wav")
Expand All @@ -224,7 +234,7 @@ To translate non-English speech to English, use the `translate` task:
<TabItemCpp>
```cpp
int main() {
ov::genai::WhisperPipeline pipe(model_path, "CPU");
ov::genai::ASRPipeline pipe(model_path, "CPU");

// Translate French audio to English
raw_speech = utils::audio::read_wav("french_sample.wav");
Expand All @@ -243,33 +253,33 @@ To translate non-English speech to English, use the `translate` task:
</TabItemJS>
</LanguageTabs>

### Timestamps Prediction
#### Timestamps Prediction

Whisper can predict timestamps for each segment of speech, which is useful for synchronization or creating subtitles:

<LanguageTabs>
<TabItemPython>
```python
pipe = ov_genai.WhisperPipeline(model_path, "CPU")
pipe = ov_genai.ASRPipeline(model_path, "CPU")

# Enable timestamp prediction
result = pipe.generate(raw_speech, return_timestamps=True)

# Print timestamps and text segments
for chunk in result.chunks:
for chunk in result.chunks[0]:
print(f"timestamps: [{chunk.start_ts:.2f}, {chunk.end_ts:.2f}] text: {chunk.text}")
```
</TabItemPython>
<TabItemCpp>
```cpp
int main() {
ov::genai::WhisperPipeline pipe(model_path, "CPU");
ov::genai::ASRPipeline pipe(model_path, "CPU");

// Enable timestamp prediction
result = pipe.generate(raw_speech, ov::genai::return_timestamps(true));

// Print timestamps and text segments
for (auto& chunk : *result.chunks) {
for (const auto& chunk : (*result.chunks)[0]) {
std::cout << "timestamps: [" << chunk.start_ts << ", " << chunk.end_ts
<< "] text: " << chunk.text << "\n";
}
Expand All @@ -292,7 +302,7 @@ Whisper can predict timestamps for each segment of speech, which is useful for s
</TabItemJS>
</LanguageTabs>

### Word-level Timestamps Prediction
#### Word-level Timestamps Prediction

Whisper can predict timestamps for each word of speech, which provides more granular timing information compared to segment-level timestamps.

Expand All @@ -301,30 +311,30 @@ Whisper can predict timestamps for each word of speech, which provides more gran
```python
# Word timestamps require decomposition of cross-attention decoder SDPA layers,
# so word_timestamps must be passed to the pipeline constructor (not just in generation config)
pipe = openvino_genai.WhisperPipeline(model_path, "CPU", word_timestamps=True)
pipe = openvino_genai.ASRPipeline(model_path, "CPU", word_timestamps=True)

# Enable word-level timestamp prediction
result = pipe.generate(raw_speech, word_timestamps=True)

# Print word-level timestamps
for word in result.words:
print(f"[{word.start_ts:.2f}, {word.end_ts:.2f}]: {word.word}")
for word in result.words[0]:
print(f"[{word.start_ts:.2f}, {word.end_ts:.2f}]: {word.text}")
```
</TabItemPython>
<TabItemCpp>
```cpp
int main() {
// Word timestamps require decomposition of cross-attention decoder SDPA layers,
// so word_timestamps must be passed to the pipeline constructor (not just in generation config)
ov::genai::WhisperPipeline pipeline(model_path, "CPU", ov::genai::word_timestamps(true));
ov::genai::ASRPipeline pipeline(model_path, "CPU", ov::genai::word_timestamps(true));

// Enable word-level timestamp prediction
auto result = pipeline.generate(raw_speech, ov::genai::word_timestamps(true));

// Print word-level timestamps
std::cout << std::fixed << std::setprecision(2);
for (auto& word : *result.words) {
std::cout << "[" << word.start_ts << ", " << word.end_ts << "]: " << word.word << "\n";
for (const auto& word : (*result.words)[0]) {
std::cout << "[" << word.start_ts << ", " << word.end_ts << "]: " << word.text << "\n";
}
}
```
Expand All @@ -347,23 +357,18 @@ Whisper can predict timestamps for each word of speech, which provides more gran
</TabItemJS>
</LanguageTabs>

:::info
NPU device requires `STATIC_PIPELINE=True` property passed to `WhisperPipeline` constructor:
`openvino_genai.WhisperPipeline(model_path, "NPU", word_timestamps=True, STATIC_PIPELINE=True)`
:::

### Long-Form Audio Processing
#### Long-Form Audio Processing

Whisper models are designed for audio segments up to 30 seconds in length.
For longer audio, the OpenVINO GenAI Whisper pipeline automatically handles the processing using a sequential chunking algorithm ("sliding window"):
For longer audio, the OpenVINO GenAI ASR pipeline automatically handles Whisper processing using a sequential chunking algorithm ("sliding window"):

1. The audio is divided into 30-second segments
2. Each segment is processed sequentially
3. Results are combined to produce the complete transcription

This happens automatically when you input longer audio files.

### Using Initial Prompts and Hotwords
#### Using Initial Prompts and Hotwords

You can improve transcription quality and guide the model's output style by providing initial prompts or hotwords using the following parameters:

Expand All @@ -377,7 +382,7 @@ Such prompts can be used to steer the model to use particular spellings or style
<LanguageTabs>
<TabItemPython>
```python
pipe = ov_genai.WhisperPipeline(model_path, "CPU")
pipe = ov_genai.ASRPipeline(model_path, "CPU")

result = pipe.generate(raw_speech)
# He has gone and gone for good answered Paul Icrom who...
Expand All @@ -389,7 +394,7 @@ Such prompts can be used to steer the model to use particular spellings or style
<TabItemCpp>
```cpp
int main() {
ov::genai::WhisperPipeline pipe(model_path, "CPU");
ov::genai::ASRPipeline pipe(model_path, "CPU");

auto result = pipeline.generate(raw_speech);
Comment on lines +397 to 399
// He has gone and gone for good answered Paul Icrom who...
Expand All @@ -414,7 +419,7 @@ Such prompts can be used to steer the model to use particular spellings or style
</LanguageTabs>

:::info
For the full list of Whisper generation parameters, refer to the [Whisper Generation Config API](https://docs.openvino.ai/2026/api/genai_api/_autosummary/openvino_genai.WhisperGenerationConfig.html).
For the ASR API, refer to the [ASR Generation Config API](https://docs.openvino.ai/2026/api/genai_api/_autosummary/openvino_genai.ASRGenerationConfig.html).
:::

<Streaming />
Loading
Loading