[Whisper]: Split on unicode non whitespace languages#4131
Open
as-suvorov wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses incorrect Whisper word-level timestamps for no-space (scriptio continua) languages by making the token-to-word grouping logic language-aware and by propagating detected language into the word-timestamping path.
Changes:
- Updates
add_word_level_timestamps()to acceptSotTokensResult(tokens + detected language) instead of rawsot_tokens. - Adds language-dependent splitting: uses unicode-based splitting for
zh/ja/th/lo/my/yue, and space-based splitting otherwise. - Updates Whisper generation call sites to pass
SotTokensResultthrough to word-timestamp generation.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/cpp/src/whisper/word_level_timestamps.hpp |
Changes add_word_level_timestamps signatures to take SotTokensResult so language can influence splitting. |
src/cpp/src/whisper/word_level_timestamps.cpp |
Introduces language-aware token splitting and uses detected language when generating word-level timestamps. |
src/cpp/src/whisper/whisper.cpp |
Threads SotTokensResult through generation and into word-timestamp processing. |
src/cpp/src/whisper/pipeline_static.cpp |
Threads SotTokensResult through the static pipeline and into word-timestamp processing. |
Comment on lines
+410
to
+414
| bool is_no_space_language(const std::string& language) { | ||
| static const std::set<std::string> no_space_languages = {"zh", "ja", "th", "lo", "my", "yue"}; | ||
|
|
||
| return no_space_languages.count(language) > 0; | ||
| } |
Comment on lines
+418
to
+422
| split_to_word_tokens(const std::vector<int64_t>& tokens, ov::genai::Tokenizer& tokenizer, const std::string& language) { | ||
| if (is_no_space_language(language)) { | ||
| return split_tokens_on_unicode(tokens, tokenizer); | ||
| } | ||
| return split_tokens_on_spaces(tokens, tokenizer); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #4113
Checklist: