[EXPERIMENT][WIP] fix(VocabDecoderStep): avoid zero-dim Const when skip_special_tokens=False - #677
Draft
mlukasze wants to merge 1 commit into
Draft
Conversation
…False
When skip_special_tokens=False (do_skip_tokens=False), the previous code
used a Slice with stop=0 to produce an empty output, which constant-folds
to a zero-dimension Constant node as VocabDecoder port-4. This zero-dim
Const triggers a crash in the Intel CPU plugin:
Check 'edge' failed at node.cpp:712:
Node ReadValue_XXXX contains dead weak ptr
when the detokenizer is loaded alongside a stateful language model in
VLMPipeline (KV-cache ReadValue/Assign nodes have their Variable registry
corrupted by the zero-dim Const compilation).
Fix: when do_skip_tokens=False, simply omit port-4 entirely. VocabDecoder
already supports 4-input form (falls back to empty m_skip_tokens attribute
== skip nothing), so this is semantically correct and avoids the crash.
This also removes the never-used "runtime on/off toggle" mechanism
(it was broken: a constant stop value means no actual runtime switch).
Tested with: opendatalab/MinerU2.5-Pro-2604-1.2B (Qwen2-based VLM)
which requires skip_special_tokens=False to preserve document-structure
tokens (<fcel>, <nl>, <|md_start|>, etc.) in VLMPipeline output.
Collaborator
|
Deleting special tokens constant from the graph will break this functionality. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When converting a detokenizer with
skip_special_tokens=False,VocabDecoderStep.get_ov_subgraph()builds aSlicewithstop=0, which constant-folds to a zero-dimensionalConstwired intoVocabDecoderport-4 (the skip-tokens list). When such a detokenizer is loaded together with a stateful model that has KV-cacheReadValue/Assignnodes (e.g. viaopenvino_genai.VLMPipeline/LLMPipeline), the Intel CPU plugin crashes with:Fix
When
do_skip_tokens=False, omit port-4 entirely and use the 4-inputVocabDecoderform (the op already treats a missing skip-tokens input as "skip nothing" via the emptym_skip_tokensattribute), instead of creating a zero-dimConst. This eliminates the zero-dim constant that triggers the plugin crash, with no behavioural change (skip_special_tokens=False already means "skip nothing").Test
Adds a regression test asserting that a detokenizer converted with
skip_special_tokens=Falseproduces no zero-dimensionalConstfeeding theVocabDecoder, and round-trips special tokens correctly.Context
Found while enabling
opendatalab/MinerU2.5-Pro-2604-1.2B(a Qwen2-VL document-parsing VLM) for OpenVINO. That model relies on custom structural special tokens (<fcel>,<ched>,<|md_start|>, …) in its generated Markdown;skip_special_tokens=Falseis required to preserve them inVLMPipelineoutput.Follow-up
A separate hardening could be made in the OpenVINO CPU plugin to tolerate zero-dim
Constinputs to custom ops gracefully rather than crashing; tracked as a follow-up issue. This PR fixes the issue at the tokenizer-graph layer, which is the correct place.