Skip to content

Update Sentencepiece Tokenizer - #703

Open
apaniukov wants to merge 6 commits into
openvinotoolkit:masterfrom
apaniukov:update-sentencepiece-022
Open

Update Sentencepiece Tokenizer#703
apaniukov wants to merge 6 commits into
openvinotoolkit:masterfrom
apaniukov:update-sentencepiece-022

Conversation

@apaniukov

Copy link
Copy Markdown
Collaborator
  • Upgraded SentencePiece to 0.2.2 for both Python and C++ builds, including CMake URL/hash, dependency metadata, and lockfile updates.

  • Updated C++ integration for SentencePiece 0.2.2: switched removed logging API usage to Abseil logging, adjusted Abseil provider configuration, and hardened precompiled charsmap generation status handling.

  • Fixed SentencePiece/HF normalization regressions:

    • CharsmapStep now defaults remove_extra_whitespaces=False.
    • HF Precompiled normalizers preserve whitespace.
    • WhitespaceSplit conversion now matches HF behavior with trim + whitespace collapse + split.
  • Fixed SentencePiece runtime errors for tokenizer conversion by preserving BOS/EOS ids when mutating tokenizer-side SentencePiece models.

  • Removed pytest-harvest from dev dependencies and replaced README coverage generation with pytest terminalreporter stats, making --update_readme -n auto work reliably under xdist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant