[EXPERIMENT][WIP] Add PiperTTSImpl backend for Piper (VITS) text-to-speech voices by mlukasze · Pull Request #4116 · openvinotoolkit/openvino.genai

mlukasze · 2026-07-08T17:21:35Z

⚠️ AUTOMATICALLY GENERATED BY OMEGA AGENT — REQUIRES HUMAN REVIEW ⚠️
This PR was created by an AI agent as part of automated model enablement.
A human maintainer must review and approve it before it can be considered for merge.
Do NOT merge without human review and sign-off.

Summary

Adds a new PiperTTSImpl backend to Text2SpeechPipeline, enabling
rhasspy/piper-voices — a catalog of 100+
single-speaker VITS-family text-to-speech voices, distributed as ONNX (non-HuggingFace
-Transformers architecture; converted to OpenVINO IR via OMEGA's custom-workflow path since
no optimum-cli export openvino path exists for this architecture).

Tracking ticket: openvinotoolkit/omega#53. Two representative voices validated end-to-end
in this change: en_US-joe-medium (English) and de_DE-thorsten-medium (German).

What's included

New src/cpp/src/speech_generation/piper_tts_model.{hpp,cpp}: single-shot (non
-autoregressive) VITS inference — loads phoneme_id_map from config.json, builds the
BOS/PAD/EOS phoneme-id sequence per Piper's convention, runs one inference pass
(input/input_lengths/scales → output), and rejects (throws on) non-empty
speaker_embedding since Piper voices are single-speaker.
SpeechGenerationConfig extended with Piper-specific noise_scale, length_scale,
noise_w (reuses the existing language field for the espeak-ng voice name — no new
field needed there).
Text2SpeechPipeline's dispatcher (resolve_backend) extended with an additive Piper
detection branch (keyed on architectures containing "Piper", or a phoneme_id_map
object in config.json) — zero changes to the existing SpeechT5/Kokoro detection logic.
misaki_cpp: the internal raw_espeak_phonemize helper gained an additive use_tie
parameter (default true, preserving exact existing Kokoro behavior at both existing call
sites) plus a new public misaki::raw_espeak_ipa_phonemize() function that reuses the
existing espeak-ng dlopen/caching infrastructure to return untouched per-codepoint IPA
output (no Misaki diphthong-tie merging/symbol remapping), which Piper's phoneme_id_map
requires.
New tests/python_tests/test_piper_pipeline.py + tests/python_tests/utils/piper_test_assets.py:
a tiny synthetic ONNX/OpenVINO fixture matching Piper's exact I/O contract (no HF/optimum
-cli export path exists for this architecture to build a fixture from) — 5 tests covering
backend dispatch, waveform generation, speaker_embedding rejection, and configurable
inference scales. All passing.

Update (rebase + CI-scope fix)

This PR was previously rejected because it accidentally included .github/workflows/*
changes (an unrelated CI-file drift picked up by an earlier automated push). This update:

Rebased enable/rhasspy-piper-voices cleanly (linear, no conflicts) onto current
openvinotoolkit/openvino.genai master (commit 2200993f).
Dropped the offending CI-sync commit entirely (rather than reverting it) — the branch now
contains exactly one commit (2358aaac), touching only the 10 Piper-scoped files
listed below. git diff against upstream/master for .github/workflows/ is empty.
Force-pushed the cleaned branch to the fork; no new PR was created.

Files changed by this PR (unchanged from original scope):
src/cpp/include/openvino/genai/speech_generation/speech_generation_config.hpp,
src/cpp/include/openvino/genai/speech_generation/text2speech_pipeline.hpp,
src/cpp/src/speech_generation/misaki_cpp/include/misaki/fallbacks.hpp,
src/cpp/src/speech_generation/misaki_cpp/src/espeak_fallback.cpp,
src/cpp/src/speech_generation/piper_tts_model.{cpp,hpp} (new),
src/cpp/src/speech_generation/speech_generation_config.cpp,
src/cpp/src/speech_generation/text2speech_pipeline.cpp,
tests/python_tests/test_piper_pipeline.py (new),
tests/python_tests/utils/piper_test_assets.py (new).

Validation (this session — real build, real models, real tests)

Build: clean C++ core + Python bindings build from the rebased branch (Release,
Ninja, pip-installed openvino==2026.3.0.dev20260708 matching openvino_tokenizers~=2026.3.0.0.dev
pin), 1186/1186 targets, no new warnings. misaki_cpp compiled with espeak-ng support.
Real model run (not mocked): downloaded the real ONNX + sidecar JSON for both scoped
voices from rhasspy/piper-voices, converted to OpenVINO IR (FP16 and INT8
weight-compressed) with ov.convert_model/nncf.compress_weights, and ran
Text2SpeechPipeline end-to-end with real espeak-ng phonemization (espeakng_loader)
for both voices, both precisions, on both CPU and GPU (Intel Arrow Lake-U iGPU) — all
8 combinations produced finite, non-empty, non-zero waveforms at the expected 22050 Hz
sample rate. IR + sample WAV artifacts stored under
enabled-models/rhasspy-piper-voices/<voice>/<precision>/ in the OMEGA repo.
Tests: test_piper_pipeline.py 5/5 passed; test_kokoro_pipeline.py (regression
check) 16/16 passed — confirms the additive dispatcher change does not affect existing
Kokoro/SpeechT5 backends. Combined: 21/21 passed, 0 failures, 0 skips (after installing
the optional kokoro extra needed only for that regression suite).
No implementation changes were required this session beyond the CI-scope cleanup — the
previously-validated PiperTTSImpl logic (phoneme-id/BOS-PAD-EOS convention, dispatcher
detection, scales wiring) worked correctly against real IR and real espeak-ng output on
first run.

Scope / non-goals

Only 2 representative voices validated — NOT the full 100+ voice Piper catalog.
Reimplemented the phoneme-ID encoding convention independently from the publicly
documented ONNX I/O contract; no code vendored from the (GPL-3.0-licensed) reference
runtime (OHF-Voice/piper1-gpl).
GPU latency variance (if any) is not the focus of this change and is not fixed here
(keeping scope narrow per model-enablement conventions).

Pull request overview

This PR adds a new Piper (VITS-family) backend to the C++ Text2SpeechPipeline, extending the speech generation configuration with Piper-specific inference scale parameters and introducing a Python test fixture + test suite to validate backend dispatch and waveform generation.

Changes:

Introduces PiperTTSImpl and extends Text2SpeechPipeline backend resolution to detect Piper via class name or phoneme_id_map in config.json.
Extends SpeechGenerationConfig with noise_scale, length_scale, and noise_w, including JSON/AnyMap parsing and validation.
Adds misaki-cpp support for raw (un-normalized) IPA output needed by Piper tokenization, plus new Python unit tests with a tiny synthetic ONNX→OpenVINO fixture.
Also includes broad GitHub Actions workflow changes (action versions, job/test selection), which are not described in the PR summary and materially change CI behavior.

Reviewed changes

Copilot reviewed 21 out of 22 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/python_tests/utils/piper_test_assets.py	Generates a tiny synthetic Piper-like ONNX graph and converts it to OpenVINO IR for fast tests.
tests/python_tests/test_piper_pipeline.py	Adds Python tests covering Piper backend dispatch, waveform generation, scales, and speaker embedding rejection.
src/cpp/src/speech_generation/text2speech_pipeline.cpp	Adds Piper backend selection and instantiation.
src/cpp/src/speech_generation/speech_generation_config.cpp	Loads/updates/validates Piper scale parameters.
src/cpp/src/speech_generation/piper_tts_model.hpp	Declares the `PiperTTSImpl` backend implementation.
src/cpp/src/speech_generation/piper_tts_model.cpp	Implements Piper config parsing, phoneme-id sequence building, and single-pass inference.
src/cpp/src/speech_generation/misaki_cpp/src/espeak_fallback.cpp	Adds a tie-toggle for raw espeak phonemization and implements raw IPA phonemization helper.
src/cpp/src/speech_generation/misaki_cpp/include/misaki/fallbacks.hpp	Exposes `raw_espeak_ipa_phonemize()` API for consumers (Piper).
src/cpp/include/openvino/genai/speech_generation/text2speech_pipeline.hpp	Updates speaker embedding shape docs to mention Piper `{0}`.
src/cpp/include/openvino/genai/speech_generation/speech_generation_config.hpp	Adds Piper scale fields and public `ov::Property<float>` keys.
.github/workflows/workflow_rerunner.yml	Changes pinned `actions/setup-python` revision.
.github/workflows/windows.yml	Updates action pins and materially changes the Python test matrix/selection.
.github/workflows/sdl.yml	Updates action pins.
.github/workflows/manylinux_2_28.yml	Updates action pins and materially changes the Python test matrix/selection.
.github/workflows/mac.yml	Updates action pins and materially changes the Python test matrix/selection.
.github/workflows/linux.yml	Updates action pins and introduces an unpinned `actions/checkout@v5.0.1` usage.
.github/workflows/lint.yml	Updates action pins.
.github/workflows/deploy_gh_pages.yml	Updates action pins.
.github/workflows/coverity.yml	Updates action pins.
.github/workflows/copilot-setup-steps.yml	Changes checkout/action references and downgrades gh-aw CLI versions.
.github/workflows/ci-doctor.md	Adds engine/version config and other CI-doctor workflow behavior updates.
.github/workflows/ci-doctor.lock.yml	Large regenerated lockfile update for ci-doctor workflow.

+    const std::array<float, 3> scales = {generation_config.noise_scale,
+                                         generation_config.length_scale,
+                                         generation_config.noise_w};
+
+    for (const std::string& text : texts) {
+#if OPENVINO_GENAI_HAS_MISAKI_CPP
+        const auto phonemized = misaki::raw_espeak_ipa_phonemize(text, espeak_voice, library_path);
+        OPENVINO_ASSERT(phonemized.has_value(),
+                        "Piper G2P failed to phonemize input text with espeak-ng voice '",
+                        espeak_voice,
+                        "'. Install espeak-ng and/or set MISAKI_ESPEAK_LIBRARY.");
+        const std::vector<int64_t> phoneme_ids = build_phoneme_id_sequence(*phonemized);
+#else
+        OPENVINO_THROW(
+            "Piper backend requires misaki-cpp for espeak-ng based G2P. Configure with ENABLE_MISAKI_CPP=ON and "
+            "provide misaki-cpp sources.");
+        const std::vector<int64_t> phoneme_ids;
+#endif
+
+        const int64_t input_length = static_cast<int64_t>(phoneme_ids.size());
+        ov::Tensor input_tensor(ov::element::i64, ov::Shape{1, phoneme_ids.size()}, const_cast<int64_t*>(phoneme_ids.data()));
+        ov::Tensor input_lengths_tensor(ov::element::i64, ov::Shape{1}, const_cast<int64_t*>(&input_length));
+        ov::Tensor scales_tensor(ov::element::f32, ov::Shape{3}, const_cast<float*>(scales.data()));
+


+    OPENVINO_ASSERT(model.inputs().size() > fallback_index,
+                    "Piper model is missing an expected input matching '",
+                    substring,
+                    "'");
+    return model.input(static_cast<int>(fallback_index)).get_any_name();


      - name: Clone openvino.genai
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        uses: actions/checkout@v5.0.1
        with:
          path: ${{ env.SRC_DIR }}
          submodules: recursive


+          - name: 'Whisper'
            # TODO: skip some tests temporary untill https://github.com/huggingface/datasets/issues/7647 dataset is fixed
            cmd: |
-              python -m pytest -s -v tests/python_tests/test_whisper_pipeline.py tests/python_tests/test_whisper_pipeline_static.py -m "not transformers_lower_v5"
+              python -m pytest -s -v tests/python_tests/test_whisper_pipeline.py tests/python_tests/test_whisper_pipeline_static.py -m "not transformers_lower_v5" -k "not test_smoke and not test_asr_constructors and not test_max_new_tokens and not test_language_mode and not test_task_mode and not test_language_autodetect and not test_asr_config_constructor and not test_initial_prompt_hotwords and not test_random_sampling"


+def _configure_espeak_from_venv() -> None:
+    """
+    Point the C++ misaki espeak fallback (reused by PiperTTSImpl for raw IPA
+    phonemization) at the shared library/data bundled with the ``espeakng_loader``
+    Python package, for CI machines without a system-wide espeak-ng install.
+    """
+    try:
+        import espeakng_loader  # type: ignore[import]
+    except ImportError:
+        logger.debug("espeakng_loader not available; skipping espeak env setup")
+        return
+
+    lib_path = espeakng_loader.get_library_path()
+    data_path = espeakng_loader.get_data_path()
+
+    if "MISAKI_ESPEAK_LIBRARY" not in os.environ:
+        os.environ["MISAKI_ESPEAK_LIBRARY"] = lib_path
+    if "ESPEAK_DATA_PATH" not in os.environ:
+        os.environ["ESPEAK_DATA_PATH"] = data_path
+
+
+_configure_espeak_from_venv()
+
+SAMPLE_RATE = 22050


Adds a new PiperTTSImpl backend to Text2SpeechPipeline supporting Piper (rhasspy/piper-voices), a single-graph, non-autoregressive VITS-family TTS architecture with no separate encoder/decoder split. - New src/cpp/src/speech_generation/piper_tts_model.{hpp,cpp}: loads phoneme_id_map from config.json, builds the BOS/PAD/EOS phoneme-id sequence per Piper's convention, runs a single inference pass through the ONNX-derived OpenVINO IR (input/input_lengths/scales -> output), and rejects non-empty speaker_embedding (Piper voices are single-speaker). - Extended SpeechGenerationConfig with Piper-specific noise_scale, length_scale, and noise_w parameters (reusing the existing language field for the espeak-ng voice name). - Extended the Text2SpeechPipeline dispatcher (resolve_backend) with an additive Piper detection branch keyed on the architectures field or the presence of a phoneme_id_map object in config.json. - misaki_cpp: added an additive use_tie parameter (default true, preserving existing Kokoro behavior) to the internal raw_espeak_phonemize helper, and a new public misaki::raw_espeak_ipa_phonemize() function that reuses the existing espeak-ng loading/caching infrastructure to return untouched IPA codepoints (no Misaki diphthong-tie merging or symbol remapping), which Piper's per-codepoint phoneme_id_map requires. - New tests/python_tests/test_piper_pipeline.py and tests/python_tests/utils/piper_test_assets.py: a tiny ONNX/OpenVINO fixture matching Piper's exact I/O contract (no HF/optimum-cli export path exists for this architecture) plus unit tests for backend dispatch, waveform generation, speaker_embedding rejection, and configurable inference scales. Validated end-to-end against the real en_US-joe-medium and de_DE-thorsten-medium IRs (FP32/INT8/INT4) on CPU and GPU. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Copilot AI review requested due to automatic review settings July 8, 2026 17:21

mlukasze added experimental Experimental agent-generated work do-not-merge Do not merge labels Jul 8, 2026

github-actions Bot added category: GHA CI based on Github actions category: CPP API Changes in GenAI C++ public headers category: GGUF GGUF file reader category: speech generation labels Jul 8, 2026

Copilot started reviewing on behalf of mlukasze July 8, 2026 17:22 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 9, 2026 05:18

omega-intel force-pushed the enable/rhasspy-piper-voices branch from 573ddf5 to 2358aaa Compare July 9, 2026 05:18

Copilot AI reviewed Jul 9, 2026

github-actions Bot removed the category: GHA CI based on Github actions label Jul 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[EXPERIMENT][WIP] Add PiperTTSImpl backend for Piper (VITS) text-to-speech voices#4116

[EXPERIMENT][WIP] Add PiperTTSImpl backend for Piper (VITS) text-to-speech voices#4116
mlukasze wants to merge 1 commit into
openvinotoolkit:masterfrom
mlukasze:enable/rhasspy-piper-voices

mlukasze commented Jul 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mlukasze commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's included

Update (rebase + CI-scope fix)

Validation (this session — real build, real models, real tests)

Scope / non-goals

Related

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mlukasze commented Jul 8, 2026 •

edited

Loading