Skip to content

Phoneme alignment support #135

Description

@JarbasAl

Goal

Add phoneme-level timing alignment so consumers can map audio samples back to individual phonemes.

What is phoneme alignment

Per-phoneme start/end sample indices extracted from the model's duration predictor output. Enables use-cases like viseme lip-sync, karaoke highlighting, and pronunciation analysis.

Upstream

Ported from piper (OHF-Voice/piper1-gpl commit eb6be6b). The alignment logic does not use espeak-ng and is covered by Apache 2.

Design

  • Second ONNX output tensor: per-frame phoneme IDs
  • hop_length (default 256) converts frames → samples
  • PhonemeAlignment dataclass: phoneme + num_samples
  • AudioChunk extended with phoneme_alignments, phoneme_ids, phoneme_id_samples
  • TTSVoice.synthesize(include_alignments=True) returns populated chunks when the model supports it
  • phoonnx_train export-onnx --add-phoneme-alignment embeds the extra output in the exported graph

PR

#125feat/phoneme-alignment

Deliverables

  • PhonemeAlignment dataclass
  • AudioChunk alignment fields
  • synthesize / phoneme_ids_to_audio alignment path
  • tests/test_alignment.py (24 hermetic unit tests)
  • docs/alignment.md (usage guide with viseme and karaoke examples)

Acceptance criteria

  • feat: phoneme alignment support #125 merged
  • ONNX export with --add-phoneme-alignment produces valid two-output graphs
  • synthesize(include_alignments=True) returns PhonemeAlignment lists when model supports it
  • synthesize(include_alignments=True) returns None fields gracefully when model does not support it

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestinferenceInference engine support tracking

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions