whisper-subs

Rust CLI that generates subtitles from video/audio files using whisper-rs (Rust bindings to whisper.cpp).

Pipeline: media file → ffmpeg → 16 kHz mono f32 PCM → whisper.cpp → SRT/VTT/TXT/JSON.

Features

• Any input ffmpeg can decode (AVI, MKV, MP4, MOV, WAV, MP3, FLAC, …)
• Output in SRT, VTT, TXT, or JSON
• Automatic language detection or explicit --language
• Transcribe or translate-to-English (--task translate)
• Line reflow (--max-line-width) and gap merging (--merge-gap-ms)
• Initial prompt (--prompt) to bias vocabulary (proper nouns, jargon)
• Backend feature flags: Metal, CoreML, CUDA, Vulkan, OpenBLAS, or plain CPU
• Progress bar driven by whisper's internal progress callback

Prerequisites

• Rust 1.88+ (needed for edition 2024 let-chains and whisper-rs-sys)

• A C++ toolchain + cmake — whisper-rs-sys builds whisper.cpp from vendored C++ source on first compile.

  • macOS: xcode-select --install then brew install cmake
  • Debian/Ubuntu: sudo apt install build-essential cmake
  • Windows: MSVC Build Tools + CMake

• ffmpeg on PATH (or pass --ffmpeg /path/to/ffmpeg)

• A ggml-*.bin Whisper model. Download from https://huggingface.co/ggerganov/whisper.cpp:

  curl -L -o ggml-small.bin \
      https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.bin

  Recommended starting points:

  • ggml-small.bin (~466 MB) — good balance, real-time-ish on CPU
  • ggml-medium.bin (~1.5 GB) — clearly better, especially for non-English
  • ggml-large-v3.bin (~3.1 GB) — best quality
  • ggml-large-v3-turbo.bin (~1.6 GB) — near-large quality, much faster

• (optional but recommended) A Silero VAD model. Drop it next to your whisper model and it gets auto-detected; it trims non-speech and cuts hallucinations on long files. Download from https://huggingface.co/ggml-org/whisper-vad:

  curl -L -o ggml-silero-v5.1.2.bin \
      https://huggingface.co/ggml-org/whisper-vad/resolve/main/ggml-silero-v5.1.2.bin
  # or the newer release:
  curl -L -o ggml-silero-v6.2.0.bin \
      https://huggingface.co/ggml-org/whisper-vad/resolve/main/ggml-silero-v6.2.0.bin

  Either ggml-silero-v5.1.2.bin or ggml-silero-v6.2.0.bin sitting next to --model is picked up automatically. To use a different path pass --vad-model <path>; to opt out entirely pass --no-vad.

Build

Pick the backend for your hardware.

macOS (Apple Silicon)

cargo build --release
# Metal is auto-enabled on Apple Silicon; no --features flag needed.
# For Apple Neural Engine too, add: --features coreml
# (CoreML also requires converting the model first; see whisper.cpp docs.)

Linux with NVIDIA GPU

# CUDA toolkit must be installed and nvcc on PATH.
cargo build --release --features cuda

Linux/Windows CPU (with BLAS)

# Requires libopenblas-dev (Debian/Ubuntu) or equivalent.
cargo build --release --features openblas

Cross-platform GPU (Vulkan)

cargo build --release --features vulkan

Plain CPU

cargo build --release

(On Apple Silicon this still pulls in Metal — see "Disabling Metal on Apple Silicon" below.)

Disabling Metal on Apple Silicon

Metal is wired into the Cargo manifest for the aarch64-apple-darwin target, so passing --no-default-features or omitting --features metal on an M-series Mac does not turn it off. This is a Cargo limitation: target-specific dependency features are unconditional. Two ways around it:

1. Comment out the auto-enable block in Cargo.toml — find the [target.'cfg(all(target_os = "macos", target_arch = "aarch64"))'.dependencies] block and comment its two lines. Remember not to commit that change.

2. Build for the Intel triple (runs under Rosetta 2) — this dodges the target_arch = "aarch64" cfg entirely:

   rustup target add x86_64-apple-darwin
   cargo build --release --target x86_64-apple-darwin

   The binary lands at target/x86_64-apple-darwin/release/whisper-subs. Expect lower throughput than native-arm64 + Metal, obviously.

The binary lands at target/release/whisper-subs.

Usage

# Simplest case — auto-detect language, output movie.srt next to input
whisper-subs movie.avi --model ./ggml-small.bin

# Italian source, 42-char line width, merge gaps under 200 ms
whisper-subs movie.avi \
    --model ./ggml-medium.bin \
    --language it \
    --max-line-width 42 \
    --merge-gap-ms 200

# Translate German audio → English subtitles
whisper-subs interview.mkv \
    --model ./ggml-large-v3.bin \
    --language de \
    --task translate \
    --format srt

# Bias vocabulary with an initial prompt
whisper-subs tech-talk.mp4 \
    --model ./ggml-small.bin \
    --prompt "Talk about Rust, ownership, borrow checker, async/await, tokio."

# JSON output for downstream pipelines
whisper-subs lecture.mp4 \
    --model ./ggml-small.bin \
    --format json \
    --output lecture.json

# Set model path via env var to avoid repeating it
export WHISPER_MODEL=$HOME/models/ggml-medium.bin
whisper-subs episode.mkv

Run whisper-subs --help for the full option list.

Performance notes

• On Apple Silicon (Metal is auto-enabled), a 2-hour film with small takes a few minutes; large-v3 is still faster than real-time.
• On CPU-only, small is roughly real-time with OpenBLAS; medium is 2–3× real-time; large-v3 is 4–8× real-time.
• The --threads flag helps on many-core CPU machines; leave at 0 for auto.

Hallucination control

Whisper's most common failure mode is a repetition loop — the same sentence dumped to the end of the transcript. whisper-subs ships with defaults tuned for quality over raw throughput. Every knob is overridable.

Defaults (all on)

Mitigation	How it works	Cost	Flag
Silero VAD	Skip non-speech; auto-detected next to --model	tiny	--no-vad, --vad-model
Chunked decoding, 15 s	Cap each decoder call so loops can't spiral	possible boundary cut	--vad-max-speech
No context carry-over	Don't feed previous-chunk tokens into the next	—	(not exposed)
Suppress non-speech	Drops the "music/thanks" hallucination family	—	(not exposed)
Beam search (width 5)	Explore multiple decoder paths	~2× slower	--beam-size (1 = greedy)
Stricter logprob (-0.5)	Re-decode low-confidence windows at higher temp	slower on hard audio	--logprob-threshold
Tail-loop detector	Truncate or fail if ≥5 identical trailing segments	—	--on-loop, --loop-threshold
Heavy-truncation tag	If the loop ate more than the last 10% of the transcript, save as movie-truncated.srt	—	--loop-near-end-pct

Spotting bad outputs in a batch. When the detector fires and a substantial chunk of the transcript gets dropped, the output is written as <stem>-truncated.<ext> instead of <stem>.<ext>, so you can spot problem files at a glance in a large batch even if you missed the warning on the console. Batch mode also skips any file that already has a -truncated sibling, so re-running the batch won't re-transcribe them. To retry one, delete the -truncated file; to accept it, rename it to the canonical <stem>.<ext>.

The "last 10%" threshold is tunable via --loop-near-end-pct: set it to 100 to never tag (every truncated output keeps its canonical name), or to 0 to always tag.

If you still see loops

Whisper's failure rate on long audio is 1–5% even with everything enabled. Over a batch you'll see the detector fire occasionally. To push the rate lower, at increasing speed cost:

# tighter VAD chunks (smaller blast radius)
whisper-subs -r ~/Videos --vad-max-speech 10

# stricter confidence gate (more windows re-decoded)
whisper-subs -r ~/Videos --logprob-threshold -0.3

# wider beam (slowest, most robust)
whisper-subs -r ~/Videos --beam-size 8

Diminishing returns beyond that. If a specific file reliably fails, try biasing the decoder with --prompt "topical words", or switch to a smaller model — paradoxically, larger models hallucinate more on noisy or music-heavy audio.

The "Transcription by CastingWords" hallucination

If the detector reports that the loop covers the entire transcript and the repeated text is something like "Transcription by CastingWords", "Thanks for watching!", "Subtitles by the Amara.org community", "Please like and subscribe", or similar credit-line phrasing, you've hit one of Whisper's trained-in hallucinations. These strings appeared all over the model's training data (YouTube-adjacent subtitle dumps), so the decoder falls back to them when it can't get a foothold on the audio. Classic triggers: old film, low-bitrate mono, tape hiss, music-heavy sections, and very quiet or muffled speech.

Standard mitigations (VAD, logprob threshold, beam search) don't help much because every VAD chunk independently converges on the same attractor — tightening them just produces more copies of the hallucination. Three things do work, in order of effectiveness:

1. Prime the decoder with an initial prompt. Biggest lever by far. Tell whisper what kind of content to expect; 20–30 tokens of plausible domain vocabulary is usually enough to pull it out of the basin:

whisper-subs \
  --prompt "Rod Serling narrates a Twilight Zone episode titled The Grave." \
  movie.avi

2. Disable VAD. When every chunk hallucinates, VAD is amplifying the problem by feeding the decoder too-short spans. Longer inputs give it more genuine speech to anchor on:

whisper-subs --no-vad movie.avi

3. Switch from turbo to full large-v3. Turbo's distilled decoder is more susceptible to this specific pattern:

whisper-subs -m models/ggml-large-v3.bin movie.avi

Combine all three for the hard cases:

whisper-subs \
  -m models/ggml-large-v3.bin \
  --no-vad \
  --beam-size 8 \
  --prompt "Rod Serling narrates a Twilight Zone episode titled The Grave." \
  movie.avi

If even that produces garbage, extract a 30-second sample and listen before tuning further — if the audio is mostly hiss or music, no decoder tweak will rescue it and you need audio preprocessing (denoise + normalize) first:

ffmpeg -ss 60 -t 30 -i movie.avi -ac 1 -ar 16000 /tmp/sample.wav
afplay /tmp/sample.wav   # macOS; use mpv or ffplay elsewhere

Trading accuracy for speed

The defaults prioritise transcription quality; if throughput matters more than the occasional loop, flip the two dominant costs off:

whisper-subs -r ~/Videos --beam-size 1 --logprob-threshold -1.0

• ~2× faster — matches whisper-cli's stock defaults.
• Higher hallucination rate — the loop detector (--on-loop) will fire noticeably more often on long batches, and subtler quality drops (phantom words, dropped trailing phrases) become more common too. The quality-first defaults exist specifically because those whisper-cli defaults miss too many bad decodes on feature-length audio.

You can dial either knob independently if you want a compromise: keep --beam-size 1 but leave --logprob-threshold at -0.5 (about 1.5× faster, middling robustness), or vice versa.

Project layout

whisper-subs/
├── Cargo.toml
├── README.md
└── src/
    ├── main.rs        # CLI entry point, orchestration
    ├── audio.rs       # ffmpeg → f32 PCM extraction
    ├── transcribe.rs  # whisper-rs wrapper
    └── srt.rs         # SRT/VTT/TXT/JSON rendering + reflow/merge

Tests

cargo test

Covers timestamp formatting, line reflow, segment merging, SRT structure, and JSON escaping. The whisper/ffmpeg paths are integration-tested by running the binary on a short sample (not included).

Troubleshooting

• ffmpeg not found — install ffmpeg (brew install ffmpeg, apt install ffmpeg) or pass --ffmpeg /full/path.
• model file not found — download from the HuggingFace link above.
• Repeated/looping output on silent sections — try --merge-gap-ms 500 or use a larger model; Whisper's known hallucination failure mode.
• Wrong language auto-detected — pass --language <code> explicitly for long files; detection runs on the first 30 s only.
• CUDA build fails — make sure nvcc --version works and CUDA_PATH / LD_LIBRARY_PATH point at your toolkit.

License

MIT OR Apache-2.0