feat(pipeline): A/V-sync hardening ported from Autocrop-vertical (study r6)

Sixth external reframe study (kamilstanuch/Autocrop-vertical). Its reframe
algorithm is simpler than ClippyMe's, but its production-hardening layer closes
real, previously-unhandled gaps in the clip/reframe render path — all confirmed
absent by an audit (no VFR detection, no start_time compensation, no corrupt-
frame guard, missing pix_fmt on two encoders).

New cv2-free, host-tested module media_probe.py (pure helpers + never-raise
ffprobe wrappers):
- VFR detection (is_vfr / probe_is_variable_frame_rate)
- stream start_time → audio seek args (parse_start_time / audio_sync_seek_args /
  probe_stream_start_time)
- frame-rate parsing (parse_frame_rate)

Wiring (gated so the proven path stays byte-identical on CFR, zero-start input):
- reframe.py: VFR→CFR pre-normalize; -pix_fmt yuv420p + -vsync cfr on the raw
  encoder; try/except per-frame → duplicate last good frame; audio start_time
  compensation; -shortest mux; temp_cfr_input cleanup.
- main.py: -pix_fmt yuv420p + -vsync cfr on the persisted source-slice cut.

Fixes the classic yt-dlp A/V desync (non-zero video start_time) and VFR drift
(phone uploads) — YouTube/phone are ClippyMe's primary inputs.

Deferred (documented): HW encoder, quality/ratio CLI flags, fps cross-check.

Verification:
- host: pytest -m "not integration" → 279 passed, 2 skipped (248 baseline + 31 new)
- Docker: pytest -m integration → 13 passed (reframe path unaffected)

Deliverables (per /goal): docs/autocrop-vertical-analysis.md (report + prioritized
list + learnings), media_probe.py + tests (implementation + coverage).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: fralapo

CLAUDE.md

@@ -11,7 +11,7 @@ ClippyMe is a self-hosted AI video platform that transforms long-form videos (Yo
 All Python backend code lives under `src/clippyme/` (src-layout, installed via `pip install -e .`):
 
 - `src/clippyme/api/` — FastAPI app (`app.py`), `schemas.py`, `security.py`
-- `src/clippyme/pipeline/` — `main.py` orchestrator (~800 LOC, was 2216), `deepgram_transcribe.py`, `gemini_parser.py`, `gemini_service.py`, plus extracted stages: `transcribe_cache.py`, `download.py`, `scene_detection.py`, `postprocess.py`, `diarization.py`, `hardware.py` (DEVICE/Whisper-model selection), `reframe.py` (cv2/YOLO/MediaPipe core: `process_video_to_vertical` + `SmoothedCameraman`/`SpeakerTracker`/`DetectionSmoother` + face/person detection; `ASPECT_RATIO` is a module global `main` sets per-job via `reframe.ASPECT_RATIO = ...`), and pure-math `reframe_ops.py`. `main` re-exports the reframe classes for back-compat. Verify pipeline changes with `docker compose run --rm -u root backend sh -lc "pip install -q pytest && pytest -m integration"`.
+- `src/clippyme/pipeline/` — `main.py` orchestrator (~800 LOC, was 2216), `deepgram_transcribe.py`, `gemini_parser.py`, `gemini_service.py`, plus extracted stages: `transcribe_cache.py`, `download.py`, `scene_detection.py`, `postprocess.py`, `diarization.py`, `hardware.py` (DEVICE/Whisper-model selection), `reframe.py` (cv2/YOLO/MediaPipe core: `process_video_to_vertical` + `SmoothedCameraman`/`SpeakerTracker`/`DetectionSmoother` + face/person detection; `ASPECT_RATIO` is a module global `main` sets per-job via `reframe.ASPECT_RATIO = ...`), pure-math `reframe_ops.py`, and `media_probe.py` (cv2-free ffprobe wrappers + pure A/V-sync helpers: VFR detection, stream `start_time` compensation, frame-rate parsing — host-unit-tested; wired into `reframe.py`/`main.py` to fix VFR drift + YouTube audio desync. Ported from `kamilstanuch/Autocrop-vertical`, see `docs/autocrop-vertical-analysis.md`). `main` re-exports the reframe classes for back-compat. Verify pipeline changes with `docker compose run --rm -u root backend sh -lc "pip install -q pytest && pytest -m integration"`.
 - `src/clippyme/domain/` — `compose.py`, `clip_endpoints.py`, `job_results.py`, `job_artifacts.py`, `job_worker.py`, `history_service.py`, `subtitles.py`, `smartcut.py`, `hooks.py`
 - `src/clippyme/integrations/` — `social_publisher.py`, `auto_editor_updater.py`
 - `src/clippyme/storage/` — `config_store.py`

docs/autocrop-vertical-analysis.md

@@ -0,0 +1,148 @@
+# Comparative analysis: `kamilstanuch/Autocrop-vertical` → ClippyMe
+
+Date: 2026-06-17
+External repo: <https://github.com/kamilstanuch/Autocrop-vertical> @ `main`
+Size: single 710-LOC `main.py` + `requirements.txt` + README (v1.4.1).
+Sixth external study, after `gauravzazz/smart-reframe`,
+`KazKozDev/auto-vertical-reframe`, `obi19999/smart-video-reframe`,
+`mfahsold/montage-ai`, and `aregrid/frame`.
+
+---
+
+## 0. Verdict up front
+
+Unlike the fifth study (`aregrid/frame`, vaporware), this is **real, mature,
+single-file code** in the same domain: horizontal → vertical social-media
+cropping (PySceneDetect → YOLOv8 person detection → per-scene TRACK/LETTERBOX →
+OpenCV frame pipe → FFmpeg encode). Its **reframe algorithm is simpler than
+ClippyMe's** (static per-scene center-crop on a person box; no camera smoothing,
+no active-speaker selection, no continuous zoom, no global trajectory pass) — so
+there is nothing to port on the reframe-*math* axis where ClippyMe already runs
+ahead (`reframe_ops.py` + five prior studies).
+
+Its real value is the **production-hardening layer** the changelog was written
+around: A/V-sync correctness on messy real-world inputs (YouTube downloads, phone
+VFR). That layer maps onto concrete, previously-unhandled gaps in ClippyMe's
+clip/reframe render path, and is what this study ports.
+
+---
+
+## 1. Architecture comparison
+
+| Aspect | Autocrop-vertical | ClippyMe | Takeaway |
+|---|---|---|---|
+| Scene detection | PySceneDetect `ContentDetector`, `--frame-skip`/`--downscale` | PySceneDetect (`scene_detection.py`) | Parity. |
+| Subject detect | YOLOv8n person + Haar-cascade face (middle frame only) | YOLOv8n + MediaPipe FaceMesh + MAR active-speaker (per frame) | ClippyMe richer. |
+| Crop decision | `decide_cropping_strategy`: 0 people→LETTERBOX, 1→track box, group→TRACK if group-width < crop-width else LETTERBOX | `analyze_scenes_strategy` → TRACK/WIDE/GENERAL/DISABLED | Comparable intent; ClippyMe adds WIDE multi-speaker. |
+| Camera | **static** center-crop per scene (no motion) | `SmoothedCameraman`: EMA/1€/spring smoothing, continuous zoom, lost-subject drift | ClippyMe far ahead. |
+| **VFR handling** | `is_variable_frame_rate` (ffprobe r vs avg rate) → re-mux `-vsync cfr` | **none** | **Gap. Ported.** |
+| **Audio start_time** | `get_stream_start_time` → `-ss` trims audio lead-in | **none** | **Gap. Ported.** |
+| **Corrupt-frame** | try/except → duplicate last good frame | loop aborts on exception | **Gap. Ported.** |
+| **Output pix_fmt** | explicit `yuv420p` | only on *later* passes; reframe core + cut omit it | **Gap. Ported.** |
+| fps source | OpenCV (same backend that reads frames) + `-vsync cfr` | PySceneDetect for `-r`, OpenCV elsewhere (mismatch risk) | Mitigated by CFR + VFR ports. |
+| HW encoder | `--encoder hw` (VideoToolbox/NVENC) + fallback | libx264 only | Deferred (see §3). |
+| Quality presets | `--quality fast/balanced/high` + CRF/preset overrides | fixed per stage | Deferred (off product axis). |
+
+---
+
+## 2. Prioritised improvement list
+
+| Pri | Improvement | Status | Notes |
+|---|---|---|---|
+| **High** | Audio `start_time` offset compensation | ✅ implemented | Classic yt-dlp A/V desync; YouTube is ClippyMe's primary input. `audio_sync_seek_args` + `probe_stream_start_time`, wired into reframe Step 5. |
+| **High** | VFR → CFR pre-normalization | ✅ implemented | Phone/odd-timebase sources. `probe_is_variable_frame_rate` gate + re-mux in reframe; CFR inputs (the normal clip slices) untouched. |
+| **Medium** | `-pix_fmt yuv420p` on the two bare libx264 encoders | ✅ implemented | reframe raw-frame encoder + main.py source-slice cut; guarantees universal decode even when later 420p passes are skipped. |
+| **Medium** | Corrupt/failed-frame resilience | ✅ implemented | try/except in the reframe render loop → duplicate last good frame; preserves frame count → no A/V drift. |
+| **Medium** | `-vsync cfr` on reframe encoder + cut | ✅ implemented | Locks constant output rate; pairs with the VFR fix. |
+| Low | fps cross-check (cv2 ↔ PySceneDetect) | ⏸ deferred | Subsumed: VFR normalize + CFR output remove the drift this guarded against. A hard assert could be added later. |
+| Low | Hardware encoder (NVENC/VideoToolbox) | ⏸ deferred | ClippyMe encodes in a Linux/Docker container where libx264 is the portable choice; HW detect adds surface for marginal CPU savings off the product axis. Documented, not wired. |
+| Low | `--quality`/`--ratio`/`--plan-only` CLI knobs | ⏸ deferred | ClippyMe's ratio/quality are driven by the dashboard + per-stage tuning, not CLI flags; not a fit. |
+
+---
+
+## 3. What was implemented
+
+New module **`src/clippyme/pipeline/media_probe.py`** (pure helpers + thin
+ffprobe wrappers, **no cv2** → host-importable), following the established
+`reframe_ops.py` "pure logic in a testable module, thin glue in the cv2 file"
+pattern:
+
+- `parse_frame_rate(str) -> float` — "30000/1001" → 29.97; malformed/zero → 0.0.
+- `is_vfr(r_rate, avg_rate, threshold=0.5) -> bool` — nominal vs average gap.
+- `parse_start_time(str) -> float` — ffprobe `start_time`; "N/A"/garbage → 0.0.
+- `audio_sync_seek_args(start, min_offset=0.05) -> list[str]` — `['-ss', …]` or `[]`.
+- `probe_stream_start_time` / `probe_is_variable_frame_rate` — ffprobe wrappers
+  that **never raise** (missing ffprobe / odd file → safe default).
+
+Wiring (glue, in the integration-tested cv2 files):
+
+- **`reframe.py`** — (a) VFR detection + `-vsync cfr` re-mux pre-step (gated on
+  `probe_is_variable_frame_rate`, so CFR inputs are byte-identical); (b)
+  `-pix_fmt yuv420p` + `-vsync cfr` on the raw-frame encoder; (c) try/except
+  around per-frame strategy → duplicate last good frame (`dropped_frames`
+  reported); (d) audio extracted with `audio_sync_seek_args(start_time)`;
+  (e) `-shortest` on the final mux; (f) `temp_cfr_input` cleanup.
+- **`main.py`** — `-pix_fmt yuv420p` + `-vsync cfr` on the persisted source-slice
+  cut, so the slice is universally decodable and CFR before reframe ever sees it.
+
+Tests: **`tests/pipeline/test_media_probe.py`** — 31 host (non-integration)
+cases covering frame-rate parsing, VFR thresholds, start_time parsing, seek-arg
+no-op boundaries, and the never-raise contract.
+
+---
+
+## 4. Conflicts & resolutions
+
+- **"Proven single-pass path must stay byte-identical" (CLAUDE.md).** Every port
+  is a *no-op on the common case*: `audio_sync_seek_args` returns `[]` for a
+  zero start_time, VFR normalization only fires when detection is confident, and
+  the corrupt-frame guard only diverges on an exception that previously aborted
+  the run. The normal pipeline feeds already-re-encoded CFR, zero-start clip
+  slices, so its bytes are unchanged — confirmed by the integration suite still
+  passing (13/13).
+- **"Don't rewrite; keep the style" (instruction).** No reframe logic was
+  rewritten; the static-crop algorithm was *not* adopted (ClippyMe's is better).
+  Only the orthogonal robustness layer was added, in the same inline-ffmpeg +
+  pure-helper-module style already used across the pipeline.
+- **fps-source mismatch (concern #3).** Rather than re-plumb every fps read, the
+  VFR normalize + CFR-locked output neutralize the drift the mismatch could
+  cause. Noted as a deferred hardening (an explicit cv2-vs-PySceneDetect assert).
+- **`_render_global_smooth` opt-in path** shares the same single-write model, so
+  the corrupt-frame guard is not applied there yet — that path is opt-in and
+  off by default; noted for a follow-up if it graduates.
+- **Heavy ports skipped.** HW encoder, quality/ratio CLI flags are real but off
+  ClippyMe's product axis (dashboard-driven, containerized libx264) — catalogued
+  as deferred, honouring "Non riscrivere tutto."
+
+---
+
+## 5. Learnings & how they were applied
+
+1. **A simpler upstream can still be a net contribution — via its hardening, not
+   its core.** Autocrop's *algorithm* is behind ClippyMe's, but its changelog is
+   a checklist of real-world A/V-sync bugs (VFR drift, start_time desync,
+   dropped-frame drift) that a more sophisticated reframe core never bothered to
+   handle. Value lived in the boring layer. *Applied:* ported the robustness,
+   ignored the algorithm.
+2. **Pure-vs-glue split makes "untestable" ffmpeg logic testable.** The decision
+   content (VFR threshold, seek-arg boundaries, rate parsing) was extracted into
+   a cv2-free module and host-unit-tested (31 cases), while only the ffmpeg
+   string-building stayed in the integration-only file — same discipline as
+   `reframe_ops.py`. *Applied:* `media_probe.py`.
+3. **Port as no-ops on the happy path.** Each fix is gated so the proven path is
+   byte-identical and only pathological inputs take the new branch. This is what
+   let a change to the integration render loop ship without re-baselining the
+   golden output. *Applied:* every wiring point above.
+
+---
+
+## 6. Verification
+
+- Host pure-helper suite: `pytest tests/pipeline/test_media_probe.py` → **31 passed**.
+- Full host (non-integration) suite: `pytest -m "not integration"` → **279 passed,
+  2 skipped** (248 prior baseline + 31 new) — no regression.
+- Integration suite in Docker:
+  `docker compose run --rm -u root backend sh -lc "pip install -q pytest && pytest -m integration"`
+  → **13 passed, 290 deselected** — reframe.py imports (cv2/mediapipe) and the
+  reframe render path are unaffected by the wiring.
+- `py_compile` on `reframe.py`, `main.py`, `media_probe.py` → clean.

src/clippyme/pipeline/main.py

@@ -883,7 +883,13 @@ def _resolve_output_dir(out: str | None, default: str) -> str:
                     '-ss', f'{float(start):.3f}',
                     '-i', input_video,
                     '-t', f'{clip_duration:.3f}',
+                    # -pix_fmt yuv420p + -vsync cfr guarantee the persisted source
+                    # slice is universally decodable and constant-frame-rate, so the
+                    # downstream reframe render (which writes raw frames at a fixed
+                    # -r) can't drift against audio even if the original download was
+                    # VFR (ported from kamilstanuch/Autocrop-vertical).
                     '-c:v', 'libx264', '-crf', '18', '-preset', 'fast',
+                    '-pix_fmt', 'yuv420p', '-vsync', 'cfr',
                     '-c:a', 'aac',
                     clip_source_path,
                 ]

src/clippyme/pipeline/media_probe.py

@@ -0,0 +1,132 @@
+"""ffprobe-backed media inspection + pure A/V-sync helpers.
+
+Robustness utilities ported from ``kamilstanuch/Autocrop-vertical`` (sixth
+external study — see ``docs/autocrop-vertical-analysis.md``). They close two
+real gaps in the clip / reframe render path:
+
+* **Variable-frame-rate (VFR) detection** — phone uploads and some YouTube
+  downloads carry a VFR video stream. The reframe loop decodes N frames with
+  OpenCV and writes them back at a *fixed* ``-r fps``, so an undetected VFR
+  source drifts against the stream-copied audio. ``is_vfr`` lets the caller
+  re-mux to constant frame rate first.
+* **Stream ``start_time`` compensation** — YouTube downloads frequently have a
+  non-zero video-stream ``start_time`` (audio at 0.0s, video at e.g. 1.8s).
+  When the video is re-encoded from frame 0 but the audio is stream-copied
+  verbatim, that container offset becomes an audible A/V desync.
+  ``audio_sync_seek_args`` trims the audio lead-in to match.
+
+The pure parsing / decision helpers (``parse_frame_rate``, ``is_vfr``,
+``parse_start_time``, ``audio_sync_seek_args``) perform no I/O and are
+host-unit-tested in ``tests/pipeline/test_media_probe.py`` — no cv2 import, so
+they run in the fast (non-integration) suite. The ``probe_*`` wrappers shell out
+to ffprobe and **degrade gracefully (never raise)**: a missing ffprobe or an odd
+file yields the "assume CFR / zero offset" default, so the pipeline keeps its
+prior behaviour instead of breaking.
+"""
+from __future__ import annotations
+
+import subprocess
+
+
+def parse_frame_rate(rate: str) -> float:
+    """Parse an ffprobe frame-rate string ("30000/1001", "25", "0/0") to fps.
+
+    Returns ``0.0`` for empty / malformed / zero-denominator input rather than
+    raising, so callers can treat "unknown" as "assume CFR / skip".
+    """
+    if not rate:
+        return 0.0
+    rate = rate.strip()
+    try:
+        if "/" in rate:
+            num, den = rate.split("/", 1)
+            den_f = float(den)
+            if den_f == 0:
+                return 0.0
+            return float(num) / den_f
+        return float(rate)
+    except (ValueError, ZeroDivisionError):
+        return 0.0
+
+
+def is_vfr(r_frame_rate: str, avg_frame_rate: str, threshold: float = 0.5) -> bool:
+    """True when nominal (``r_frame_rate``) and average (``avg_frame_rate``) frame
+    rates differ enough to indicate a variable frame rate.
+
+    ffprobe reports both as "num/den". A gap above ``threshold`` fps means the
+    container's real timing wanders from its nominal rate (classic for
+    phone-recorded clips). Unknown / zero rates → ``False`` (assume CFR — the
+    safer default, since a needless re-encode is worse than a no-op).
+    """
+    r = parse_frame_rate(r_frame_rate)
+    avg = parse_frame_rate(avg_frame_rate)
+    if r <= 0 or avg <= 0:
+        return False
+    return abs(r - avg) > threshold
+
+
+def parse_start_time(value: str) -> float:
+    """Parse an ffprobe stream ``start_time`` to seconds; "N/A"/""/garbage → 0.0."""
+    if not value:
+        return 0.0
+    try:
+        return float(value.strip())
+    except ValueError:
+        return 0.0
+
+
+def audio_sync_seek_args(video_start_time: float, min_offset: float = 0.05) -> list[str]:
+    """Return ffmpeg input-seek args dropping the audio lead-in equal to the video
+    stream's ``start_time`` — or ``[]`` when the offset is negligible.
+
+    Placed BEFORE ``-i`` (fast input seek) so the stream-copied audio is trimmed
+    to begin where the (re-encoded-from-frame-0) video begins. Offsets under
+    ``min_offset`` (≈1.5 frames @30fps) are ignored — not worth a re-seek, and it
+    keeps the common zero-start case byte-identical to the old behaviour.
+    """
+    if video_start_time is None or video_start_time <= min_offset:
+        return []
+    return ["-ss", f"{video_start_time:.3f}"]
+
+
+def probe_stream_start_time(video_path: str, stream: str = "v:0") -> float:
+    """ffprobe a stream's ``start_time`` in seconds (0.0 if unavailable).
+
+    Never raises — a missing ffprobe or unreadable file returns 0.0 so the caller
+    simply skips the sync adjustment.
+    """
+    try:
+        result = subprocess.run(
+            ["ffprobe", "-v", "error", "-select_streams", stream,
+             "-show_entries", "stream=start_time", "-of", "csv=p=0", video_path],
+            capture_output=True, text=True,
+        )
+        if result.returncode == 0:
+            return parse_start_time(result.stdout)
+    except (FileNotFoundError, OSError):
+        pass
+    return 0.0
+
+
+def probe_is_variable_frame_rate(video_path: str, threshold: float = 0.5) -> bool:
+    """ffprobe whether the first video stream is VFR.
+
+    Never raises — a missing ffprobe / unreadable file / odd output returns
+    ``False`` (assume CFR), so the pipeline only ever does *extra* work when it is
+    confident the source is variable-rate.
+    """
+    try:
+        result = subprocess.run(
+            ["ffprobe", "-v", "error", "-select_streams", "v:0",
+             "-show_entries", "stream=r_frame_rate,avg_frame_rate",
+             "-of", "csv=p=0", video_path],
+            capture_output=True, text=True,
+        )
+        if result.returncode != 0:
+            return False
+        parts = result.stdout.strip().split(",")
+        if len(parts) < 2:
+            return False
+        return is_vfr(parts[0], parts[1], threshold=threshold)
+    except (FileNotFoundError, OSError):
+        return False