ReelForge

An AI pipeline that forges scroll-stopping vertical reels from talking-head video.

📐 Architecture diagram & design deep-dive →

Overview

ReelForge takes a single speaker clip (a podcast cut, an interview, a webinar moment) plus its word-level transcript and automatically produces a polished, captioned, color-graded, music-scored 9:16 short — the kind of clip you'd post to TikTok, Reels, or Shorts. It plans the edit with a vision LLM, generates or sources B-roll, tracks the speaker's face for dynamic zoom, renders kinetic captions, masters the audio, grades the color, scores it with ducked music, and finally grades its own output with an AI quality check.

It is built as a sequence of ten independent, individually-skippable stages, each with graceful fallbacks so a missing API key or unavailable service degrades quality rather than crashing the run.

Note — This is a self-contained portfolio/reference implementation. No API keys, credentials, or media are included. The ingest layer reads a local sample transcript (samples/sample_transcript.json) instead of any production backend, so you can read and run the pipeline without external infrastructure. Bring your own input.mp4 and API keys to generate real reels.

Key features

• LLM-planned edit — one video upload, three Gemini passes: B-roll segments, camera zoom events, and emphasis words.
• Cost-aware, quality-gated B-roll routing — try Sora 2, grade each clip, retry or auto-demote to free Pexels stock, and track estimated dollars saved.
• Anchor-word timing snapping — AI-proposed cut points are fuzzy-matched against real transcript word timings for ~0.01 s precision.
• Face-tracked digital zoom — MediaPipe detection smoothed with an exponential moving average for a steady "camera operator" punch-in.
• Kinetic captions — word-chunked, bold, stroke-outlined captions with emphasis words popped in color.
• Frame-accurate compositing — base video + B-roll (cut / fade / crossfade / swipe) + captions, all snapped to frame boundaries.
• Broadcast audio mastering — loudness-normalized voice with transition SFX mixed in without crushing speech.
• Cinematic color grading — .cube LUT presets (warm / cool / vibrant / vintage), with sample LUTs generated procedurally.
• Content-aware background music — mood-classified, fetched from Freesound, and sidechain-ducked under the voice.
• Self-evaluating output — the finished reel is scored by Gemini across six dimensions with a pass/fail gate.
• Graceful degradation everywhere — every external dependency has a fallback.

Architecture

The orchestrator (reelforge/cli.py) runs ten ordered stages, passing typed Pydantic models between them. Any stage can be disabled with a --skip-* flag.

 input.mp4 + word-level transcript
              │
              ▼
   ┌──────────────────────┐
   │ 1 · Ingest           │  ClipData (context + transcript + video)
   └──────────┬───────────┘
              ▼
   ┌──────────────────────┐
   │ 2 · Analysis         │  Gemini "Director" — upload once, 3 passes:
   │   (The Director)     │  segments · zoom events · emphasis words
   └──────────┬───────────┘
              ▼
   ┌──────────────────────┐
   │ 3 · Routing          │  per segment: Sora 2 → quality gate → stock
   │   (The Brain)        │  fallback · or internal zoom / text
   └───┬───────────┬──────┘
       │           │
       ▼           ▼
  ┌─────────┐  ┌─────────────┐
  │ 4 · Zoom│  │ 5 · Captions│   MediaPipe EMA-smoothed zoom +
  │ engine  │  │ engine      │   kinetic typography (Pillow)
  └────┬────┘  └──────┬──────┘
       └─────┬────────┘
             ▼
   ┌──────────────────────┐
   │ 6 · Stitch           │  MoviePy composite: base + B-roll
   │                      │  (+ transitions) + captions, frame-snapped
   └──────────┬───────────┘
              ▼
   ┌──────────────────────┐
   │ 7 · Audio mastering  │  FFmpeg loudnorm (-14 LUFS) + SFX (amix)
   └──────────┬───────────┘
              ▼
   ┌──────────────────────┐
   │ 8 · Color grade      │  FFmpeg lut3d (.cube preset)
   └──────────┬───────────┘
              ▼
   ┌──────────────────────┐
   │ 9 · Music            │  Gemini classify · Freesound · sidechain duck
   └──────────┬───────────┘
              ▼
   ┌──────────────────────┐
   │ 10 · Quality eval    │  Gemini scores 6 dimensions · pass if avg ≥ 75
   │   (QA)               │
   └──────────┬───────────┘
              ▼
   Final 9:16 reel + quality report

#	Stage	What it does
1	Ingest	Load the clip's word-level transcript + source video into typed models via a pluggable loader (local file / URL).
2	Analysis — "The Director"	One video upload → three Gemini passes: B-roll segments, camera zoom events, and emphasis words.
3	Routing — "The Brain"	Per segment, choose the cheapest sufficient source: Sora (cinematic) → quality gate → stock fallback; or internal zoom/text.
4	Zoom engine	Detect & track the speaker's face, apply smoothed digital "punch-in" zoom.
5	Caption engine	Render word-chunked kinetic captions; emphasis words pop in color.
6	Stitch	Composite base video + B-roll (with fade/crossfade/swipe transitions) + captions, frame-snapped.
7	Audio mastering	Normalize the voice to broadcast loudness, then inject transition SFX without crushing speech.
8	Color grade	Apply a cinematic .cube LUT (warm / cool / vibrant / vintage).
9	Music	Classify the clip's mood/genre, fetch a matching CC track, duck it under speech.
10	Quality eval — "QA"	Score the finished reel across six dimensions; pass when the average ≥ 75.

See docs/architecture.md for the full, stage-by-stage technical breakdown.

Tech stack

Layer	Technology
Language	Python 3.10+
Domain models	Pydantic v2
Edit planning · QA · music classification	Google Gemini (google-genai)
Cinematic B-roll generation	OpenAI Sora 2 (openai)
Stock B-roll	Pexels API
Background music	Freesound API (Creative Commons)
Face detection	MediaPipe · OpenCV
Caption rendering	Pillow
Compositing	MoviePy (v1/v2 compatible)
Audio / color / ducking	FFmpeg + ffprobe (loudnorm, lut3d, sidechaincompress, amix)
Numerics	NumPy
HTTP	requests

Project structure

reelforge/
├── reelforge/
│   ├── cli.py              # orchestrator — drives all 10 stages
│   ├── __main__.py         # `python -m reelforge` entry point
│   ├── config.py           # environment-based settings (no hardcoded secrets)
│   ├── models.py           # shared Pydantic domain models
│   ├── ingest/             # transcript + video -> ClipData (pluggable backend)
│   │   ├── source.py
│   │   └── transcript_loader.py
│   ├── analysis/           # Gemini "Director": segments, zoom, emphasis
│   │   └── director.py
│   ├── routing/            # "Brain": Sora / stock routing + quality gate
│   │   ├── router.py
│   │   ├── sora_generator.py
│   │   └── stock_fetcher.py
│   ├── effects/            # kinetic captions + face-tracked zoom
│   │   ├── captions.py
│   │   └── zoom.py
│   ├── compose/            # stitching + audio mastering + color + music
│   │   ├── stitcher.py
│   │   ├── audio.py
│   │   ├── color.py
│   │   └── music.py
│   └── quality/            # Gemini quality evaluation
│       └── evaluator.py
├── assets/luts/            # sample cinematic color LUT
├── samples/                # sample transcript for a runnable demo
├── docs/                   # architecture deep-dive
├── tests/                  # self-contained unit tests (no network required)
├── pyproject.toml
├── requirements.txt
├── .env.example
└── LICENSE

Getting started

Prerequisites: Python 3.10+ and FFmpeg (ffmpeg + ffprobe) installed separately and available on your PATH.

# 1. Install dependencies
pip install -r requirements.txt
#    ...or install as a package:  pip install -e .

# 2. Configure credentials (all optional; a missing key -> that stage is
#    skipped or falls back to a free/local alternative)
cp .env.example .env        # then edit .env with your own keys

# 3. Add a source video named input.mp4 in the project root

# 4. Run the pipeline
python -m reelforge --clip_id demo-001
#    ...or, if installed:  reelforge --clip_id demo-001

Environment variables (see .env.example): GEMINI_API_KEY, OPENAI_API_KEY, PEXELS_API_KEY, FREESOUND_API_KEY, plus optional path overrides REELFORGE_INPUT_VIDEO and REELFORGE_TRANSCRIPT.

Useful CLI flags: --skip-zoom, --skip-captions, --skip-audio, --skip-color, --skip-music, --skip-evaluation, --color-preset {cinematic_warm,cinematic_cool,vibrant,vintage,neutral}, --music-energy {low,medium,high}.

Run the tests

python -m unittest discover -s tests -v

The included tests exercise the transcript model, the ingest layer, caption chunking, and routing math using synthetic fixtures — no network or API keys needed.

How it works

These are the design decisions that separate "calls an API" from "a production-minded pipeline":

• Single-upload, three-pass analysis. The source video is uploaded to the Gemini Files API once and the file handle is reused across the segment, zoom, and emphasis passes — a deliberate cost/latency optimization over three separate uploads, with each prompt kept narrow for more reliable structured (JSON-schema) output.
• Anchor-word timing snapping. LLMs give loose timestamps. Instead of trusting them, each AI-proposed segment carries an anchor word that is fuzzy-matched (case-insensitive, punctuation-stripped) against the real transcript word timings, pulling cut points to ~0.01 s precision.
• Cost-aware, quality-gated routing. Expensive Sora generation is tried first, each clip is graded by a Gemini evaluator, REGENERATE retries up to a cap, and USE_FALLBACK automatically demotes to free Pexels stock — with estimated dollar savings tracked per run.
• Audio mastering in the right order. The voice is normalized to −14 LUFS first, then SFX are mixed in at a fixed relative level with amix:normalize=0 — deliberately avoiding a second normalization pass that would re-trigger the limiter and squash the speech.
• Content-aware sidechain ducking. Background music is ducked under the voice with FFmpeg sidechaincompress, using per-energy presets so a calm clip and a high-energy clip duck differently.
• EMA-smoothed face tracking. Per-frame MediaPipe detections are smoothed with an exponential moving average to mimic a real camera operator and kill jitter, with graceful fallback to the last known position / center crop.
• Deterministic safety nets over the LLM. Spacing rules, zoom/segment overlap removal, and frame-boundary snapping correct the model's output in code rather than hoping the prompt was obeyed.

Notes / limitations

• FFmpeg is required for stages 6–9 (audio, color, music) and must be installed separately.
• No media, keys, or proprietary data ship with this repo. The ingest layer is a swappable adapter (ClipSource) that defaults to local files; point it at your own backend by subclassing it and overriding its resolution hooks — nothing downstream changes.
• API access (Gemini, Sora 2, Pexels, Freesound) is required to generate a real reel; without keys the corresponding stages skip or fall back so the pipeline still runs end-to-end.

License

MIT