Video Upload Pipeline

A high-throughput video processing microservice written in Go, powered by ffmpeg. Can handle single more (upto thousands even!) uploads. it queues, probes, transcodes, thumbnails, and previews each one concurrently.

---

Architecture

HTTP POST /upload (multipart, N files)
        │
        ▼
  ┌─────────────┐     buffered channel (512)     ┌──────────────────────┐
  │  API Handler│ ──────────────────────────────► │  Worker Pool         │
  │  (goroutine)│                                 │  (N=CPU core goroutines)│
  └─────────────┘                                 └──────────┬───────────┘
                                                             │
                                                             ▼
                                                   ┌─────────────────┐
                                                   │   Processor     │
                                                   │  1. ffprobe     │
                                                   │  2. transcode   │
                                                   │  3. thumbnails  │
                                                   │  4. GIF preview │
                                                   │  5. MP4 clip    │
                                                   │  6. meta.json   │
                                                   └─────────────────┘

What happens per video

Step	Tool	Output
Probe	ffprobe	Duration, resolution, codecs, FPS
Transcode	ffmpeg libx264	{id}_720p.mp4, {id}_480p.mp4, {id}_360p.mp4
Thumbnails	ffmpeg (1 frame, 3 sizes)	thumb_small.jpg, thumb_medium.jpg, thumb_large.jpg
Preview GIF	ffmpeg (3 s, 10 fps, palettegen)	preview.gif
Preview Clip	ffmpeg (5 s MP4)	preview_clip.mp4
Metadata	Go JSON	meta.json

---

Output directory structure

processed/
└── {uuid}/
    ├── meta.json
    ├── {uuid}_720p.mp4
    ├── {uuid}_480p.mp4
    ├── {uuid}_360p.mp4
    ├── thumb_small.jpg      (160×90)
    ├── thumb_medium.jpg     (320×180)
    ├── thumb_large.jpg      (640×360)
    ├── preview.gif          (480px wide, 3 s, 10 fps)
    └── preview_clip.mp4     (640px wide, 5 s)

---

meta.json example

{
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "original_file": "Jane Smith - Episode 12 720p.mp4",
  "clean_title": "Jane Smith - Episode 12",
  "artist": "Jane Smith",
  "scene": "Episode 12",
  "upload_size_bytes": 734003200,
  "upload_size_mb": 699.97,
  "duration_sec": 2743.5,
  "duration_human": "45:43",
  "width": 1280,
  "height": 720,
  "fps": 29.97,
  "codec": "h264",
  "audio_codec": "aac",
  "resolutions": [
    { "label": "720p",  "width": 1280, "height": 720,  "bitrate": "2500k", "file": "..._720p.mp4", "size_mb": 143.2 },
    { "label": "480p",  "width": 854,  "height": 480,  "bitrate": "1200k", "file": "..._480p.mp4", "size_mb":  68.5 },
    { "label": "360p",  "width": 640,  "height": 360,  "bitrate": "700k",  "file": "..._360p.mp4", "size_mb":  39.1 }
  ],
  "thumbnails": [
    { "size": "small",  "file": "thumb_small.jpg",  "width": 160, "height": 90  },
    { "size": "medium", "file": "thumb_medium.jpg", "width": 320, "height": 180 },
    { "size": "large",  "file": "thumb_large.jpg",  "width": 640, "height": 360 }
  ],
  "preview_gif":  "preview.gif",
  "preview_clip": "preview_clip.mp4",
  "processed_at": "2026-02-18T14:22:10Z"
}

---

API

POST /upload

Upload one or more videos for processing.

# Single file
curl -X POST http://localhost:8080/upload \
  -F "video=@/path/to/video.mp4"

# Multiple files at once
curl -X POST http://localhost:8080/upload \
  -F "video=@clip1.mp4" \
  -F "video=@clip2.mkv" \
  -F "video=@episode03.avi"

Response 202 Accepted:

{
  "uploaded": 2,
  "jobs": [
    { "job_id": "abc-123", "filename": "clip1.mp4", "status": "queued" },
    { "job_id": "def-456", "filename": "clip2.mkv", "status": "queued" }
  ]
}

---

GET /status/{job_id}

Poll the status of a specific job.

curl http://localhost:8080/status/abc-123

Returns the full Job object including meta once done.

---

GET /jobs

List all jobs (in-memory).

curl http://localhost:8080/jobs

---

Running

With Docker (recommended)

docker compose up --build

Locally (requires Go 1.22+ and ffmpeg)

cd vidpipeline
go mod tidy
go run .

---

Configuration

Env var	Default	Description
OUTPUT_DIR	./processed	Root directory for processed videos

Worker count = number of CPU cores (min 2). Each worker runs FFmpeg jobs sequentially, so N workers means N videos processed in parallel.

---

Title parsing

The pipeline parses messy upload filenames into clean Artist - Scene titles:

Input filename	Artist	Scene
Jane.Smith-Episode.12.720p.x264.mp4	Jane Smith	Episode 12
The.Band_Live.At.Madison.Square.Garden.mkv	—	The Band Live At Madison Square Garden
artist - scene name [1080p][BluRay].avi	Artist	Scene Name

---

Extending

• Persistent job store: Replace model.Store with a Redis or PostgreSQL backend
• Webhook callbacks: Add a callback URL to the job, POST to it on completion
• S3/GCS upload: After proc.Run(), upload p.dir contents to object storage
• Priority queues: Use separate channels per priority tier
• Hardware encoding: Replace libx264 with h264_nvenc (NVIDIA) or h264_videotoolbox (macOS)