Skip to content

S12.md

Latest commit

 

History

History
97 lines (62 loc) · 6.88 KB

S12.md

File metadata and controls

97 lines (62 loc) · 6.88 KB
Raw

Now, we are Deconstructing Netflix.

As a Engineer, you know how to move text and JSON. But moving a 500GB raw 4K video file, processing it into 120 different formats, and streaming it concurrently to 100 million living rooms on a Friday night without a single second of buffering? That requires a completely different architectural mindset.

1. EASY: The "Studio Master" (Massive File Ingestion)

Scenario: A Hollywood studio has just finished editing the season finale of Stranger Things. They need to upload the raw, uncompressed "Master" file (500GB) to your servers. The Problem: Uploading 500GB over standard HTTP is guaranteed to fail. A network blip at 99% will force them to restart the 10-hour upload. Task: Design a highly reliable, resumable ingestion layer for massive blobs of data.

Solution: S3 Multipart Upload & Transfer Acceleration

We use AWS S3 (Simple Storage Service) as our foundational data lake, but we optimize the network path.

The Workflow:

  1. Transfer Acceleration: Instead of routing the upload over the public internet all the way to our Virginia data center, the studio's client uploads to the nearest AWS Edge Location (e.g., Los Angeles). From there, it rides AWS's private, high-speed fiber backbone to Virginia.
  2. Multipart Upload: The 500GB file is chopped into 10,000 smaller parts (50MB each) on the studio's machine.
  3. Parallel Upload: The client uploads 20 parts simultaneously.
  4. Assembly: Once all 10,000 parts arrive, S3 mathematically verifies their checksums and stitches them back into the single 500GB Master file.

Production Grade Architecture

The "Event-Driven" Kickoff & Lifecycle: Storing petabytes of 500GB master files will bankrupt you if left on standard SSD storage.

  • Event Trigger: The moment S3 finishes assembling the Master file, it emits an s3:ObjectCreated event to Amazon EventBridge, which triggers an AWS Lambda function to start the Transcoding Pipeline (our next step).
  • Lifecycle Policies: We implement a strict S3 Lifecycle Rule.
  • Day 0-30: S3 Standard (for immediate processing).
  • Day 31-90: S3 Infrequent Access (cheaper).
  • Day 91+: Move to S3 Glacier Deep Archive (costs pennies, but takes 12 hours to retrieve).

2. MEDIUM: The "Baking Process" (Distributed Video Transcoding)

Scenario: You have the 500GB 4K Master file. The Problem: A user watching on a 3G mobile network in rural India cannot stream a 500GB file. A user on an Apple TV needs 4K Dolby Vision. A user on a laptop needs 1080p stereo. Task: You must convert the single Master file into ~120 different combinations of resolutions, bitrates, and audio codecs. Doing this on a single server would take 3 days per episode. You need it done in 30 minutes.

Solution: The "MapReduce" Pattern for Video (Chunking)

We do not process the video sequentially. We use a massive distributed worker fleet.

The Workflow:

  1. Inspection & Chunking (The Split): A microservice reads the Master file and splits it into 3-minute chunks (e.g., a 60-minute episode becomes twenty 3-minute files).
  2. The Queue: We push 20 "Chunk Transcode Tasks" into an Amazon SQS queue (or Kafka) for each of the 120 formats. (Total: 2,400 tasks).
  3. The Worker Fleet: An Auto-Scaling Group of 1,000 EC2 instances running FFmpeg (a video processing engine) pulls tasks from the queue.
  4. Parallel Processing: Because the chunks are small, 2,400 servers can process all 2,400 chunks simultaneously in a few minutes.
  5. Assembly (The Merge): Once all chunks for the "1080p MP4" format are done, a final worker stitches them back together into the final streaming file.

Production Grade Architecture

Cost Optimization with Spot Instances & Priority Queues: Video encoding is 100% CPU-bound. Paying retail price for 1,000 high-CPU servers is extremely expensive.

  • Compute: We use AWS Spot Instances (unused AWS capacity sold at a 70-90% discount).
  • Fault Tolerance: AWS can terminate a Spot Instance with exactly 2 minutes' notice. If a server dies mid-encode, the SQS message "Visibility Timeout" expires, and the task simply pops back onto the queue for another server to pick up. No data is corrupted.
  • Priority Routing: We use two queues: High_Priority_Queue (for a new release dropping tonight) and Low_Priority_Queue (for re-encoding a 1990s movie into a new codec).

3. HARD: The "Friday Night Premiere" (Global Edge Delivery)

Scenario: It's 8:00 PM on Friday. 100 million users hit "Play" on the new season. The Problem: You have 120 versions of the video sitting in AWS S3 in Virginia. If 100 million people stream directly from AWS, you will consume a massive chunk of the entire world's internet bandwidth, causing global outages and incurring hundreds of millions of dollars in AWS egress fees. Task: Design the Edge Delivery architecture to stream this video with zero buffering.

Solution: Netflix Open Connect (Custom CDN) & Adaptive Bitrate Streaming

Netflix does not use standard cloud providers (like AWS or GCP) to serve video. They built their own hardware Content Delivery Network called Open Connect.

The Architecture:

  1. Open Connect Appliances (OCAs): Netflix builds custom red server racks packed with massive hard drives. They literally ship these boxes for free to ISPs all over the world (Comcast, AT&T, Vodafone). The ISP plugs the box directly into their local network.
  2. Proactive Caching: During the night (when internet traffic is low), Netflix pushes the newly transcoded files from AWS to these OCAs.
  3. The Play Button: When you hit "Play", the Netflix app talks to the AWS backend (control plane) to verify your password and get recommendations.
  4. The Handoff: AWS looks at your IP address and tells your TV: "Your video is waiting for you on the Comcast Open Connect box 3 miles from your house. Go get it." The massive video files never cross the public internet backbone.

Production Grade Architecture

Adaptive Bitrate Streaming (DASH/HLS) & Client-Side Telemetry: What happens if someone in the house starts downloading a huge game, and your TV's internet speed suddenly drops by 50%?

  • The Manifest File: Before the video starts, your TV downloads a tiny text file called a Manifest (.m3u8 or .mpd). This file lists the URLs for every chunk of video in every resolution.

  • Dynamic Switching: Your TV downloads the video in 3-second chunks.

  • Chunk 1: TV measures 50Mbps internet speed. Requests 4K chunk.

  • Chunk 2: Someone starts a download. Speed drops to 5Mbps.

  • Chunk 3: The TV detects the drop. It instantly looks at the Manifest and requests the next 3-second chunk in 720p instead of 4K.

  • The Result: The video quality drops slightly, but the video never buffers. This logic lives entirely on the Client (the TV/App), keeping the Backend stateless and massively scalable.