Skip to content

S52.md

Latest commit

 

History

History
72 lines (45 loc) · 5.76 KB

S52.md

File metadata and controls

72 lines (45 loc) · 5.76 KB
Raw

We look at the physical foundation of everything we've built. We have spent weeks talking about Spark, Flink, and the Medallion Architecture. But what is physically sitting on those hard drives in AWS S3?

Historically, Data Lakes were just massive folders full of Parquet or CSV files. But raw S3 buckets have a fatal flaw: they do not understand transactions.

Now, we conquer Data Lakehouse Formats: Apache Iceberg and Delta Lake. We are going to bring strict, relational database guarantees to raw cloud storage.

1. EASY: The "Corrupted Read" (The Problem with Raw Data Lakes)

Scenario: Your nightly Apache Spark job is writing 500 new Parquet files to the sales_data S3 bucket. The Problem: Halfway through the write, an AWS EC2 Spot Instance is terminated. The Spark cluster crashes. It only successfully wrote 250 files. The Disaster: S3 is just a dumb object store; it has no "rollback" feature. Downstream business analysts running queries in Tableau will blindly read those 250 partial files mixed with old data. The daily revenue report is mathematically corrupted, and nobody knows until the CFO complains. Task: Bring ACID (Atomicity, Consistency, Isolation, Durability) guarantees to an object store.

Solution: The Metadata Layer (The Lakehouse)

Formats like Delta Lake (by Databricks) and Apache Iceberg (by Netflix/Apple) are not execution engines. They are not storage hardware. They are metadata layers that sit between your compute (Spark) and your storage (S3).

The Architecture: They enforce Atomicity. A write job either succeeds 100%, or it fails completely and is invisible to readers. There are no partial reads.

2. MEDIUM: The "Concurrent Collision" (The Transaction Log)

Scenario: You have a real-time Flink streaming job inserting new users into the Silver layer every second. Simultaneously, a batch dbt job is running an UPDATE statement to mask PII (Personally Identifiable Information) on that exact same table. The Problem: Parquet files are completely immutable. You cannot "update" a row inside a Parquet file. To update a record, the engine must read the file, rewrite a brand new Parquet file with the changes, and delete the old one. If Flink and dbt try to do this to the same S3 bucket at the exact same millisecond, they will overwrite each other, causing massive data loss. Task: Safely manage high-concurrency INSERT, UPDATE, and DELETE operations on immutable files.

Solution: Snapshot Isolation and The Log

Delta Lake and Iceberg solve this by forcing every single engine to check a central ledger before touching the data.

The Architecture: Alongside your data files, the format maintains a strictly ordered Transaction Log (e.g., the _delta_log folder).

The Workflow:

  1. The Write: When Spark runs an UPDATE, it does not touch the existing Parquet files. It writes brand new Parquet files containing the updated rows to S3.
  2. The Commit: Spark then writes a tiny JSON file to the transaction log that says: "Commit 005: Add file_v2.parquet, Remove file_v1.parquet."
  3. The Read (Snapshot Isolation): When an analyst queries the table, the query engine (like ClickHouse or Athena) does not blindly scan S3. It reads the transaction log first. The log tells it exactly which Parquet files make up the current "Snapshot." It completely ignores file_v1.parquet, even though the physical file is still sitting on the hard drive.

Production Grade Architecture (Optimistic Concurrency Control): What if Flink and dbt try to write Commit 005 at the exact same time? The Lakehouse uses Optimistic Concurrency Control. It allows both to write their Parquet files to S3. But whoever writes to the log first wins. If dbt wins, Flink realizes Commit 005 is taken, looks at what changed, and quietly retries its operation as Commit 006. No data is lost.

3. HARD: The "Undo Button" (Time Travel & Compaction)

Scenario: A junior data engineer runs a cleanup script: DELETE FROM users WHERE region = 'US'. They meant to type 'UK'. They just wiped out 80% of your company's user base in production. The Problem: In a traditional Postgres database, if you don't have a backup from 5 minutes ago, you are updating your resume. Restoring a multi-terabyte Data Lake from a cold backup would take 24 hours. Task: Instantly revert a massive data deletion without moving a single byte of data.

Solution: Time Travel (Zero-Copy Rollback)

Because Lakehouse formats never actually delete the old Parquet files immediately (they just mark them as "removed" in the transaction log), the historical data is still physically sitting on the hard drive.

The Workflow:

  1. The Panic: The junior engineer realizes their mistake.
  2. The Fix: You simply run a SQL command: RESTORE TABLE users TO TIMESTAMP AS OF '2026-03-08 10:00:00'
  3. The Magic: The engine creates Commit 006 in the log, which simply points the metadata back to the original Parquet files from before the delete. The data is instantly "restored" in 200 milliseconds.

Production Grade Architecture (Vacuum & Optimize): You cannot keep every file forever, or your S3 bill will bankrupt the company (as we discussed in Day 47). A Senior Data Engineer must orchestrate two critical maintenance jobs on the Lakehouse:

  • OPTIMIZE (Bin-Packing): Streaming jobs create thousands of tiny 1MB Parquet files, which destroys read performance. OPTIMIZE safely reads these tiny files in the background and rewrites them into massive, highly efficient 1GB files, updating the log seamlessly.
  • VACUUM: This physically deletes old Parquet files that are no longer referenced by the current transaction log, enforcing your Time Travel retention window (e.g., "Only keep history for 7 days").