Skip to content

S67.md

Latest commit

 

History

History
62 lines (38 loc) · 5.44 KB

S67.md

File metadata and controls

62 lines (38 loc) · 5.44 KB
Raw

We return to the streaming engine block. We spent tearing apart Apache Spark's physical execution network. But batch processing is forgiving. If a Spark job fails, you just run it again.

Stream processing is ruthless. When the Sentinel fraud detection engine is evaluating thousands of live transactions per second using Apache Flink, a server crash is guaranteed to happen eventually. When it does, how do you ensure a user isn't double-charged for a transaction, or worse, that a fraudulent event isn't completely dropped during the reboot?

Now, we master Apache Flink Internals: The Chandy-Lamport Algorithm and Exactly-Once Semantics. We are going to look at how distributed systems take perfect photographs of themselves while moving at a million miles an hour.

1. EASY: The "Double Charge" (At-Least-Once vs. Exactly-Once)

Scenario: Sentinel processes a Kafka event: "User A transfers $500 to User B." Flink updates its internal state to reflect the transfer and sends an alert. The Problem: Exactly one millisecond after sending the alert, but before Flink can tell Kafka "I am done with this message," the Flink server loses power. The Disaster: When a new Flink server boots up, it looks at Kafka. Kafka says, "You never finished processing the $500 transfer." Flink reads the message again, updates its state again, and sends a second alert. User A is effectively charged $1,000. This is "At-Least-Once" processing, and in financial systems, it is unacceptable. Task: Guarantee that every single event affects the internal state of the engine exactly one time, even if the hardware explodes.

Solution: Distributed State Snapshots

Flink achieves Exactly-Once processing by periodically saving the entire state of the cluster (every variable, every timer, every Kafka offset) into a durable storage layer like AWS S3. If a node dies, the entire cluster rolls back to the last successful snapshot.

2. MEDIUM: The "Flock of Birds" (The Distributed Snapshot Problem)

Scenario: Taking a snapshot of a single database on one server is easy; you just lock the table and copy the file. The Problem: Flink is running across 50 different EC2 instances. Data is flowing between them constantly. Trying to capture a mathematically perfect, globally consistent state of 50 moving servers is like trying to take a panoramic photograph of a flock of birds using 50 different cameras. If Camera 1 snaps a picture at 12:00:00, and Camera 2 snaps a picture at 12:00:01, a bird might have flown from Camera 1's view into Camera 2's view. The Disaster: The final stitched photograph shows the exact same bird twice. The state is corrupted. Task: Coordinate a snapshot across a distributed network without pausing the multi-terabyte data stream.

Solution: The Chandy-Lamport Algorithm

Invented in 1985, this algorithm allows a distributed system to record a consistent global state asynchronously. Flink’s founders adapted this brilliant mathematical concept to work on continuous data streams.

The Architecture: Flink does not stop the world to take a picture. It weaves the camera into the data stream itself.

3. HARD: The "Checkpoint Barrier" (Asynchronous State Alignment)

Scenario: We need to capture the exact state of the Kafka reader (Source), the fraud logic (Operator), and the database writer (Sink) at a specific logical point in time, without any of them talking to each other.

The Workflow (How Flink actually does it):

  1. The Injection: The Flink JobManager (the brain) wakes up every 10 seconds. It injects a special, invisible message called a Checkpoint Barrier directly into the raw Kafka data stream.
  2. The Flow: This Barrier flows through the pipes exactly like a normal piece of data. It does not overtake the transaction events; it waits in line.
  • (Data A) -> (Data B) -> [BARRIER 1] -> (Data C)
  1. The Source Snapshot: When the Kafka Source node sees [BARRIER 1], it instantly realizes: "Ah, this is the dividing line." It saves its current Kafka offset to AWS S3, forwards the Barrier downstream, and immediately goes back to reading Data C.
  2. The Operator Snapshot: The Barrier hits the CEP Operator holding the complex fraud sequences. The Operator sees the Barrier. It pauses for a microsecond, copies its current RocksDB state (which includes Data A and Data B, but absolutely not Data C) to AWS S3, forwards the Barrier to the Sink, and resumes processing.
  3. The Alignment (The Hard Part): What if the fraud Operator is receiving data from two different Kafka topics? It will receive two different Barriers.
  • It receives [BARRIER 1] from Topic X.
  • It must pause processing any more data from Topic X. It buffers it.
  • It waits patiently until [BARRIER 1] arrives from Topic Y.
  • Once both barriers align, it takes its snapshot and releases the buffered data. This guarantees perfect mathematical consistency across multiple chaotic network streams.
  1. The Commit: Once the Sink node sees the Barrier and saves its state, the JobManager declares Checkpoint 1 successful.

Why this is best: If the server dies 4 seconds later while processing Data C, Flink simply reboots, downloads Checkpoint 1 from S3, resets the Kafka offsets to exactly where they were at the Barrier, and processes Data C again. The internal state is completely untouched by the crash. You have achieved Exactly-Once semantics without ever pausing the data pipeline.