Skip to content

S17.md

Latest commit

 

History

History
108 lines (67 loc) · 7.16 KB

S17.md

File metadata and controls

108 lines (67 loc) · 7.16 KB
Raw

As a Engineer, you work with Apache Spark, Apache Kafka, and Kubernetes. But what keeps those systems alive when network cables get unplugged? How does a Spark cluster know which node is the "Master" if the network splits in half? How does Kafka guarantee you don't lose messages when a broker's motherboard fries?

Now, we solve the Split-Brain Problem and dive into the mathematics of truth: Zookeeper, etcd, and the Raft Algorithm.

1. EASY: The "Split-Brain" (Leader Election Basics)

Scenario: You have a critical database with two nodes: Node A (Leader) and Node B (Follower). They communicate via a network switch. The Problem: The network switch dies. Node A and Node B are both healthy, but they can no longer talk to each other.

  • Node B says: "I can't reach the Leader! I must become the new Leader."
  • Node A says: "I am still the Leader." The Disaster: You now have a "Split-Brain." Both nodes accept write requests from different web servers. Data diverges, causing massive, unresolvable corruption. Task: Design a fail-safe mechanism to ensure there is never more than one Leader.

Solution: Quorum (The Majority Vote)

We cannot rely on just two nodes. In distributed systems, 2 is the most dangerous number. We need an odd number of nodes (e.g., 3, 5, or 7) to achieve a Quorum.

The Workflow:

  1. The Math: To elect a leader or commit a write, a strict majority of nodes must agree. The formula is $Q = \lfloor N/2 \rfloor + 1$. For a 3-node cluster, Quorum is 2.
  2. The Network Split: Let's say we have Nodes A, B, and C. A network partition isolates Node A from B and C.
  3. The Election: * Node A tries to be leader but only has 1 vote (itself). It fails to reach Quorum ($1 < 2$). It steps down and refuses writes.
  • Nodes B and C talk to each other. They have 2 votes ($2 \ge 2$). They elect B as the new Leader.
  • Result: Split-Brain is mathematically impossible.

Production Grade Architecture

Fencing Tokens (The Storage Failsafe): Even with Quorum, a "zombie" Node A might pause due to a heavy Java Garbage Collection cycle, wake up 30 seconds later, still think it's the leader, and try to write a file to your S3 bucket or SAN storage just as Node B is also writing to it.

The Fix:

  • Every time a Leader is elected, it gets a mathematically increasing Fencing Token (e.g., Epoch: 5).
  • When Node B is elected, it gets Epoch: 6.
  • The Storage Layer (like HDFS or AWS S3 conditional writes) is configured to only accept writes with a token greater than or equal to the highest one it has ever seen.
  • When zombie Node A wakes up and tries to write with Epoch: 5, the storage layer rejects it: "Access Denied, I've already seen Epoch 6."

2. MEDIUM: The "Cluster Coordinator" (etcd & Zookeeper)

Scenario: You have a massive compute cluster running 5,000 workers. The Problem: The workers need to know shared configurations, which nodes are dead, and who holds the distributed locks for specific files. You cannot use a standard relational database (like Postgres) for this because if Postgres goes down, the entire 5,000-node cluster becomes paralyzed. Task: Design the highly available "Control Plane" for the cluster.

Solution: Distributed Key-Value Stores (etcd)

We use a specialized, lightweight, strongly consistent Key-Value store. For Hadoop/Kafka, this was traditionally Apache Zookeeper. For modern systems like Kubernetes, it is etcd.

The Workflow:

  1. The Architecture: We deploy etcd as a 5-node cluster. It stores tiny amounts of critical metadata (not heavy data logs).
  2. The "Watch" Pattern: Instead of 5,000 workers constantly querying etcd every second ("Did the config change?"), etcd uses long-polling/gRPC streams.
  3. The Trigger: * A worker establishes a Watch on the key /config/spark/memory_limit.
  • When an admin updates that key, etcd instantly pushes the update event down the open connection to all 5,000 workers simultaneously.

Production Grade Architecture

Managing Ephemeral State (Heartbeats): How does the cluster know if a worker node suddenly loses power?

  • Leases and Keep-Alives: When a worker boots up, it writes its IP address to etcd and attaches a "Lease" with a 10-second Time-To-Live (TTL).
  • The Heartbeat: The worker runs a background thread that sends a ping to etcd every 3 seconds: "Renew my lease!"
  • The Failure: If the server loses power, the heartbeats stop. After 10 seconds, the Lease expires. etcd automatically deletes the worker's IP from the directory and triggers a "Watch" event to the Master node, which then reschedules that worker's tasks onto a healthy machine.

3. HARD: The "Internal Brain" (The Raft Consensus Algorithm)

Scenario: You are modernizing a high-throughput event streaming platform. The Problem: Managing an external Zookeeper cluster just to keep your message brokers alive is an operational nightmare. If Zookeeper goes down, the brokers crash. Task: You must remove the external dependency. You need to build consensus directly into the message brokers themselves so they can elect their own leaders and replicate data safely. (This is exactly what Kafka did when they created KRaft to replace Zookeeper).

Solution: The Raft Algorithm

Raft is the modern standard for distributed consensus. It relies on three states: Follower, Candidate, and Leader.

The Workflow (Leader Election):

  1. The Baseline: All nodes start as Followers. They wait for a heartbeat from a Leader.
  2. The Timeout: If a Follower doesn't hear a heartbeat within a randomized timeout (e.g., 150ms - 300ms), it promotes itself to a Candidate.
  3. The Campaign: The Candidate increments the Term number (Epoch) and sends a RequestVote RPC to all other nodes.
  4. The Rule: A node will grant its vote to the first Candidate that asks during that Term, provided the Candidate's data logs are as up-to-date as its own.
  5. The Victory: If the Candidate gets votes from a majority, it becomes the Leader and immediately starts blasting heartbeats to suppress new elections.

Production Grade Architecture

Log Replication (The Write Path): How does Raft guarantee that a message sent to the Leader is never lost, even if the Leader crashes a millisecond later?

  • The Two-Phase Commit:
  1. AppendEntries: A user sends a message to the Leader. The Leader writes it to its own local log (but does not commit it to the application state yet).
  2. Replication: The Leader sends the log entry to the Followers.
  3. The Quorum Ack: Once a majority of Followers reply, "We have written this to our disk," the Leader knows the data is safely replicated across the cluster.
  4. The Commit: The Leader officially "Commits" the message, updates its internal state machine, and replies 200 OK to the user. In the next heartbeat, it tells the Followers to commit it as well.
  • The Crash: If the Leader crashes right after Step 1, the data was never replicated, and the user gets a timeout. A new Leader is elected, and the uncommitted log on the old dead Leader is eventually overwritten. The system remains 100% mathematically consistent.