Skip to content

S20.md

Latest commit

 

History

History
104 lines (65 loc) · 6.29 KB

S20.md

File metadata and controls

104 lines (65 loc) · 6.29 KB
Raw

Let us build foundational phase with one of the most notoriously difficult problems in distributed architecture: Distributed Transactions.

When you break a monolith into microservices, you also shatter the database. You lose the magical SQL COMMIT and ROLLBACK that guarantees "all or nothing." If the Billing Service succeeds but the Inventory Service crashes, you have a massive data inconsistency.

Now, we learn how to keep distributed systems perfectly synchronized using Two-Phase Commit (2PC) and the Saga Pattern.

1. EASY: The "All or Nothing" Lock (Two-Phase Commit)

Scenario: You have a legacy system with two separate relational databases. Database A handles Orders. Database B handles Payments. The Problem: An order must be saved, and the payment must be deducted. If either fails, neither can be saved. Task: Guarantee ACID (Atomicity, Consistency, Isolation, Durability) properties across two physically separate network nodes.

Solution: Two-Phase Commit (2PC)

We introduce a central "Coordinator" node to manage a strict, synchronous transaction.

The Workflow:

  1. Phase 1 (Prepare): The Coordinator asks both DB A and DB B: "Are you ready to commit?" 2. The Lock: DB A and DB B lock their respective rows. They write the intent to their local logs and reply: "Yes, prepared."
  2. Phase 2 (Commit): Once the Coordinator gets a "Yes" from both, it sends the final command: "Commit!" Both databases finalize the write and release their locks.

Production Grade Architecture

The Blocking Vulnerability (Why we don't use 2PC at scale): 2PC is conceptually easy but architecturally dangerous. It is a blocking protocol.

  • If DB A says "Yes", it locks the row.
  • If the Coordinator crashes immediately after receiving that "Yes" (before sending the final "Commit"), DB A is stuck. It cannot release the lock. It cannot accept new writes to that row.
  • The Verdict: 2PC is fine for internal, low-throughput systems, but it is too brittle and slow for massive, high-concurrency cloud architectures. We need something asynchronous.

2. MEDIUM: The "Collaborative File System" (Choreography Saga)

Scenario: You are building the backend for a collaborative, lightweight file system application where colleagues can share and build rules around documents. The Problem: When a user uploads a file, a complex chain must happen:

  1. Storage Service saves the raw file to S3.
  2. Security Service scans it for malware.
  3. Metadata Service updates the collaborative folder view. The Disaster: The file is saved, the malware scan passes, but the Metadata Service crashes. The file exists in storage, but nobody can see it in the UI. We cannot use 2PC here because S3 doesn't support it. Task: Design an eventually consistent transaction that can roll itself back if a step fails.

Solution: The Saga Pattern (Choreography)

We use an event-driven architecture powered by Apache Kafka. There is no central coordinator. Each service reacts to events and emits new ones.

The Workflow (Happy Path):

  1. Storage Service saves the file and publishes an event to Kafka: FileUploaded.
  2. Security Service consumes FileUploaded, scans it, and publishes ScanPassed.
  3. Metadata Service consumes ScanPassed and updates the UI database.

The Workflow (Failure & Compensation):

  1. Storage Service publishes FileUploaded.
  2. Security Service consumes it, finds a virus, and publishes ScanFailed.
  3. The Compensation: The Storage Service is configured to listen for ScanFailed. When it sees this event, it executes a compensating transaction: it permanently deletes the raw file from S3.

Production Grade Architecture

The "Phantom Read" Problem: In a Saga, the transaction isn't isolated. For a few milliseconds, the file is in S3 before the malware scan finishes.

  • The Fix: We use state flags. The Storage Service saves the file with status = PENDING. The UI is programmed to completely ignore files in the PENDING state. Only when the final MetadataUpdated event loops back does the status change to AVAILABLE.

3. HARD: The "Complex Pipeline" (Orchestration Saga)

Scenario: You are executing a massive data pipeline involving multiple clusters.

  • Step 1: Provision temporary AWS EC2 nodes.
  • Step 2: Run an Apache Flink streaming job.
  • Step 3: Write results to ClickHouse. The Problem: If Choreography (Medium scenario) gets too complex, it turns into "Event Spagetti." If Step 3 fails, figuring out which service needs to trigger which compensation event to tear down the EC2 nodes becomes a debugging nightmare because the logic is scattered across 10 different codebases. Task: Design a transaction that is asynchronous but centrally managed and strictly tracked.

Solution: The Saga Pattern (Orchestration)

We introduce an Orchestrator (often implemented via a state machine or a DAG runner). The Orchestrator tells the microservices what to do, rather than the services reacting to each other.

The Architecture:

  • We use a central commander (while tools like Apache Airflow are excellent for scheduling batch DAGs, for transactional micro-second orchestration we use systems like AWS Step Functions, Temporal.io, or Netflix Conductor).

The Workflow:

  1. Command: The Orchestrator sends a command via a message queue: ProvisionNodes.
  2. Reply: The Node Service provisions them and replies to the Orchestrator: NodesReady.
  3. Command: Orchestrator sends StartFlinkJob.
  4. Failure Reply: Flink Service replies: JobFailed (Out of Memory).
  5. The Rollback: The Orchestrator looks at its internal state machine, sees that it needs to undo Step 1, and explicitly sends a TerminateNodes command to the Node Service.

Production Grade Architecture

The Idempotency Requirement: What if the Orchestrator sends the TerminateNodes command, but the network drops the acknowledgment? The Orchestrator will retry and send TerminateNodes again.

  • Every single microservice participating in an Orchestrated Saga must be perfectly idempotent (which we solved back on Day 14). If the Node Service receives TerminateNodes twice, it must safely ignore the second request rather than throwing an error that could halt the rollback process.