Skip to content

S6.md

Latest commit

 

History

History
142 lines (89 loc) · 6.44 KB

S6.md

File metadata and controls

142 lines (89 loc) · 6.44 KB
Raw

Now, we focus on Processing Architectures: The battle between Batch (High Throughput, High Latency) and Stream (Low Latency, Complex State).

As a Engineer, this is your home turf, so we will push the complexity to "Lead Architect" levels.

1. EASY: The "End-of-Day" Bank Reconciliation (Reliable Batch)

Scenario: You work for a Digital Bank. Every night at 00:00 UTC, you must reconcile 50 million transactions from the Core Banking System against the Payment Gateway Logs (Stripe/Visa) to ensure no money went missing. Constraint: Accuracy must be 100%. If the job fails, it must resume exactly where it left off. Task: Design the batch pipeline.

Solution: Spark on Kubernetes with Delta Lake

We use Apache Spark for its massive parallel processing capabilities and Delta Lake to provide ACID transactions on top of raw S3 files.

The Architecture:

  • Orchestrator: Apache Airflow (manages the DAG/Dependencies).
  • Compute: Spark Submit $\rightarrow$ Kubernetes Cluster (Ephemeral Pods).
  • Storage: S3 (with Delta Lake format).

The Workflow:

  1. Ingest: Airflow triggers a Spark job to read raw CSVs from Visa and JSONs from Core Banking.
  2. Bronze Layer (Raw): Save raw data into Delta Tables (s3://bank/bronze/visa/).
  3. Silver Layer (Clean): Spark performs the JOIN on transaction_id.
  • Logic: CASE WHEN visa.amount != core.amount THEN 'FLAGGED' ELSE 'MATCHED' END.
  1. Gold Layer (Agg): Write the "Flagged" transactions to a report table for the manual review team.

Production Grade Architecture:

  • Idempotency: We use Delta Lake's MERGE INTO command. If the job runs twice effectively, it won't duplicate data; it will just update existing rows.
MERGE INTO gold_report USING new_batch
ON gold_report.id = new_batch.id
WHEN MATCHED THEN UPDATE ...
WHEN NOT MATCHED THEN INSERT ...
  • Spot Instances: To save 60% on cost, we run Spark Executors on AWS Spot Instances. If a node is reclaimed, Spark's DAG scheduler automatically retries the task on a new node.

Why this is best:

  • Delta Lake: Standard Parquet files (used in basic Hadoop) can leave corrupt data if a job crashes mid-write. Delta Lake uses a transaction log (_delta_log) to guarantee atomicity.
  • Airflow: Provides clear visibility into failure points (retries, alerts).

2. MEDIUM: The "Twitter" Trending Topics (Windowed Streaming)

Scenario: You are building the "Trending Now" sidebar for Twitter. Input: A Kafka Stream of 50,000 Tweets/sec. Task: Calculate the Top 10 Hashtags for the last 1 hour, updated every 1 minute. Problem: Data arrives late. A tweet timestamped 10:00 might arrive at 10:05 due to network lag.

Solution: Spark Structured Streaming (Micro-Batch)

We use Spark Structured Streaming because we don't need sub-millisecond latency (updating every minute is fine), and Spark's API is excellent for windowing.

The Workflow:

  1. Read: Spark connects to Kafka topic tweets.
  2. Window Aggregation:
hashtags.groupBy(
    window(col("timestamp"), "1 hour", "1 minute"), # Sliding Window
    col("hashtag")
).count().orderBy("count", ascending=False)
  1. Watermarking (Crucial): We tell Spark to wait 10 minutes for late data.
  • .withWatermark("timestamp", "10 minutes")
  • If a tweet from 09:00 arrives at 10:15, it is dropped (too late to affect the count).
  1. Sink: Write the Top 10 lists to a Redis Sorted Set for fast retrieval by the frontend.

Production Grade Architecture:

  • Checkpointing: Spark saves the "offset" (where it is in the Kafka stream) to S3 every batch. If the cluster crashes, it restarts, reads the checkpoint, and resumes processing without losing a single tweet.
  • Trigger: .trigger(processingTime='1 minute'). This defines the micro-batch interval.

Why this is best:

  • Throughput: Spark micro-batches are higher throughput than row-at-a-time processing.
  • Simplicity: You write code that looks like SQL/Batch, and the engine handles the streaming complexity.

3. HARD: The "Credit Card" Fraud Detection (Complex Event Processing)

Scenario: You are the architect for Visa. Requirement: You must block a transaction before it completes (latency budget: < 200ms). The Pattern (Impossible Travel):

  • Event A: Card swiped in New York at 10:00 AM.
  • Event B: Card swiped in London at 10:05 AM.
  • Result: Decline Event B immediately. Challenge: Spark Streaming is too slow (seconds latency). You need Stateful Event Driven processing.

Solution: Apache Flink (Stateful Streaming)

We use Apache Flink because it processes row-by-row (True Streaming) and manages massive distributed state locally.

The Architecture:

  • Ingest: Kafka.
  • Engine: Flink Cluster.
  • State Backend: RocksDB (Embedded NoSQL DB inside Flink).

The Workflow:

  1. KeyBy: Flink partitions the stream by card_id. All events for "Card X" go to the same worker node.
  2. State: When Event A (NY) arrives:
  • The Flink operator saves last_location = "NY" and last_time = 10:00 into its local RocksDB.
  1. Process: When Event B (London) arrives at 10:05:
  • Flink retrieves last_location from RocksDB (Local RAM/Disk speed, microseconds).
  • Calculates speed: Distance(NY, London) / 5 minutes = 50,000 mph.
  • Action: If Speed > 600mph, emit "FRAUD_ALERT".
  1. Update: Overwrite state with new location (if valid).

Production Grade Architecture (The "Checkpoints" are Key):

  • RocksDB Incremental Checkpoints: Flink snapshots the state (the locations of all active cards) to S3 asynchronously every 10 seconds.
  • Exactly-Once Semantics: Flink uses the Chandy-Lamport algorithm. It injects "Barriers" into the data stream. When a barrier passes through, the state is snapshotted. This guarantees that even if a node crashes, every transaction is processed exactly once (no double billing, no skipped fraud checks).
  • Side Output: If data is extremely late, route it to a "Side Output" stream for manual investigation later, rather than crashing the pipeline.

Why this is best:

  • Low Latency: Unlike Spark (which waits to fill a batch), Flink reacts the instant the event hits the wire.
  • State Management: Handling "state" (history) is the hardest part of streaming. Flink abstracts this away but keeps it local (fast) rather than querying an external Redis (slow network hop) for every single transaction.