Skip to content

S24.md

Latest commit

 

History

History
98 lines (61 loc) · 6.85 KB

S24.md

File metadata and controls

98 lines (61 loc) · 6.85 KB
Raw

We have designed the storage, the streaming, and the batch layers. But who conducts this massive orchestra?

If you rely on standard Linux cron jobs to trigger a complex data pipeline, your system is a ticking time bomb. A single failure will leave you manually untangling broken dependencies at 3:00 AM. Today, we master Advanced Workflow Orchestration using platforms like Apache Airflow.

As an architect designing pipelines that handle Spark clusters, ClickHouse ingestions, and machine learning models, you must understand how to scale the orchestrator itself and, more importantly, how to gracefully handle the hardest problem in data engineering: Time Travel (Backfilling).

1. EASY: The "Fragile Cron Job" (Idempotency & DAGs)

Scenario: You have a Python script that pulls yesterday's sales data from an API and inserts it into a Postgres database. It runs every night at 1:00 AM via cron. The Problem: At 1:05 AM, the API rate-limits your script, and it crashes halfway through. 500 rows were inserted. You wake up at 8:00 AM, fix the rate limit, and manually rerun the script. The script blindly inserts all 1,000 rows. You now have 500 duplicate rows, and the CFO's dashboard is wrong. Task: Design a robust orchestration pattern that can safely be retried infinite times without duplicating data.

Solution: The DAG and the Idempotent Task

We replace cron with a Directed Acyclic Graph (DAG) in Apache Airflow. But more importantly, we change the nature of the operation.

The Workflow:

  1. The Graph: We break the monolithic script into isolated Tasks (Extract $\rightarrow$ Transform $\rightarrow$ Load). If "Transform" fails, Airflow knows to only retry "Transform", not "Extract".
  2. The Execution Date: In Airflow, a run isn't tied to "now". It is tied to a specific logical_date (e.g., 2026-03-07).
  3. The Idempotent Write: * Bad: INSERT INTO sales VALUES (...)
  • Good: We use an UPSERT (or a MERGE in Spark/ClickHouse).
  • The Logic: INSERT INTO sales VALUES (...) ON CONFLICT (order_id) DO UPDATE SET ...

Production Grade Architecture

The "Partition Replace" Pattern: If you are writing to a Data Lake (S3) instead of a database, you cannot do a row-level UPSERT easily.

  • The Fix: We write the data into a specific folder: s3://lake/sales/ds=2026-03-07/.
  • If the job fails and we retry it, the Airflow task is programmed to execute an AWS S3 rm -rf command on that specific ds=2026-03-07 folder before writing the new Parquet files. This guarantees that a retry completely overwrites the failed state, achieving perfect idempotency.

2. MEDIUM: The "Midnight Traffic Jam" (Scaling the Executor)

Scenario: Your company has grown. You now have 500 different DAGs. At exactly midnight, 300 of them trigger simultaneously to process the day's data. The Problem: If you run Airflow on a single EC2 instance (using the LocalExecutor), the server's CPU hits 100%, memory exhausts, and tasks start randomly failing or hanging forever. Task: Design an orchestration architecture that can instantly scale from 0 to 1,000 parallel tasks and back to 0.

Solution: The KubernetesExecutor

We decouple the Airflow Scheduler from the execution environment.

The Architecture:

  1. The Scheduler: A lightweight pod that constantly reads the DAG files and decides what needs to run and when.
  2. The Meta-Database: A highly available PostgreSQL database that stores the state of every task (QUEUED, RUNNING, SUCCESS).
  3. The Execution: When a task is ready to run, the Scheduler does not run the Python code locally. It talks to the Kubernetes API.

The Workflow:

  • The Scheduler says: "K8s, spin up a brand new, isolated Pod specifically to run Task A."
  • Kubernetes provisions the Pod, runs the Spark-submit job or Python script, reports SUCCESS back to the database, and then instantly destroys the Pod.

Production Grade Architecture

Resource Isolation & Dependency Hell: In a monolithic orchestrator, if Team A needs pandas==1.0 and Team B needs pandas==2.0, you have a dependency conflict that breaks the server.

  • The KubernetesPodOperator Fix: With K8s, every single task can specify its own Docker image. Team A's task spins up a Pod using their specific Python environment. Team B's task spins up a completely different container.
  • Compute Profiling: You can assign strict limits. A simple API fetch task gets requested 256MB of RAM. A heavy Pandas transformation task gets 16GB of RAM. Kubernetes places these Pods on the underlying EC2 nodes perfectly, maximizing cost efficiency.

3. HARD: The "Time Traveler" (Backfilling & Catchup)

Scenario: You have a massive Spark pipeline that calculates "Daily Active Users" and writes it to a reporting table. It has been running perfectly for a year. The Problem: The business decides to change the definition of an "Active User." They want this new definition applied retroactively to the last 6 months of data. The Disaster: You cannot just run a for loop in a script to recalculate 180 days of Spark jobs. It will overload the cluster, mix historical data with today's live data, and take down the BI dashboard. Task: Design a safe, controlled architecture to execute a massive backfill without disrupting production SLAs.

Solution: Airflow Backfill CLI with Concurrency Limits

Airflow was explicitly designed for this exact scenario. Because every DAG run is tied to a specific logical_date, the code doesn't need to change. We just tell Airflow to travel back in time.

The Architecture:

  1. Deploy the Code: You push the new Spark logic to production.
  2. The Command: You SSH into the Airflow webserver and execute the backfill command: airflow dags backfill -s 2025-09-08 -e 2026-03-08 daily_active_users
  3. The Execution: Airflow generates 180 distinct DAG runs in the database.

Production Grade Architecture

Protecting the Cluster (Pools & Priority Weights): If Airflow suddenly tries to launch 180 massive Spark jobs simultaneously, your AWS EMR or Databricks cluster will melt, and today's critical production jobs will fail.

  • Airflow Pools: We create a resource pool called spark_heavy_jobs and set slots = 10. We assign the DAG to this pool.
  • The Throttling: When you trigger the backfill, Airflow will only spin up 10 Spark jobs at a time. As one finishes for Sept 8th, it queues up Sept 9th.
  • Priority Weights: We assign the daily production run a priority_weight = 999 and the backfill runs a weight of 1.
  • Result: The backfill churns quietly in the background at a safe pace (say, 10 days of data per hour). When midnight hits, the live production job jumps to the absolute front of the queue, runs immediately, and then the backfill seamlessly resumes. Zero downtime, perfectly recalculated history.