Skip to content

S27.md

Latest commit

 

History

History
101 lines (65 loc) · 7.38 KB

S27.md

File metadata and controls

101 lines (65 loc) · 7.38 KB
Raw

Lets dive into the silent killer of data engineering: Data Quality & Observability.

You can build the most elegant Apache Spark pipelines, orchestrate them flawlessly with Airflow, and stream them via Kafka. But what happens when the pipeline executes perfectly (green checkmarks across the board), but the data itself is absolute garbage? If a frontend bug starts sending NULL for a critical user_id, your pipeline won't crash. It will just happily ingest millions of broken rows, corrupt your ClickHouse dashboards, and cause executives to make decisions based on lies.

Now, we stop flying blind. We are building mathematical circuit breakers to catch "Silent Data Failures."

1. EASY: The "Silent Null" (Data Unit Testing)

Scenario: You have a nightly batch job parsing JSON logs from a mobile app. The Problem: The iOS team releases an update. A bug causes the device_type field to occasionally send as NULL or as an empty string "". Your Spark job doesn't care; it just writes the data. Three days later, the Marketing team panics because the "Revenue by Device" dashboard shows a 20% drop in iPhone sales. Task: Design a system to detect logical data errors before they are loaded into the final analytical tables.

Solution: Data Contracts & Circuit Breakers (dbt tests / Great Expectations)

Just as software engineers write unit tests for code, Data Engineers must write unit tests for data.

The Workflow:

  1. The Staging Area: Spark writes the parsed JSON into a temporary staging table, not the final production table.
  2. The Assertions: We define strict rules (expectations) for the data.
  • device_type must NOT BE NULL.
  • device_type must be in the set ['iOS', 'Android', 'Web'].
  • transaction_amount must be $\ge 0$.
  1. The Execution: An evaluation engine (like Great Expectations or dbt's built-in tests) runs SQL queries against the staging table to verify these rules.
  2. The Circuit Breaker: If the test queries return even a single row that violates the rules, the pipeline halts. The data is quarantined, and an alert is fired.

Production Grade Architecture

The "Warning vs. Error" Thresholds: If 1 row out of 50 million has a NULL device type, should you really halt the entire company's revenue dashboard? Probably not.

  • The Fix: We implement threshold-based testing.
  • We configure the test: fail_threshold = 0.01 (1%).
  • If 0.001% of rows are bad, the test emits a "Warning" to a Slack channel for the engineers to investigate, but allows the Airflow DAG to proceed and load the 99.999% of good data into ClickHouse.
  • If the bad rows exceed 1%, the circuit breaker trips, returning a hard "Error" and halting the pipeline immediately to prevent dashboard corruption.

2. MEDIUM: The "Volume Drop" (Automated Data Observability)

Scenario: You ingest data from a third-party payment gateway (like Stripe) using a nightly Airflow DAG. The Problem: Stripe silently changes their API pagination logic. Instead of returning 100,000 rows, the API hits a limit and only returns 10,000 rows. The Disaster: The pipeline doesn't crash. There are no NULL values. The schema hasn't changed. Your unit tests from the Easy scenario all pass perfectly. But you just lost 90,000 transactions. Task: Design a system that detects anomalies in the macro-behavior of the data without you having to write thousands of manual volume tests.

Solution: Machine Learning Anomaly Detection (Monte Carlo / Datafold)

We move from manual rule-based testing (Data Quality) to automated metadata monitoring (Data Observability).

The Architecture: We do not query the raw data itself (too expensive). We query the database's internal metadata (the information_schema).

The Workflow:

  1. Metadata Harvesting: A lightweight agent continuously polls your data warehouse's metadata tables. It records exactly how many rows were inserted, the total byte size of the table, and the timestamp of the last update.
  2. Time-Series Forecasting: The Observability engine uses machine learning (like ARIMA models or Prophet) to build a baseline of "normal" behavior. It knows that on a typical Tuesday, the payments table grows by 95,000 to 105,000 rows.
  3. The Anomaly Alert: When the pipeline only inserts 10,000 rows, the ML model flags it as a statistically significant anomaly ($>$ 3 standard deviations from the mean) and fires a high-priority PagerDuty alert.

Production Grade Architecture

Data Lineage (The Blast Radius): When the volume drop alert fires, the immediate question is: "Which dashboards are broken because of this?"

  • Automated Lineage Parsing: Tools like Monte Carlo or open-source DataHub parse the raw SQL queries running in your warehouse to build a dependency graph.
  • Incident Triage: The alert doesn't just say "Stripe data dropped." It says: "Stripe data dropped. This feeds the int_payments table, which feeds the fct_revenue table, which powers the CEO's 'Daily Earnings' Tableau Dashboard. I have automatically paused the dashboard refresh to prevent the CEO from seeing partial data."

3. HARD: The "Semantic Drift" (Continuous Distribution Tracking)

Scenario: You maintain the feature store for a massive Machine Learning recommendation engine (like the one we built on Day 15). The Problem: The ML model's accuracy slowly degrades over three months. Users are clicking less. The Investigation: You check the pipelines. No failed DAGs. No volume drops. No NULL values. The schema is perfect. The Threat: The distribution of the data has shifted. For example, a marketing campaign brought in millions of teenagers. The average user age dropped from 35 to 19. The ML model was trained on 35-year-olds and is now making terrible predictions for 19-year-olds. Task: Design a system to catch "Data Drift" mathematically in real-time.

Solution: Statistical Distribution Profiling (KL Divergence)

We cannot just check max/min values. We must continuously calculate the shape of the data using statistical distance metrics.

The Workflow:

  1. The Baseline (Training Data): When the ML model is trained, we save the statistical profile of every feature. (e.g., Age: Mean 35, StdDev 10, Skewness 0.2).
  2. The Operational Stream: As new live data flows through Kafka and Flink, we maintain a rolling sketch of the incoming data distribution (using algorithms like HyperLogLog or T-Digest for memory efficiency).
  3. The Distance Calculation: We mathematically compare the live distribution to the training distribution. We use metrics like Kullback-Leibler (KL) Divergence or the Wasserstein Distance.
  4. The Trigger: If the KL Divergence exceeds a pre-defined mathematical threshold, we know the data has drifted too far from what the model understands.

Production Grade Architecture

Automated Retraining Pipelines: You don't just alert an engineer when drift occurs; you automate the response.

  • The Event Bus: The Drift Detection engine publishes a DriftDetected event to Kafka.
  • The Orchestrator: Airflow listens for this event via a sensor.
  • The Action: When triggered, Airflow automatically spins up a heavy Spark job to extract the last 30 days of new data, triggers a fresh model training pipeline on the GPU cluster, validates the new model's accuracy, and shadow-deploys it. The system heals its own intelligence.