Skip to content

S25.md

Latest commit

 

History

History
87 lines (54 loc) · 6.37 KB

S25.md

File metadata and controls

87 lines (54 loc) · 6.37 KB
Raw

Now, we conquer the Data Mesh.

We have built incredible technology: Kafka streams, Flink processors, and massive Snowflake/Delta Lake storage. But at a massive enterprise scale (like Netflix or Uber), the technology isn't what breaks. The organization breaks.

If 50 different software teams are dumping data into one centralized Data Lake, and a single centralized Data Engineering team of 10 people is responsible for cleaning, transforming, and serving all of it, that team becomes a massive bottleneck. Nothing gets done.

Now, we shift from a centralized architecture to a decentralized, domain-driven architecture.

1. EASY: The "Bottleneck" (Domain-Oriented Ownership)

Scenario: You are the Lead Architect at an e-commerce giant. The Checkout Team, the Inventory Team, and the Marketing Team all push raw logs into the Data Lake. The Problem: The Marketing Team asks the central Data Engineering team to build a "Customer Churn" dashboard. The central team looks at the checkout_logs table and has no idea what the status_flag_4 column means because they didn't write the checkout code. They spend 3 weeks in meetings just trying to understand the data. Task: Redesign the organizational and technical architecture to eliminate this bottleneck.

Solution: Domain-Oriented Data Ownership

We apply the microservices concept of "Bounded Contexts" to analytical data. We completely dissolve the centralized Data Engineering team.

The Architecture:

  • Decentralization: The Checkout Team is now responsible for both their operational database (Postgres) AND their analytical data.
  • Embedded Engineers: We move Data Engineers out of the central team and embed them directly inside the Checkout Team and Marketing Team.
  • The Result: The engineers writing the microservice code are the exact same engineers transforming the analytical data. When status_flag_4 changes, they update the analytical pipeline in the same sprint.

Why this is best: It aligns the architecture with Conway's Law. Teams move at their own speed without waiting on a central ticketing system.

2. MEDIUM: The "Broken Pipeline" (Data as a Product & Contracts)

Scenario: You successfully implemented Domain Ownership. The Checkout Team owns their data. The Finance Team consumes it to calculate daily revenue. The Problem: A software engineer on the Checkout Team decides to rename a column in their Postgres database from total_amount to cart_total to make the code cleaner. They deploy the code. The operational app works fine, but the analytical pipeline feeding the Finance Team instantly breaks. Revenue reporting goes down for 2 days. Task: Design a technical failsafe so domains can evolve independently without breaking downstream consumers.

Solution: Data as a Product & Data Contracts

If a domain's data is going to be used by other teams, it must be treated with the same rigor as a public API. You wouldn't arbitrarily change a REST API payload without versioning it. We must do the same for data.

The Architecture:

  • The Data Contract: The Checkout Team and Finance Team sign a strict, machine-readable Data Contract (using JSON Schema, Protobuf, or YAML).
  • The contract states: "The sales_events data product will ALWAYS contain a column named total_amount of type DECIMAL."

The Workflow (CI/CD Enforcement):

  1. The Checkout engineer opens a Pull Request (PR) changing total_amount to cart_total in their database schema (or dbt models).
  2. The CI/CD pipeline runs an automated check against the central Data Contract Registry.
  3. The pipeline detects a contract violation.
  4. The Failsafe: The CI/CD pipeline strictly blocks the deployment. The PR turns red. The engineer is forced to either version the data product (e.g., sales_events_v2) or map cart_total back to total_amount in their outgoing analytical pipeline before merging.

Why this is best: It stops bad data at the source (the producer) rather than forcing the consumer to scramble and fix broken dashboards downstream.

3. HARD: The "Global View" (Federated Computational Governance)

Scenario: You have 50 independent domains operating beautifully.

  • Marketing built their data products in Google BigQuery.
  • Inventory built theirs in AWS S3 (Iceberg).
  • Checkout built theirs in Snowflake. The Problem: The CEO wants a global dashboard that JOINs Marketing campaign data with Checkout revenue data. Furthermore, both datasets contain Personally Identifiable Information (PII) like email addresses, which must be strictly masked according to GDPR. Task: How do you execute a secure, cross-platform JOIN and enforce global security policies without forcing everyone to move their data back into a single, centralized database?

Solution: The Federated Query Engine & Global Governance Plane

We leave the data physically where it is (Decentralized Storage) but create a unified layer for querying and security (Federated Access).

The Architecture:

  1. The Federated Engine: We deploy a massive Trino (formerly Presto) cluster. Trino doesn't store data; it reaches into Snowflake, S3, and BigQuery simultaneously, pulls the necessary chunks into its own RAM, and executes the JOIN.
  2. The Central Catalog: We use a tool like DataHub or Amundsen. Every domain registers their Data Products here. The catalog tags columns (e.g., checkout.user_email is tagged PII: TRUE).
  3. The Policy Engine: We use Open Policy Agent (OPA) or Apache Ranger to define global rules: "If a user is not in the Legal team, mask all columns tagged PII: TRUE."

The Workflow:

  • A data scientist writes: SELECT m.campaign, c.user_email FROM bigquery.marketing m JOIN snowflake.checkout c ON... and sends it to Trino.
  • Trino intercepts the query and checks the Policy Engine.
  • The Policy Engine sees the data scientist is not in Legal, and sees user_email is tagged as PII.
  • Trino dynamically rewrites the query on the fly to apply a hashing function: SHA256(c.user_email).
  • Trino queries BigQuery and Snowflake, joins the data, and returns the result with securely masked emails.

Why this is best: The domains retain absolute freedom over their tech stack (Snowflake vs. BigQuery), but the Chief Information Security Officer (CISO) retains absolute control over global compliance and security. It is the ultimate balance of speed and safety.