Consolidate 12 categories into 9

shiningflash · shiningflash · commit 7f9e9a3d745f · 2026-06-03T07:18:49.000+02:00
Old buckets that were either too narrow or too overlapping have been
merged so every category lands at six or more problems and the index
reads as one coherent map instead of a grab-bag.

Merges:
  Database Internals + Database Operations -&gt; Databases &amp; Operations
  Batch Pipelines + ops items from Debugging &amp; Reliability -&gt;
    Batch Pipelines &amp; Orchestration
  Cloud Services + Cost Optimization -&gt; Cloud &amp; Cost
  Storage &amp; Formats -&gt; Storage &amp; Lakehouse (renamed, same scope for now)

Dissolved:
  Debugging &amp; Reliability now splits across Batch Pipelines &amp;
  Orchestration (operational issues) and Databases &amp; Operations (the
  one query-plan-flip problem). Each item lives where readers actually
  look for it.

Untouched: SQL &amp; Querying, Data Modeling, Streaming, System Design,
People &amp; Process.

41 question.md files updated. PROBLEMS.md regenerated from the new
legend.
diff --git a/PROBLEMS.md b/PROBLEMS.md
diff --git a/problems/001-log-file-error-analysis/question.md b/problems/001-log-file-error-analysis/question.md
@@ -1,7 +1,7 @@
 ---
 id: 1
 title: Log File Error Analysis
-category: Batch Pipelines
+category: Batch Pipelines & Orchestration
 topics: [file streaming, counters, top-N, IoT logs]
 difficulty: Easy
 interview_value: strong
diff --git a/problems/003-transform-and-clean-raw-data-for-analytics/question.md b/problems/003-transform-and-clean-raw-data-for-analytics/question.md
@@ -1,7 +1,7 @@
 ---
 id: 3
 title: Transform and Clean Raw Data for Analytics
-category: Batch Pipelines
+category: Batch Pipelines & Orchestration
 topics: [CSV, validation, regex, date checks]
 difficulty: Medium
 interview_value: optional
diff --git a/problems/005-merging-messy-csvs-from-multiple-partners/question.md b/problems/005-merging-messy-csvs-from-multiple-partners/question.md
@@ -1,7 +1,7 @@
 ---
 id: 5
 title: Merging Messy CSVs from Multiple Partners
-category: Batch Pipelines
+category: Batch Pipelines & Orchestration
 topics: [CSV, column mapping, date parsing, file walk]
 difficulty: Medium
 interview_value: optional
diff --git a/problems/006-partitioning-vs-clustering-in-bigquery/question.md b/problems/006-partitioning-vs-clustering-in-bigquery/question.md
@@ -1,7 +1,7 @@
 ---
 id: 6
 title: Partitioning vs Clustering in BigQuery
-category: Storage & Formats
+category: Storage & Lakehouse
 topics: [BigQuery, partitioning, clustering, cost]
 difficulty: Easy
 interview_value: strong
diff --git a/problems/007-etl-vs-elt-and-why-elt-won/question.md b/problems/007-etl-vs-elt-and-why-elt-won/question.md
@@ -1,7 +1,7 @@
 ---
 id: 7
 title: ETL vs ELT and Why ELT Won
-category: Batch Pipelines
+category: Batch Pipelines & Orchestration
 topics: [ETL, ELT, dbt, warehouse]
 difficulty: Easy
 interview_value: must-have
diff --git a/problems/008-oltp-vs-olap/question.md b/problems/008-oltp-vs-olap/question.md
@@ -1,7 +1,7 @@
 ---
 id: 8
 title: OLTP vs OLAP
-category: Database Internals
+category: Databases & Operations
 topics: [OLTP, OLAP, column store, row store]
 difficulty: Easy
 interview_value: must-have
diff --git a/problems/009-idempotency-in-data-pipelines/question.md b/problems/009-idempotency-in-data-pipelines/question.md
@@ -1,7 +1,7 @@
 ---
 id: 9
 title: Idempotency in Data Pipelines
-category: Batch Pipelines
+category: Batch Pipelines & Orchestration
 topics: [idempotency, retries, MERGE, partitions]
 difficulty: Medium
 interview_value: must-have
diff --git a/problems/011-data-contracts-in-plain-words/question.md b/problems/011-data-contracts-in-plain-words/question.md
@@ -1,7 +1,7 @@
 ---
 id: 11
 title: Data Contracts in Plain Words
-category: Batch Pipelines
+category: Batch Pipelines & Orchestration
 topics: [data contracts, schema registry, ownership]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/012-parquet-vs-csv-vs-json/question.md b/problems/012-parquet-vs-csv-vs-json/question.md
@@ -1,7 +1,7 @@
 ---
 id: 12
 title: Parquet vs CSV vs JSON
-category: Storage & Formats
+category: Storage & Lakehouse
 topics: [Parquet, CSV, JSON, columnar storage]
 difficulty: Easy
 interview_value: must-have
diff --git a/problems/013-data-lake-vs-warehouse-vs-lakehouse/question.md b/problems/013-data-lake-vs-warehouse-vs-lakehouse/question.md
@@ -1,7 +1,7 @@
 ---
 id: 13
 title: Data Lake vs Warehouse vs Lakehouse
-category: Storage & Formats
+category: Storage & Lakehouse
 topics: [lake, warehouse, lakehouse, Iceberg, Delta]
 difficulty: Medium
 interview_value: must-have
diff --git a/problems/029-daily-report-quietly-wrong-for-two-weeks/question.md b/problems/029-daily-report-quietly-wrong-for-two-weeks/question.md
@@ -1,7 +1,7 @@
 ---
 id: 29
 title: Daily Report Quietly Wrong for Two Weeks
-category: Debugging & Reliability
+category: Batch Pipelines & Orchestration
 topics: [incident, postmortem, comms, data quality]
 difficulty: Medium
 interview_value: must-have
diff --git a/problems/030-warehouse-cost-doubled-in-two-months/question.md b/problems/030-warehouse-cost-doubled-in-two-months/question.md
@@ -1,7 +1,7 @@
 ---
 id: 30
 title: Warehouse Cost Doubled in Two Months
-category: Cost Optimization
+category: Cloud & Cost
 topics: [cost, governance, comms, INFORMATION_SCHEMA]
 difficulty: Medium
 interview_value: must-have
diff --git a/problems/031-the-dashboard-is-wrong/question.md b/problems/031-the-dashboard-is-wrong/question.md
@@ -1,7 +1,7 @@
 ---
 id: 31
 title: The Dashboard is Wrong
-category: Debugging & Reliability
+category: Batch Pipelines & Orchestration
 topics: [trust, comms, vague reports]
 difficulty: Easy
 interview_value: strong
diff --git a/problems/034-three-days-of-data-lost/question.md b/problems/034-three-days-of-data-lost/question.md
@@ -1,7 +1,7 @@
 ---
 id: 34
 title: Three Days of Data Lost
-category: Debugging & Reliability
+category: Batch Pipelines & Orchestration
 topics: [Kafka retention, replay, recovery, postmortem]
 difficulty: Hard
 interview_value: must-have
diff --git a/problems/035-lambda-vs-cloud-function-vs-cloud-run/question.md b/problems/035-lambda-vs-cloud-function-vs-cloud-run/question.md
@@ -1,7 +1,7 @@
 ---
 id: 35
 title: Lambda vs Cloud Function vs Cloud Run
-category: Cloud Services
+category: Cloud & Cost
 topics: [serverless, AWS, GCP, runtime limits]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/036-scheduled-pipeline-pay-only-when-run/question.md b/problems/036-scheduled-pipeline-pay-only-when-run/question.md
@@ -1,7 +1,7 @@
 ---
 id: 36
 title: Scheduled Pipeline Pay Only When Run
-category: Cloud Services
+category: Cloud & Cost
 topics: [scheduled jobs, Cloud Run Jobs, AWS Batch]
 difficulty: Easy
 interview_value: optional
diff --git a/problems/037-bigquery-vs-snowflake-for-new-team/question.md b/problems/037-bigquery-vs-snowflake-for-new-team/question.md
@@ -1,7 +1,7 @@
 ---
 id: 37
 title: BigQuery vs Snowflake for New Team
-category: Cloud Services
+category: Cloud & Cost
 topics: [BigQuery, Snowflake, pricing model]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/038-store-partner-files-in-s3-or-warehouse/question.md b/problems/038-store-partner-files-in-s3-or-warehouse/question.md
@@ -1,7 +1,7 @@
 ---
 id: 38
 title: Store Partner Files in S3 or Warehouse
-category: Cloud Services
+category: Cloud & Cost
 topics: [S3, raw layer, audit, schema evolution]
 difficulty: Easy
 interview_value: optional
diff --git a/problems/039-managed-airflow-vs-self-hosted/question.md b/problems/039-managed-airflow-vs-self-hosted/question.md
@@ -1,7 +1,7 @@
 ---
 id: 39
 title: Managed Airflow vs Self Hosted
-category: Cloud Services
+category: Cloud & Cost
 topics: [Airflow, MWAA, Composer, Astronomer, Dagster]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/040-bigquery-access-control-for-50-person-company/question.md b/problems/040-bigquery-access-control-for-50-person-company/question.md
@@ -1,7 +1,7 @@
 ---
 id: 40
 title: BigQuery Access Control for 50 Person Company
-category: Cloud Services
+category: Cloud & Cost
 topics: [IAM, datasets, groups, RLS, audit]
 difficulty: Medium
 interview_value: optional
diff --git a/problems/046-region-suddenly-shows-zero-revenue/question.md b/problems/046-region-suddenly-shows-zero-revenue/question.md
@@ -1,7 +1,7 @@
 ---
 id: 46
 title: Region Suddenly Shows Zero Revenue
-category: Debugging & Reliability
+category: Batch Pipelines & Orchestration
 topics: [dashboard, joins, SCD, time zones]
 difficulty: Medium
 interview_value: must-have
diff --git a/problems/047-airflow-green-but-output-empty/question.md b/problems/047-airflow-green-but-output-empty/question.md
@@ -1,7 +1,7 @@
 ---
 id: 47
 title: Airflow Green but Output Empty
-category: Debugging & Reliability
+category: Batch Pipelines & Orchestration
 topics: [silent success, idempotency, anomaly checks]
 difficulty: Medium
 interview_value: must-have
diff --git a/problems/048-query-suddenly-80x-slower/question.md b/problems/048-query-suddenly-80x-slower/question.md
@@ -1,7 +1,7 @@
 ---
 id: 48
 title: Query Suddenly 80x Slower
-category: Debugging & Reliability
+category: Databases & Operations
 topics: [EXPLAIN, statistics, plan flip, join strategy]
 difficulty: Medium
 interview_value: must-have
diff --git a/problems/049-user-says-data-is-wrong/question.md b/problems/049-user-says-data-is-wrong/question.md
@@ -1,7 +1,7 @@
 ---
 id: 49
 title: User Says Data Is Wrong
-category: Debugging & Reliability
+category: Batch Pipelines & Orchestration
 topics: [comms, vague reports, triage]
 difficulty: Easy
 interview_value: optional
diff --git a/problems/050-partition-always-ten-percent-smaller/question.md b/problems/050-partition-always-ten-percent-smaller/question.md
@@ -1,7 +1,7 @@
 ---
 id: 50
 title: Partition Always Ten Percent Smaller
-category: Debugging & Reliability
+category: Batch Pipelines & Orchestration
 topics: [anomaly, baselines, patterns, judgement]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/051-bigquery-bill-eight-times-higher/question.md b/problems/051-bigquery-bill-eight-times-higher/question.md
@@ -1,7 +1,7 @@
 ---
 id: 51
 title: BigQuery Bill Eight Times Higher
-category: Cost Optimization
+category: Cloud & Cost
 topics: [INFORMATION_SCHEMA, top queries, slot reservation]
 difficulty: Medium
 interview_value: must-have
diff --git a/problems/052-four-hour-spark-job-under-one-hour/question.md b/problems/052-four-hour-spark-job-under-one-hour/question.md
@@ -1,7 +1,7 @@
 ---
 id: 52
 title: Four Hour Spark Job Under One Hour
-category: Cost Optimization
+category: Cloud & Cost
 topics: [Spark UI, skew, AQE, broadcast joins]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/053-hourly-scan-on-daily-data/question.md b/problems/053-hourly-scan-on-daily-data/question.md
@@ -1,7 +1,7 @@
 ---
 id: 53
 title: Hourly Scan on Daily Data
-category: Cost Optimization
+category: Cloud & Cost
 topics: [summary tables, MV, refresh, BI tool]
 difficulty: Easy
 interview_value: strong
diff --git a/problems/054-just-throw-more-memory-at-it/question.md b/problems/054-just-throw-more-memory-at-it/question.md
@@ -1,7 +1,7 @@
 ---
 id: 54
 title: Just Throw More Memory At It
-category: Cost Optimization
+category: Cloud & Cost
 topics: [upsize, plan inspection, optimization]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/055-partitioning-clustering-materialized-views/question.md b/problems/055-partitioning-clustering-materialized-views/question.md
@@ -1,7 +1,7 @@
 ---
 id: 55
 title: Partitioning Clustering Materialized Views
-category: Storage & Formats
+category: Storage & Lakehouse
 topics: [partitioning, clustering, MV, BigQuery]
 difficulty: Easy
 interview_value: strong
diff --git a/problems/065-4000-dag-airflow-at-90-percent-cpu/question.md b/problems/065-4000-dag-airflow-at-90-percent-cpu/question.md
@@ -1,7 +1,7 @@
 ---
 id: 65
 title: 4000 DAG Airflow at 90 Percent CPU
-category: Debugging & Reliability
+category: Batch Pipelines & Orchestration
 topics: [Airflow, scheduler, parsing, scale-out]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/066-indexes-when-to-add-and-when-they-hurt/question.md b/problems/066-indexes-when-to-add-and-when-they-hurt/question.md
@@ -1,7 +1,7 @@
 ---
 id: 66
 title: Indexes When to Add and When They Hurt
-category: Database Internals
+category: Databases & Operations
 topics: [indexes, B-tree, write cost, EXPLAIN]
 difficulty: Easy
 interview_value: must-have
diff --git a/problems/067-transactions-and-acid/question.md b/problems/067-transactions-and-acid/question.md
@@ -1,7 +1,7 @@
 ---
 id: 67
 title: Transactions and ACID
-category: Database Internals
+category: Databases & Operations
 topics: [transactions, ACID, durability, atomicity]
 difficulty: Easy
 interview_value: must-have
diff --git a/problems/068-isolation-levels-in-plain-words/question.md b/problems/068-isolation-levels-in-plain-words/question.md
@@ -1,7 +1,7 @@
 ---
 id: 68
 title: Isolation Levels in Plain Words
-category: Database Internals
+category: Databases & Operations
 topics: [isolation, snapshot, anomalies, MVCC]
 difficulty: Medium
 interview_value: must-have
diff --git a/problems/069-normalization-and-when-to-denormalize/question.md b/problems/069-normalization-and-when-to-denormalize/question.md
@@ -1,7 +1,7 @@
 ---
 id: 69
 title: Normalization and When to Denormalize
-category: Database Internals
+category: Databases & Operations
 topics: [normalization, 3NF, denormalization, star schema]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/070-b-tree-vs-hash-vs-lsm-tree/question.md b/problems/070-b-tree-vs-hash-vs-lsm-tree/question.md
@@ -1,7 +1,7 @@
 ---
 id: 70
 title: B-Tree vs Hash vs LSM Tree
-category: Database Internals
+category: Databases & Operations
 topics: [B-tree, hash, LSM, storage engines]
 difficulty: Medium
 interview_value: must-have
diff --git a/problems/071-read-replicas-and-replication-lag/question.md b/problems/071-read-replicas-and-replication-lag/question.md
@@ -1,7 +1,7 @@
 ---
 id: 71
 title: Read Replicas and Replication Lag
-category: Database Operations
+category: Databases & Operations
 topics: [replicas, replication lag, read after write]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/072-sharding-and-picking-a-shard-key/question.md b/problems/072-sharding-and-picking-a-shard-key/question.md
@@ -1,7 +1,7 @@
 ---
 id: 72
 title: Sharding and Picking a Shard Key
-category: Database Operations
+category: Databases & Operations
 topics: [sharding, shard key, hot shards, hash]
 difficulty: Hard
 interview_value: must-have
diff --git a/problems/073-database-connection-pooling/question.md b/problems/073-database-connection-pooling/question.md
@@ -1,7 +1,7 @@
 ---
 id: 73
 title: Database Connection Pooling
-category: Database Operations
+category: Databases & Operations
 topics: [connection pool, PgBouncer, sizing, Postgres]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/074-deadlocks-and-lock-escalation/question.md b/problems/074-deadlocks-and-lock-escalation/question.md
@@ -1,7 +1,7 @@
 ---
 id: 74
 title: Deadlocks and Lock Escalation
-category: Database Operations
+category: Databases & Operations
 topics: [deadlocks, locks, retries, lock escalation]
 difficulty: Medium
 interview_value: strong
diff --git a/problems/075-sql-vs-nosql/question.md b/problems/075-sql-vs-nosql/question.md
@@ -1,7 +1,7 @@
 ---
 id: 75
 title: SQL vs NoSQL
-category: Database Internals
+category: Databases & Operations
 topics: [SQL, NoSQL, KV, document, wide column, graph]
 difficulty: Medium
 interview_value: strong
diff --git a/scripts/build_index.py b/scripts/build_index.py
@@ -27,18 +27,15 @@
 INDEX = REPO / "PROBLEMS.md"
 
 CATEGORY_LEGEND = [
-    ("SQL & Querying",          "Writing, reading and reasoning about SQL like a senior engineer"),
-    ("Data Modeling",           "Star schemas, history tracking, grain, dimensions, SCDs"),
-    ("Database Internals",      "Engines, ACID, isolation levels, indexes, B-tree vs LSM, normalization"),
-    ("Database Operations",     "Running databases at scale: replicas, sharding, connection pools, deadlocks"),
-    ("Batch Pipelines",         "ETL/ELT, idempotency, data cleaning, contracts, orchestration"),
-    ("Streaming",               "Kafka, watermarks, exactly-once, ordering, consumer lag"),
-    ("Storage & Formats",       "Parquet, lakehouse, partitioning, clustering, materialized views"),
-    ("System Design",           "End-to-end pipelines for real consumer and energy-sector products"),
-    ("Cloud Services",          "Picking between AWS, GCP and Azure services with clear trade-offs"),
-    ("Cost Optimization",       "Finding waste in queries, jobs, and infrastructure when the bill spikes"),
-    ("Debugging & Reliability", "Step-by-step investigation when the number is wrong or the job died"),
-    ("People & Process",        "Mentoring, comms, postmortems, ownership, rollouts"),
+    ("SQL & Querying",                 "Writing, reading and reasoning about SQL like a senior engineer"),
+    ("Data Modeling",                  "Star schemas, history tracking, grain, dimensions, SCDs"),
+    ("Databases & Operations",         "Engines, ACID, indexes, replication, sharding, pools, deadlocks"),
+    ("Batch Pipelines & Orchestration","ETL/ELT, idempotency, contracts, Airflow ops, pipeline debugging"),
+    ("Streaming",                      "Kafka, watermarks, exactly-once, ordering, consumer lag"),
+    ("Storage & Lakehouse",            "Parquet, lakehouse formats, partitioning, clustering, materialized views"),
+    ("Cloud & Cost",                   "Picking AWS/GCP/Azure services and finding waste when bills spike"),
+    ("System Design",                  "End-to-end pipelines for real consumer and energy-sector products"),
+    ("People & Process",               "Mentoring, comms, postmortems, ownership, rollouts"),
 ]
 
 
