Skip to content

Commit 9d0e6ad

Browse files
shiningflashshiningflash
committed
Merge branch 'problems' into main
- Restructure all 65 problems into problems/NNN-slug/ - Add notify-blog workflow (auto-syncs to amirulislamalmamun.com/practice)
2 parents a61d613 + 0356123 commit 9d0e6ad

135 files changed

Lines changed: 985 additions & 163 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/notify-blog.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
name: Notify Blog
2+
3+
# Fires a repository_dispatch event at shiningflash/shiningflash.github.io
4+
# whenever problems or PROBLEMS.md change on main, so the blog's
5+
# sync-practice workflow runs and the new content goes live within ~1 minute.
6+
7+
on:
8+
push:
9+
branches: [main]
10+
paths:
11+
- "problems/**"
12+
- "PROBLEMS.md"
13+
- "scripts/build_index.py"
14+
15+
workflow_dispatch:
16+
17+
jobs:
18+
dispatch:
19+
runs-on: ubuntu-latest
20+
steps:
21+
- name: Dispatch to blog
22+
env:
23+
# A fine-grained PAT with "Contents: write" on shiningflash.github.io.
24+
# Add it as a repo secret named BLOG_DISPATCH_TOKEN.
25+
GH_TOKEN: ${{ secrets.BLOG_DISPATCH_TOKEN }}
26+
run: |
27+
if [ -z "$GH_TOKEN" ]; then
28+
echo "BLOG_DISPATCH_TOKEN secret not set — skipping dispatch."
29+
echo "Add it under Settings → Secrets and variables → Actions."
30+
exit 0
31+
fi
32+
curl -fsSL -X POST \
33+
-H "Accept: application/vnd.github+json" \
34+
-H "Authorization: Bearer $GH_TOKEN" \
35+
-H "X-GitHub-Api-Version: 2022-11-28" \
36+
https://api.github.com/repos/shiningflash/shiningflash.github.io/dispatches \
37+
-d '{"event_type":"practice-problems-updated","client_payload":{"source_repo":"${{ github.repository }}","sha":"${{ github.sha }}"}}'
38+
echo "Dispatched practice-problems-updated to blog."

‎CONTRIBUTION.md‎

Lines changed: 38 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,44 @@ Thanks for helping expand the Data Engineering Practice Problems collection. Thi
1919

2020
## Repository Structure Standards
2121

22-
- Each problem lives in its own folder at the repository root, e.g. `Problem X: <Descriptive Title>/`.
22+
- Every problem lives in `problems/NNN-kebab-case-title/` where `NNN` is a zero-padded id (`001`, `065`, ...). Pick the next free id.
2323
- Inside each problem directory:
24-
- `question.md` – prompt or requirements written in Markdown; include context, inputs, outputs, and any bonus objectives.
25-
- `solution.py` – reference implementation using standard library where possible; include a short module docstring explaining the approach.
26-
- Optional subdirectories (`data/`, `tests/`) if the problem demands bespoke assets. Keep paths relative (e.g., `../data/` for shared sample data).
24+
- `question.md` – the problem prompt. **Must start with YAML frontmatter** (see below), followed by a single H1 `# Problem N — Title`, then the body.
25+
- `solution.md` – written walkthrough (preferred for design / discussion problems), **or** `solution.py` for runnable code problems. Pick one; the frontmatter declares which.
26+
- Optional subdirectories (`data/`, `tests/`) if the problem demands bespoke assets. Keep paths relative (e.g., `../../data/` for shared sample data).
2727
- Shared datasets belong in the top-level `data/` directory. Provide a `README` snippet or inline comments describing their provenance if they are newly added.
2828

29+
### Required frontmatter
30+
31+
Every `question.md` begins with this block — `PROBLEMS.md` is generated from it:
32+
33+
```yaml
34+
---
35+
id: 21
36+
title: Data Platform for an Electricity Retailer
37+
category: System Design
38+
topics: [smart meter, IoT, warehouse, batch]
39+
difficulty: Hard
40+
solution: solution.md
41+
---
42+
```
43+
44+
Allowed `category` values match the legend in `PROBLEMS.md`. `difficulty` is one of `Easy`, `Medium`, `Hard`. `solution` is the filename of the answer file in the same folder.
45+
46+
### Regenerating the index
47+
48+
After adding or editing a problem, regenerate the top-level index:
49+
50+
```bash
51+
python3 scripts/build_index.py
52+
```
53+
54+
CI (or a pre-PR check) can verify staleness with:
55+
56+
```bash
57+
python3 scripts/build_index.py --check
58+
```
59+
2960
## Problem Design Guidelines
3061

3162
- Aim for realistic data engineering themes: streaming ingestion, batch transformation, data quality, orchestration, storage optimisation, monitoring, or reliability.
@@ -45,8 +76,10 @@ Thanks for helping expand the Data Engineering Practice Problems collection. Thi
4576

4677
Before opening a pull request, confirm:
4778

79+
- [ ] `question.md` has valid frontmatter (id, title, category, topics, difficulty, solution).
4880
- [ ] `question.md` reads cleanly with correct Markdown formatting.
49-
- [ ] `solution.py` runs locally using the provided instructions.
81+
- [ ] The declared `solution` file exists and is the right type (`.md` or `.py`).
82+
- [ ] `python3 scripts/build_index.py --check` passes.
5083
- [ ] README references (if touched) remain accurate.
5184

5285
Thank you for sharing your expertise and helping fellow engineers level up. Star the repository to stay notified about new challenges and updates!

‎PROBLEMS.md‎

Lines changed: 85 additions & 83 deletions
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,12 +47,28 @@ And many more coming ...
4747
python -m venv venv && source venv/bin/activate
4848

4949
# 2. Use Python 3.10+
50-
# 3. Pick a problem
51-
# Each folder has a question.md and a reference solution.py
50+
# 3. Browse PROBLEMS.md for the full index, or open any folder in problems/.
51+
# Each problem lives in problems/NNN-slug/ and contains:
52+
# - question.md (problem statement, with YAML frontmatter)
53+
# - solution.md (written walkthrough) OR solution.py (runnable code)
5254
```
5355

5456
Inputs live in `data/`, outputs are generated beside them for easy inspection. Data files are excluded intentionally to keep the repo lightweight.
5557

58+
### Repo layout
59+
60+
```
61+
problems/
62+
001-log-file-error-analysis/
63+
question.md
64+
solution.py
65+
...
66+
data/ # sample input files (gitignored where large)
67+
scripts/
68+
build_index.py # regenerates PROBLEMS.md from question.md frontmatter
69+
PROBLEMS.md # generated index — do not edit by hand
70+
```
71+
5672
---
5773

5874
## How to Contribute

Problem 1: Log File Error Analysis/question.md renamed to problems/001-log-file-error-analysis/question.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,13 @@
1-
### Problem 1: Log File Error Analysis
1+
---
2+
id: 1
3+
title: Log File Error Analysis
4+
category: Logs and Monitoring
5+
topics: [file streaming, counters, top-N, IoT logs]
6+
difficulty: Easy
7+
solution: solution.py
8+
---
9+
10+
# Problem 1 — Log File Error Analysis
211

312
**Scenario:**
413
You have a huge log file from an IoT platform. Each line follows this structure:

Problem 1: Log File Error Analysis/solution.py renamed to problems/001-log-file-error-analysis/solution.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
from collections import defaultdict
99
from typing import Dict
1010

11-
LOG_FILE_PATH = "../data/sensor_data.log"
11+
LOG_FILE_PATH = "../../data/sensor_data.log"
1212

1313

1414
def analyze_log(file_path: str) -> Dict[str, Dict[str, int]]:

Problem 2: Rolling Average of Sensor Readings/question.md renamed to problems/002-rolling-average-of-sensor-readings/question.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,13 @@
1-
## Problem 2: Rolling Average of Sensor Readings (Real-Time Processing)
1+
---
2+
id: 2
3+
title: Rolling Average of Sensor Readings
4+
category: Streaming
5+
topics: [rolling window, deque, IoT sensors, real-time]
6+
difficulty: Easy
7+
solution: solution.py
8+
---
9+
10+
# Problem 2 — Rolling Average of Sensor Readings
211

312
**Scenario:**
413
You are building a data pipeline for IoT sensors (like BESS, PV inverters, weather stations).

Problem 2: Rolling Average of Sensor Readings/solution.py renamed to problems/002-rolling-average-of-sensor-readings/solution.py

File renamed without changes.

Problem 3: Transform and Clean Raw Data for Analytics/question.md renamed to problems/003-transform-and-clean-raw-data-for-analytics/question.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,13 @@
1-
## Problem 3: Transform and Clean Raw Data for Analytics
1+
---
2+
id: 3
3+
title: Transform and Clean Raw Data for Analytics
4+
category: Data Cleaning
5+
topics: [CSV, validation, regex, date checks]
6+
difficulty: Medium
7+
solution: solution.py
8+
---
9+
10+
# Problem 3 — Transform and Clean Raw Data for Analytics
211

312
**Scenario:**
413
You receive raw user activity data from a partner API in a messy CSV file. The company wants to load it into a data warehouse (e.g., BigQuery or PostgreSQL) for analytics.

Problem 3: Transform and Clean Raw Data for Analytics/solution.py renamed to problems/003-transform-and-clean-raw-data-for-analytics/solution.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,9 @@
1111
from datetime import datetime
1212
from typing import Dict, Iterator
1313

14-
DIRTY_WAREHOUSE_DATA_FILE_PATH = "../data/dirty_warehouse_data.csv"
15-
CLEANED_WAREHOUSE_DATA_FILE_PATH = "../data/cleaned_warehouse_data.csv"
16-
INVALID_ROWS_FILE_PATH = "../data/invalid_warehouse_data.csv"
14+
DIRTY_WAREHOUSE_DATA_FILE_PATH = "../../data/dirty_warehouse_data.csv"
15+
CLEANED_WAREHOUSE_DATA_FILE_PATH = "../../data/cleaned_warehouse_data.csv"
16+
INVALID_ROWS_FILE_PATH = "../../data/invalid_warehouse_data.csv"
1717

1818
EMAIL_REGEX = re.compile(r"^[\w\.-]+@[\w\.-]+\.\w+$")
1919

0 commit comments

Comments
 (0)