Skip to content

Commit 625bf79

Browse files
andkretandkret
committed
added claude instructions to help me write effiently
1 parent d58e9a7 commit 625bf79

3 files changed

Lines changed: 295 additions & 3 deletions

File tree

‎BUILD-PDF.md‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Building the Cookbook PDF
2+
3+
How to turn the multi-file cookbook into a single PDF **without** merging files
4+
or hand-fixing links. We use a book compiler (mdBook) that resolves cross-file
5+
links automatically. You keep editing section files and pushing to GitHub as
6+
normal — the PDF is just a build output.
7+
8+
## The idea
9+
10+
- **Content** lives in `sections/*.md` — edit normally, push to GitHub normally.
11+
- **Order** is declared once in `sections/SUMMARY.md`.
12+
- **Styling** lives in CSS/theme, never in the content.
13+
- **PDF** is a disposable output — regenerate with one command, never hand-edit.
14+
15+
---
16+
17+
## One-time setup
18+
19+
### 1. Install mdBook + PDF backend
20+
```bash
21+
brew install mdbook
22+
cargo install mdbook-pdf # needs Rust (https://rustup.rs)
23+
```
24+
25+
### 2. Create `book.toml` in the repo root
26+
```toml
27+
[book]
28+
title = "The Data Engineering Cookbook"
29+
authors = ["Andreas Kretz"]
30+
src = "sections"
31+
32+
[output.html]
33+
[output.pdf]
34+
```
35+
36+
### 3. Create `sections/SUMMARY.md` (declares chapter order)
37+
```markdown
38+
# Summary
39+
40+
[Introduction](01-Introduction.md)
41+
- [Basic Engineering Skills](02-BasicSkills.md)
42+
- [Advanced Engineering Skills](03-AdvancedSkills.md)
43+
- [Free Hands-On Courses](04-HandsOnCourse.md)
44+
- [Case Studies](05-CaseStudies.md)
45+
- [Best Practices Cloud Platforms](06-BestPracticesCloud.md)
46+
- [130+ Data Sources](07-DataSources.md)
47+
- [1001 Interview Questions](08-InterviewQuestions.md)
48+
- [Recommended Books & Courses](09-BooksAndCourses.md)
49+
- [Updates](10-Updates.md)
50+
```
51+
52+
### 4. Two one-time fix-ups
53+
- **Links:** make in-file links use `XX-File.md#anchor` (drop the `sections/`
54+
prefix — mdBook's `src` is already `sections`).
55+
- **Images:** make sure the `images/` folder is reachable from `sections/`
56+
(copy or symlink it so paths like `images/foo.jpg` resolve).
57+
58+
---
59+
60+
## Every time you want a fresh PDF
61+
62+
```bash
63+
mdbook build
64+
# website -> ./book
65+
# PDF -> ./book/pdf/output.pdf
66+
```
67+
68+
Add a video/article to a section, push to GitHub, rerun `mdbook build`. Done.
69+
Cross-file links resolve automatically — no merging, no sed, nothing to break.
70+
71+
---
72+
73+
## Workflow reminder
74+
75+
1. **Update content first** (videos, articles, courses into the right sections).
76+
2. Push to GitHub as usual.
77+
3. Run `mdbook build` to regenerate the PDF.
78+
79+
---
80+
81+
## Alternatives (if mdBook ever doesn't fit)
82+
83+
- **Honkit** — maintained GitBook fork, same `SUMMARY.md` model:
84+
`npm i -g honkit && honkit pdf ./ cookbook.pdf`
85+
- **Quarto** — more powerful, multi-format, but its PDF path uses LaTeX which
86+
fights the raw HTML (`<div align>`, `<img>`) in these files. Only worth it if
87+
you rewrite the HTML bits as plain Markdown.

‎EDITORIAL-RULEBOOK.md‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
# Cookbook Editorial Rulebook
2+
3+
The rules for what goes into the Data Engineering Cookbook, what stays out, and
4+
how everything cross-links. Read this before adding or restructuring content.
5+
If an edit breaks a rule here, the rule wins — or we change the rule on purpose.
6+
7+
---
8+
9+
## 1. What the Cookbook IS
10+
11+
The Cookbook is the **wide map of the field** and the **on-ramp**. Its job is to
12+
help a reader understand the 2026 data-engineering landscape, locate themselves
13+
in it, and learn *how to think* — then route them to the right deeper resource.
14+
15+
It is the **discovery layer**: the broad, browsable, GitHub/SEO surface that
16+
pulls strangers in and turns them into email subscribers.
17+
18+
The Cookbook owns:
19+
- **The 2026 worldview** — how the field splits (data / analytics / ML / AI
20+
engineering), what changed, where a reader fits.
21+
- **Roadmaps / "Start Here"** — the on-ramp paths by role and level.
22+
- **Judgment & decisions** — "lake vs warehouse vs lakehouse," "stream vs
23+
batch," "do you even need big data." The *why* and *how to choose*.
24+
- **Patterns & concepts** — mental models, architecture thinking, trade-offs.
25+
- **The breadth net** — case studies, interview questions, data sources. The
26+
surface that attracts traffic.
27+
28+
## 2. What the Cookbook is NOT
29+
30+
- **NOT a tool catalog.** It never lists "all the tools for the Process phase."
31+
That is the Tools Guide's job (see §3). When a reader asks "but which tool?",
32+
the Cookbook *links to the Tools Guide* — it does not answer inline.
33+
- **NOT a build manual.** Step-by-step pipeline builds live in courses /
34+
hands-on content, not here.
35+
- **NOT a dumping ground for links.** Resources are woven in at the point of
36+
curiosity (see §5), not collected in an end chapter.
37+
38+
> **The one-sentence test:** if a paragraph could be replaced by a row in the
39+
> Tools Guide matrix, it does not belong in the Cookbook.
40+
41+
---
42+
43+
## 3. Division of labor (Cookbook vs Tools Guide vs the rest)
44+
45+
| Layer | Document | Owns | Funnel role |
46+
|---|---|---|---|
47+
| **Map** | **Cookbook** | Worldview, roadmaps, decisions, concepts, breadth net | Discovery → email capture |
48+
| **Mechanics** | **Tools Guide** | Blueprint phases, tool matrix, pipeline patterns, project→course links | Focused magnet → Academy |
49+
| **Depth** | **Free content** (YT / LinkedIn / Udemy / free courses) | How-to, demos, tutorials | Woven inline everywhere |
50+
| **Paid apex** | **Academy → Masterclass** | Full courses, certification | Apex CTA |
51+
52+
**Rule:** each fact has exactly one home. The blueprint *mechanics and tool
53+
catalog* live in the Tools Guide. The Cookbook references them; it never
54+
re-creates them.
55+
56+
When the Cookbook touches the blueprint (Connect → Buffer → Process → Store →
57+
Visualize), it stays at the **decision/concept** altitude and hands off:
58+
59+
> *"…for the full tool catalog and cloud-native vs open-source alternatives for
60+
> this phase, grab the Tools Guide → {{TOOLS_GUIDE_CTA}}"*
61+
62+
That hand-off is also a funnel step (email capture / Academy jump). One move,
63+
both goals.
64+
65+
---
66+
67+
## 4. The funnel ladder (where every CTA points)
68+
69+
Every piece of content should make the next step obvious. The ladder:
70+
71+
1. **Cookbook** (free, broad) → capture the **email via download**.
72+
2. **Tools Guide** (focused magnet) → another capture / Academy jump.
73+
3. **Free content** (YT / LinkedIn / courses) → woven inline at each decision.
74+
4. **Academy → Masterclass** (paid apex) → the deep "go further" CTA.
75+
76+
Goals, in order: build trust → capture email → drive views to (sponsored) free
77+
content → upsell to the paid tier.
78+
79+
---
80+
81+
## 5. The inline-link pattern (how resources are woven in)
82+
83+
Resources appear **at the point of curiosity**, never in a separate chapter.
84+
Model: the Tools Guide's example-projects move — concept → diagram → "here's the
85+
course."
86+
87+
After a concept or decision, attach the relevant exits:
88+
89+
```markdown
90+
## Storing your data: lake, warehouse, or lakehouse?
91+
92+
[2–3 paragraphs of opinionated judgment — when each makes sense, the trade-off
93+
that trips people up, what I'd pick and why.]
94+
95+
> 🎥 Watch: [video]
96+
> 📝 Read: [article / LinkedIn post]
97+
> 🧰 Full tool catalog for this phase: {{TOOLS_GUIDE_CTA}}
98+
> 🎓 Build it hands-on: [free course]
99+
> 🚀 Go deeper: {{MASTERCLASS_CTA}}
100+
```
101+
102+
Rules for inline links:
103+
- Only attach links that genuinely fit the topic. No padding.
104+
- A topic with no link yet = a visible gap = your content backlog. Leave the
105+
block; it tells you what to record next.
106+
- Prose carries the value. Links are exits, not the substance.
107+
108+
---
109+
110+
## 6. CTA tokens (define once, reuse everywhere)
111+
112+
Never hard-code campaign links inline. Use tokens so a single edit re-points the
113+
whole book — critical for the planned Academy→free / Masterclass→paid switch.
114+
115+
| Token | Means | Today points to |
116+
|---|---|---|
117+
| `{{DOWNLOAD_CTA}}` | Get the PDF / join the mailing list | email-capture page |
118+
| `{{TOOLS_GUIDE_CTA}}` | Get the Tools Guide | Tools Guide download |
119+
| `{{MASTERCLASS_CTA}}` | The paid apex | Academy (later: Masterclass) |
120+
| `{{ACADEMY_CTA}}` | The course platform | LearnDataEngineering.com |
121+
122+
When the business model changes, update the token definitions in one place — not
123+
the content.
124+
125+
---
126+
127+
## 7. Modernization rules (keeping it "2026")
128+
129+
- **Lead with current.** Modern Data Stack, lakehouse + table formats
130+
(Iceberg / Delta / Hudi), the AI-data layer (vector DBs, embeddings, RAG,
131+
feature stores), DataOps / observability / contracts / catalogs.
132+
- **Don't delete history — relabel it.** Legacy tech (MapReduce, Sqoop, Flume,
133+
Storm, Samza, Drill, Solr) becomes a short *"Legacy / know-it-exists"* note,
134+
not a deep section.
135+
- **Reframe "Data Science" → the four destination roles** (data / analytics /
136+
ML / AI engineering) wherever the old framing appears. See §9 for how these
137+
relate to the origin-based roadmaps — don't conflate the two.
138+
- **Surface, don't bury** (see §10). Hot, in-demand topics — AI engineering
139+
above all — get direct entry points near the top, not a subsection three
140+
levels deep.
141+
- **Opinion over completeness.** A clear recommendation beats an exhaustive
142+
survey. Surveys are the Tools Guide's job.
143+
144+
---
145+
146+
## 8. Quick decision guide ("where does this go?")
147+
148+
- Explaining *why* or *how to choose* → **Cookbook**.
149+
- Listing *which tools exist* for a phase → **Tools Guide** (link to it).
150+
- Step-by-step *how to build* it → **course / free content** (link to it).
151+
- A real resource (video / article / course) → **inline, at the topic**.
152+
- "All the resources in one place" → **no.** See §5.
153+
154+
---
155+
156+
## 9. Paths & roadmaps: two axes (don't conflate them)
157+
158+
Two different questions, two different structures. Mixing them is the most
159+
likely structural mistake — keep them separate.
160+
161+
| Axis | Answers | Keyed on | Owner |
162+
|---|---|---|---|
163+
| **Destination** | "What are the DE-adjacent roles in 2026, which do I want?" | where you're **going** | Cookbook-owned field map |
164+
| **Origin** | "I'm an analyst / dev / grad — how do I get in from here?" | where you're **coming from** | mirrors the LDE roadmaps 1:1 |
165+
166+
**Destination → the field map.** A Cookbook-owned worldview section: data /
167+
analytics / ML / AI engineering as distinct roles. Pure orientation, no funnel
168+
pressure — it serves the reader who doesn't yet know what they want. No LDE
169+
dependency.
170+
171+
**Origin → the "Start Here" roadmaps.** These mirror the LDE roadmap selector
172+
1:1 (Beginner / Data Analyst / Data Scientist / Software Developer). The Cookbook
173+
gives the *free orientation* for each ("the path from where you are"); each one's
174+
apex CTA deep-links to the matching **paid** LDE roadmap. The Cookbook never
175+
reproduces the week-by-week curriculum (§2).
176+
177+
- Roadmap selector: https://learndataengineering.com/p/roadmap-selector
178+
- Canonical origin roadmaps = Beginner · Data Analyst · Data Scientist ·
179+
Software Developer. Name them to match LDE so the funnel is 1:1.
180+
181+
**AI Engineering straddles both.** On LDE it's a *destination* ("become an AI
182+
engineer"), not an origin like the other four. So it appears in the field map as
183+
a first-class role **and** as a surfaced roadmap on-ramp — flagged as "where
184+
you're going," not "where you came from."
185+
186+
---
187+
188+
## 10. Surface, don't bury
189+
190+
On a five-year-old wide map, the topics people actually search for get buried
191+
under accumulated structure unless we deliberately surface them. Don't let that
192+
happen — give in-demand topics direct entry points near the top, the same way
193+
the LDE site routes people straight to AI engineering instead of hiding it.
194+
195+
**Worked example — AI engineering appears in three surfaced places, not one:**
196+
- In the **field map** as a first-class destination role (§9).
197+
- As a **direct on-ramp / CTA** to the AI Engineering roadmap.
198+
- As the **AI-data layer** woven through the blueprint — vector DBs, embeddings,
199+
RAG, feature stores — connecting to the existing Ollama/RAG tutorials in
200+
`sections/04-HandsOnCourse.md`.
201+
202+
**Why it matters for the funnel:** the surfaced AI on-ramp *is* a top-of-funnel
203+
magnet, because it matches what strangers are searching for in 2026. Burying it
204+
wastes the exact traffic we most want to capture.
205+
206+
**Rule of thumb:** if a topic is in high demand and a reader would expect to find
207+
it fast, it gets a surfaced entry point — not a deep subsection.

‎README.md‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
<br>
2020

2121
## If You Like This Book & Need More Help
22-
Check out my Data Engineering Academy at LearnDataEngineering.com trusted by almost 2,000 students!
22+
Check out my Data Engineering Academy at LearnDataEngineering.com trusted by over 2,000 students!
2323

2424
**Visit learndataengineering.com:** [Click Here](https://learndataengineering.com)
2525

@@ -28,8 +28,6 @@ Check out my Data Engineering Academy at LearnDataEngineering.com trusted by alm
2828
- Proven process based on years of experience and hundreds of hours of personal coaching
2929
- Over 30 prepared courses on the most important techniques, fundamental tools and platforms plus our
3030
- Associate Data Engineer Certification
31-
- Academy Discord server with over 1,000 members
32-
3331

3432

3533
## Support This Book For Free!

0 commit comments

Comments
 (0)