Skip to content

Commit 2c53aeb

Browse files
committed
Update template-readme.md
1 parent e9dc69e commit 2c53aeb

1 file changed

Lines changed: 10 additions & 3 deletions

File tree

Code Examples/template-readme.md

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -92,10 +92,19 @@ For each layer, also say what you considered and rejected. A choice with a rejec
9292
Not every project has all five layers. If you have no buffer, say so and say why you did not need one. An empty section with an honest sentence in it is better than pretending.
9393

9494
## Connect
95+
*Everything for data ingestion — pulling data or receiving it, and the tools and processing that make that happen.*
96+
9597
## Buffer
98+
*Everything to do with message queues (e.g. Kafka, Kinesis, Pub/Sub) that sit between ingestion and processing.*
99+
96100
## Processing
101+
*Everything with processing — Python scripts, Spark, dockerized apps, orchestration and observability.*
102+
97103
## Storage
104+
*Where the data lands — OLAP and OLTP databases, data lakes and lakehouses.*
105+
98106
## Visualization
107+
*How people and systems consume the data — apps, dashboards, BI tools and, of course, APIs for clients.*
99108

100109
# Pipelines
101110

@@ -107,8 +116,6 @@ Link to the files, do not paste the whole pipeline in here. The code already liv
107116
Say what the pipeline does when something goes wrong. Retries, what happens to a bad record, whether the job fails loudly or writes garbage quietly. This is the part people forget and it is exactly the part that separates a project from a tutorial.
108117

109118
## Stream Processing
110-
### Storing Data Stream
111-
### Processing Data Stream
112119
## Batch Processing
113120
## Visualizations
114121

@@ -151,4 +158,4 @@ Add the link to your LinkedIn profile.
151158

152159
# Appendix
153160

154-
[Markdown Cheat Sheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)
161+
This template comes from Andreas Kretz's Data Engineering Cookbook. If it helped you, point people back to it: [The Data Engineering Cookbook](https://github.com/andkret/Cookbook)

0 commit comments

Comments
 (0)