Production-style PySpark data pipeline implementing a Bronze–Silver–Gold architecture to transform raw e-commerce event data into analytics-ready business metrics.
- Bronze: Raw ingestion with enforced schema + partitioned Parquet storage
- Silver: Data cleaning and validation (business rules applied)
- Gold: Aggregated KPIs for reporting (revenue, conversion metrics, breakdowns)
- Daily revenue
- Daily conversion rate
- Revenue by device type
- Conversion by traffic source
Note: Synthetic data is used to simulate production event logs.
In a real system, data would be sourced from cloud storage or streaming platforms.
notebooks/ → End-to-end pipeline notebook
images/ → Project visuals
README.md → Documentation
- Open the notebook in Google Colab
- Run all cells from top to bottom
- The pipeline generates synthetic event data and produces Bronze, Silver, and Gold layer outputs
- PySpark
- Parquet
- Google Colab