Bank Simulation Kafka Real Time, Apache Spark Streaming Data Engineering Project

Introduction

This project involves executing an end-to-end real-time data engineering pipeline using simulated bank transaction data. Leveraging technologies such as Python, Apache Kafka, PySpark, DigitalOcean, Elasticsearch, and Kibana, the project focuses on real-time data streaming and processing with Apache Spark. The pipeline enables real-time data ingestion, transformation, and visualization to deliver actionable insights and support critical decision-making processes.

Architecture

Technology Used

• Programming Language - Python
• DigitalOcean
• Apache Kafka
• Apache Spark
• Kibana
• ElasticSearch

DashBoard

Kafka-Broker-View

Project Overview

Overview.mp4