This project handles the upload of CSV files to MinIO — a local simulation of Amazon S3. After uploading, it stores the file's storage path in a MySQL database and publishes a message to a Kafka topic to notify downstream services.
In a real-world scenario, I encountered the need to upload CSV files into an internal system. These files could be quite large (up to 7MB with ~100,000 rows), making real-time validation inefficient and performance-heavy.
To address this, a more scalable and efficient approach was adopted:
- ✅ Validate the MIME type to ensure it's a genuine CSV file
- ✅ Validate the CSV header for required structure
- ✅ Ensure the file size does not exceed 7MB
After this initial validation and upload, a background service (System B) will:
- Consume messages from the Kafka topic
- Check whether the file has already been processed (based on the
Statuscolumn in the database) - Process the file's content according to business rules asynchronously
- .NET – Main backend framework
- Swagger – API documentation
- MySQL – Relational database to store metadata
- MinIO – Local S3-compatible object storage
- Apache Kafka – Event streaming and message queuing
- Project structure diagram created using GitDiagram by @ahmedkhaleel2004