Data Processor Solution with Node.js for Revieve technical test

.eslintrc.json

@@ -0,0 +1,22 @@
+{
+    "env": {
+      "browser": true,
+      "es2021": true,
+      "node": true
+    },
+    "extends": [
+      "eslint:recommended",
+      "plugin:import/errors",
+      "plugin:import/warnings",
+      "plugin:import/typescript",
+      "plugin:prettier/recommended"
+    ],
+    "parserOptions": {
+      "ecmaVersion": 12,
+      "sourceType": "module"
+    },
+    "rules": {
+      "prettier/prettier": "error"
+    }
+  }
+

.github/workflows/data-processing.yml

@@ -0,0 +1,49 @@
+name: Data Processing and Testing
+
+on:
+  push:
+    branches:
+      - main
+      - '**'
+  pull_request:
+    branches:
+      - main
+      - '**'
+  workflow_dispatch:
+
+jobs:
+  build-and-run:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v2
+
+    - name: Log in to Docker Hub
+      uses: docker/login-action@v2
+      with:
+        username: ${{ secrets.DOCKER_USERNAME }}
+        password: ${{ secrets.DOCKER_PASSWORD }}
+
+    - name: Build Docker image
+      run: docker build -t my-node-app .
+
+    - name: Run Docker container in detached mode with custom command
+      run: docker run -d --name data-processing-container my-node-app tail -f /dev/null
+
+    - name: Check container status
+      run: docker ps -a
+
+    - name: Wait for the container to be ready
+      run: |
+        sleep 10
+        docker logs data-processing-container
+
+    - name: Run tests
+      run: docker exec data-processing-container npm test
+
+    - name: Cleanup
+      run: docker rm -f data-processing-container

.gitignore

@@ -0,0 +1,35 @@
+# Node modules
+node_modules/
+
+# Logs
+logs/
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# Dependency directories
+jspm_packages/
+
+# Build directories
+dist/
+build/
+
+# Runtime data
+pids/
+*.pid
+*.seed
+*.pid.lock
+
+# Compiled source
+lib/
+
+# Coverage
+coverage/
+
+# System files
+.DS_Store
+Thumbs.db
+
+# Environment variables
+.env

.prettierrc

@@ -0,0 +1,4 @@
+{
+  "singleQuote": true,
+  "semi": false
+}

Dockerfile

@@ -0,0 +1,17 @@
+# Use the official Node.js image
+FROM node:14
+
+# Create and change to the app directory
+WORKDIR /usr/src/app
+
+# Copy package.json and package-lock.json
+COPY package*.json ./
+
+# Install app dependencies
+RUN npm install
+
+# Copy the rest of the application code
+COPY . .
+
+# Command to run your data processing script
+CMD ["node", "index.js"]

GUIDELINE.md

@@ -0,0 +1,101 @@
+
+# GUIDELINE.md
+
+## Introduction
+
+This document provides instructions on how to set up and run the script for this Node.js project.
+
+## Implementation
+
+1. This is a simple project using Node.js to process and transform a CSV data based on the requirements.
+2. This project csv-parser, csv-writer, and file stream library for the main logic.
+3. As for additional task, a simple unit test is also being added using mocha chai testing framework that are quite popular and easy to use.
+4. On the logger feature, I am using winston logging library because it supports different logging levels, transports, and formats.
+5. Lastly, on the linter library, I am using the eslint and prettier library to ensure code quality and adherence to best practices.
+
+## Code Optimization
+
+In the solution code, there are also some code technique being used for a better performance and to ensure that we can handle a large datasets:
+
+1. Stream Processing:
+   * The use of fs.createReadStream and csv-parser allows processing large files in chunks rather than loading the entire file into memory.
+2. Asynchronous Processing:
+   * The parseCsv function is asynchronous, which helps in processing files without blocking the event loop, enhancing performance.
+3. Batch Processing:
+   * Writing all processed data at once using createObjectCsvWriter minimizes the number of I/O operations, which is more efficient than writing data one record at a time.
+
+## Prerequisites
+
+Ensure you have the following installed on your system:
+
+- [Node.js](https://nodejs.org/) (version 14.x or higher)
+- [npm](https://www.npmjs.com/) (comes with Node.js)
+
+## Installation
+
+1. **Clone the repository** (if you haven't already):
+    \`\`\`sh
+    git clone https://github.com/aditioagungnugroho/devops-interview.git
+    \`\`\`
+
+2. **Navigate to the project directory**:
+    \`\`\`sh
+    cd devops-interview
+    \`\`\`
+
+3. **Install the dependencies**:
+    \`\`\`sh
+    npm install
+    \`\`\`
+
+## Running the Script
+
+### Production Mode
+
+To run the data processing script:
+
+\`\`\`sh
+node index.js
+\`\`\`
+
+## Project Structure
+
+- **index.js**: The main entry point of the application.
+- **logger.js**: A module for logging functionalities.
+- **data/**: Directory containing data files used by the script.
+- **scripts/**: Directory containing additional scripts.
+- **test/**: Directory containing test files.
+- **.eslintrc.json**: Configuration file for ESLint.
+- **.gitignore**: Git ignore file.
+- **.prettierrc**: Configuration file for Prettier.
+- **package.json**: Contains metadata about the project and its dependencies.
+- **package-lock.json**: Describes the exact dependency tree that was generated.
+
+## Testing
+
+To run the tests:
+
+\`\`\`sh
+npm test
+\`\`\`
+
+## Linting
+
+To check for linting errors:
+
+\`\`\`sh
+npm run lint
+\`\`\`
+
+## Additional Notes
+
+- Ensure that the `data/` directory contains all necessary data files required by the script.
+- Adjust any configuration settings in the `.eslintrc.json` and `.prettierrc` files as needed.
+
+## Troubleshooting
+
+If you encounter any issues:
+
+1. Ensure all dependencies are installed correctly by running `npm install`.
+2. Check the logs for any error messages and follow the hints provided.
+3. Refer to the existing `README.md` for additional context and instructions.

data/customer_ranking.csv

@@ -0,0 +1,37 @@
+ID,FIRSTNAME,LASTNAME,TOTAL_EUROS
+34,Samuel,Lavoisier,145.25
+44,Mary,Fermi,138.41
+24,Ringo,Maxwell,136.83
+22,Paul,Lavoisier,124.71
+32,Samuel,Fermi,119.16
+5,John,Pasteur,99.14
+10,James,Lavoisier,73.18
+29,Ringo,Pasteur,70.03
+38,Sam,Fermi,65.07
+41,Sam,Pasteur,60.78
+21,Paul,Feynman,54.64
+15,Peter,Feynman,54.27
+59,Jane,Pasteur,52.64
+50,Sarah,Fermi,51.17
+17,Peter,Pasteur,50.91
+28,Ringo,Lavoisier,45.54
+3,John,Feynman,40.34
+9,James,Feynman,38.54
+47,Mary,Pasteur,38.28
+37,Sam,Heisenberg,37.19
+46,Mary,Lavoisier,36.67
+45,Mary,Feynman,35.79
+51,Sarah,Feynman,34.59
+20,Paul,Fermi,34.40
+19,Paul,Heisenberg,33.66
+54,Jane,Maxwell,28.20
+57,Jane,Feynman,23.15
+6,James,Maxwell,22.81
+25,Ringo,Heisenberg,20.87
+0,John,Maxwell,18.94
+35,Samuel,Pasteur,16.30
+40,Sam,Lavoisier,15.36
+8,James,Fermi,12.71
+36,Sam,Maxwell,9.81
+58,Jane,Lavoisier,6.49
+48,Sarah,Maxwell,5.88

data/customers.csv

@@ -0,0 +1,61 @@
+id,firstname,lastname
+0,John,Maxwell
+1,John,Heisenberg
+2,John,Fermi
+3,John,Feynman
+4,John,Lavoisier
+5,John,Pasteur
+6,James,Maxwell
+7,James,Heisenberg
+8,James,Fermi
+9,James,Feynman
+10,James,Lavoisier
+11,James,Pasteur
+12,Peter,Maxwell
+13,Peter,Heisenberg
+14,Peter,Fermi
+15,Peter,Feynman
+16,Peter,Lavoisier
+17,Peter,Pasteur
+18,Paul,Maxwell
+19,Paul,Heisenberg
+20,Paul,Fermi
+21,Paul,Feynman
+22,Paul,Lavoisier
+23,Paul,Pasteur
+24,Ringo,Maxwell
+25,Ringo,Heisenberg
+26,Ringo,Fermi
+27,Ringo,Feynman
+28,Ringo,Lavoisier
+29,Ringo,Pasteur
+30,Samuel,Maxwell
+31,Samuel,Heisenberg
+32,Samuel,Fermi
+33,Samuel,Feynman
+34,Samuel,Lavoisier
+35,Samuel,Pasteur
+36,Sam,Maxwell
+37,Sam,Heisenberg
+38,Sam,Fermi
+39,Sam,Feynman
+40,Sam,Lavoisier
+41,Sam,Pasteur
+42,Mary,Maxwell
+43,Mary,Heisenberg
+44,Mary,Fermi
+45,Mary,Feynman
+46,Mary,Lavoisier
+47,Mary,Pasteur
+48,Sarah,Maxwell
+49,Sarah,Heisenberg
+50,Sarah,Fermi
+51,Sarah,Feynman
+52,Sarah,Lavoisier
+53,Sarah,Pasteur
+54,Jane,Maxwell
+55,Jane,Heisenberg
+56,Jane,Fermi
+57,Jane,Feynman
+58,Jane,Lavoisier
+59,Jane,Pasteur

data/order_prices.csv

@@ -0,0 +1,51 @@
+ID,EUROS
+0,18.94
+1,61.43
+2,23.15
+3,34.40
+4,45.54
+5,15.36
+6,73.09
+7,25.22
+8,18.87
+9,37.19
+10,54.64
+11,34.59
+12,22.81
+13,39.21
+14,32.32
+15,40.34
+16,51.17
+17,65.35
+18,28.20
+19,52.64
+20,9.81
+21,12.71
+22,51.37
+23,20.87
+24,54.64
+25,43.67
+26,64.53
+27,69.26
+28,16.93
+29,2.90
+30,33.66
+31,60.78
+32,38.28
+33,5.88
+34,36.67
+35,29.14
+36,10.44
+37,46.07
+38,16.30
+39,18.30
+40,23.15
+41,71.48
+42,73.18
+43,50.91
+44,38.54
+45,63.29
+46,51.74
+47,9.39
+48,46.91
+49,6.49