Text Preprocessing, Language Modeling & Spelling Correction

Overview

This repository implements a complete NLP pipeline for news article analysis in three sequential parts:

Part 1: Data preparation with Porter stemming

Part 2: N-gram language modeling with add-k smoothing

Part 3: Spelling correction system optimized for stemmed text

Project Structure

N-Gram Language Modeling/
├── dataPreparation.py                 # Part 1: Data preparation (stemming required)
├── nGramLanguageModeling.py                 # Part 2: N-gram language modeling
├── spellingCorrection.py                 # Part 3: Spelling correction for stemmed text
├── data/
│   └── news.csv          # Input dataset 
├── processed_data/       # Generated outputs 
│   ├── processed_sentences.json
│   ├── train_corpus.txt
│   ├── val_corpus.txt
│   ├── test_corpus.txt
│   └── processed_corpora.json
├── README.md             

Academic Context

This project was developed for an advanced NLP course. All preprocessing and modeling follow assignment specifications precisely, while the critical analysis demonstrates deeper understanding expected at graduate level.