HDB Resale Pipeline

Pulls HDB resale flat transaction data from data.gov.sg, cleans it, and generates derived fields (remaining lease, price outlier flags, resale identifiers).

Setup

pip install -r requirements.txt

Run

python main.py

This runs the full pipeline end to end:

1. Fetch both HDB resale datasets from the data.gov.sg API
2. Clean data (check nulls/blanks, drop invalid floor area / storey rows)
3. Calculate remaining lease years
4. Remove duplicates (keep highest resale price per group)
5. Flag price-per-sqm outliers by town / flat type / lease band
6. Generate resale identifiers

Progress and validation results are printed to the console via logging.

Output files

Each stage writes a CSV, used as input by the next stage:

File	Produced by
combined.csv	fetch + clean
combined_failValidation.csv	rows that failed cleaning
combined_csv_leaseremainding.csv	lease calculation
combined_passed.csv	deduplicated rows
combined_failed.csv	discarded lower-price duplicates
combined_passed_with_anomaly_flags.csv	outlier detection
combined_passed_with_resale_id.csv	final output with resale identifiers

These are all generated locally and are not committed to git (see .gitignore).

Configuration

All filenames, API parameters, and thresholds live in hdb_pipeline/config.py (PipelineConfig). Edit that file, or construct a custom PipelineConfig(...) in your own script, to change behavior without touching pipeline logic.