NodeCommerce Backend — Demand Intelligence Platform

Bangladesh-wide demand forecasting + agent signal aggregation across 64 districts and ~495 upazilas

---

What is this?

NodeCommerce forecasts what stock is needed where across Bangladesh — at every upazila × product level — using LightGBM on 94 features, then adjusts predictions with trust-weighted agent signals from local resellers. Output feeds a planned hierarchical fulfillment routing system (see routing plan).

Daily pipeline:
  Transactions → Feature engineering → LightGBM predict → SHAP decomposition
                                                              ↓
                                              Trust-weighted agent adjustment
                                                              ↓
                                              demand_forecasts table
                                                              ↓
                                (future) Hierarchical routing → delivery

---

Quick Start

Prerequisites

• Docker + Docker Compose
• Python 3.14+ with uv

1. Start services

docker compose up -d postgres redis

2. Seed dashboard data (fast, non-destructive)

make seed-dashboard

This seeds demand_forecasts (~30K rows), inventory_state (~10K rows), lgb_model, and pipeline_checkpoints from existing nodes/products. Runs in ~7 seconds.

3. Start the API

make run-api

4. Open the dashboard

• Map dashboard: http://localhost:8000 — full-screen Leaflet map with surplus/deficit circles, click for inventory + SHAP breakdown
• Old internal dashboard: http://localhost:8000/internal (v1 — being replaced)

---

Key Commands

make seed-data-small      # Full seed: ~1.9M transactions, all tables
make seed-dashboard       # Dashboard tables only (fast, non-destructive)
make run-api              # Start FastAPI server
make run-pipeline-job     # Run a single pipeline job (e.g. daily_features)
make test                 # Run tests

Seed flags (via uv run python -m pipeline.jobs.seed):

Flag	Effect
--drop	Truncate all tables and re-seed
--small	467 upazilas × 20 products × ~188 days
--dashboard	Dashboard tables only from existing data
--months N	Override date range to N months

---

Architecture (v3)

[Resellers] → [Agent] → [Trust-weighted adjustment] ───┐
                                                         ↓
[Transactions] → [Features] → [LightGBM] → [F_base] → [F_final]
                                                         ↓
                                                   demand_forecasts
                                                         ↓
                                              ┌────(future)────┐
                                              │ Hub routing     │
                                              │ Packaging       │
                                              │ Delivery        │
                                              └─────────────────┘

Layers

Layer	What it does	Status
1 — Raw Data	Node, Product, Transaction, Customer, InventoryState	✅ v3
2 — Forecasting	DemandForecast, FeatureStore (94 typed columns), RecsysScore, LgbModel, GraphRagCache	✅ v3
5 — Central	CentralFinding — spike detection, event classification	✅ v3
4 — Agentic	LocalSignal — agent signal store (partial v3 migration)	🟡 Partial
6 — ACO	FulfillmentQueue, InventoryReservation, etc.	⛔ v2 (not planned)

Pipeline jobs

Job	Schedule	What it does
daily_features	Daily	Impute censored demand → compute 94 features → write to feature_store
daily_inference	Daily	LightGBM predict + SHAP + trust-weighted agent adjustment → demand_forecasts
daily_central_agent	Daily	Spike detection, event classification → CentralFinding
daily_event_features	Daily	Compute holiday/weather/supply event features
hourly_graphrag	Hourly	Neo4j graph features → feature_store
weekly_recsys	Weekly	Product affinity scores → recsys_scores
weekly_train	Weekly	Retrain LightGBM → lgb_model
monitor	Daily	Coverage checks, pipeline status

API endpoints

Endpoint	Purpose
GET /api/v1/dashboard/map?product_id=X	Upazila-level surplus/deficit for map markers
GET /api/v1/dashboard/upazila/{id}/inventory	Per-product inventory for side panel
GET /api/v1/dashboard/upazila/{id}/explain?product_id=X	SHAP forecast decomposition (8 components)
GET /api/v1/dashboard/business-metrics	Aggregate ৳X at risk across all products
GET /api/v1/dashboard/products	Product list for dropdown
GET /forecast/{upazila_id}/{product_id}?explain=true	Full forecast history with SHAP (requires token)

Frontend

• Map dashboard (investor.html at /): Leaflet with CartoDB Dark tiles, upazila circles colored by surplus/deficit, click for side panel with inventory table + SHAP breakdown bar chart
• Upazila coordinates: Pre-processed from Wikidata (496 entries) at static/data/upazilas.json. Compound keyed by district|name to handle 9 duplicate names across districts
• Business metrics badge: Red badge in top bar showing total value at risk in BDT

---

Seed vs Real Pipeline

The seed generates realistic demo data so the dashboard works without running the full ML pipeline:

Aspect	Seed (--dashboard)	Real pipeline
SHAP deltas	Generated for ALL rows (synthetic proportions of F_P50)	shap.TreeExplainer but only first 200 rows
Demand	Synthetic log-normal noise	Real transactions → censoring imputation
Features	Not generated	daily_features → 94 typed columns
Model	Dummy (predicts 50.0)	LightGBM trained on actual data
Quantiles	Random uniform multipliers	Derived from product cv_error

---

Planned: Hierarchical Fulfillment Routing

v3 answers "what stock is needed where." The next layer answers "how to get it there."

Surplus upazilas → District Hub → Smart packaging → Deficit upazilas

9-layer implementation plan

#	Layer	What it adds
0	Foundation	Hub assignment, surplus collection, deficit matching, transfer order tracking
1	Route optimization	Vehicle routing (CVRP), distance matrix, daily delivery schedule
2	Consolidation & packaging	Product grouping (ambient/冷藏), crate packing, pallet building
3	Cross-district	Surplus from District A → Hub B when District A has excess
4	Return logistics	Empty crate return, reverse surplus flow, packaging lifecycle
5	Hub inventory policy	Buffer stock, reorder triggers, slow-mover flags
6	Dynamic reoptimization	Intra-day triggers (weather, agent surge), GPS ETA, rerouting
7	Learning	Route effectiveness, packaging efficiency, cost model calibration
8	Full autonomy	Self-tuning, predictive positioning, multi-modal, carbon accounting

See ARCHITECTURE.md for full detail.

Relationship to v2 ACO

The v2 packages/aco/ was point-to-point single-product routing between individual nodes — combinatorial explosion, no consolidation. This design is fundamentally different:

• Tree structure: upazila → district hub → upazila
• Multi-product: hub consolidates 20 products into one pallet per destination
• Not a starting point: this is a from-scratch design

---

Project Structure

├── packages/
│   ├── core/          ✅ DB models, config, engine
│   ├── api/           ✅ FastAPI app, routers, static frontend
│   ├── pipeline/      ✅ Pipeline jobs + Airflow DAGs
│   ├── forecasting/   ✅ Feature engineering, inference, training
│   ├── agents/        ✅ Local + central agents
│   ├── feature_store/ ✅ Calendar, graph, holiday features
│   ├── knowledge_graph/ ✅ Neo4j EKG, GraphRAG
│   ├── monitoring/    ✅ Drift, data quality, health checks
│   ├── aco/           ⛔ v2 — not planned for v3
│   └── router/        ⛔ v2 — not planned for v3
├── scripts/           ✅ Utilities (pipeline test runner, coord preprocessing)
├── ARCHITECTURE.md    ✅ Full v3 architecture + routing plan
└── Makefile           ✅ Common commands

Broken Airflow DAGs

Three DAGs have v2 column references and will fail if enabled:

DAG	Problem
dag_daily_cleanup.py	Imports layer6_execution (doesn't exist)
dag_daily_drift.py	Uses r.node_id, r.date, r.features (v3 FeatureStore has none of these)
dag_daily_data_quality.py	Renames upazila_id → node_id, monitoring expects upazila_id

---

Tech Stack

Component	Technology
API server	FastAPI (uvicorn)
Database	PostgreSQL 16 (postgis)
ML	LightGBM (CPU)
LLM	Groq API (llama-3.1-8b-instant) — zero calls at prediction time
Graph	Neo4j Community
Cache	Redis 7
Monitoring	Prometheus + Grafana
Orchestration	Cron + Redis workers (Airflow DAGs exist but not actively used)

---

Development

Environment

cp .env.example .env
# Edit DATABASE_URL and other settings

Install dependencies

uv sync

Run a pipeline job manually

uv run python -c "from pipeline.jobs.daily_features import main; main()"

Run the full pipeline test

make seed-data-small
uv run python scripts/test_full_pipeline.py

---

Project Status

Component	Status
v3 forecasting pipeline	✅ Complete
Map dashboard + SHAP explain	✅ Complete
Business metrics	✅ Complete
Seed system (--small, --dashboard)	✅ Complete
Agent signal pipeline	🟡 Needs v3 column migration in LocalSignal
Graph features wiring	🟡 graph_features.py exists but not called by pipeline
FAISS cold-start	🟡 Uses sklearn fallback
Hierarchical routing	🔲 Planned (see architecture doc)

---

License

MIT

---

_{ARCHITECTURE.md has the full detail — this README is the entry point.}