diff --git a/ROADMAP.md b/ROADMAP.md index d5075ea..8912295 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -13,10 +13,12 @@ Order is approximate. Each item is sized so the surrounding system remains shipp - **Bulk recompute task.** Celery beat schedule that walks the catalog in shards, emits suggestions, applies the auto-approve set, queues the rest for review. - **Webhook ingest.** Inbound `products/update` webhook with HMAC verification (already implemented in `shopify/client.py:verify_webhook`) drives a recompute task for the affected SKU. - **Rate-limit-aware Shopify client.** Backoff on `X-Shopify-Shop-Api-Call-Limit`. Batch variant updates where possible. +- **Per-merchant automation suspend.** Emergency-stop flag on the merchant row. Suspended merchants skip the auto-apply path; the audit trail still records manual changes. Complements rollback โ rollback fixes the past, suspend prevents the next move. ## Later - **Pricing-signal layer.** Replace the toy `demand_signal - inventory_pressure` proposal with something that reads real signals. The guardrail layer is the safety surface โ the proposal is swappable. - **Per-merchant policy.** Today `MerchantPolicy` is constructed with module defaults. Move to a per-merchant config row. +- **Asymmetric guardrail caps.** Split `max_change_pct` into `max_increase_pct` and `max_decrease_pct`. Most merchants want prices to fall slower than they rise. - **A/B price testing.** Treatment / control groups, lift measurement against an audited baseline. - **Margin-based reporting.** Slice realized margin by SKU, by category, by guardrail-fired rate. diff --git a/ecom-dynamic-pricing/.env.example b/ecom-dynamic-pricing/.env.example deleted file mode 100644 index 11e3518..0000000 --- a/ecom-dynamic-pricing/.env.example +++ /dev/null @@ -1,28 +0,0 @@ -# FastAPI -APP_ENV=local -APP_DEBUG=true -APP_HOST=0.0.0.0 -APP_PORT=8000 - -# Database -POSTGRES_USER=ecom -POSTGRES_PASSWORD=ecom_password -POSTGRES_DB=ecom_db -POSTGRES_HOST=db -POSTGRES_PORT=5432 - -# Redis / Celery -REDIS_URL=redis://redis:6379/0 - -# Shopify -SHOPIFY_APP_API_KEY=your_shopify_public_key_here -SHOPIFY_APP_API_SECRET=your_shopify_api_secret_here -SHOPIFY_APP_SCOPES=read_products,write_products,read_orders -SHOPIFY_APP_REDIRECT_URI=https://your-ngrok-or-domain.com/api/shopify/oauth/callback - -# App -JWT_SECRET=super_secret_change_me -JWT_ALGORITHM=HS256 - -# Error Monitoring (optional - leave empty to disable) -SENTRY_DSN= diff --git a/ecom-dynamic-pricing/.gitignore b/ecom-dynamic-pricing/.gitignore deleted file mode 100644 index 2e0fe40..0000000 --- a/ecom-dynamic-pricing/.gitignore +++ /dev/null @@ -1,54 +0,0 @@ -# Environment variables (contains secrets) -.env - -# Python -__pycache__/ -*.py[cod] -*$py.class -*.so -.Python -env/ -venv/ -ENV/ -build/ -develop-eggs/ -dist/ -downloads/ -eggs/ -.eggs/ -lib/ -lib64/ -parts/ -sdist/ -var/ -wheels/ -*.egg-info/ -.installed.cfg -*.egg - -# IDEs -.vscode/ -.idea/ -*.swp -*.swo -*~ - -# OS -.DS_Store -Thumbs.db - -# Docker -docker-compose.override.yml - -# Postgres data -*.sql -pgdata/ - -# Logs -*.log -logs/ - -# Testing -.pytest_cache/ -.coverage -htmlcov/ diff --git a/ecom-dynamic-pricing/HANDOFF.md b/ecom-dynamic-pricing/HANDOFF.md deleted file mode 100644 index 02476f6..0000000 --- a/ecom-dynamic-pricing/HANDOFF.md +++ /dev/null @@ -1,566 +0,0 @@ -# System Handoff Document - -**Date**: 2025-01-13 -**Version**: 1.0.0 -**Status**: Production Ready โ - ---- - -## ๐ฏ Executive Summary - -This is a **complete, production-ready SaaS platform** for automated Shopify product pricing. It's been built with one goal in mind: - -> **"Make the system stable enough that someone else can run it without bothering you."** - -### โ What's Complete - -| Component | Status | Files | Lines of Code | -|-----------|--------|-------|---------------| -| Backend API | โ Complete | 15+ files | ~2,500 LOC | -| Frontend UI | โ Complete | 20 files | ~1,400 LOC | -| Database Schema | โ Complete | 3 models | ~150 LOC | -| Documentation | โ Complete | 4 docs | ~3,500 lines | -| Docker Setup | โ Complete | 2 files | ~100 LOC | -| **Total** | **โ Ready** | **50+ files** | **~7,650 LOC** | - -### ๐ก Key Achievement - -**Zero Hand-Holding Required** - -- Merchants understand the system in 5 minutes (MERCHANT_QUICKSTART.md) -- Operators can diagnose 95% of issues (OPERATOR_RUNBOOK.md) -- Developers can extend features easily (README.md + inline docs) - ---- - -## ๐ฆ What You're Getting - -### **1. Backend Infrastructure** - -``` -FastAPI + PostgreSQL + Redis + Celery + Docker -``` - -**Capabilities:** -- Shopify OAuth integration -- Product sync (background jobs) -- Dynamic pricing engine with guardrails -- Preview mode (test before applying) -- Rollback functionality (undo changes) -- Suspend/resume automation -- Rate limiting (Shopify API) -- Retry logic (3 attempts, exponential backoff) -- Timeout guardrails (no infinite loops) -- Health checks (Postgres, Redis, Celery) -- Structured logging -- Sentry integration (error monitoring) - -**Stability Features:** -- โ Tasks never hang (timeout limits) -- โ API never crashes (retry logic) -- โ Logs tell you exactly what happened -- โ Errors auto-forward to Sentry -- โ Health checks show system status in real-time - -### **2. Frontend Dashboards** - -``` -React 18 + TypeScript + Tailwind CSS + Vite -``` - -**Merchant Dashboard** (`/merchant`): -- Control panel (suspend/resume/preview/rollback) -- Guardrails editor (min/max prices, protection days) -- Pricing history (full audit trail) -- Real-time stats - -**Operator Panel** (`/operator`): -- System health monitor -- Worker status -- Recent activity across stores -- Quick actions -- Runbook reference - -### **3. Documentation** - -| Document | Purpose | Audience | -|----------|---------|----------| -| `README.md` | Complete system overview | Everyone | -| `MERCHANT_QUICKSTART.md` | 5-minute setup guide | Merchants | -| `OPERATOR_RUNBOOK.md` | Operations manual | Support team | -| `frontend/README.md` | Frontend development | Developers | -| `HANDOFF.md` | This document | New team lead | - ---- - -## โ ๏ธ Before You Start - -**Read [TECHNICAL_DEBT.md](./TECHNICAL_DEBT.md) end-to-end. Treat it as the primary backlog for hardening.** - -This document gives you complete transparency on all known gaps, workarounds, and the prioritized remediation plan. No surprises. - ---- - -## ๐ Quick Start for New Owner - -### **Day 1: Get It Running (30 minutes)** - -```bash -# 1. Clone repo -git clone https://github.com/MacFall7/E-Commerce.git -cd E-Commerce/ecom-dynamic-pricing - -# 2. Configure -cp .env.example .env -# Edit .env with Shopify credentials - -# 3. Start backend -docker-compose up -d --build - -# 4. Verify health -curl http://localhost:8000/api/health/detailed - -# 5. Start frontend (optional) -cd frontend && npm install && npm run dev - -# 6. Access dashboards -open http://localhost:3000 -``` - -### **Day 2: Understand Architecture (1 hour)** - -1. Read [`README.md`](./README.md) - Architecture section -2. Review [`backend/app/main.py`](./backend/app/main.py) - See how routing works -3. Review [`backend/app/pricing_engine.py`](./backend/app/pricing_engine.py) - Understand pricing logic -4. Review [`frontend/src/pages/merchant/MerchantDashboard.tsx`](./frontend/src/pages/merchant/MerchantDashboard.tsx) - See UI structure - -### **Day 3: Connect Test Store (30 minutes)** - -1. Create Shopify Partner account (free) -2. Create test app -3. Add credentials to `.env` -4. Install on test store -5. Watch products sync -6. Run preview mode -7. Apply pricing -8. Test rollback - -### **Week 1: Make Your First Change (2 hours)** - -**Suggested First Change**: Add email notifications - -1. Install `sendgrid` package -2. Add email config to `config.py` -3. Create `utils/email.py` with send function -4. Call from `tasks.py` after pricing cycle -5. Test locally -6. Deploy - ---- - -## ๐๏ธ Repository Structure - -``` -ecom-dynamic-pricing/ -โ -โโโ backend/ # FastAPI backend -โ โโโ app/ -โ โ โโโ main.py # Entry point -โ โ โโโ config.py # Settings -โ โ โโโ database.py # SQLAlchemy -โ โ โโโ models.py # DB models (Store, Product, PricingEvent) -โ โ โโโ schemas.py # Pydantic schemas -โ โ โโโ pricing_engine.py # Pricing logic โญ -โ โ โโโ shopify_client.py # Shopify API wrapper -โ โ โโโ celery_app.py # Celery config -โ โ โโโ tasks.py # Background tasks โญ -โ โ โโโ routes/ -โ โ โโโ shopify.py # OAuth + webhooks -โ โ โโโ operator.py # Operator controls โญ -โ โ โโโ health.py # Health checks -โ โโโ requirements.txt -โ โโโ Dockerfile -โ -โโโ frontend/ # React frontend -โ โโโ src/ -โ โ โโโ components/ -โ โ โ โโโ merchant/ # Merchant UI โญ -โ โ โ โโโ operator/ # Operator UI โญ -โ โ โ โโโ shared/ # Reusable components -โ โ โโโ lib/ -โ โ โ โโโ api.ts # API client โญ -โ โ โ โโโ types.ts # TypeScript types -โ โ โโโ pages/ -โ โ โ โโโ merchant/ -โ โ โ โโโ operator/ -โ โ โโโ App.tsx -โ โโโ package.json -โ โโโ vite.config.ts -โ -โโโ docker-compose.yml # Service orchestration โญ -โโโ .env.example # Environment template -โโโ README.md # Main documentation โญ -โโโ MERCHANT_QUICKSTART.md # Merchant guide -โโโ OPERATOR_RUNBOOK.md # Operator manual -โโโ HANDOFF.md # This document - -โญ = Critical files to understand -``` - ---- - -## ๐ Critical Concepts - -### **1. Pricing Flow** - -``` -Product Sync (Celery) โ Pricing Analysis (Engine) โ Preview (UI) โ Apply (API) โ Event Log (DB) - โ - Rollback (if needed) -``` - -### **2. Safety Mechanisms** - -| Mechanism | Location | Purpose | -|-----------|----------|---------| -| **Min/Max Limits** | `pricing_engine.py` | Never go too low/high | -| **Cost Floor** | `pricing_engine.py` | Protect margins (cost ร 1.1) | -| **New Product Protection** | `pricing_engine.py` | Don't touch new items | -| **Preview Mode** | `tasks.py` | Test before apply | -| **Rollback** | `routes/operator.py` | Undo mistakes | -| **Suspend Toggle** | `models.py` | Emergency stop | - -### **3. Data Models** - -**Store**: -```python -id, shop_domain, access_token, automation_suspended, -min_price_limit, max_price_limit, new_product_protection_days -``` - -**Product**: -```python -id, store_id, shopify_variant_id, title, current_price, -cost, inventory_quantity, first_seen_at -``` - -**PricingEvent**: -```python -id, store_id, shopify_variant_id, old_price, new_price, -reason, rolled_back, created_at -``` - -### **4. Background Tasks** - -| Task | Schedule | What It Does | -|------|----------|-------------| -| `sync_store_products` | Manual or periodic | Fetches products from Shopify, updates DB | -| `run_pricing_cycle` | Manual or periodic | Analyzes products, applies price changes | - ---- - -## ๐ ๏ธ Common Operations - -### **Add a New Store** - -```bash -# Get install URL -curl "http://localhost:8000/api/shopify/oauth/install?shop=new-store.myshopify.com" - -# Merchant visits URL and authorizes -# System auto-creates store record and syncs products -``` - -### **Change Pricing Logic** - -**File**: `backend/app/pricing_engine.py` - -```python -# Current logic (lines 70-81): -if sales_velocity > 5 and inventory > 20: - # increase price - new_price = min(price + delta, price * (1 + self.max_increase_pct)) - reason = "fast_seller_increase" - -# To modify: -# 1. Change thresholds (5, 20) -# 2. Change step_pct (default 0.05 = 5%) -# 3. Add new conditions (e.g., seasonality, competitor prices) -``` - -### **Suspend Automation for All Stores** - -```bash -# Loop through all stores -for store_id in $(seq 1 10); do - curl -X POST http://localhost:8000/api/operator/stores/$store_id/suspend -done -``` - -### **View Recent Errors** - -```bash -# Check logs -docker-compose logs --tail=100 backend | grep ERROR - -# Or use Sentry dashboard -open https://sentry.io/organizations/your-org/issues/ -``` - -### **Scale Workers** - -```bash -# Increase to 3 workers -docker-compose up -d --scale worker=3 - -# Verify -curl http://localhost:8000/api/health/detailed -# Should show 3 workers -``` - ---- - -## ๐ Monitoring Checklist - -### **Daily (2 minutes)** - -- [ ] Check system health: `curl http://localhost:8000/api/health/detailed` -- [ ] Review Sentry errors: Visit Sentry dashboard -- [ ] Check Celery queue depth: `docker-compose exec redis redis-cli LLEN celery` (should be < 100) - -### **Weekly (10 minutes)** - -- [ ] Review pricing events: Check for anomalies -- [ ] Check disk usage: `docker system df` -- [ ] Review merchant feedback -- [ ] Update dependencies (security patches) - -### **Monthly (30 minutes)** - -- [ ] Database vacuum: `docker-compose exec db psql -U ecom -d ecom_db -c "VACUUM ANALYZE;"` -- [ ] Backup database: `docker-compose exec db pg_dump -U ecom ecom_db > backup_$(date +%Y%m%d).sql` -- [ ] Review and optimize slow queries -- [ ] Update roadmap based on merchant requests - ---- - -## ๐จ Emergency Procedures - -### **System Down** - -```bash -# 1. Check container status -docker-compose ps - -# 2. Check logs -docker-compose logs --tail=100 - -# 3. Restart all services -docker-compose restart - -# 4. If still failing, rebuild -docker-compose down -docker-compose up -d --build - -# 5. Verify health -curl http://localhost:8000/api/health/detailed -``` - -### **Database Corrupted** - -```bash -# 1. Stop services -docker-compose down - -# 2. Restore from backup -cat backup_20250113.sql | docker-compose exec -T db psql -U ecom -d ecom_db - -# 3. Restart -docker-compose up -d - -# 4. Verify -curl http://localhost:8000/api/operator/stores/1/status -``` - -### **Merchant Reports Wrong Prices** - -```bash -# 1. Suspend automation -curl -X POST http://localhost:8000/api/operator/stores/{store_id}/suspend - -# 2. Check recent events -curl http://localhost:8000/api/operator/stores/{store_id}/pricing-history - -# 3. Rollback if needed -curl -X POST "http://localhost:8000/api/operator/stores/{store_id}/rollback?limit=10" - -# 4. Investigate logs -docker-compose logs worker | grep "store_id={store_id}" -``` - ---- - -## ๐ฎ Future Enhancements - -### **Short Term (1-2 weeks each)** - -- [ ] Add email notifications on pricing changes -- [ ] Scheduled pricing cycles (cron-like) -- [ ] Export pricing history to CSV -- [ ] Multi-store dashboard view -- [ ] Advanced filters on pricing history - -### **Medium Term (1-2 months each)** - -- [ ] Machine learning price optimization -- [ ] Competitor price monitoring -- [ ] A/B testing for pricing strategies -- [ ] Seasonal pricing rules -- [ ] Custom pricing rules per product tag - -### **Long Term (3-6 months each)** - -- [ ] Multi-channel support (Amazon, eBay) -- [ ] Mobile app (React Native) -- [ ] White-label solution for agencies -- [ ] Marketplace for pricing strategies -- [ ] Advanced analytics dashboard - ---- - -## ๐ฅ Team Recommendations - -### **Ideal Team Structure** - -| Role | Hours/Week | Responsibilities | -|------|------------|------------------| -| **DevOps Engineer** | 5-10 | Maintain infrastructure, monitor health, handle incidents | -| **Backend Developer** | 10-20 | Add features, optimize pricing engine, API enhancements | -| **Frontend Developer** | 5-10 | UI improvements, new dashboards, mobile app (future) | -| **Customer Success** | 10-20 | Merchant support, feedback collection, documentation updates | - -**Total**: 30-60 hours/week for active development + support - -### **Skills Needed** - -- **Must Have**: Python, Docker, PostgreSQL -- **Should Have**: React, TypeScript, FastAPI -- **Nice to Have**: Celery, Redis, Shopify API - ---- - -## ๐ Learning Resources - -### **For Backend** - -- [FastAPI Tutorial](https://fastapi.tiangolo.com/tutorial/) -- [Celery Documentation](https://docs.celeryq.dev/) -- [SQLAlchemy ORM](https://docs.sqlalchemy.org/en/20/orm/) - -### **For Frontend** - -- [React Documentation](https://react.dev/) -- [TypeScript Handbook](https://www.typescriptlang.org/docs/handbook/intro.html) -- [Tailwind CSS](https://tailwindcss.com/docs) - -### **For Shopify** - -- [Shopify API Reference](https://shopify.dev/api) -- [OAuth Flow](https://shopify.dev/apps/auth/oauth) - ---- - -## โ Handoff Checklist - -Before final handoff, ensure: - -- [ ] All code is committed and pushed -- [ ] `.env.example` is up to date -- [ ] Documentation is current (README, runbooks) -- [ ] Production environment is configured -- [ ] Sentry is set up and receiving errors -- [ ] Database backups are automated -- [ ] At least one test merchant is onboarded -- [ ] Support email is set up -- [ ] Monitoring alerts are configured -- [ ] New team has access to: - - [ ] GitHub repository - - [ ] Shopify Partner account - - [ ] Sentry account - - [ ] Production servers - - [ ] Database credentials - ---- - -## ๐ Knowledge Transfer Sessions - -### **Recommended Schedule** - -**Session 1 (1 hour)**: System Overview -- Architecture walkthrough -- Data flow explanation -- Key files review - -**Session 2 (1 hour)**: Backend Deep Dive -- Pricing engine logic -- Celery tasks -- Database models -- API endpoints - -**Session 3 (1 hour)**: Frontend Deep Dive -- Component structure -- API integration -- State management -- Routing - -**Session 4 (30 minutes)**: Operations -- Health monitoring -- Common issues -- Emergency procedures -- Deployment process - -**Session 5 (30 minutes)**: Next Steps -- Roadmap review -- Feature prioritization -- Questions & answers - ---- - -## ๐ Point of Contact - -**Original Developer**: [Your Name] -**Handoff Date**: 2025-01-13 -**Contact**: [Your Email] - -**Availability for Questions**: 30 days after handoff - -**Emergency Contact**: Same as above (for critical production issues only) - ---- - -## ๐ Final Words - -This system has been built with care to be: - -โ **Stable** - Retry logic, timeouts, health checks -โ **Safe** - Guardrails, preview mode, rollback -โ **Clear** - Documentation, logging, error messages -โ **Extensible** - Clean architecture, TypeScript types -โ **Handoff-Ready** - Everything needed to run independently - -**You're inheriting a production-ready system.** The hard work is done. Your job is to: -1. Keep it running smoothly -2. Add value through new features -3. Support merchants - -**Good luck! ๐** - ---- - -