🏗️ [EPIC] Architectural Refactor: Modern CLI-First Design with Plugin System

[dannycab]

Overview

Comprehensive architectural refactor to improve modularity, user experience, security, and extensibility of WikiAccess.

Current Issues

• Multi-step manual workflow requires 5+ separate Python scripts
• Hard-coded configuration throughout codebase
• No plugin system for adding new wiki formats
• Limited error recovery and retry logic
• No concurrent processing for large batches
• Security vulnerabilities (rate limiting, path traversal, etc.)

Proposed Architecture

Hybrid approach combining:

1. CLI-First Design - Single unified command interface
2. Plugin System - Extensible architecture for sources/formats
3. Async Pipeline - High-performance concurrent processing
4. Configuration-Driven - YAML/JSON config files

Implementation Phases

• Phase 1: CLI-First refactor with config files (Phase 1: CLI-First Refactor with Configuration System #18)
• Phase 2: Plugin system for extensibility (Phase 2: Plugin System for Extensibility #19)
• Phase 3: Async pipeline for performance (Phase 3: Async Pipeline for Performance #20)
• Phase 4: API layer (optional, for web UI) (Phase 4: REST API and Web Dashboard (Optional) #21)

Related Issues

Security:

• 🔒 Security: Add Rate Limiting and Request Throttling #22 - Rate limiting and request throttling
• 🔒 Security: Path Traversal Risk in Image Downloads #23 - Path traversal risk in image downloads
• 🔒 Security: Secrets Management for Authentication #24 - Secrets management for authentication

Technical Debt:

• 🧹 Technical Debt: Split Monolithic database.py Module #25 - Split monolithic database.py module
• 🧹 Technical Debt: Add Dependency Version Locking #26 - Add dependency version locking

Features:

• ✨ Feature: Add Caching System for Images and Pages #27 - Caching system for images and pages
• ✨ Feature: Add URL Validation and Pre-flight Checks #28 - URL validation and pre-flight checks

Success Metrics

• Single command replaces 7+ scripts
• 10-100x performance improvement for large batches
• Easy to add new wiki formats (MediaWiki, Confluence, etc.)
• Configuration-driven (no hardcoded values)
• Comprehensive error handling and recovery

[dannycab]

Phase 1 Status: ✅ COMPLETE

Phase 1 (CLI-First Refactor) is complete and merged in v2.0.0.

Completed Phases:

• Phase 1: CLI-First refactor with config files (Phase 1: CLI-First Refactor with Configuration System #18)

In Progress:

• Phase 2: Plugin system for extensibility (Phase 2: Plugin System for Extensibility #19)

Planned:

• Phase 3: Async pipeline for performance (Phase 3: Async Pipeline for Performance #20)
• Phase 4: API layer (optional, for web UI) (Phase 4: REST API and Web Dashboard (Optional) #21)

Moving to Phase 2 implementation now.