refactor: Enhance email processing configuration and connection management

Author: taoi11

.appLogic/emailModule.md

@@ -1,91 +1,88 @@
 # Email Processing Module
 
 ## Overview
-Handles IMAP email retrieval and routing for two inboxes:
-1. `pacenotefoo@caf-gpt.com` - Routes to Pace Notes system
-2. `policyfoo@caf-gpt.com` - Routes to Policy Foo system
+Handles IMAP email retrieval and routing for two specific folders in ProtonMail:
+1. `CAF-GPT/PaceNoteFoo` - Routes to Pace Notes system
+2. `CAF-GPT/PolicyFoo` - Routes to Policy Foo system
 
 ## Architecture
 
 ### Data Flow
-0. Initialization
+1. Initialization
    - On startup: Connect to IMAP
-   - Reload all unread messages into queue
-   - Maintain original received order
+   - Select specific mailboxes for processing
+   - Load configuration from environment
+   - Setup logging based on development mode
 
-1. Queue Management
-   - Thread-safe in-memory storage using Python's deque
-   - Cold start protection via IMAP reload
-   - No persistent storage of queue state
-   - Lock-based concurrency control
+2. Connection Management
+   - Single IMAP connection with health monitoring
+   - Automatic reconnection with exponential backoff
+   - Connection status tracking
+   - Clean error handling
 
-2. Processing Flow
-   - Emails added to queue as received
-   - **Routing Logic**:
-     - Add `system` metadata key based on `To:` field:
-       - `pacenotefoo@caf-gpt.com` → `pace_notes`
-       - `policyfoo@caf-gpt.com` → `policy_foo`
-     - Unknown recipients are logged and skipped
-   - Async processing loop with health checks
-   - Success confirmation required before next item
-   - Mark email as read after processing
-   - Failed emails are retried with exponential backoff
+3. Email Processing
+   - Continuous processing loop
+   - Mailbox-specific routing (pace_notes/policy_foo)
+   - Mark-as-read confirmation
+   - Error handling with logging
 
-## Technical Implementation
+### Implementation Details
 
-### Environment Variables
-```
-EMAIL_HOST=127.0.0.1
+#### Configuration
+```python
+# Environment Variables
+EMAIL_HOST=100.99.136.75
 EMAIL_PASSWORD=****
 IMAP_PORT=1143
 SMTP_PORT=1025
-```
-
-### System Design
-- **Queue Characteristics**:
-  - Pure Python deque with maxlen=100
-  - Thread-safe operations
-  - Messages stored as EmailMessage objects
-  - Order preserved from IMAP UID sequence
 
-- **Connection Management**:
-  - Single IMAP connection with health monitoring
-  - Automatic reconnection with exponential backoff
-  - Connection status exposed via health check
-
-- **Health Monitoring**:
-  - Queue statistics (size, processing state)
-  - Connection health (status, retry count)
-  - Integrated with FastAPI health check endpoint
+# Hardcoded Mailboxes
+MAILBOXES = {
+    "pace_notes": "CAF-GPT/PaceNoteFoo",
+    "policy_foo": "CAF-GPT/PolicyFoo"
+}
+```
 
-### Message Parsing
-- **Headers Extracted**:
-  - `From`: Sender's email address
-  - `To`: Recipient address(es)
-  - `Subject`: Email subject line
-  - `Date`: Received timestamp
-- **Body Handling**:
-  - Only process `text/plain` content
-  - Ignore HTML and attachments
+### Mailbox Handling
+- **Folder Selection**: Explicitly select each mailbox before processing
+- **Folder Switching**: Switch between mailboxes during processing loop
+- **Folder Monitoring**: Track last processed message for each folder
+- **Error Handling**: Handle folder access errors gracefully
 
 ### Error Handling
-- **IMAP Connection Failures**:
-  - Exponential backoff (1s to 1 hour)
-  - Health status monitoring
-  - Automatic reconnection attempts
+- Connection failures with backoff
+- Folder access errors with logging
+- Graceful shutdown on interrupts
+- Development mode detailed logging
 
-- **Processing Errors**:
-  - Failed messages marked for retry
-  - Retry count tracking
-  - Error reason logging
+### Health Monitoring
+- **Connection status**:
+  - Last successful connection time
+  - Connection error count
+  - Current connection state
+- **Queue statistics**:
+  - Current queue size
+  - Messages processed
+  - Messages failed
+  - Average processing time
+- **System metrics**:
+  - CPU/memory usage
+  - Thread count
+  - Active connections
+- **Alerting**:
+  - Email processing failures
+  - Queue capacity warnings
+  - Connection errors
 
-- **Queue Management**:
-  - Full queue handling (drop new messages)
-  - Thread-safe operations
+### Queue Implementation
+- **Thread-safe in-memory storage** using Python's deque
+- **Max capacity**: 100 messages
+- **Message ordering**: Preserved from IMAP UID sequence
+- **Retry mechanism**:
+  - Failed messages are requeued
+  - Exponential backoff between retries
+  - Max retry attempts: 5
+- **Message tracking**:
+  - UID-based message identification
   - Processing state tracking
-
-### Integration
-- Async startup/shutdown methods
-- Health check status reporting
-- Background processing loop
-- Clean process lifecycle management
+  - Error history for failed messages

.appLogic/overview.md

@@ -1,7 +1,22 @@
 # CAF-GPT Application Plan
 
 ## Overview
-A collection of AI tools and agents for army personnel, packaged as a Python Docker container. Triggers on news emails via IMAP. Uses LLMs to formulate a response to the email.
+A collection of AI tools and agents for army personnel, packaged as a Python application. Processes emails via IMAP and routes them to appropriate systems.
+
+## Core Components
+
+### Email Processing
+1. **IMAPConnection**
+   - Manages IMAP server connection
+   - Handles authentication
+   - Retrieves unread messages
+   - Tracks connection health
+
+2. **EmailProcessor**
+   - Manages processing workflow
+   - Routes messages to appropriate systems
+   - Handles errors and retries
+   - Maintains processing queue
 
 ## Core Principles
 - Keep user messages browser-side only
@@ -41,4 +56,24 @@ A collection of AI tools and agents for army personnel, packaged as a Python Doc
   - Read-only access configuration
 
 ## Project Structure
-See [dirStructure.yaml](dirStructure.yaml) for details.
+See [dirStructure.yaml](dirStructure.yaml) for details.
+
+## Development vs Production Mode
+
+### Development Mode
+- **Enabled by**: DEVELOPMENT=true
+- **Features**:
+  - Detailed debug logging
+  - Automatic application restart on changes
+  - Reduced rate limiting
+  - Mock services for testing
+  - Verbose error messages
+
+### Production Mode
+- **Enabled by**: DEVELOPMENT=false
+- **Features**:
+  - Info-level logging only
+  - Strict rate limiting
+  - Real service connections
+  - Generic error messages
+  - Performance optimizations

.cursor/rules/base.mdc

@@ -6,14 +6,15 @@ globs: src/*
 - **Primary Goal**: Simplify and streamline app development while maintaining functionality.
 - **Core Philosophy**: Simplicity is key. Prioritize built-in libraries and frameworks first.
 - **User-Centric**: Always wait for user confirmation before making changes to the code base.
+- **Code-Docs**: Use single line comment for code level documentation.
 
 ## Best Practices
 1. **Clarity Over Complexity**:
   - Always explain reasoning behind decisions.
   - Use simple language and examples when possible.
 
 2. **Transparency**:
-  - Document all changes and thought processes in `notePad/`.
+  - Document all changes and thought processes in `notepad/`.
   - Maintain a clear audit trail of decisions.
 
 3. **User Control**:

.env.example

@@ -2,13 +2,14 @@
 PORT=8080
 DEVELOPMENT=true
 
-# LLM
-OLLAMA_HOST=http://localhost:11434  # Default Ollama port
-PACE_NOTE_MODEL=meta-llama/llama-3.3-70b-instruct
-DOAD_FINDER_MODEL=amazon/nova-pro-v1
-DOAD_CHAT_MODEL=anthropic/claude-3.5-sonnet:beta
+# Email
+EMAIL_HOST=100.99.136.75
+EMAIL_PASSWORD=****
+IMAP_PORT=1143
+SMTP_PORT=1025
+
 
 # S3
-S3_BUCKET_NAME=your-bucket-name
+S3_BUCKET_NAME=your-bucket
 S3_ACCESS_KEY=your-access-key
-S3_SECRET_KEY=your-secret-key
+S3_SECRET_KEY=your-secret-key

src/emails/connection.py

@@ -15,6 +15,7 @@ class IMAPConnection:
     def __init__(self):
         self.host = EMAIL_CONFIG['host']
         self.port = EMAIL_CONFIG['imap_port']
+        self.username = EMAIL_CONFIG['username']
         self.password = EMAIL_CONFIG['password']
         self.connection: Optional[imaplib.IMAP4] = None
         self.is_healthy = False
@@ -31,8 +32,11 @@ def connect(self) -> bool:
                 except:
                     self.connection = None
 
+            logger.debug(f"Attempting IMAP connection to {self.host}:{self.port}")
             self.connection = imaplib.IMAP4(self.host, self.port)
-            self.connection.login('user', self.password)
+            
+            logger.debug(f"Attempting login with username: {self.username}")
+            self.connection.login(self.username, self.password)
 
             self.is_healthy = True
             self.retry_count = 0
@@ -41,11 +45,11 @@ def connect(self) -> bool:
             logger.info("IMAP connection established")
             return True
 
+        except imaplib.IMAP4.error as e:
+            logger.error(f"IMAP login failed: {str(e)}")
+            logger.debug(f"Login details - Host: {self.host}:{self.port}, User: {self.username}")
+            return False
         except Exception as e:
-            self.is_healthy = False
-            self.retry_count += 1
-            self.last_error = str(e)
-            
             logger.error(f"IMAP connection failed: {str(e)}")
             return False
 
@@ -125,4 +129,10 @@ def get_health_check(self) -> dict:
             "is_healthy": self.is_healthy,
             "retry_count": self.retry_count,
             "last_error": self.last_error
-        } 
+        }
+
+    def is_connected(self) -> bool:
+        """Check if the IMAP connection is active and healthy."""
+        return self.connection is not None and self.is_healthy
+
+    # Add this method to the IMAPConnection class 

src/emails/processor.py

@@ -1,6 +1,7 @@
 import asyncio
 from typing import Optional, Callable
 from datetime import datetime
+import time
 
 from src.utils.logger import logger
 from src.types import EmailMessage, EmailHealthCheck
@@ -40,7 +41,14 @@ async def stop(self) -> None:
 
     async def _processing_loop(self) -> None:
         # Main processing loop.
+        retry_delay = 5  # seconds
         while self.running:
+            if not self.connection.is_connected():
+                logger.debug(f"Waiting {retry_delay}s before connection retry")
+                time.sleep(retry_delay)
+                retry_delay = min(retry_delay * 2, 300)  # Cap at 5 minutes
+                continue
+
             try:
                 # Check connection health
                 if not self.connection.is_healthy:

src/main.py

@@ -18,9 +18,6 @@
 )
 logger = logging.getLogger(__name__)
 
-# Initialize email processor
-email_processor = EmailProcessor()
-
 async def main():
     """Main application entry point."""
     try:
@@ -35,6 +32,11 @@ async def main():
     except KeyboardInterrupt:
         logger.info("Shutting down...")
         await email_processor.stop()
+    except Exception as e:
+        logger.error(f"Unexpected error: {str(e)}")
+        await email_processor.stop()
+        raise
 
 if __name__ == "__main__":
+    email_processor = EmailProcessor()
     asyncio.run(main())

src/utils/config.py

@@ -1,5 +1,6 @@
 import os
 from dotenv import load_dotenv
+from typing import Dict, Any
 
 # Load environment variables
 load_dotenv()
@@ -15,8 +16,11 @@
         raise ValueError(f"Missing required environment variable: {key}")
 
 # Export environment helpers
-IS_DEV = os.getenv('DEVELOPMENT', 'false').lower() == 'true'
-PORT = int(os.getenv('PORT', '8080'))
+def _get_env(key: str, default: Any = None) -> str:
+    value = os.getenv(key, default)
+    if value is None:
+        raise ValueError(f"Missing required environment variable: {key}")
+    return value
 
 # Export model configurations
 MODELS = {
@@ -27,18 +31,34 @@
     'paceNote': os.getenv('PACE_NOTE_MODEL')
 }
 
-# Email configuration
-EMAIL_INBOXES = {
-    'pace_notes': 'pacenotefoo@caf-gpt.com',
-    'policy_foo': 'policyfoo@caf-gpt.com'
+# Email Configuration
+EMAIL_CONFIG: Dict[str, Any] = {
+    # From environment
+    "host": _get_env("EMAIL_HOST"),
+    "password": _get_env("EMAIL_PASSWORD"),
+    "imap_port": int(_get_env("IMAP_PORT")),
+    "smtp_port": int(_get_env("SMTP_PORT")),
+    
+    # Hardcoded values
+    "username": "pacenotefoo@caf-gpt.com",  # Default inbox
+    
+    # System inboxes mapping
+    "inboxes": {
+        "pace_notes": "pacenotefoo@caf-gpt.com",
+        "policy_foo": "policyfoo@caf-gpt.com"
+    }
+}
+
+# Server Configuration
+SERVER_CONFIG = {
+    "port": int(_get_env("PORT", "8080")),
+    "development": _get_env("DEVELOPMENT", "false").lower() == "true"
 }
 
-# Add email configuration section
-EMAIL_CONFIG = {
-    'host': os.getenv('EMAIL_HOST', '127.0.0.1'),
-    'password': os.getenv('EMAIL_PASSWORD'),
-    'imap_port': int(os.getenv('IMAP_PORT', '1143')),
-    'smtp_port': int(os.getenv('SMTP_PORT', '1025')),
-    'inboxes': EMAIL_INBOXES
+# S3 Configuration
+S3_CONFIG = {
+    "bucket_name": _get_env("S3_BUCKET_NAME"),
+    "access_key": _get_env("S3_ACCESS_KEY"),
+    "secret_key": _get_env("S3_SECRET_KEY")
 }
 

src/utils/logger.py

@@ -4,8 +4,7 @@
 from enum import Enum
 from typing import Any, Dict, Optional
 from uuid import uuid4
-
-from .config import IS_DEV
+from .config import SERVER_CONFIG
 
 class LogLevel(Enum):
     DEBUG = 0
@@ -15,15 +14,27 @@ class LogLevel(Enum):
 
 class Logger:
     def __init__(self):
-        self.current_level = LogLevel.DEBUG if IS_DEV else LogLevel.INFO
+        self.current_level = LogLevel.DEBUG if SERVER_CONFIG['development'] else LogLevel.INFO
         self.llm_requests: Dict[str, float] = {}  # Track request start times
 
         # Configure logging
-        logging.basicConfig(
-            level=logging.DEBUG if IS_DEV else logging.INFO,
-            format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+        self.logger = logging.getLogger('caf-gpt')
+        self.logger.setLevel(logging.DEBUG if SERVER_CONFIG['development'] else logging.INFO)
+
+        # Create console handler
+        ch = logging.StreamHandler()
+        ch.setLevel(logging.DEBUG if SERVER_CONFIG['development'] else logging.INFO)
+
+        # Create formatter
+        formatter = logging.Formatter(
+            '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
         )
-        self.logger = logging.getLogger(__name__)
+
+        # Add formatter to ch
+        ch.setFormatter(formatter)
+
+        # Add ch to logger
+        self.logger.addHandler(ch)
 
     def _trim_system_message(self, content: str) -> str:
         max_length = 200
@@ -80,7 +91,7 @@ def _format_llm_response(self, data: Dict[str, Any], request_id: str, duration_m
         return json.dumps({**response, **extra_metadata}, indent=2)
 
     async def log_llm_interaction(self, data: Dict[str, Any]) -> None:
-        if not IS_DEV:
+        if not SERVER_CONFIG['development']:
             return
 
         request_id = data.get('metadata', {}).get('requestId')
@@ -123,7 +134,7 @@ def error(self, message: str, metadata: Optional[Dict[str, Any]] = None) -> None
         self.logger.error(self._format_message(message, metadata))
 
     def log_request(self, method: str, url: str, status_code: int, metadata: Optional[Dict[str, Any]] = None) -> None:
-        if not IS_DEV:
+        if not SERVER_CONFIG['development']:
             return
 
         path = url.split('?')[0]
@@ -144,7 +155,23 @@ def _format_message(self, message: str, metadata: Optional[Dict[str, Any]] = Non
         return f"{message} {json.dumps(metadata)}"
 
     def _should_log(self, level: LogLevel) -> bool:
-        return IS_DEV or level.value >= self.current_level.value
+        return SERVER_CONFIG['development'] or level.value >= self.current_level.value
 
 # Export singleton instance
 logger = Logger()
+
+def log_error(message: str, **kwargs: Any) -> None:
+    """Log error with additional context"""
+    logger.error(message, extra=kwargs)
+
+def log_warning(message: str, **kwargs: Any) -> None:
+    """Log warning with additional context"""
+    logger.warning(message, extra=kwargs)
+
+def log_info(message: str, **kwargs: Any) -> None:
+    """Log info with additional context"""
+    logger.info(message, extra=kwargs)
+
+def log_debug(message: str, **kwargs: Any) -> None:
+    """Log debug with additional context"""
+    logger.debug(message, extra=kwargs)