Skip to content

ARCHITECTURE_V2.md

Latest commit

 

History

History
2065 lines (1747 loc) · 78.1 KB

ARCHITECTURE_V2.md

File metadata and controls

2065 lines (1747 loc) · 78.1 KB
Raw

RunAnywhere Swift SDK - Architecture Overview v2.0

Table of Contents

  1. Introduction
  2. Core Design Principles
  3. Architecture Overview
  4. Layer Architecture
  5. Capabilities System
  6. Core Infrastructure
  7. Voice System Architecture
  8. Data Flow
  9. Key Components
  10. Detailed Component Analysis
  11. Core Interaction Flows
  12. File-Level Details
  13. Edge Cases and Error Scenarios
  14. Implementation Status
  15. Developer Guide
  16. Extension Points

Introduction

The RunAnywhere Swift SDK is a sophisticated on-device AI platform that provides intelligent routing between on-device and cloud AI models. Built with a clean 5-layer architecture, the SDK emphasizes privacy-first design, cost optimization, and developer experience.

Current State (v2.0):

  • Files: 303 Swift files organized across 5 architectural layers
  • Capabilities: 20+ modular capability systems with 60+ models
  • Frameworks: Support for CoreML, TensorFlow Lite, GGUF, MLX, ONNX, ExecuTorch, PicoLLM, MLC, WhisperKit, and more
  • Voice Support: Complete voice pipeline with STT, TTS, VAD, and wake word detection
  • Platforms: iOS 13.0+, macOS 10.15+, tvOS 13.0+, watchOS 6.0+
  • Concurrency: Modern Swift async/await throughout with sophisticated error handling
  • Configuration: Device-only execution with comprehensive analytics and monitoring

Core Design Principles

1. Modular Architecture

  • Self-contained capabilities with clear boundaries
  • No circular dependencies between modules
  • Plugin-based extensibility through protocol-based design

2. Privacy-First Design

  • On-device execution as enforced default (cloud routing disabled)
  • Configurable privacy policies with strict mode
  • Zero data leakage by design - currently hardcoded to device-only execution
  • Comprehensive telemetry consent management

3. Developer Experience

  • Simple, intuitive public API with singleton pattern
  • Modern Swift concurrency patterns (async/await)
  • Comprehensive error handling with recovery suggestions
  • Type-safe structured output support through Generatable protocol

4. Performance Optimization

  • Intelligent hardware detection and optimization
  • Advanced memory management with pressure handling
  • Real-time performance monitoring and analytics
  • Cost tracking and optimization with detailed breakdowns

5. SOLID Principles

  • Single Responsibility per component
  • Open for extension through protocols and adapters
  • Interface segregation with focused protocols
  • Dependency inversion via ServiceContainer

Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│                         PUBLIC API LAYER                            │
│  RunAnywhereSDK • Configuration • GenerationOptions • ModelInfo     │
│  Structured Output • Error Types • Framework Availability           │
│  Voice API (transcribe, processVoiceQuery)                          │
└─────────────────────────────────────────────────────────────────────┘
                                    │
┌─────────────────────────────────────────────────────────────────────┐
│                        CAPABILITIES LAYER                           │
│  TextGeneration • ModelLoading • Routing • Memory • Downloading    │
│  Validation • Tokenization • Storage • Monitoring • ABTesting      │
│  ErrorRecovery • StructuredOutput • Progress • Registry • Profiling │
│  GenerationAnalytics • Benchmarking • Compatibility                │
│  DeviceCapability (Hardware Detection) • Voice (STT/TTS/VAD)        │
└─────────────────────────────────────────────────────────────────────┘
                                    │
                     ┌──────────────┴──────────────┐
                     ▼                             ▼
┌─────────────────────────────────────┐ ┌─────────────────────────────┐
│           CORE LAYER                │ │         DATA LAYER          │
│  Domain Models • Protocols          │ │  Repositories • Storage     │
│  Configuration Types                │ │  Network • DTOs • Entities  │
│  ModelInfo • LLMFramework          │ │  Database Implementation    │
│  LLMService • FrameworkAdapter      │ │                            │
└─────────────────────────────────────┘ └─────────────────────────────┘
                     │                             │
                     └──────────────┬──────────────┘
                                    │
┌─────────────────────────────────────────────────────────────────────┐
│                      INFRASTRUCTURE LAYER                           │
│  Platform-Specific Implementations • Framework Adapters             │
│  Service Lifecycle Management                                       │
└─────────────────────────────────────────────────────────────────────┘
                                    │
┌─────────────────────────────────────────────────────────────────────┐
│                        FOUNDATION LAYER                             │
│  Logging • Error Types • Utilities • Constants                      │
│  AsyncQueue • WeakCollection • Remote Logging • SDKConstants        │
│  Dependency Injection (ServiceContainer)                            │
└─────────────────────────────────────────────────────────────────────┘

Layer Architecture

1. Public API Layer

Purpose: Clean, user-facing interface that hides internal complexity

Key Components:

  • /Public/RunAnywhereSDK.swift - Main singleton entry point (841 lines)
  • /Public/RunAnywhereSDK+StructuredOutput.swift - Structured output extensions
  • /Public/RunAnywhereSDK+Voice.swift - Voice API extensions (transcribe, processVoiceQuery)
  • /Public/Configuration/ - SDK setup and policies (7 files)
  • /Public/Models/ - Public data models (8 files)
  • /Public/Models/Voice/ - Voice-specific models (TranscriptionResult, VoiceSession, etc.)
  • /Public/Errors/ - User-facing error types (2 files)
  • /Public/StructuredOutput/ - Type-safe structured generation (1 file)

Design Patterns:

  • Singleton pattern for SDK access (RunAnywhereSDK.shared)
  • Async/await for all operations
  • Builder pattern for configuration
  • Extension-based API organization

Key Features:

  • Model management (load/unload/list/download/delete)
  • Text generation (standard, streaming, structured)
  • Voice processing (transcription, voice queries, TTS integration)
  • Configuration management (dynamic settings)
  • Analytics and monitoring access
  • Storage management

2. Capabilities Layer

Purpose: Feature-specific business logic in self-contained modules

Core Capabilities:

Text Generation & Streaming

  • Path: /Capabilities/TextGeneration/
  • Files: 4 services including GenerationService.swift (296 lines)
  • Features: Context management, thinking content parsing, structured output
  • Dependencies: RoutingService, ContextManager, PerformanceMonitor

Model Management

  • ModelLoading: /Capabilities/ModelLoading/
    • ModelLoadingService.swift - Central coordinator
    • LoadedModel.swift - Model state representation
  • ModelValidation: /Capabilities/ModelValidation/
    • 15 files including validators for each format (GGUF, CoreML, ONNX, etc.)
    • Comprehensive metadata extraction and checksum validation
  • Registry: /Capabilities/Registry/
    • Model catalog and discovery with caching
  • Downloading: /Capabilities/Downloading/
    • Robust model acquisition with resume support using Alamofire

Resource Management

  • Memory: /Capabilities/Memory/ (13 files)
    • Advanced memory management with pressure handling
    • LRU eviction strategy and configurable thresholds
    • Platform-specific monitoring with thermal state awareness
  • Storage: /Capabilities/Storage/ (12 files)
    • File system management and cleanup
    • Storage analytics and recommendations
  • Routing: /Capabilities/Routing/
    • Intelligent execution path decisions (currently device-only)

Analytics & Testing

  • ABTesting: /Capabilities/ABTesting/ (23 files)
    • Statistical A/B testing framework with significance calculation
    • Performance comparison and metrics collection
  • Monitoring: /Capabilities/Monitoring/ (12 files)
    • Real-time performance monitoring with alert management
  • Analytics: /Capabilities/Analytics/ (structured by domain)
    • Generation: /Capabilities/Analytics/Generation/ - Text generation analytics
    • STT: /Capabilities/Analytics/STT/ - Speech-to-text analytics
    • Voice: /Capabilities/Analytics/Voice/ - Voice pipeline analytics
    • Monitoring: /Capabilities/Analytics/Monitoring/ - System monitoring analytics
    • Unified analytics architecture with centralized queue management
  • Benchmarking: /Capabilities/Benchmarking/ (18 files)
    • Comprehensive benchmarking suite with multiple export formats

Support Services

  • Tokenization: /Capabilities/Tokenization/ (6 files)
    • Multi-format tokenizer management with caching
  • ErrorRecovery: /Capabilities/ErrorRecovery/ (8 files)
    • Fault tolerance and recovery strategies
  • Progress: /Capabilities/Progress/ (6 files)
    • Multi-stage operation progress tracking
  • Profiling: /Capabilities/Profiling/ (15 files)
    • Memory profiling and leak detection

Device Capability

  • DeviceCapability: /Capabilities/DeviceCapability/
    • Hardware detection and monitoring
    • Services:
      • HardwareDetectionService - Central hardware capability detection
      • ThermalMonitorService - Thermal state monitoring with ThermalState enum
      • BatteryMonitorService - Battery state monitoring with BatteryState enum
      • Platform-specific detectors (Processor, Neural Engine, GPU)
    • Models:
      • DeviceCapabilities - Unified device capability information
      • BatteryInfo - Battery state and level
      • ThermalState - Thermal throttling states
    • Features:
      • Cross-platform hardware detection
      • Real-time thermal monitoring
      • Battery state tracking (iOS)
      • Memory pressure detection

Voice Processing (Integrated with Orchestrator)

  • Voice Protocols: /Core/Protocols/Voice/
    • Core STT/TTS service protocols with streaming support
    • Voice activity detection and wake word protocols
    • Performance monitoring for voice operations
    • NEW: VoiceOrchestrator for complete pipeline management
  • Voice Models: /Public/Models/Voice/
    • Transcription results and options
    • Voice session management
    • Audio streaming data structures
    • NEW: VoicePipelineEvent, VoicePipelineConfig, VoicePipelineResult
  • Voice Services: /Core/Services/Voice/
    • NEW: DefaultVoiceOrchestrator implementation
  • Voice Integration:
    • Unified framework adapter pattern for voice services
    • Complete STT → LLM → TTS pipeline orchestration
    • Real-time streaming events with progress tracking
    • Seamless integration with existing model loading pipeline
    • Full participation in memory and error management
    • Comprehensive analytics and performance tracking

3. Core Layer

Purpose: Shared domain models, protocols, and business logic

Key Components:

Models (/Core/Models/ - 20+ files)

  • Execution Models: ExecutionTarget, RoutingDecision, RoutingReason
  • Hardware Models: HardwareConfiguration, ProcessorInfo, ResourceAvailability
  • Model Information: ModelInfo, ModelFormat, LLMFramework
  • Request Models: InferenceRequest, RequestPriority
  • Configuration Models (/Core/Models/Configuration/):
    • ConfigurationData: Main configuration structure with composed sub-configurations
    • RoutingConfiguration: Routing behavior settings (using RoutingPolicy enum)
    • AnalyticsConfiguration: Analytics and telemetry settings (using AnalyticsLevel enum)
    • GenerationConfiguration: Text generation settings with DefaultGenerationSettings
    • StorageConfiguration: Storage behavior settings with CacheEvictionPolicy enum
    • AnalyticsLevel: Enum for analytics collection levels
    • CacheEvictionPolicy: Enum for cache eviction strategies

Protocols (/Core/Protocols/ - 20+ protocols)

  • Service Interfaces: LLMService, FrameworkAdapter, ModelRegistry
  • Hardware Detection: HardwareDetector
  • Lifecycle Management: ModelLifecycleProtocol
  • Authentication: AuthProvider
  • Memory Management: MemoryManager
  • Storage: ModelStorageManager
  • Tokenization: UnifiedTokenizerProtocol

Compatibility (/Core/Compatibility/ - 6 files)

  • Device capability detection
  • Framework compatibility checking
  • Architecture support validation

4. Data Layer

Purpose: Centralized data persistence and network operations

Key Components:

Protocols (/Data/Protocols/)

  • Repository: Base repository protocol for data operations
  • ConfigurationRepository: Configuration-specific persistence
  • ModelMetadataRepository: Model metadata storage
  • TelemetryRepository: Telemetry data storage
  • GenerationAnalyticsRepository: Analytics data storage

Storage (/Data/Storage/)

  • Database: SQLite implementation for local persistence
  • Database protocols and core operations
  • Transaction support and query builders

Network (/Data/Network/)

  • APIClient: REST API client for cloud operations
  • DataSyncService: Data synchronization service
  • APIEndpoint: Endpoint definitions

Models (/Data/Models/)

  • Entities: Database entities (ConfigurationData, ModelMetadataData, TelemetryData)
  • DTOs: Network transfer objects with strong typing:
    • ConfigurationDTO
    • TelemetryDTO (with TelemetryEventType enum)
    • ModelMetadataDTO (using LLMFramework and ModelFormat enums)

Repositories (/Data/Repositories/)

  • Concrete implementations of repository protocols
  • Database operations for each entity type

5. Infrastructure Layer

Purpose: Platform-specific implementations

Key Components:

  • Framework adapters for ML frameworks
  • Platform-specific service implementations

6. Foundation Layer

Purpose: Cross-cutting utilities and platform extensions

Key Components:

Dependency Injection (/Foundation/DependencyInjection/ - 3 files)

  • ServiceContainer: Central service registry with 25+ services (503 lines)
  • ServiceFactory: Type-safe service creation
  • ServiceLifecycle: Service startup/shutdown management

Logging System (/Foundation/Logging/ - 8 files)

  • Multi-level logging (debug, info, warning, error, fault)
  • Local and remote logging capabilities with batch submission
  • Privacy-aware device metadata
  • Configurable log formatters

Utilities (/Foundation/Utilities/ - 2 files)

  • AsyncQueue: Thread-safe sequential task execution
  • WeakCollection: Memory-safe object collections

Constants (/Foundation/Constants/ - 2 files)

  • SDKConstants: SDK configuration defaults using enums (RoutingPolicy, AnalyticsLevel)
  • ErrorCodes: Standardized error categorization

Capabilities System

Design Pattern

Each capability follows a consistent structure:

Capabilities/{CapabilityName}/
├── Protocols/       # Interfaces and contracts
├── Services/        # Main business logic
├── Models/          # Data structures
├── Strategies/      # Algorithm implementations (optional)
├── Tracking/        # Analytics and metrics (optional)
├── Analysis/        # Advanced analysis (optional)
├── Operations/      # Specific operations (optional)
└── Extensions/      # Integration helpers (optional)

Key Capabilities Deep Dive

1. TextGeneration

Services:

  • GenerationService: Main generation orchestrator (296 lines)
  • ContextManager: Conversation context management
  • ThinkingParser: Extracts reasoning from model outputs (DeepSeek-style)
  • StreamingService: Real-time text streaming

Features:

  • Structured output support with schema validation
  • Thinking/reasoning extraction with configurable patterns
  • Context trimming and management
  • Performance tracking with detailed metrics
  • Error recovery with timeout handling

2. ModelLoading

Services:

  • ModelLoadingService: Central loading coordinator
  • Multi-framework adapter support
  • Memory registration and tracking

Flow:

  1. Registry lookup for model information
  2. Validation pipeline execution
  3. Download if model not present locally
  4. Memory allocation check
  5. Framework adapter selection
  6. Model loading and service creation

3. Memory Management

Services:

  • MemoryService: Central memory coordinator (262 lines)
  • MemoryMonitor: Real-time usage tracking
  • PressureHandler: Memory pressure response
  • AllocationManager: Model memory allocation
  • CacheEviction: LRU eviction strategy

Features:

  • Configurable thresholds (warning: 500MB, critical: 200MB)
  • Platform-specific monitoring with thermal state
  • Automatic model eviction under pressure
  • Memory statistics and health monitoring

4. Storage Management

Services:

  • SimplifiedFileManager: File system operations
  • Framework-specific organization
  • Automatic cleanup capabilities

Directory Structure:

RunAnywhere/
├── Models/
│   ├── CoreML/
│   ├── TensorFlowLite/
│   ├── GGUF/
│   └── [other frameworks]/
├── Cache/
└── Temp/

5. Analytics & Monitoring

GenerationAnalytics (10 files):

  • Session-based tracking with UUIDs
  • Live metrics streaming
  • Performance aggregation
  • Database integration (SQLite)

Monitoring (12 files):

  • Real-time performance tracking
  • Alert management with thresholds
  • System metrics collection
  • Report generation in multiple formats

Voice System Architecture

Overview

The RunAnywhere SDK implements a comprehensive voice processing system that seamlessly integrates speech-to-text (STT), text-to-speech (TTS), and voice activity detection (VAD) capabilities. The voice system follows the SDK's modular architecture using unified framework adapters and protocol-based design.

Voice Pipeline Flow

Audio Input → VAD → STT (WhisperKit) → LLM Processing → TTS → Audio Output
     ↓              ↓                        ↓               ↓
AudioCapture → SimpleVAD → WhisperKitService → RunAnywhereSDK → SystemTTS

Voice Components

SDK Core Voice Components

Protocols (/Core/Protocols/Voice/):

  • VoiceService - Core STT protocol with streaming support
  • TextToSpeechService - Comprehensive TTS protocol with voice management
  • VoiceActivityDetector - VAD protocol for speech detection
  • WakeWordDetector - Wake word detection capabilities
  • VoicePerformanceMonitor - Voice-specific performance tracking

Models (/Public/Models/Voice/):

  • VoiceSession - Session management with state transitions
  • TranscriptionResult - STT output structure with confidence scores
  • TranscriptionOptions - STT configuration (language, task, etc.)
  • TranscriptionSegment - Streaming transcription chunks
  • AudioChunk - Audio data container for streaming

Public API (/Public/Extensions/):

  • RunAnywhereSDK+Voice - Voice API extensions
    • transcribe(audio:modelId:) - Single audio transcription
    • processVoiceQuery(audio:voiceModelId:llmModelId:) - End-to-end voice processing

Framework Modality System

Voice is integrated through the unified framework modality system:

public enum FrameworkModality: String, CaseIterable {
    case textToText = "text-to-text"
    case voiceToText = "voice-to-text"      // STT
    case textToVoice = "text-to-voice"      // TTS
    case imageToText = "image-to-text"
    case textToImage = "text-to-image"
    case multimodal = "multimodal"
}

Voice Adapter Pattern

Voice frameworks use the same unified adapter pattern as LLM frameworks:

public protocol UnifiedFrameworkAdapter {
    var supportedModalities: Set<FrameworkModality> { get }
    func createService(for modality: FrameworkModality) -> Any?
}

// Example: WhisperKit supports both STT and TTS
class WhisperKitAdapter: UnifiedFrameworkAdapter {
    let supportedModalities: Set<FrameworkModality> = [.voiceToText, .textToVoice]

    func createService(for modality: FrameworkModality) -> Any? {
        switch modality {
        case .voiceToText: return WhisperKitService()
        case .textToVoice: return SystemTTSServiceWrapper()
        default: return nil
        }
    }
}

Voice Session Management

The SDK provides comprehensive voice session management:

public enum VoiceSessionState {
    case idle        // Session created but not started
    case listening   // Actively recording audio
    case processing  // Transcribing or processing
    case speaking    // TTS playback
    case ended       // Session completed
}

public class VoiceSession {
    let id: String
    let configuration: VoiceSessionConfig
    var state: VoiceSessionState
    var transcripts: [TranscriptionResult]
    var duration: TimeInterval
}

Voice Processing Flow

Complete Voice Query Processing

// High-level voice query processing in SDK
func processVoiceQuery(audio: Data, voiceModelId: String, llmModelId: String?) async throws -> VoiceResponse {
    // 1. Service Discovery
    let voiceService = findVoiceService(for: voiceModelId)

    // 2. Transcription (STT)
    try await voiceService.initialize(modelPath: voiceModelId)
    let transcription = try await voiceService.transcribe(audio: audio, options: TranscriptionOptions())

    // 3. LLM Processing
    let textResponse = try await generate(prompt: transcription.text, options: GenerationOptions())

    // 4. Return structured response
    return VoiceResponse(
        inputText: transcription.text,
        outputText: textResponse.text
    )
}

Streaming Support

The voice system includes comprehensive streaming support for real-time processing:

// Streaming transcription
func transcribeStream(
    audioStream: AsyncStream<AudioChunk>,
    options: TranscriptionOptions
) -> AsyncThrowingStream<TranscriptionSegment, Error>

// Streaming VAD
func detectActivityStream(
    audioStream: AsyncStream<AudioChunk>
) -> AsyncStream<VADSegment>

// Streaming TTS
func speakStream(
    textStream: AsyncStream<String>,
    voice: TTSVoice?
) -> AsyncThrowingStream<AudioData, Error>

Voice Performance Monitoring

Voice operations are tracked with specialized metrics:

public protocol VoicePerformanceMonitor {
    func trackTranscription(duration: TimeInterval, audioLength: TimeInterval, model: String)
    func trackTTS(duration: TimeInterval, textLength: Int, voice: String)
    func trackVAD(processingTime: TimeInterval, audioLength: TimeInterval)
    func getMetrics() -> VoicePerformanceMetrics
}

public struct VoicePerformanceMetrics {
    let averageTranscriptionRTF: Float    // Real-time factor (< 1.0 is good)
    let averageTTSLatency: TimeInterval
    let averageVADLatency: TimeInterval
    let totalTranscriptions: Int
}

Voice Orchestrator (NEW)

The SDK now includes a comprehensive Voice Orchestrator system that manages the complete voice pipeline with streaming events and real-time feedback.

Orchestrator Components

Core Protocol (/Core/Protocols/Voice/VoiceOrchestrator.swift):

public protocol VoiceOrchestrator {
    func processVoicePipeline(audio: Data, config: VoicePipelineConfig)
        -> AsyncThrowingStream<VoicePipelineEvent, Error>
    func processVoiceQuery(audio: Data, config: VoicePipelineConfig)
        async throws -> VoicePipelineResult
}

Pipeline Events (/Public/Models/Voice/VoicePipelineEvent.swift):

  • Real-time pipeline event notifications
  • Progress tracking for each stage
  • Stage-specific error reporting

Pipeline Configuration (/Public/Models/Voice/VoicePipelineConfig.swift):

  • Configurable timeouts per stage (STT: 30s, LLM: 60s, TTS: 30s)
  • Generation options integration
  • Streaming enable/disable control

Pipeline Event Flow

1. started(sessionId) → Pipeline initialization
2. transcriptionStarted → transcriptionProgress → transcriptionCompleted
3. llmGenerationStarted → llmGenerationProgress → llmGenerationCompleted
4. ttsStarted → ttsProgress → ttsCompleted (if enabled)
5. completed(result) → Final result with timing metrics

Key Features

  • Unified Pipeline Management: Single orchestrator handles STT → LLM → TTS
  • Streaming Events: Real-time updates for UI responsiveness
  • Stage Timing: Performance metrics for each pipeline stage
  • Error Recovery: Stage-specific error handling and timeouts
  • Flexible Configuration: Per-stage timeout and option customization

Integration with Existing Capabilities

Voice seamlessly integrates with existing SDK capabilities:

  • Model Loading: Voice models use the same loading pipeline
  • Memory Management: Voice services participate in memory pressure handling
  • Error Recovery: Voice errors use the same recovery strategies
  • Analytics: Voice operations tracked alongside text generation
  • Configuration: Voice settings managed through unified configuration
  • Service Container: Voice orchestrator registered in dependency injection container

Sample App Implementation

The sample iOS app demonstrates complete voice integration:

Audio Services (/Core/Services/Audio/):

  • AudioCapture - AVAudioEngine-based recording at 16kHz mono
  • SimpleVAD - Energy-based voice activity detection
  • AudioProcessor - Audio preprocessing utilities

Voice Framework (/Core/Services/WhisperKit/):

  • WhisperKitAdapter - UnifiedFrameworkAdapter implementation
  • WhisperKitService - VoiceService with dual-mode (actual/simulated)

TTS Services (/Core/Services/TTS/):

  • SystemTTSService - AVSpeechSynthesizer implementation
  • SystemTTSServiceWrapper - Protocol adapter for TTS

UI Components (/Features/Voice/):

  • VoiceAssistantView - SwiftUI interface with microphone controls
  • VoiceAssistantViewModel - Business logic coordinator

Core Infrastructure

ServiceContainer Architecture

The ServiceContainer implements sophisticated dependency injection with:

// Service Registration Pattern
private(set) lazy var serviceName: ServiceType = {
    ServiceImplementation(dependencies...)
}()

Registered Services (25+ services):

  • Core Services: ConfigurationValidator, ModelRegistry, FrameworkAdapterRegistry
  • Capability Services: ModelLoadingService, GenerationService, StreamingService, ContextManager, ValidationService, DownloadService, ProgressTracker, FileManager, RoutingService, PerformanceMonitor, BenchmarkRunner, ABTestRunner
  • Infrastructure Services: HardwareManager, MemoryService, ErrorRecoveryService, CompatibilityService, TokenizerService, ConfigurationService
  • Analytics Services: GenerationAnalytics, DataSyncService (optional)

Health Monitoring:

  • Periodic health checks every 30 seconds
  • Service-specific health validators
  • Unhealthy service detection and logging

Bootstrap Process:

  1. Configuration validation
  2. Database initialization (currently disabled due to JSON corruption)
  3. Service registration and configuration
  4. Health monitoring startup

Hardware Detection System

Components:

  • ProcessorDetector: Apple Silicon vs Intel detection
  • NeuralEngineDetector: A12+/M1+ chip detection with performance tiers
  • GPUDetector: Metal support and memory configuration
  • CapabilityAnalyzer: Unified capability assessment

Detection Results:

struct DeviceCapabilities {
    let processorInfo: ProcessorInfo
    let totalMemory: Int64
    let availableMemory: Int64
    let hasNeuralEngine: Bool
    let hasGPU: Bool
    let supportedAccelerators: [HardwareAcceleration]
    let memoryPressureLevel: MemoryPressureLevel
}

Data Flow

1. SDK Initialization Flow

User → SDK.initialize(config)
  ↓
  ConfigurationValidator.validate()
  ↓
  ServiceContainer.bootstrap(config)
    ├── Database initialization (disabled)
    ├── ConfigurationService setup (in-memory)
    ├── API client creation (if API key provided)
    ├── ModelRegistry initialization
    ├── Hardware capability detection
    ├── Memory threshold configuration
    └── Performance monitoring startup
  ↓
  Health monitoring start
  ↓
  Success/Error response

2. Model Loading Flow

User → SDK.loadModel(identifier)
  ↓
  ModelRegistry.lookup(identifier)
  ↓
  ValidationService.validate(model)
    ├── Format detection
    ├── Metadata extraction
    ├── Checksum verification
    └── Dependency checking
  ↓
  Download if needed
    ├── DownloadService.downloadModel()
    ├── Progress tracking
    └── Storage management
  ↓
  Memory allocation check
    ├── MemoryService.canAllocate()
    ├── Pressure evaluation
    └── Eviction if needed
  ↓
  Framework adapter selection
    ├── FrameworkAdapterRegistry.findBestAdapter()
    └── Compatibility verification
  ↓
  Model loading
    ├── LLMService.initialize()
    ├── Memory registration
    └── Context preparation
  ↓
  LoadedModel creation and registration

3. Text Generation Flow

User → SDK.generate(prompt, options)
  ↓
  Configuration merge (user options + defaults)
  ↓
  Structured output preparation (if configured)
  ↓
  ContextManager.prepareContext()
  ↓
  RoutingService.determineRouting()
    └── Currently returns .onDevice (cloud disabled)
  ↓
  GenerationService.generateOnDevice()
    ├── LoadedModel validation
    ├── Context setting
    ├── LLMService.generate() with error handling
    ├── Thinking content parsing (if supported)
    ├── Performance metrics calculation
    └── Memory usage tracking
  ↓
  Structured output validation (if configured)
  ↓
  Analytics recording (if enabled)
  ↓
  GenerationResult with metrics

Key Components

Public API Design

Main SDK Interface (RunAnywhereSDK.swift - 841 lines + Voice Extensions):

RunAnywhereSDK.shared
├── Initialization
│   └── initialize(configuration:) async throws
├── Model Management
│   ├── loadModel(_:) async throws -> ModelInfo
│   ├── unloadModel() async throws
│   ├── listAvailableModels() async throws -> [ModelInfo]
│   ├── downloadModel(_:) async throws -> DownloadTask
│   ├── deleteModel(_:) async throws
│   └── addModelFromURL(name:url:framework:) -> ModelInfo
├── Generation
│   ├── generate(prompt:options:) async throws -> GenerationResult
│   ├── generateStream(prompt:options:) -> AsyncThrowingStream<String, Error>
│   └── generateStructured(_:prompt:options:) async throws -> T
├── Voice Processing (Enhanced with Orchestrator)
│   ├── transcribe(audio:modelId:) async throws -> VoiceTranscriptionResult
│   ├── processVoiceQuery(audio:voiceModelId:llmModelId:ttsEnabled:) async throws -> VoicePipelineResult
│   ├── processVoiceStream(audio:config:) -> AsyncThrowingStream<VoicePipelineEvent, Error>
│   └── transcribeStream(audioStream:options:) -> AsyncThrowingStream<TranscriptionSegment, Error>
├── Configuration Management
│   ├── setTemperature(_:) async
│   ├── setMaxTokens(_:) async
│   ├── setTopP(_:) async
│   ├── getGenerationSettings() async -> DefaultGenerationSettings
│   └── resetGenerationSettings() async
├── Framework Management
│   ├── registerFrameworkAdapter(_:)
│   ├── getRegisteredAdapters() -> [LLMFramework: FrameworkAdapter]
│   └── getFrameworkAvailability() -> [FrameworkAvailability]
├── Analytics Access
│   ├── getAnalyticsSession(_:) async -> GenerationSession?
│   ├── getAllAnalyticsSessions() async -> [GenerationSession]
│   └── observeLiveMetrics(for:) -> AsyncStream<LiveGenerationMetrics>
└── Storage Management
    ├── getStorageInfo() async -> StorageInfo
    ├── getStoredModels() async -> [StoredModel]
    └── clearCache() async throws

Error Handling System

Comprehensive Error Types:

  1. RunAnywhereError (174 lines) - Primary user-facing errors:

    • Initialization errors (notInitialized, invalidConfiguration)
    • Model errors (modelNotFound, modelLoadFailed, modelValidationFailed)
    • Generation errors (generationFailed, generationTimeout, contextTooLong)
    • Hardware errors (hardwareUnsupported, memoryPressure, thermalStateExceeded)
    • Each error includes recovery suggestions

  2. SDKError - Internal SDK errors

  3. UnifiedModelError - Model operation errors

  4. LLMServiceError - Service-level errors with framework context

  5. FrameworkError - Framework-specific errors with context

Error Recovery System:

  • Automatic retry strategies
  • Fallback framework switching
  • Memory pressure handling
  • Timeout management with helpful messages

Configuration System

SDKConfiguration (Configuration struct):

struct Configuration {
    let apiKey: String
    var baseURL: URL
    var enableRealTimeDashboard: Bool
    var routingPolicy: RoutingPolicy  // Forced to .deviceOnly
    var telemetryConsent: TelemetryConsent
    var privacyMode: PrivacyMode
    var preferredFrameworks: [LLMFramework]
    var memoryThreshold: Int64
    var downloadConfiguration: DownloadConfig
    var defaultGenerationSettings: DefaultGenerationSettings
}

Dynamic Configuration:

  • Runtime setting updates (temperature, tokens, etc.)
  • Persistent storage (when database enabled)
  • Cloud synchronization capabilities
  • User preference management

Structured Output System

Type-Safe Generation:

// Protocol for generatable types
protocol Generatable: Codable {
    static var generationSchema: String { get }
    static var generationInstructions: String { get }
}

// Usage example
let result = try await SDK.shared.generateStructured(
    PersonInfo.self,
    prompt: "Extract person information from: John Doe, 30 years old",
    validationMode: .strict
)

Features:

  • Schema-based validation
  • JSON schema generation
  • Validation modes (strict, lenient, none)
  • Custom instruction generation

Detailed Component Analysis

1. Memory Management Deep Dive

Architecture:

MemoryService (Central Coordinator)
├── AllocationManager (Model Memory Tracking)
├── PressureHandler (Memory Pressure Response)
├── CacheEviction (LRU Strategy)
├── MemoryMonitor (Real-time Monitoring)
└── ThresholdWatcher (Threshold Management)

Key Features:

  • Allocation Tracking: Precise memory tracking per loaded model
  • Pressure Handling: Multi-level pressure response (low/medium/high/warning/critical)
  • Eviction Strategy: LRU-based model unloading with priority consideration
  • Platform Integration: iOS memory warnings, macOS thermal throttling
  • Health Monitoring: Continuous memory health assessment

Configuration Options:

struct Config {
    var memoryThreshold: Int64 = 500_000_000 // 500MB
    var criticalThreshold: Int64 = 200_000_000 // 200MB
    var monitoringInterval: TimeInterval = 5.0
    var unloadStrategy: UnloadStrategy = .leastRecentlyUsed
}

2. Model Validation System

Validation Pipeline:

ValidationService
├── FormatDetector (GGUF, CoreML, ONNX, TFLite, MLX detection)
├── MetadataExtractor (Model metadata parsing)
├── ChecksumValidator (File integrity verification)
├── DependencyChecker (Framework requirements)
└── Format-Specific Validators
    ├── GGUFValidator
    ├── CoreMLValidator
    ├── ONNXValidator
    ├── TFLiteValidator
    └── MLXValidator

Validation Results:

struct ValidationResult {
    let isValid: Bool
    let errors: [ValidationError]
    let warnings: [ValidationWarning]
    let metadata: ModelMetadata?
    let requirements: ModelRequirements
    let missingDependencies: [MissingDependency]
}

3. Download Management

Download System (AlamofireDownloadService):

  • Features: Resume support, progress tracking, concurrent downloads
  • Error Handling: Network failures, storage issues, corruption detection
  • Progress Reporting: Real-time download progress with bandwidth monitoring
  • Validation: Post-download checksum verification

Download Flow:

  1. URL validation and reachability check
  2. Storage space verification
  3. Resume capability detection
  4. Download initiation with progress tracking
  5. Real-time progress updates
  6. Checksum verification
  7. File organization and metadata storage

4. Analytics & Telemetry

GenerationAnalytics Architecture:

GenerationAnalyticsService
├── GenerationAnalyticsRepository (Database operations)
├── TelemetryRepository (Remote sync)
├── PerformanceTracker (Real-time metrics)
└── LiveMetricsStreaming (Real-time updates)

Data Models:

  • Generation: Individual generation record with full metrics
  • GenerationSession: Grouped generations with session metadata
  • GenerationPerformance: Performance-specific metrics
  • LiveGenerationMetrics: Real-time streaming metrics

Capabilities:

  • Session-based tracking with UUID identification
  • Live metrics streaming for real-time monitoring
  • Performance aggregation and analytics
  • Export capabilities in multiple formats

5. A/B Testing Framework

Statistical Engine:

ABTestService
├── TestLifecycleManager (Test state management)
├── VariantManager (A/B variant handling)
├── MetricsCollector (Performance metrics)
├── ResultAnalyzer (Statistical analysis)
├── SignificanceCalculator (Statistical significance)
└── WinnerDeterminer (Test conclusion)

Features:

  • Statistical significance calculation
  • Multiple variant support
  • Performance comparison metrics
  • Automated winner determination
  • Detailed result reporting

Core Interaction Flows

1. SDK Initialization Flow (Detailed)

┌─ User calls SDK.initialize(config) ─┐
│                                     │
├─ STEP 1: Configuration Validation   │
│  ├─ ConfigurationValidator.validate()
│  ├─ API key validation              │
│  ├─ Routing policy enforcement      │
│  └─ Hardware compatibility check    │
│                                     │
├─ STEP 2: Service Container Bootstrap│
│  ├─ Database initialization         │
│  │  └─ Currently disabled (JSON corruption)
│  ├─ ConfigurationService setup      │
│  │  └─ InMemoryConfigurationService │
│  ├─ API client creation            │
│  │  └─ Only if API key provided     │
│  ├─ ModelRegistry initialization    │
│  ├─ Hardware capability detection   │
│  └─ Memory threshold configuration  │
│                                     │
├─ STEP 3: Service Health Monitoring  │
│  ├─ Health check registration       │
│  ├─ Periodic monitoring start (30s) │
│  └─ Unhealthy service detection     │
│                                     │
├─ STEP 4: Performance Monitoring     │
│  ├─ Real-time dashboard (if enabled)│
│  ├─ Performance tracking start      │
│  └─ Analytics initialization        │
│                                     │
└─ SUCCESS: SDK ready for operations  │
   └─ All services operational        │

Error Scenarios:

  • Configuration Validation Failure: Invalid API key, unsupported routing policy
  • Hardware Incompatibility: Insufficient memory, unsupported architecture
  • Service Initialization Failure: Database corruption, file system issues
  • Recovery: Fallback to minimal configuration, error reporting

2. Model Loading Flow (Comprehensive)

┌─ User calls SDK.loadModel(identifier) ─┐
│                                        │
├─ STEP 1: Model Discovery               │
│  ├─ ModelRegistry.lookup(identifier)   │
│  ├─ Local model scanning               │
│  ├─ Repository query (if available)    │
│  └─ ERROR: ModelNotFound if missing    │
│                                        │
├─ STEP 2: Model Validation              │
│  ├─ Format detection                   │
│  │  ├─ File extension analysis         │
│  │  ├─ Magic number detection          │
│  │  └─ Content inspection              │
│  ├─ Metadata extraction                │
│  │  ├─ Model size calculation          │
│  │  ├─ Architecture detection          │
│  │  └─ Requirements parsing            │
│  ├─ Checksum verification              │
│  ├─ Dependency checking                │
│  │  ├─ Framework availability          │
│  │  ├─ Hardware requirements           │
│  │  └─ System compatibility            │
│  └─ ERROR: ValidationFailed if invalid │
│                                        │
├─ STEP 3: Download (if needed)          │
│  ├─ Local path existence check         │
│  ├─ Download URL validation            │
│  ├─ Storage space verification         │
│  ├─ AlamofireDownloadService.download()│
│  │  ├─ Progress tracking               │
│  │  ├─ Resume capability               │
│  │  └─ Error handling                  │
│  ├─ Post-download validation           │
│  └─ ERROR: DownloadFailed if issues    │
│                                        │
├─ STEP 4: Memory Allocation             │
│  ├─ Memory requirement calculation     │
│  ├─ Available memory check             │
│  ├─ Memory pressure evaluation         │
│  ├─ Model eviction (if needed)         │
│  │  ├─ LRU selection                   │
│  │  ├─ Priority consideration          │
│  │  └─ Graceful unloading              │
│  ├─ Memory reservation                 │
│  └─ ERROR: InsufficientMemory if full  │
│                                        │
├─ STEP 5: Framework Selection           │
│  ├─ Preferred framework check          │
│  ├─ Compatible frameworks enumeration  │
│  ├─ FrameworkAdapterRegistry.findBest()│
│  ├─ Hardware compatibility validation  │
│  └─ ERROR: NoCompatibleFramework       │
│                                        │
├─ STEP 6: Model Loading                 │
│  ├─ FrameworkAdapter.loadModel()       │
│  ├─ LLMService.initialize()            │
│  ├─ Model memory registration          │
│  ├─ Service health verification        │
│  ├─ Context preparation                │
│  └─ ERROR: LoadingFailed if issues     │
│                                        │
├─ STEP 7: Registration & Finalization   │
│  ├─ LoadedModel creation               │
│  ├─ GenerationService.setCurrentModel()│
│  ├─ Memory service registration        │
│  ├─ Analytics tracking (if enabled)    │
│  ├─ Last used timestamp update         │
│  └─ SUCCESS: Model ready for inference │
│                                        │
└─ RETURN: ModelInfo with local path     │

Decision Points:

  • Format Detection: Determines validation strategy
  • Memory Pressure: Triggers eviction or fails loading
  • Framework Selection: Chooses optimal execution path
  • Error Recovery: Automatic retry, framework switching, memory cleanup

3. Text Generation Flow (Detailed)

┌─ User calls SDK.generate(prompt, options) ─┐
│                                            │
├─ STEP 1: Pre-Generation Setup             │
│  ├─ SDK initialization check               │
│  ├─ Model loading verification             │
│  ├─ Configuration merge                    │
│  │  ├─ User options priority               │
│  │  ├─ Configuration defaults              │
│  │  └─ SDK constants fallback              │
│  └─ Analytics enabled check                │
│                                            │
├─ STEP 2: Structured Output Preparation    │
│  ├─ Structured output config check         │
│  ├─ Schema generation (if needed)          │
│  ├─ Prompt modification                    │
│  │  ├─ Schema injection                    │
│  │  ├─ Instructions addition               │
│  │  └─ Format specification                │
│  └─ Validation mode configuration          │
│                                            │
├─ STEP 3: Context Management               │
│  ├─ ContextManager.prepareContext()        │
│  ├─ Historical context loading             │
│  ├─ Context length validation              │
│  ├─ Context trimming (if needed)           │
│  │  ├─ Message prioritization              │
│  │  ├─ Smart truncation                    │
│  │  └─ Essential content preservation      │
│  └─ Context optimization                   │
│                                            │
├─ STEP 4: Routing Decision                 │
│  ├─ RoutingService.determineRouting()      │
│  ├─ Privacy policy enforcement             │
│  ├─ HARDCODED: Return .onDevice            │
│  │  └─ Cloud routing disabled for privacy │
│  ├─ Framework selection validation         │
│  └─ Execution path determination           │
│                                            │
├─ STEP 5: On-Device Generation             │
│  ├─ LoadedModel validation                 │
│  ├─ Service readiness check                │
│  ├─ Context setting on service             │
│  ├─ Performance tracking start             │
│  ├─ LLMService.generate()                  │
│  │  ├─ Prompt processing                   │
│  │  ├─ Inference execution                 │
│  │  ├─ Token generation                    │
│  │  ├─ Error handling                      │
│  │  │  ├─ Timeout detection                │
│  │  │  ├─ Framework error processing       │
│  │  │  └─ Recovery attempt                 │
│  │  └─ Response preparation                │
│  ├─ Memory usage tracking                  │
│  └─ Generation completion                  │
│                                            │
├─ STEP 6: Thinking Content Processing      │
│  ├─ Model thinking support check           │
│  ├─ Thinking pattern configuration         │
│  │  ├─ Default: <think>...</think>         │
│  │  ├─ Alternative: <thinking>...</thinking>│
│  │  └─ Custom patterns supported           │
│  ├─ ThinkingParser.parse()                 │
│  │  ├─ Pattern matching                    │
│  │  ├─ Content extraction                  │
│  │  └─ Thinking separation                 │
│  ├─ Final content preparation              │
│  └─ Thinking content preservation          │
│                                            │
├─ STEP 7: Performance Metrics              │
│  ├─ Latency calculation                    │
│  ├─ Token counting (estimation)            │
│  ├─ Tokens per second calculation          │
│  ├─ Memory usage measurement               │
│  ├─ Cost calculation                       │
│  └─ Metrics aggregation                    │
│                                            │
├─ STEP 8: Structured Output Validation     │
│  ├─ Structured config presence check       │
│  ├─ JSON parsing attempt                   │
│  ├─ Schema validation                      │
│  │  ├─ Type validation                     │
│  │  ├─ Required field checking             │
│  │  └─ Format verification                 │
│  ├─ Validation result creation             │
│  └─ Result metadata enhancement            │
│                                            │
├─ STEP 9: Analytics Recording              │
│  ├─ Analytics enabled verification         │
│  ├─ Generation record creation             │
│  │  ├─ Session identification              │
│  │  ├─ Performance metrics                 │
│  │  ├─ Model information                   │
│  │  └─ Context metadata                    │
│  ├─ Repository storage                     │
│  ├─ Live metrics update                    │
│  └─ Telemetry transmission (if enabled)    │
│                                            │
└─ RETURN: GenerationResult                 │
   ├─ Generated text (final content)         │
   ├─ Thinking content (if extracted)        │
   ├─ Performance metrics                    │
   ├─ Cost breakdown                         │
   ├─ Model identification                   │
   ├─ Execution target (onDevice)            │
   └─ Structured output validation           │

Error Handling Paths:

  • Timeout Errors: Helpful messages about model size/complexity
  • Framework Errors: Automatic framework switching if available
  • Memory Pressure: Automatic cleanup and retry
  • Validation Errors: Detailed schema validation feedback

File-Level Details

Core Service Files

/Public/RunAnywhereSDK.swift (841 lines)

Purpose: Main SDK singleton providing complete public API Key Classes: RunAnywhereSDK Key Methods:

  • initialize(configuration:) - SDK setup with comprehensive service bootstrapping
  • loadModel(_:) - Model loading with validation and memory management
  • generate(prompt:options:) - Text generation with analytics and error handling
  • generateStream(prompt:options:) - Real-time streaming generation
  • generateStructured(_:prompt:options:) - Type-safe structured output
  • Configuration methods (setTemperature, setMaxTokens, etc.)
  • Analytics access methods (getAnalyticsSession, observeLiveMetrics)
  • Storage management methods (getStorageInfo, deleteModel)

Dependencies: ServiceContainer, Configuration validation, Error handling Error Handling: Comprehensive error catching with user-friendly messages

/Infrastructure/DependencyInjection/ServiceContainer.swift (503 lines)

Purpose: Central dependency injection container with lazy service initialization Key Features:

  • Lazy service initialization with dependency injection
  • Service health monitoring every 30 seconds
  • Database integration (currently disabled due to JSON corruption)
  • Bootstrap process with configuration validation
  • 25+ registered services with proper lifecycle management

Service Categories:

  • Core Services: ConfigurationValidator, ModelRegistry, FrameworkAdapterRegistry
  • Capability Services: ModelLoading, Generation, Streaming, Validation
  • Infrastructure Services: Hardware detection, Memory management
  • Analytics Services: Generation analytics, Performance monitoring

/Capabilities/TextGeneration/Services/GenerationService.swift (296 lines)

Purpose: Core text generation orchestration with context management Key Features:

  • Multi-routing support (currently device-only)
  • Thinking content parsing with configurable patterns
  • Structured output integration
  • Comprehensive error handling with timeout detection
  • Performance metrics calculation
  • Memory usage tracking

Flow: Context preparation → Routing decision → On-device generation → Thinking parsing → Result creation

/Capabilities/Memory/Services/MemoryService.swift (262 lines)

Purpose: Advanced memory management with pressure handling Key Components:

  • AllocationManager: Model memory tracking
  • PressureHandler: Multi-level pressure response
  • CacheEviction: LRU-based model eviction
  • MemoryMonitor: Real-time memory monitoring

Configuration:

  • Memory threshold: 500MB (warning)
  • Critical threshold: 200MB (critical action)
  • Monitoring interval: 5 seconds
  • Eviction strategy: LRU with priority consideration

/Infrastructure/Hardware/HardwareCapabilityManager.swift (251 lines)

Purpose: Unified hardware detection and capability management Features:

  • Cross-platform capability detection (iOS/macOS/tvOS/watchOS)
  • Cached capability results with 1-minute validity
  • Neural Engine detection (A12+/M1+ chips)
  • GPU capability assessment
  • Thermal state monitoring
  • Battery information (mobile platforms)

Configuration and Model Files

/Public/Configuration/SDKConfiguration.swift (70 lines)

Purpose: SDK configuration with privacy-first defaults Key Settings:

  • Routing policy: Hardcoded to .deviceOnly for privacy
  • Cloud routing: Disabled by default
  • Analytics: Fully enabled with live metrics
  • Privacy mode: Standard with upgrade options
  • Memory threshold: 500MB default

/Core/Models/ModelInfo.swift (87 lines)

Purpose: Comprehensive model information with thinking support Key Features:

  • Thinking tag pattern support (DeepSeek-style)
  • Multiple download URL support
  • Hardware requirement specification
  • Framework compatibility matrix
  • Metadata and checksum validation

/Public/Errors/RunAnywhereError.swift (174 lines)

Purpose: User-facing error types with recovery suggestions Error Categories:

  • Initialization: Configuration and setup errors
  • Model: Loading, validation, and compatibility errors
  • Generation: Runtime inference errors
  • Hardware: Memory pressure and thermal errors
  • Network: Download and connectivity errors

Recovery System: Each error includes specific recovery suggestions

Foundation and Utility Files

/Foundation/Constants/SDKConstants.swift (85 lines)

Purpose: SDK-wide constants and configuration defaults Key Constants:

  • Version and user agent information
  • Timeout configurations (API: 60s, Download: 300s)
  • Memory thresholds (Warning: 80%, Critical: 90%)
  • Default generation settings (Temperature: 0.7, Tokens: 256)
  • Directory names and routing policies

/Foundation/Logging/Logger/SDKLogger.swift

Purpose: Comprehensive logging system with remote capabilities Features:

  • Multi-level logging (debug through fault)
  • Privacy-aware metadata collection
  • Batch submission for efficiency
  • Remote logging support
  • Structured log formatting

/Foundation/Utilities/AsyncQueue.swift

Purpose: Thread-safe sequential task execution Usage: Ensures ordered execution of async operations

/Foundation/Utilities/WeakCollection.swift

Purpose: Memory-safe object collections preventing retain cycles Usage: Observer patterns and delegate management

Protocol Definition Files

/Core/Protocols/Services/LLMService.swift (128 lines)

Purpose: Primary protocol for ML framework integration Key Methods:

  • initialize(modelPath:) - Model loading
  • generate(prompt:options:) - Synchronous generation
  • streamGenerate(prompt:options:onToken:) - Streaming generation
  • getModelMemoryUsage() - Memory monitoring
  • Context management methods

Error Types: LLMServiceError, FrameworkError with detailed context

/Core/Protocols/Frameworks/FrameworkAdapter.swift

Purpose: Framework integration protocol Responsibilities: Model loading, service creation, compatibility checking

/Core/Protocols/Registry/ModelRegistry.swift

Purpose: Model catalog and discovery interface Features: Model lookup, filtering, registration, metadata management

Edge Cases and Error Scenarios

1. Database Disabled Scenarios

Current State: Database is temporarily disabled due to JSON corruption issues Impact:

  • Configuration stored in-memory only (lost on restart)
  • Analytics use no-op service
  • Model metadata not persisted

Fallback Behavior:

// ServiceContainer.swift line 190
logger.warning("Database disabled - using in-memory configuration only")
return nil

// Results in InMemoryConfigurationService usage
_configurationService = InMemoryConfigurationService()

Error Recovery: Graceful degradation to minimal functionality

2. Memory Pressure Handling

Pressure Levels:

  • Low/Medium: Standard monitoring
  • High: Increase memory threshold by 1.5x
  • Warning: Increase threshold by 2x, consider eviction
  • Critical: Increase threshold by 3x, force eviction

Eviction Strategy:

// MemoryService.swift - LRU eviction with priority
enum UnloadStrategy {
    case leastRecentlyUsed
    case largestFirst
    case oldestFirst
    case priorityBased
}

Platform Integration:

  • iOS: UIApplication memory warnings
  • macOS: Thermal state monitoring
  • Automatic model unloading under pressure

3. Network Failures

Download Resilience:

  • Automatic resume capability
  • Alternative URL fallback
  • Progress preservation
  • Integrity verification
  • Timeout handling with exponential backoff

Cloud Sync Failures (when enabled):

  • Local storage fallback
  • Offline mode operation
  • Conflict resolution on reconnect
  • Retry mechanisms

4. Model Compatibility Issues

Detection Strategy:

// ValidationService comprehensive checking
├── Format detection (magic numbers, extensions)
├── Metadata extraction (architecture, requirements)
├── Hardware compatibility (Neural Engine, GPU, memory)
├── Framework availability (adapter registration)
└── Dependency verification (system libraries)

Error Recovery:

  • Alternative framework suggestion
  • Hardware upgrade recommendations
  • Model alternative suggestions
  • Detailed compatibility reports

5. Concurrent Operation Handling

Thread Safety:

  • ServiceContainer: Lazy initialization with locks
  • Memory management: Atomic operations
  • Configuration updates: Synchronized access
  • Analytics: Concurrent-safe repositories

Resource Contention:

  • Single model loading at a time
  • Memory allocation queuing
  • Download request throttling
  • Generation request ordering

6. Framework Unavailability

Scenario: Required framework adapter not registered Error: RunAnywhereError.hardwareUnsupported Recovery:

  • Alternative framework suggestions
  • Adapter registration guidance
  • Capability-based recommendations

7. Thermal State Management

Detection:

ProcessInfo.processInfo.thermalState

Response Strategy:

  • Normal: Standard operation
  • Fair: Performance monitoring
  • Serious: Generation throttling
  • Critical: Operation suspension with user notification

8. Storage Exhaustion

Prevention:

  • Pre-download space verification
  • Cache size limits (100MB default)
  • Automatic cleanup recommendations
  • Storage analytics and alerts

Recovery:

  • Temporary file cleanup
  • Cache eviction
  • Model deletion suggestions
  • Storage optimization guidance

Recent Architectural Improvements (Phases 1-3 Refactoring)

Phase 1: Foundation & Core Cleanup

  1. Moved Dependency Injection to Foundation Layer: Better alignment with cross-cutting concerns
  2. Enum-Based Configuration: Replaced string constants with type-safe enums:
    • RoutingPolicy enum for routing decisions
    • AnalyticsLevel enum for analytics configuration
    • CacheEvictionPolicy enum for storage management
  3. Structured Configuration: Decomposed monolithic ConfigurationData into focused sub-configurations:
    • RoutingConfiguration for routing behavior
    • AnalyticsConfiguration for analytics settings
    • GenerationConfiguration for text generation
    • StorageConfiguration for storage management
  4. Consolidated Models: Moved configuration models to Core layer for better organization

Phase 2: Data Module Creation

  1. New Data Layer: Created dedicated module for all data operations
    • Repository protocols for clean interfaces
    • Storage implementations (SQLite)
    • Network operations (API client, sync)
    • DTOs with strong typing
  2. Separated Concerns: Moved all data access out of capabilities
  3. Type-Safe DTOs: Created strongly-typed data transfer objects

Phase 3: Capability Cleanup

  1. Clean Capabilities: Removed all repository/data code from capabilities
  2. DeviceCapability: Moved hardware detection from Infrastructure to a proper capability
    • Added ThermalMonitorService with ThermalState enum
    • Added BatteryMonitorService with BatteryState enum
    • Consolidated all hardware detection services
  3. Type Safety: Updated remaining string-based APIs to use enums
    • setAnalyticsLevel/getAnalyticsLevel now use AnalyticsLevel enum
    • TelemetryEventType enum for telemetry events

Phase 4: Final Cleanup (Completed December 2024)

  1. Resolved Compilation Issues: Fixed duplicate file names and protocol conflicts
    • Removed duplicate HardwareDetector protocol from DeviceCapability
    • Renamed repository implementations to avoid name conflicts (*RepositoryImpl.swift)
    • Moved BatteryInfo model to proper location in DeviceCapability/Models
  2. Build System: Successfully compiling with Swift 5.9+
    • All 303 Swift source files compile successfully
    • Resolved ambiguous type lookup issues
    • Fixed protocol/implementation naming conflicts
  3. Architecture Alignment: Ensured all components follow the documented architecture
    • Protocols remain in Core layer
    • Implementations use distinct names
    • Models properly organized within their respective modules

Implementation Status

✅ Fully Implemented (Production Ready)

Core Infrastructure

  • SDK Architecture: Complete 5-layer architecture with 303 Swift files
  • Dependency Injection: ServiceContainer with 25+ services and health monitoring
  • Hardware Detection: Cross-platform capability detection with caching
  • Memory Management: Advanced memory management with pressure handling and LRU eviction
  • Error Handling: Comprehensive error system with recovery suggestions
  • Configuration System: Dynamic configuration with persistence (when database enabled)

Voice Processing (Enhanced with Orchestrator)

  • Voice Orchestrator: Complete pipeline management with streaming events
  • Voice Pipeline: End-to-end STT → LLM → TTS processing
  • Speech-to-Text: WhisperKit integration with streaming support
  • Text-to-Speech: System TTS with multiple voice options (protocol ready)
  • Voice Activity Detection: Energy-based VAD implementation
  • Session Management: Comprehensive voice session tracking
  • Audio Processing: Professional-grade audio preprocessing
  • Pipeline Events: Real-time progress tracking and error reporting
  • Configurable Timeouts: Per-stage timeout management (STT: 30s, LLM: 60s, TTS: 30s)
  • Unified Adapters: Voice integrated through unified framework adapter pattern

Model Management

  • Model Validation: Complete validation pipeline for all supported formats
    • GGUF, CoreML, ONNX, TensorFlow Lite, MLX validators
    • Metadata extraction and checksum verification
    • Dependency checking and compatibility validation
  • Model Registry: Model catalog with discovery and filtering
  • File Management: Organized storage with framework-specific directories

Networking & Downloads

  • Download System: Robust downloading with Alamofire integration
    • Resume capability and progress tracking
    • Multiple URL fallback support
    • Integrity verification and error recovery

Analytics & Monitoring

  • Performance Monitoring: Real-time performance tracking with alerts
  • Generation Analytics: Comprehensive analytics with session tracking
    • Live metrics streaming and database integration
    • Export capabilities and aggregation
  • A/B Testing: Statistical testing framework with significance calculation
  • Benchmarking: Complete benchmarking suite with multiple export formats

Storage & Utilities

  • Storage Management: File system operations with cleanup and analytics
  • Logging System: Multi-level logging with remote capabilities
  • Utility Classes: AsyncQueue, WeakCollection, and foundational utilities

🚧 Partially Implemented (Functional but Limited)

Text Generation

Status: Simulated generation with complete infrastructure Implemented:

  • Generation orchestration with context management
  • Thinking content parsing (DeepSeek-style patterns)
  • Structured output support with schema validation
  • Streaming generation infrastructure
  • Performance metrics and analytics integration

Missing:

  • Actual ML framework integrations (adapters are protocol-only)
  • Real inference execution (currently returns simulated responses)

Model Loading

Status: Complete loading pipeline without actual framework execution Implemented:

  • Model discovery and validation
  • Memory allocation and tracking
  • Framework adapter selection
  • Service lifecycle management

Missing:

  • Framework adapter implementations (CoreML, TensorFlow Lite, etc.)
  • Actual model loading into ML frameworks

Tokenization

Status: Infrastructure complete, adapters needed Implemented:

  • Tokenizer service architecture
  • Format detection and caching
  • Multi-format support framework

Missing:

  • Actual tokenizer implementations for each format
  • Framework-specific tokenizer adapters

Database Integration

Status: Temporarily disabled due to JSON corruption Implemented:

  • SQLite database integration architecture
  • Repository pattern implementation
  • Data sync service framework

Issue: JSON corruption causing database failures Current Workaround: InMemoryConfigurationService for configuration

📋 Planned (Architecture Ready)

Cloud Integration

Architecture: Complete routing and API client framework Status: Intentionally disabled for privacy-first approach Ready For: Cloud provider integration when privacy requirements met

Real ML Framework Integration

Architecture: Complete adapter system with protocol definitions Planned Frameworks:

  • CoreML integration for Apple ecosystem optimization
  • TensorFlow Lite for cross-platform compatibility
  • MLX for Apple Silicon optimization
  • ONNX for model interoperability
  • ExecuTorch for mobile deployment
  • LlamaCpp for GGUF model support

Advanced Features

  • Model Conversion: Tools for converting between formats
  • Advanced Caching: Sophisticated caching strategies beyond LRU
  • Distributed Processing: Multi-device coordination capabilities
  • Custom Training: On-device fine-tuning capabilities

Enhanced Analytics

  • Predictive Analytics: Usage pattern prediction
  • Cost Optimization: Advanced cost-benefit analysis
  • User Behavior Analytics: Usage pattern insights
  • Performance Predictions: Model performance forecasting

Database Status Update

Current Issue: Database temporarily disabled due to JSON corruption

// ServiceContainer.swift lines 177-193
private var database: DatabaseCore? {
    get async {
        // COMMENTED OUT: Database temporarily disabled to avoid JSON corruption issues
        // Return nil to force in-memory configuration
        logger.warning("Database disabled - using in-memory configuration only")
        return nil
    }
}

Impact:

  • Configuration stored in-memory only (reset on app restart)
  • Analytics use NoOpGenerationAnalyticsService
  • Model metadata not persisted between sessions

Workaround: Complete in-memory operation with graceful degradation

Developer Guide

Adding New Capabilities

1. Capability Structure

Create new capability following the standard pattern:

Capabilities/NewCapability/
├── Protocols/          # Service interfaces
├── Services/           # Main implementations
├── Models/             # Data structures
├── Strategies/         # Algorithm implementations
└── Tracking/           # Analytics integration

2. Service Registration

Add to ServiceContainer.swift:

private(set) lazy var newCapabilityService: NewCapabilityProtocol = {
    NewCapabilityService(dependencies...)
}()

3. Protocol Definition

Define clear interfaces in Core/Protocols/:

protocol NewCapabilityProtocol {
    func performOperation() async throws -> Result
    func isHealthy() -> Bool
}

Implementing Framework Adapters

1. Adapter Protocol Implementation

class CustomFrameworkAdapter: FrameworkAdapter {
    let framework: LLMFramework = .custom

    func loadModel(_ model: ModelInfo) async throws -> LLMService {
        // Implementation
        return CustomLLMService(model: model)
    }

    func canHandle(model: ModelInfo) -> Bool {
        return model.compatibleFrameworks.contains(.custom)
    }
}

2. LLMService Implementation

class CustomLLMService: LLMService {
    func initialize(modelPath: String) async throws {
        // Load model using custom framework
    }

    func generate(prompt: String, options: GenerationOptions) async throws -> String {
        // Implement actual inference
    }

    func streamGenerate(prompt: String, options: GenerationOptions, onToken: @escaping (String) -> Void) async throws {
        // Implement streaming inference
    }
}

3. Registration

SDK.shared.registerFrameworkAdapter(CustomFrameworkAdapter())

Extending Error Handling

1. Custom Error Types

enum CustomCapabilityError: LocalizedError {
    case specificError(String)

    var errorDescription: String? {
        switch self {
        case .specificError(let detail):
            return "Custom error: \(detail)"
        }
    }

    var recoverySuggestion: String? {
        switch self {
        case .specificError:
            return "Try specific recovery action"
        }
    }
}

2. Error Recovery Strategies

class CustomRecoveryStrategy: ErrorRecoveryStrategy {
    func canRecover(from error: Error) -> Bool {
        return error is CustomCapabilityError
    }

    func recover(from error: Error) async throws {
        // Implement recovery logic
    }
}

Performance Optimization Tips

1. Memory Management

  • Always register loaded models with MemoryService
  • Use memory priorities for critical vs optional models
  • Implement proper cleanup in service destructors
  • Monitor memory usage during development

2. Async/Await Best Practices

  • Use structured concurrency with TaskGroup for parallel operations
  • Implement proper cancellation support
  • Avoid blocking calls in async contexts
  • Use AsyncQueue for ordered operations

3. Error Handling

  • Provide specific error messages with recovery suggestions
  • Use structured error types with context
  • Implement proper error propagation chains
  • Log errors with appropriate levels

4. Configuration Management

  • Use configuration service for runtime settings
  • Implement proper default value handling
  • Support configuration validation
  • Enable configuration persistence when database available

Testing Framework Extensions

1. Mock Services

class MockLLMService: LLMService {
    var responses: [String] = []
    var currentIndex = 0

    func generate(prompt: String, options: GenerationOptions) async throws -> String {
        guard currentIndex < responses.count else {
            throw LLMServiceError.generationFailed("No more responses")
        }
        defer { currentIndex += 1 }
        return responses[currentIndex]
    }
}

2. Test Utilities

extension RunAnywhereSDK {
    static func createTestInstance() -> RunAnywhereSDK {
        // Create SDK instance with test configuration
    }
}

Debugging and Troubleshooting

1. Logging Configuration

// Enable debug logging
LoggingManager.shared.setLogLevel(.debug)

// Configure remote logging
LoggingManager.shared.configureRemoteLogging(endpoint: URL(...))

2. Health Monitoring

// Check service health
let health = await ServiceContainer.shared.checkServiceHealth()
print("Unhealthy services: \(health.filter { !$0.value })")

3. Memory Debugging

// Get memory statistics
let stats = memoryService.getMemoryStatistics()
print("Memory usage: \(stats.usedMemoryPercentage)%")
print("Model memory: \(stats.modelMemoryPercentage)%")

4. Performance Analysis

// Access performance monitor
let monitor = SDK.shared.performanceMonitor
let snapshot = await monitor.getCurrentSnapshot()

Extension Points

Custom Framework Adapters

protocol FrameworkAdapter {
    var framework: LLMFramework { get }
    func loadModel(_ model: ModelInfo) async throws -> LLMService
    func canHandle(model: ModelInfo) -> Bool
}

Implementation Guidance:

  • Implement canHandle() for model compatibility checking
  • Use async/await for loadModel() to support async initialization
  • Return LLMService implementation for inference operations
  • Handle framework-specific errors appropriately

Custom Recovery Strategies

protocol ErrorRecoveryStrategy {
    func canRecover(from error: Error) -> Bool
    func recover(from error: Error) async throws
}

Use Cases:

  • Framework-specific error handling
  • Network failure recovery
  • Memory pressure response
  • Hardware limitation workarounds

Model Providers

protocol ModelProvider {
    func searchModels(query: String) async throws -> [ModelInfo]
    func downloadModel(_ model: ModelInfo) async throws -> URL
    func validateModel(_ model: ModelInfo) async throws -> ValidationResult
}

Integration Points:

  • Registry service integration
  • Download service coordination
  • Validation pipeline integration
  • Metadata management

Custom Analytics Collectors

protocol AnalyticsCollector {
    func recordGeneration(_ generation: Generation) async
    func recordPerformance(_ metrics: PerformanceMetrics) async
    func export(format: ExportFormat) async throws -> Data
}

Hardware Detectors

protocol HardwareDetector {
    func detectCapabilities() -> DeviceCapabilities
    func getAvailableMemory() -> Int64
    func hasNeuralEngine() -> Bool
    func hasGPU() -> Bool
}

Summary

The RunAnywhere Swift SDK v2.0 represents a sophisticated, production-ready foundation for on-device AI with comprehensive voice capabilities:

Architectural Excellence

  • Clean 6-Layer Architecture: 303 Swift files organized with clear separation of concerns
  • Modular Design: 20+ independent capability modules with 60+ data models
  • Voice Integration: Complete voice pipeline with STT, TTS, VAD, and wake word support
  • Protocol-Based Extensibility: Easy integration of new frameworks and capabilities
  • Unified Framework Adapters: Single adapter pattern for all modalities (text, voice, image)
  • Dependency Injection: ServiceContainer in Foundation layer managing 25+ services with health monitoring
  • Type-Safe Configuration: Enum-based configuration system replacing string constants
  • Dedicated Data Layer: Complete separation of data access from business logic
  • Device Capability: Hardware detection as a first-class capability
  • Successfully Compiling: All refactoring phases completed with clean builds

Privacy-First Implementation

  • Device-Only Execution: Cloud routing disabled by design for maximum privacy
  • Comprehensive Analytics: Detailed metrics without data transmission
  • Configurable Privacy: Multiple privacy modes with strict enforcement
  • Zero Data Leakage: Architecture prevents accidental cloud communication

Production-Ready Features

  • Advanced Memory Management: Sophisticated pressure handling with LRU eviction
  • Robust Error Handling: Comprehensive error system with recovery suggestions
  • Real-Time Monitoring: Performance tracking with alert management
  • Statistical Testing: A/B testing framework with significance calculation

Developer Experience

  • Modern Swift APIs: Async/await throughout with structured concurrency
  • Type-Safe Operations: Structured output with compile-time safety
  • Comprehensive Documentation: Detailed architecture and implementation guides
  • Extensible Design: Clear extension points for custom implementations

Current Limitations and Opportunities

  • ML Framework Integration: Architecture complete, awaiting framework adapters
  • Database Integration: Temporarily disabled, architecture ready for re-enablement
  • Cloud Capabilities: Infrastructure ready for privacy-compliant cloud integration

The SDK is architected to scale from simple text generation to complex multi-model workflows while maintaining performance, privacy, and developer productivity. The comprehensive capability system provides a solid foundation for AI application development with room for extensive customization and extension.