Skip to content

DOCUMENTATION.md

Latest commit

 

History

History
316 lines (241 loc) · 9.81 KB

DOCUMENTATION.md

File metadata and controls

316 lines (241 loc) · 9.81 KB
Raw

AgentaFlow SRO - AI Infrastructure Tooling & Optimization

A comprehensive toolkit for deploying and managing AI infrastructure more efficiently, featuring GPU orchestration, model serving optimization, and observability tools for AI systems.

Features

1. GPU Orchestration & Scheduling

Optimize GPU utilization across workloads, reducing waste through intelligent scheduling:

  • Multiple Scheduling Strategies: Round-robin, least-utilized, best-fit, and priority-based scheduling
  • Resource Management: Track GPU memory, utilization, temperature, and power usage
  • Workload Queue Management: Efficient queuing and assignment of AI workloads
  • Utilization Metrics: Real-time monitoring of GPU utilization and memory usage
  • Flexible Priority System: Priority-based scheduling for critical workloads

Key Benefits:

  • Reduce GPU idle time by up to 40%
  • Optimize memory allocation with best-fit algorithms
  • Automatic workload distribution across GPU clusters
  • Real-time utilization tracking and optimization

2. AI Model Serving Optimization

Reduce inference costs through intelligent batching, caching, and routing:

  • Request Batching: Automatically batch inference requests for improved throughput
  • Intelligent Caching: Cache inference results with configurable TTL to reduce redundant computation
  • Routing Strategies: Load balancing across model instances using round-robin, least-latency, or least-load strategies
  • Cost Optimization: Minimize inference costs through efficient resource utilization

Key Benefits:

  • Reduce inference latency by up to 50% through caching
  • Improve throughput by 3-5x with request batching
  • Automatic load balancing across model instances
  • Configurable batch sizes and wait times

3. Observability Tools for AI Systems

Comprehensive monitoring, debugging, and cost tracking for LLM applications and training:

  • Metrics Collection: Counter, gauge, and histogram metrics for all operations
  • Event Tracking: Record and query system events with severity levels
  • Cost Tracking: Track costs for inference and training operations including GPU hours and token usage
  • Distributed Tracing: Trace requests across distributed AI systems
  • Debug Logging: Multi-level logging with filtering and analysis
  • Performance Analysis: Automatic bottleneck detection and performance profiling

Key Benefits:

  • Full visibility into AI system performance
  • Detailed cost tracking per operation and model
  • Root cause analysis with distributed tracing
  • Performance bottleneck identification

Installation

go get github.com/Finoptimize/agentaflow-sro-community

Quick Start

Running the Demo

cd cmd/agentaflow
go run main.go

This will demonstrate all three components working together.

GPU Scheduling Example

import "github.com/Finoptimize/agentaflow-sro-community/pkg/gpu"

// Create scheduler with least-utilized strategy
scheduler := gpu.NewScheduler(gpu.StrategyLeastUtilized)

// Register GPUs
gpu1 := &gpu.GPU{
    ID:          "gpu-0",
    Name:        "NVIDIA A100",
    MemoryTotal: 40960, // 40GB
    Available:   true,
}
scheduler.RegisterGPU(gpu1)

// Submit workload
workload := &gpu.Workload{
    ID:             "job-1",
    Name:           "Training Job",
    MemoryRequired: 32768, // 32GB
    Priority:       1,
}
scheduler.SubmitWorkload(workload)

// Schedule workloads
scheduler.Schedule()

// Get metrics
metrics := scheduler.GetUtilizationMetrics()

Model Serving Example

import "github.com/Finoptimize/agentaflow-sro-community/pkg/serving"

// Create serving manager with batching and caching
batchConfig := &serving.BatchConfig{
    MaxBatchSize: 32,
    MaxWaitTime:  100 * time.Millisecond,
}
servingMgr := serving.NewServingManager(batchConfig, 5*time.Minute)

// Register model
model := &serving.Model{
    ID:        "gpt-model",
    Name:      "GPT-3.5",
    Framework: "PyTorch",
}
servingMgr.RegisterModel(model)

// Submit inference request
request := &serving.InferenceRequest{
    ID:      "req-1",
    ModelID: "gpt-model",
    Input:   []byte("Your prompt here"),
}
response, _ := servingMgr.SubmitInferenceRequest(request)

// Check if response was cached
if response.CacheHit {
    fmt.Println("Response served from cache!")
}

Observability Example

import "github.com/Finoptimize/agentaflow-sro-community/pkg/observability"

// Create monitoring service
monitor := observability.NewMonitoringService(10000)

// Record metrics
monitor.RecordMetric(observability.Metric{
    Name:  "inference_latency",
    Type:  observability.MetricHistogram,
    Value: 45.5,
    Labels: map[string]string{"model": "gpt"},
})

// Record costs
monitor.RecordCost(observability.CostEntry{
    Operation:  "inference",
    ModelID:    "gpt-model",
    TokensUsed: 1000,
    GPUHours:   0.5,
    Cost:       2.50,
})

// Get cost summary
summary := monitor.GetCostSummary(startTime, endTime)
fmt.Printf("Total cost: $%.2f\n", summary["total_cost"])

// Create debugger for tracing
debugger := observability.NewDebugger(observability.DebugLevelInfo)

// Start trace
debugger.StartTrace("trace-1", "inference", map[string]string{
    "model": "gpt-3.5",
})

// Add logs to trace
debugger.AddTraceLog("trace-1", observability.DebugLevelInfo,
    "Processing request", nil)

// End trace
debugger.EndTrace("trace-1", "success")

Architecture

agentaflow-sro-community/
├── pkg/
│   ├── gpu/                    # GPU orchestration and scheduling
│   │   ├── types.go           # Core types and constants
│   │   └── scheduler.go       # Scheduling algorithms
│   ├── serving/               # AI model serving optimization
│   │   ├── manager.go         # Serving manager with batching/caching
│   │   └── router.go          # Request routing strategies
│   └── observability/         # Observability and monitoring
│       ├── monitoring.go      # Metrics and cost tracking
│       └── debugger.go        # Debug logging and tracing
├── cmd/
│   └── agentaflow/            # Main CLI application
│       └── main.go
└── examples/                  # Usage examples

Scheduling Strategies

GPU Scheduling

  • Least Utilized: Assigns workloads to the GPU with the lowest utilization
  • Best Fit: Finds the GPU with just enough free memory for the workload
  • Priority: Schedules high-priority workloads first
  • Round Robin: Distributes workloads evenly across all GPUs

Model Serving Routing

  • Least Latency: Routes to the instance with lowest average latency
  • Least Load: Routes to the instance with lowest current load
  • Round Robin: Distributes requests evenly across instances

Performance Optimization

GPU Utilization

  • Target 80% GPU utilization by default (configurable)
  • Automatic workload placement to minimize fragmentation
  • Memory-aware scheduling to prevent OOM errors

Inference Optimization

  • Request batching reduces per-request overhead by 50-70%
  • Caching eliminates redundant computations
  • Intelligent routing minimizes latency

Cost Reduction

  • Track costs per operation, model, and time period
  • Identify expensive operations for optimization
  • Monitor GPU hours and token usage

Use Cases

  1. ML Training Clusters: Optimize GPU utilization across training jobs
  2. LLM Inference Services: Reduce costs with batching and caching
  3. Multi-Model Serving: Load balance across model instances
  4. Cost Optimization: Track and optimize AI infrastructure spending
  5. Performance Debugging: Identify bottlenecks in AI pipelines

Configuration

All components support flexible configuration:

// GPU Scheduler
scheduler := gpu.NewScheduler(gpu.StrategyLeastUtilized)
// Utilization goal is 80% by default

// Serving Manager
batchConfig := &serving.BatchConfig{
    MaxBatchSize: 32,           // Maximum requests per batch
    MaxWaitTime:  100 * time.Millisecond,  // Max wait before processing
    MinBatchSize: 1,            // Minimum batch size
}
servingMgr := serving.NewServingManager(batchConfig, 5*time.Minute)

// Monitoring
monitor := observability.NewMonitoringService(10000)  // Keep last 10K entries
debugger := observability.NewDebugger(observability.DebugLevelInfo)

Metrics and Monitoring

GPU Metrics

  • Total/Active GPUs
  • Average utilization
  • Memory usage
  • Pending workloads

Serving Metrics

  • Cache hit rate
  • Batch sizes
  • Request latency
  • Model instance health

Cost Metrics

  • Total cost by operation type
  • GPU hours consumed
  • Token usage
  • Cost trends over time

License

Apache License 2.0 - see LICENSE file for details.

Contributing

Contributions are welcome! This is a community edition focused on providing core AI infrastructure optimization capabilities.

Roadmap

  • Add Kubernetes integration for GPU scheduling
  • Support for multi-cloud GPU resources
  • Advanced cost prediction and budgeting
  • Real-time GPU metrics collection
  • Model auto-scaling based on load
  • Enhanced tracing with OpenTelemetry support
  • Prometheus/Grafana integration
  • Web dashboard for monitoring

🚀 Enterprise Edition (Coming Soon)

Looking for advanced features for production environments? Our Enterprise Edition will include:

  • Multi-cluster Orchestration: Manage GPU resources across multiple Kubernetes clusters
  • Advanced Scheduling Algorithms: Cost optimization algorithms and priority queues for enterprise workloads
  • RBAC and Audit Logs: Role-based access control and comprehensive audit logging
  • Enterprise Integrations: Slack alerts, DataDog monitoring, and other enterprise tools
  • SLA Support: Guaranteed service levels with dedicated support
  • Usage-based Billing Features: Advanced cost tracking and billing automation

Contact us for early access and enterprise pricing.