Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
189 lines (143 loc) · 5.24 KB

ARCHITECTURE.md

File metadata and controls

189 lines (143 loc) · 5.24 KB

Gopher Workflow & Architecture

Step-by-step explanation of Gopher's job lifecycle, architecture, and real-world usage

Core Architecture

Components

Component Description Role
Server RESTful API Accepts job submissions and provides status endpoints
Worker Processing daemons Execute jobs from queues with configurable concurrency
CLI Command-line interface Administrative tools and job management
Redis Message broker Central job storage, queue management, and communication

Complete Workflow

1. Job Submission

Clients submit jobs via HTTP API or CLI. Server validates the request, generates a unique job ID, and enqueues it in Redis.

POST /api/jobs
{
  "type": "email",
  "payload": {
    "to": "user@example.com",
    "subject": "Welcome!",
    "body": "Thank you for signing up."
  },
  "priority": "high",
  "retries": 3
}

Response:
{
  "job_id": "e52fe7d2-8be1-4d9f-a27d-2f02cb16ad00",
  "status": "enqueued"
}

2. Queue Management

Priority System

  • High Priority Queue: Critical, time-sensitive jobs
  • Normal Priority Queue: Standard business operations
  • Low Priority Queue: Background tasks and maintenance

Scheduling

Delayed jobs are stored in Redis sorted sets for precise timing:

ZADD scheduled_jobs 1695916800 '{"job_id":"abc123","type":"reminder",...}'

3. Worker Processing

Workers implement a sophisticated polling mechanism:

  1. Priority-based polling with configurable ratios (e.g., 3:2:1)
  2. Atomic job retrieval preventing duplicate processing
  3. Handler routing based on job type
  4. Automatic retry with exponential backoff
// Handler registration
registry.RegisterHandler("email", emailHandler)
registry.RegisterHandler("image_processing", imageHandler)
registry.RegisterHandler("data_export", exportHandler)

4. Job States

State Description
enqueued Job waiting in queue
processing Currently being executed
completed Successfully finished
failed Failed after all retries
scheduled Waiting for scheduled time

Monitoring & Observability

Key Metrics

Queue Health

  • Queue depth by priority level
  • Average wait time before processing
  • Processing rate per worker

Performance

  • Job execution time percentiles (p50, p95, p99)
  • Success and failure rates
  • Worker utilization and throughput

System Health

  • Redis connection status
  • Worker pool capacity
  • Memory and CPU usage

Real-World Problem Solving

Web Application Responsiveness

Offload blocking tasks to workers to improve response times:

func SignupHandler(w http.ResponseWriter, r *http.Request) {
    client.SubmitJob("email", EmailPayload{
        To: newUser.Email,
        Subject: "Welcome!",
        Template: "welcome",
        Data: map[string]interface{}{ "username": newUser.Name },
    })
    w.WriteHeader(http.StatusCreated)
}

Microservice Coordination

Decoupled job submission for multi-service workflows:

  1. Service A submits job
  2. Worker executes
  3. Another job enqueued to notify user
  4. Worker processes notification

Resource Management

  • Priority queues ensure critical tasks first
  • Rate limiting prevents overload
  • Worker pools scale with system resources

System Reliability

  • Retries with exponential backoff
  • DLQ stores jobs that exceed retries
  • Operators can requeue after fixing issues

Real-World Applications

E-commerce Platform

func ProcessOrderHandler(w http.ResponseWriter, r *http.Request) {
    // Submit multiple jobs for order processing
    client.SubmitJob("payment_processing", paymentData)
    client.SubmitJob("inventory_update", inventoryData)
    client.SubmitJob("order_confirmation_email", emailData)
    
    // Return immediately to user
    w.WriteHeader(http.StatusAccepted)
    json.NewEncoder(w).Encode(map[string]string{
        "status": "order_received",
        "order_id": orderID,
    })
}

Media Processing Pipeline

  • Video transcoding for multiple formats
  • Thumbnail generation at various resolutions
  • Metadata extraction and storage
  • CDN distribution preparation

Financial Services

  • Transaction processing and validation
  • Regulatory report generation
  • Risk assessment calculations
  • Customer notification workflows

Technical Implementation

  • Redis atomic operations: Exactly-once processing
  • Worker pool pattern: Concurrency management
  • Circuit breaker pattern: Fault tolerance
  • Prometheus metrics: Real-time monitoring
  • OpenTelemetry: Distributed tracing

Conclusion

Gopher provides a robust, scalable solution for distributed job processing that addresses common challenges in modern applications. Its architecture ensures reliable task execution while maintaining system responsiveness and providing comprehensive monitoring capabilities. By implementing proven patterns like priority queues, retry mechanisms, and circuit breakers, Gopher enables developers to build resilient, high-performance systems with confidence.