Skip to content

DEPLOY.md

Latest commit

 

History

History
87 lines (62 loc) · 4.49 KB

DEPLOY.md

File metadata and controls

87 lines (62 loc) · 4.49 KB
Raw

Deployment & Infrastructure Guide

This guide provides technical specifications and step-by-step instructions for deploying the Hyperion AI Gateway in various environments. Hyperion is designed as a distributed system of microservices, ensuring performance, reliability, and modularity.

Deployment Architecture

A complete Hyperion deployment consists of several specialized services working in orchestration:

1. Edge & Proxy Services

  • Gateway (Go): The primary entry point. Handles TLS termination (optional), authentication, tenant enforcement, rate limiting, and exact-match L1 caching.
  • Dashboard (React/Node): The administrative interface for viewing analytics, managing organizations, and configuring API keys.

2. Processing & Intelligence

  • Embedder (Python/gRPC): Generates high-dimensional vector embeddings for all incoming prompts. This service is critical for L2 Semantic Caching.
  • Predictor/Intelligence (Python/FastAPI): Orchestrates smart routing logic by classifying request complexity and selecting the most efficient model/provider.

3. Persistence & Data Layer

  • Postgres: The primary relational database for platform state (Users, Organizations, API Keys, Pricing).
  • Redis: High-speed in-memory store for L1 caching, request quotas, and rate limiting counters.
  • Qdrant: Vector database optimized for high-performance similarity searches during semantic cache lookups.
  • ClickHouse: Columnar database designed for real-time analytics and high-volume request logging.

Hardware Requirements

Recommendation CPU RAM Storage
Minimum 2 Cores 4 GB 20 GB SSD
Recommended 4 Cores 8 GB 60 GB NVMe
High Traffic 8+ Cores 16+ GB 100+ GB NVMe

Standard Deployment (Docker Compose)

The easiest way to stand up a production-ready Hyperion instance is via Docker Compose.

1. Environment Configuration

Copy the provided .env.example to .env and configure your credentials.

cp .env.example .env

Required Infrastructure Keys:

  • ADMIN_API_KEY: Strong secret for administrative endpoints.
  • JWT_SECRET: Secret key for dashboard session signing.
  • CACHE_MASTER_SECRET: 64-character hex string for encrypting upstream API keys at rest.
  • DATABASE_URL: Postgres connection string (e.g., postgres://user:pass@postgres:5432/db).

2. Launching Services

Ensure Docker and Docker Compose (v2.20+) are installed.

docker-compose up -d --build

The gateway will automatically run required database migrations and initialize Qdrant collections on first boot.

Service Verification

Once deployed, verify the health of the individual components:

Endpoint Method Description
http://localhost:8080/v1/admin/health GET Global Gateway Health
http://localhost:3000 GET Admin Dashboard UI
http://localhost:6333/dashboard GET Qdrant Vector Console
http://localhost:8080/swagger/index.html GET Interactive API Documentation

Production Recommendations

Security & Networking

  • Reverse Proxy: Deploy Hyperion behind a professional reverse proxy like Nginx, Traefik, or HAProxy to handle SSL termination and advanced load balancing.
  • Subnet Isolation: Use internal Docker networks (inference-net) to ensure that databases are not exposed publicly. Only the Gateway (8080) and Dashboard (3000) ports should be mapped to the host.
  • Secret Rotation: Regularly rotate your CACHE_MASTER_SECRET and JWT_SECRET.

Performance Tuning

  • Redis Persistence: For mission-critical deployments, enable AOF (Append Only File) in Redis to prevent cache data loss during restarts.
  • ClickHouse Retention: Configure ClickHouse TTL policies to manage storage costs for request logs (e.g., set to 30 days).

Maintenance

  • Updates: Pull the latest images and restart with docker-compose pull && docker-compose up -d. Hyperion maintains backwards compatibility across minor versions.
  • Monitoring: Integrate the gateway's OpenTelemetry output with a collector (e.g., Prometheus/Jaeger) for continuous performance monitoring.

Troubleshooting

  • Database Connection Refused: Check that DATABASE_URL matches the internal hostname in Docker Compose (usually postgres).
  • Semantic Cache Misses: Ensure the embedder service is running and accessible on port 50051.
  • 401 Unauthorized: Verify that your X-Admin-Key matches the ADMIN_API_KEY defined in your environment.