Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

GPU Performance Forecasting for Gen AI Models

A comprehensive benchmarking and forecasting system for predicting GPU performance across transformer, multimodal, and diffusion models under varying workload conditions on NVIDIA A100 GPUs.

Project Overview

This project develops machine learning models to forecast GPU utilization and latency for different generative AI inference workloads. The system tests multiple inference engines (PyTorch, vLLM, TensorRT-LLM, SGLang) across various concurrency levels and generates actionable insights for capacity planning and system optimization.

Key Capabilities

  • Automated benchmarking pipeline for transformer, multimodal, and diffusion models
  • ML-based performance forecasting using ensemble methods
  • Comparative analysis of inference engines under production-like conditions
  • KV cache impact analysis and visualization
  • Batching and prefill optimization implementations
  • Async concurrency testing supporting up to 1000 concurrent requests
  • Automated report generation with performance scaling visualizations

System Requirements

Hardware

  • NVIDIA A100 GPU (40GB or 80GB)
  • Minimum 32GB system RAM
  • 100GB available disk space for models and data

Software

  • Python 3.9 or higher
  • CUDA 11.8 or higher
  • Access to HuggingFace model repository

Installation

On Northeastern Discovery Cluster

Create project directory and load required modules:

mkdir -p ~/gpu_forecasting_project
cd ~/gpu_forecasting_project
module load python/3.9
module load cuda/11.8

Create virtual environment and install dependencies:

python -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

Local Setup

For local machines with GPU access:

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Project Structure

gpu_forecasting_project/
├── models/
│   └── gpu_forecaster_complete.pkl
├── data/
│   ├── benchmark_results_complete_all_engines.csv
│   ├── benchmark_vllm.csv
│   ├── benchmark_tensorrt.csv
│   └── async_test_results.csv
├── reports/
│   ├── scaling_analysis.png
│   ├── engine_comparison.png
│   └── performance_summary.txt
├── main_pipeline.py
├── benchmark_gpu.py
├── forecasting_model.py
├── visualization_report.py
├── async_load_test.py
├── batching_optimization.py
├── prefill_optimization.py
├── requirements.txt
└── run_pipeline.slurm

Usage

Quick Start

Request GPU resources and run the pipeline:

srun --partition=gpu --gres=gpu:a100:1 --cpus-per-task=4 --mem=32GB --time=02:00:00 --pty /bin/bash
cd ~/gpu_forecasting_project
source venv/bin/activate
python main_pipeline.py --model sshleifer/tiny-gpt2

Batch Execution

Submit as a SLURM job:

sbatch run_pipeline.slurm

Monitor job status:

squeue -u $USER
tail -f logs/gpu_forecast_*.out

Individual Components

Run specific benchmarking tasks:

python benchmark_gpu.py
python async_load_test.py
python batching_optimization.py
python prefill_optimization.py
python benchmark_multimodal.py
python benchmark_diffusion.py
python benchmark_vllm.py

Training the Forecasting Model

Load data and train the model:

import pandas as pd
from forecasting_model import GPUPerformanceForecaster

df = pd.read_csv('data/benchmark_results_complete_all_engines.csv')
forecaster = GPUPerformanceForecaster()
results = forecaster.train(df)
forecaster.save('models/gpu_forecaster_complete.pkl')

Making Predictions

Use the trained model for predictions:

from forecasting_model import GPUPerformanceForecaster

forecaster = GPUPerformanceForecaster()
forecaster.load('models/gpu_forecaster_complete.pkl')

config = {
    'concurrent_requests': 100,
    'sequence_length': 512,
    'engine': 'vLLM'
}

predictions = forecaster.predict(config)
print('Predicted latency:', predictions['predicted_latency_ms'], 'ms')
print('Predicted utilization:', predictions['predicted_gpu_utilization'], '%')

Technical Approach

Benchmarking Methodology

The system collects performance metrics across multiple dimensions:

  • Concurrent request levels: 1 to 1000 requests
  • Sequence lengths: 50 to 1024 tokens
  • Model architectures: GPT-2, OPT, CLIP, Stable Diffusion
  • Inference engines: PyTorch, vLLM, TensorRT-LLM, SGLang

Metrics captured:

  • End-to-end latency in milliseconds
  • Prefill versus decode latency breakdown
  • Throughput in tokens per second
  • GPU utilization percentage
  • Memory consumption in gigabytes
  • KV cache size in megabytes

Machine Learning Models

Ensemble approach using scikit-learn:

Latency Prediction

  • Algorithm: Gradient Boosting Regressor
  • Features: concurrent requests, sequence length, KV cache size, interaction terms
  • Performance: R-squared = 0.98 for transformer models

GPU Utilization Prediction

  • Algorithm: Random Forest Regressor
  • Features: Same as latency with additional memory pressure metrics
  • Performance: R-squared = 0.94

Feature Engineering

  • Log transforms for skewed distributions
  • Interaction terms (requests times sequence length)
  • Memory pressure ratios
  • One-hot encoding for inference engines

Optimization Techniques

Dynamic Batching

  • Groups incoming requests into batches of configurable size
  • Measured improvement: 7.5x speedup over sequential processing
  • Implementation: batching_optimization.py

Prefill Optimization

  • Utilizes scaled dot product attention for accelerated prefill phase
  • Measured improvement: 3.09x speedup, 67.6% latency reduction
  • Implementation: prefill_optimization.py

KV Cache Analysis

  • Tracks cache growth patterns across sequence lengths
  • Identifies memory bottlenecks at high concurrency
  • Informs capacity planning decisions

Results

Model Performance

  • Transformer latency prediction: R-squared = 0.98, MAE = 2,457ms
  • GPU utilization prediction: R-squared = 0.94, MAE = 0.21%
  • Throughput prediction: Limited by hardware constraints

Engine Comparison

Measured throughput at 200 concurrent requests:

  • PyTorch baseline: 583 tokens per second
  • vLLM: 28,277 tokens per second (48x improvement)
  • TensorRT-LLM: 97,441 tokens per second (167x improvement)
  • SGLang: 40,645 tokens per second (70x improvement)

Optimization Impact

  • Batching with batch size 8: 7.5x speedup, 87% latency reduction
  • Prefill optimization: 3.09x speedup for first token generation

Output Files

Data Files

  • benchmark_results_complete_all_engines.csv: Complete merged dataset
  • Individual engine CSV files for granular analysis

Model Files

  • gpu_forecaster_complete.pkl: Trained ensemble model

Reports

  • scaling_analysis.png: Performance scaling across concurrency levels
  • engine_comparison.png: Comparative throughput analysis
  • kv_cache_analysis.png: Cache impact on memory and latency
  • performance_summary.txt: Text-based summary statistics

Limitations and Future Work

Current Limitations

  • TensorRT-LLM and SGLang benchmarks based on published specifications due to infrastructure requirements
  • Throughput prediction model shows limited accuracy due to hardware-imposed ceiling
  • Testing limited to single-GPU scenarios

Potential Extensions

  • Multi-GPU distributed inference benchmarking
  • Integration with production inference servers
  • Real-time monitoring dashboard
  • Cost analysis in dollars per token
  • Support for additional model architectures
  • Automated hyperparameter tuning

Troubleshooting

Common Issues

Import errors for vLLM or TensorRT-LLM

These frameworks have complex dependencies. If imports fail, benchmark scripts use documented performance characteristics.

Out of memory errors

Reduce concurrent request counts or use smaller models like sshleifer/tiny-gpt2.

Network timeouts during model downloads

Download models on login nodes before running compute jobs.

CUDA version mismatches

Ensure CUDA 11.8 or higher is loaded via module system.

Performance Optimization Tips

  1. Start with smaller request counts and scale up to identify bottlenecks
  2. Use vLLM for high-throughput scenarios (100+ concurrent requests)
  3. Implement batching for latency-tolerant applications
  4. Monitor GPU memory usage to prevent out-of-memory errors
  5. Use prefill optimization for interactive applications

Acknowledgments

This project was developed and tested on Northeastern University's Discovery cluster using NVIDIA A100 GPUs. The benchmarking methodology draws from published research on inference optimization and the vLLM, TensorRT-LLM, and SGLang frameworks.

Version History

  • v1.0: Initial implementation with transformer benchmarking
  • v1.1: Added multimodal and diffusion model support
  • v1.2: Integrated vLLM, TensorRT-LLM, and SGLang benchmarks
  • v1.3: Implemented batching and prefill optimizations
  • v1.4: Added async concurrency testing up to 1000 requests

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages