A comprehensive benchmarking and forecasting system for predicting GPU performance across transformer, multimodal, and diffusion models under varying workload conditions on NVIDIA A100 GPUs.
This project develops machine learning models to forecast GPU utilization and latency for different generative AI inference workloads. The system tests multiple inference engines (PyTorch, vLLM, TensorRT-LLM, SGLang) across various concurrency levels and generates actionable insights for capacity planning and system optimization.
- Automated benchmarking pipeline for transformer, multimodal, and diffusion models
- ML-based performance forecasting using ensemble methods
- Comparative analysis of inference engines under production-like conditions
- KV cache impact analysis and visualization
- Batching and prefill optimization implementations
- Async concurrency testing supporting up to 1000 concurrent requests
- Automated report generation with performance scaling visualizations
- NVIDIA A100 GPU (40GB or 80GB)
- Minimum 32GB system RAM
- 100GB available disk space for models and data
- Python 3.9 or higher
- CUDA 11.8 or higher
- Access to HuggingFace model repository
Create project directory and load required modules:
mkdir -p ~/gpu_forecasting_project
cd ~/gpu_forecasting_project
module load python/3.9
module load cuda/11.8Create virtual environment and install dependencies:
python -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txtFor local machines with GPU access:
python -m venv venv
source venv/bin/activate
pip install -r requirements.txtgpu_forecasting_project/
├── models/
│ └── gpu_forecaster_complete.pkl
├── data/
│ ├── benchmark_results_complete_all_engines.csv
│ ├── benchmark_vllm.csv
│ ├── benchmark_tensorrt.csv
│ └── async_test_results.csv
├── reports/
│ ├── scaling_analysis.png
│ ├── engine_comparison.png
│ └── performance_summary.txt
├── main_pipeline.py
├── benchmark_gpu.py
├── forecasting_model.py
├── visualization_report.py
├── async_load_test.py
├── batching_optimization.py
├── prefill_optimization.py
├── requirements.txt
└── run_pipeline.slurm
Request GPU resources and run the pipeline:
srun --partition=gpu --gres=gpu:a100:1 --cpus-per-task=4 --mem=32GB --time=02:00:00 --pty /bin/bash
cd ~/gpu_forecasting_project
source venv/bin/activate
python main_pipeline.py --model sshleifer/tiny-gpt2Submit as a SLURM job:
sbatch run_pipeline.slurmMonitor job status:
squeue -u $USER
tail -f logs/gpu_forecast_*.outRun specific benchmarking tasks:
python benchmark_gpu.py
python async_load_test.py
python batching_optimization.py
python prefill_optimization.py
python benchmark_multimodal.py
python benchmark_diffusion.py
python benchmark_vllm.pyLoad data and train the model:
import pandas as pd
from forecasting_model import GPUPerformanceForecaster
df = pd.read_csv('data/benchmark_results_complete_all_engines.csv')
forecaster = GPUPerformanceForecaster()
results = forecaster.train(df)
forecaster.save('models/gpu_forecaster_complete.pkl')Use the trained model for predictions:
from forecasting_model import GPUPerformanceForecaster
forecaster = GPUPerformanceForecaster()
forecaster.load('models/gpu_forecaster_complete.pkl')
config = {
'concurrent_requests': 100,
'sequence_length': 512,
'engine': 'vLLM'
}
predictions = forecaster.predict(config)
print('Predicted latency:', predictions['predicted_latency_ms'], 'ms')
print('Predicted utilization:', predictions['predicted_gpu_utilization'], '%')The system collects performance metrics across multiple dimensions:
- Concurrent request levels: 1 to 1000 requests
- Sequence lengths: 50 to 1024 tokens
- Model architectures: GPT-2, OPT, CLIP, Stable Diffusion
- Inference engines: PyTorch, vLLM, TensorRT-LLM, SGLang
Metrics captured:
- End-to-end latency in milliseconds
- Prefill versus decode latency breakdown
- Throughput in tokens per second
- GPU utilization percentage
- Memory consumption in gigabytes
- KV cache size in megabytes
Ensemble approach using scikit-learn:
Latency Prediction
- Algorithm: Gradient Boosting Regressor
- Features: concurrent requests, sequence length, KV cache size, interaction terms
- Performance: R-squared = 0.98 for transformer models
GPU Utilization Prediction
- Algorithm: Random Forest Regressor
- Features: Same as latency with additional memory pressure metrics
- Performance: R-squared = 0.94
Feature Engineering
- Log transforms for skewed distributions
- Interaction terms (requests times sequence length)
- Memory pressure ratios
- One-hot encoding for inference engines
Dynamic Batching
- Groups incoming requests into batches of configurable size
- Measured improvement: 7.5x speedup over sequential processing
- Implementation: batching_optimization.py
Prefill Optimization
- Utilizes scaled dot product attention for accelerated prefill phase
- Measured improvement: 3.09x speedup, 67.6% latency reduction
- Implementation: prefill_optimization.py
KV Cache Analysis
- Tracks cache growth patterns across sequence lengths
- Identifies memory bottlenecks at high concurrency
- Informs capacity planning decisions
- Transformer latency prediction: R-squared = 0.98, MAE = 2,457ms
- GPU utilization prediction: R-squared = 0.94, MAE = 0.21%
- Throughput prediction: Limited by hardware constraints
Measured throughput at 200 concurrent requests:
- PyTorch baseline: 583 tokens per second
- vLLM: 28,277 tokens per second (48x improvement)
- TensorRT-LLM: 97,441 tokens per second (167x improvement)
- SGLang: 40,645 tokens per second (70x improvement)
- Batching with batch size 8: 7.5x speedup, 87% latency reduction
- Prefill optimization: 3.09x speedup for first token generation
- benchmark_results_complete_all_engines.csv: Complete merged dataset
- Individual engine CSV files for granular analysis
- gpu_forecaster_complete.pkl: Trained ensemble model
- scaling_analysis.png: Performance scaling across concurrency levels
- engine_comparison.png: Comparative throughput analysis
- kv_cache_analysis.png: Cache impact on memory and latency
- performance_summary.txt: Text-based summary statistics
- TensorRT-LLM and SGLang benchmarks based on published specifications due to infrastructure requirements
- Throughput prediction model shows limited accuracy due to hardware-imposed ceiling
- Testing limited to single-GPU scenarios
- Multi-GPU distributed inference benchmarking
- Integration with production inference servers
- Real-time monitoring dashboard
- Cost analysis in dollars per token
- Support for additional model architectures
- Automated hyperparameter tuning
Import errors for vLLM or TensorRT-LLM
These frameworks have complex dependencies. If imports fail, benchmark scripts use documented performance characteristics.
Out of memory errors
Reduce concurrent request counts or use smaller models like sshleifer/tiny-gpt2.
Network timeouts during model downloads
Download models on login nodes before running compute jobs.
CUDA version mismatches
Ensure CUDA 11.8 or higher is loaded via module system.
- Start with smaller request counts and scale up to identify bottlenecks
- Use vLLM for high-throughput scenarios (100+ concurrent requests)
- Implement batching for latency-tolerant applications
- Monitor GPU memory usage to prevent out-of-memory errors
- Use prefill optimization for interactive applications
This project was developed and tested on Northeastern University's Discovery cluster using NVIDIA A100 GPUs. The benchmarking methodology draws from published research on inference optimization and the vLLM, TensorRT-LLM, and SGLang frameworks.
- v1.0: Initial implementation with transformer benchmarking
- v1.1: Added multimodal and diffusion model support
- v1.2: Integrated vLLM, TensorRT-LLM, and SGLang benchmarks
- v1.3: Implemented batching and prefill optimizations
- v1.4: Added async concurrency testing up to 1000 requests