CUDA/C++ project for transformer-style inference performance: memory hierarchy modeling, kernel profiling/optimization, throughput benchmarking, and power-performance tradeoff analysis.
- CUDA Toolkit (11.x or 12.x)
- C++17 compiler
- Single GPU (no multi-GPU)
CMake (recommended):
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
makeMakefile (if cmake not available):
makeAdjust CMAKE_CUDA_ARCHITECTURES in CMakeLists.txt or NVCCARCH in Makefile for your GPU (e.g. sm_75 for Turing, sm_86 for Ampere).
./gpu_inference_benchmark-
Transformer inference modeling
Models memory hierarchy (global + shared), estimates bytes and FLOPs for GEMM and attention, and classifies regions as compute-bound vs memory-bound across batch sizes (arithmetic intensity vs threshold). -
Kernel profiling and optimization
Runs baseline (16x16 tiles, no shared memory for GEMM; 16x16 attention tiles) and optimized (32x32 tiles, shared memory, coalesced access) inference, and reports end-to-end latency reduction (target ~18%). -
Throughput benchmarking
Sweeps batch sizes and sequence lengths, reports latency, throughput (samples/s), kernel time, and memory bandwidth, and identifies bandwidth saturation (batch/seq where performance plateaus). -
Power-performance tradeoff
Runs several parallelism configurations (batch/seq combinations), estimates relative power cost (active blocks, memory bandwidth proxy), and selects an optimal configuration (best throughput per estimated cost).
========== 1. Transformer Inference Modeling (Memory Hierarchy) ==========
Batch | SeqLen | GEMM_bytes | ATTN_bytes | GEMM_flops | Bound(GEMM) | Bound(ATTN)
1 | 512 | 2097152 | 4194304 | 268435456 | ComputeBound | MemoryBound
2 | 512 | 4194304 | 8388608 | 536870912 | ComputeBound | MemoryBound
...
Representative (batch=8, seq=512):
GEMM: time=0.42 ms, memory_throughput=312.00 GB/s, compute_util=0.0523, ComputeBound
ATTN: time=2.10 ms, memory_throughput=85.00 GB/s, compute_util=0.0121, MemoryBound
========== 2. Kernel Profiling and Optimization ==========
Baseline end-to-end latency: 2.850 ms
Optimized end-to-end latency: 2.340 ms
Latency reduction: 17.9% (target ~18%)
========== 3. Throughput Benchmarking ==========
...
========== 4. Power-Performance Tradeoff Analysis ==========
...
Optimal configuration (best throughput per estimated power): batch=8, seq_len=512
Actual numbers depend on GPU and driver. The ~18% improvement comes from larger tiles, shared memory, and better memory access patterns in the optimized kernels.