Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GPU Inference Performance Optimization Engine

CUDA/C++ project for transformer-style inference performance: memory hierarchy modeling, kernel profiling/optimization, throughput benchmarking, and power-performance tradeoff analysis.

Requirements

  • CUDA Toolkit (11.x or 12.x)
  • C++17 compiler
  • Single GPU (no multi-GPU)

Build

CMake (recommended):

mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make

Makefile (if cmake not available):

make

Adjust CMAKE_CUDA_ARCHITECTURES in CMakeLists.txt or NVCCARCH in Makefile for your GPU (e.g. sm_75 for Turing, sm_86 for Ampere).

Run

./gpu_inference_benchmark

What the program does

  1. Transformer inference modeling
    Models memory hierarchy (global + shared), estimates bytes and FLOPs for GEMM and attention, and classifies regions as compute-bound vs memory-bound across batch sizes (arithmetic intensity vs threshold).

  2. Kernel profiling and optimization
    Runs baseline (16x16 tiles, no shared memory for GEMM; 16x16 attention tiles) and optimized (32x32 tiles, shared memory, coalesced access) inference, and reports end-to-end latency reduction (target ~18%).

  3. Throughput benchmarking
    Sweeps batch sizes and sequence lengths, reports latency, throughput (samples/s), kernel time, and memory bandwidth, and identifies bandwidth saturation (batch/seq where performance plateaus).

  4. Power-performance tradeoff
    Runs several parallelism configurations (batch/seq combinations), estimates relative power cost (active blocks, memory bandwidth proxy), and selects an optimal configuration (best throughput per estimated cost).

Example output (illustrative)

========== 1. Transformer Inference Modeling (Memory Hierarchy) ==========
Batch | SeqLen | GEMM_bytes | ATTN_bytes | GEMM_flops | Bound(GEMM) | Bound(ATTN)
    1 |    512 |    2097152 |    4194304 |   268435456 | ComputeBound | MemoryBound
    2 |    512 |    4194304 |    8388608 |   536870912 | ComputeBound | MemoryBound
   ...

Representative (batch=8, seq=512):
  GEMM: time=0.42 ms, memory_throughput=312.00 GB/s, compute_util=0.0523, ComputeBound
  ATTN: time=2.10 ms, memory_throughput=85.00 GB/s, compute_util=0.0121, MemoryBound

========== 2. Kernel Profiling and Optimization ==========
Baseline end-to-end latency:   2.850 ms
Optimized end-to-end latency:  2.340 ms
Latency reduction:             17.9% (target ~18%)

========== 3. Throughput Benchmarking ==========
...

========== 4. Power-Performance Tradeoff Analysis ==========
...
Optimal configuration (best throughput per estimated power): batch=8, seq_len=512

Actual numbers depend on GPU and driver. The ~18% improvement comes from larger tiles, shared memory, and better memory access patterns in the optimized kernels.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages