There are 2 instance pools: cpu-pool & gpu-pool. Only gpu-worker deployment runs in gpu-pool
This is main server to accept requests. This server redirects vllm request to gpu-pool. It should push latency to redis.
There multiple or single gpu-worker. There is one running gpu at least.
It is a thread that is running on app. It calculates average latency during the last 10 seconds. If average latency is bigger than threshold, it increase number of gpu-workers.