Skip to content

Adding support for G4 nvidia-rtx-6000 GPUs for vLLM inference-ref-arch #345

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
syeda-anjum merged 17 commits into from
Jan 15, 2026
Merged

Adding support for G4 nvidia-rtx-6000 GPUs for vLLM inference-ref-arch #345

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
94c44fa
Enable G4 with Qwen on vllm
syeda-anjum Dec 16, 2025
1f70dbb
update tf for g4
syeda-anjum Dec 17, 2025
e660c63
update RTX Pro 6000 instead of G4 naming
syeda-anjum Dec 18, 2025
fcf771b
add more model support for RTX PRO 6000
syeda-anjum Dec 18, 2025
3c42563
Merge branch 'main' into sanjum-g4-gpus
syeda-anjum Dec 18, 2025
d311566
update readme
syeda-anjum Dec 18, 2025
7317f20
Changing GPU names to rtx-pro-6000 instead of rtxpro6000
syeda-anjum Dec 18, 2025
0a81854
Update platforms/gke/base/core/custom_compute_class/templates/manifes…
syeda-anjum Dec 19, 2025
dd54a09
Update platforms/gke/base/core/custom_compute_class/templates/manifes…
syeda-anjum Dec 19, 2025
5870b7d
Adding DWS and updating rtx-pro-6000 compute class naming by machine …
syeda-anjum Jan 6, 2026
be2d85b
adding support for llama models on RTX PRO 6000
syeda-anjum Jan 6, 2026
7989468
use NCCL P2P feature when G4 RTX-PRO-6000 is 2 or more GPUs
syeda-anjum Jan 6, 2026
307bcf8
Merge branch 'main' into sanjum-g4-gpus
syeda-anjum Jan 6, 2026
95027b0
adjust gpu count in compute class
syeda-anjum Jan 6, 2026
54b5329
update max-model-len to more convervative value to serve more users p…
syeda-anjum Jan 14, 2026
8052d36
fix read me
syeda-anjum Jan 14, 2026
c0bc920
Merge branch 'main' into sanjum-g4-gpus
syeda-anjum Jan 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions ...ke/base/use-cases/inference-ref-arch/online-inference-gpu/vllm-with-hf-model.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,15 +162,15 @@ This example is built on top of the

- Select an accelerator.

| Model | l4 | h100 | h200 |
| ------------------------------ | --- | ---- | ---- |
| gemma-3-1b-it | ✅ | ❌ | ❌ |
| gemma-3-4b-it | ✅ | ❌ | ❌ |
| gemma-3-27b-it | ✅ | ✅ | ✅ |
| gpt-oss-20b | ✅ | ✅ | ✅ |
| llama-3.3-70b-instruct | ❌ | ✅ | ✅ |
| llama-4-scout-17b-16e-instruct | ❌ | ✅ | ✅ |
| qwen3-32b | ✅ | ✅ | ✅ |
| Model | l4 | h100 | h200 | g4 |
| ------------------------------ | --- | ---- | ---- | --- |
| gemma-3-1b-it | ✅ | ❌ | ❌ | |
| gemma-3-4b-it | ✅ | ❌ | ❌ | |
| gemma-3-27b-it | ✅ | ✅ | ✅ | |
| gpt-oss-20b | ✅ | ✅ | ✅ | |
| llama-3.3-70b-instruct | ❌ | ✅ | ✅ | |
| llama-4-scout-17b-16e-instruct | ❌ | ✅ | ✅ | |
| qwen3-32b | ✅ | ✅ | ✅ | ✅ |

- **NVIDIA Tesla L4 24GB**:

Expand All @@ -190,6 +190,12 @@ This example is built on top of the
export ACCELERATOR_TYPE="h200"
```

- **NVIDIA RTX 6000 180GB**:

```shell
export ACCELERATOR_TYPE="g4"
```

Ensure that you have enough quota in your project to provision the selected
accelerator type. For more information, see about viewing GPU quotas, see
[Allocation quotas: GPU quota](https://cloud.google.com/compute/resource-usage#gpu_quota).
Expand Down
Loading
Loading