Skip to content

Azure matrix 1gpu #6298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
vofish wants to merge 52 commits into
base: master
Choose a base branch
from
Open

Azure matrix 1gpu #6298

vofish wants to merge 52 commits into from
+50 −0

Conversation

@vofish
Copy link
Collaborator

@vofish vofish commented Dec 15, 2025

Add support for kubernetes_scale to 1 GPU on AKS Standard:

  • Update container/kubernetes_scale/kubernetes_scale.yaml.j2 to intall the NVIDIA device plugin
  • Add cloud yaml_docs manifest

Command to run:

./pkb.py --cloud=Azure --benchmarks=kubernetes_scale \
--config_override='kubernetes_scale.container_cluster.vm_spec.Azure.zone="westus2"' \
--config_override='kubernetes_scale.container_cluster.type="Kubernetes"' \
--config_override=kubernetes_scale.container_cluster.max_vm_count=4 \
--config_override='kubernetes_scale.container_cluster.vm_spec.Azure.machine_type="Standard_NC8as_T4_v3"' \
--gpu_count=1 --gpu_type=t4 \
--kubernetes_scale_num_replicas=2 --kubernetes_scale_pod_cpus=4 --kubernetes_scale_pod_memory=4G \
--kubernetes_scale_report_individual_latencies=True --kubernetes_scale_report_latency_percentiles=False \
--metadata=cloud:Azure  --timeout_minutes=236

rsgowman
rsgowman reviewed Dec 22, 2025
View reviewed changes
Copy link

@rsgowman rsgowman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, though let's get Zach to review too.

@bvliu bvliu requested a review from hubatish January 5, 2026 15:37
hubatish
hubatish approved these changes Jan 6, 2026
View reviewed changes
Copy link
Collaborator

@hubatish hubatish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the tests are failing. In cloud Build logs (which you probably don't have access to?) I see:
INFO 2025-12-24T16:11:52.906172833Z Step #1: ERROR: testSetStorageSizeGCP (tests.disk_iops_to_capacity_test.DiskIOPSToCapacityTest.testSetStorageSizeGCP)
....
INFO 2025-12-24T16:11:52.906179565Z Step #1: self._cpu_count = int(
INFO 2025-12-24T16:11:52.906180155Z Step #1: ^^^^
INFO 2025-12-24T16:11:52.906180893Z Step #1: TypeError: only 0-dimensional arrays can be converted to Python scalars

..this doesn't look like your code. Possibly just syncing forward will fix it since this is 2 weeks old.

vofish and others added 22 commits January 7, 2026 16:05
@vofish
Put NVIDIA plugin yaml to data/container/azure folder
6173672
@vofish
Add precommit pyink formatter
dbe0c2b
@vofish
Add newline
7532b2c
@vofish
Update rev., add args
d84a68d
@vofish
Update pre-commit hook args
6baa2f7
@vofish
Update the Postgres sysbench configuration logic.
9ae39a3 
PiperOrigin-RevId: 844816397
@vofish
Remove the node selector for (new) vpa test
b957e2b 
This is no longer used and causes the test to fail as of
GoogleCloudPlatform#6291

PiperOrigin-RevId: 844830523
@hubatish @vofish
Fix Vertex AI DNS endpoint & remove beta
b2eb291 
model-garden CLI service is properly live now.

PiperOrigin-RevId: 844840901
@vofish
Fix to rampup test to ensure correct metric collector runs
be9f67a 
See GoogleCloudPlatform#6272. While
refactoring the Run() method, the KubernetesMetricCollector was omitted,
causing HPA's KMC to be run during VPA's test. This caused the primary metrics
of the test to not be collected.

PiperOrigin-RevId: 844844745
@vofish
Add Azure support to ai-inference
6909b40
@vofish
Address comments
843eb01
@vofish
Fix tests
21868f1
@vofish
Fix tests
c1ea5bb
@vofish
Add _ProvisionGPUNodePool method
ecf6162
@vofish
Revert blob-public-access
bc2f146
@vofish
Fix linting issues
f151363
@vofish
Fix linting issues
85b9239
@raymond13513 @vofish
Support triggering migration multiple times.
5868fe4 
PiperOrigin-RevId: 845340374
@ScottLinnn @vofish
Implement backup restore for DSQL
01373cf 
PiperOrigin-RevId: 845429012
@jacklacey11 @vofish
Update base Windows mixin to offer RemoteCommandWithReturnCode and …
16df725 
…the existing `RemoteCommand`.

PiperOrigin-RevId: 845429895
@hubatish @vofish
Run llama4 16e-instruct rather than 16e (without).
e719647 
PiperOrigin-RevId: 845508131
@andyz422 @vofish
Add '.j2' to Azure blobfuse2 config file.
2d49929 
PiperOrigin-RevId: 845833015
p3rf Team and others added 24 commits January 7, 2026 16:30
@vofish
Move container_service into it's own directory/module
589836c 
Preparation for refactoring this file.

PiperOrigin-RevId: 845900151
@vofish
Refactor kubernetes items out of container_service/__init__.py
6489719 
PiperOrigin-RevId: 845929610
@vofish
Refactor base items out of container_service/__init__.py
85464b9 
PiperOrigin-RevId: 845943103
@raymond13513 @vofish
Default capture_live_migration_timestamps to true.
b4f887a 
PiperOrigin-RevId: 845944984
@vofish
Refactor BUILD file to create container_service library
37d046e 
Also fix resulting pytype errors that this exposed.

PiperOrigin-RevId: 846252648
@hubatish @vofish
Move yaml code from container_service -> vm_util
305f713 
PiperOrigin-RevId: 846416972
@pmkc @vofish
Update Linux VM metadata to:
5239ccb 
1. Always show the kernel command line
2. Show if RT kernel was enabled

PiperOrigin-RevId: 846435753
@hubatish @vofish
Allow configuring worker count in Trino
a90ede8 
PiperOrigin-RevId: 846453023
@vofish
Support gs:// URLs in --ycsb_tar_url
b22e16d 
PiperOrigin-RevId: 846466524
@bvliu @vofish
Add support for specifying storage type, IOPS, and throughput for Azu…
eeb5acf 
…re SQL databases.

PiperOrigin-RevId: 846591475
@bvliu @vofish
Refactor relational DB metrics collection.
c5bbbd9 
PiperOrigin-RevId: 846606398
@bvliu @vofish
Add Azure Flexible Server metrics implementation.
84285f1 
PiperOrigin-RevId: 846734952
@bvliu @vofish
Fix sysbench sleep duration.
86f7c04 
PiperOrigin-RevId: 846749098
@vofish
Added GCE SQL Server 2025 images to PKB
0f93c5c 
PiperOrigin-RevId: 847010068
@jellyfishcake @vofish
Remove unused disk_iops_to_capacity module.
a0748fc 
PiperOrigin-RevId: 851449538
@vofish
Add the option to enable --redis_aof_verify at the end of Run pha…
536e4ca 
…se for Redis Memtier Benchmark

PiperOrigin-RevId: 851458383
@vofish
Add support for MSSQL 2025 on Linux and update SQL Server configurati…
f5a8ccc 
…on to include trace flag 9944.

PiperOrigin-RevId: 852305770
@ScottLinnn @vofish
Pass region to aws command
429e3b9 
PiperOrigin-RevId: 852383875
@raymond13513 @vofish
Support exporting metrics for multiple triggers for disruption.
002e015 
PiperOrigin-RevId: 852420089
@jacklacey11 @vofish
Reclassify timeouts on startup script retrieval to indicate the poten…
75a3d7c 
…tial source issue from the startup script service

PiperOrigin-RevId: 852450925
@Arushi-07 @vofish
Allow configurable compaction strategy for cassandra
a970e51 
PiperOrigin-RevId: 852495812
@vofish
Fix Create test
ee7dfd7
@vofish
Merge branch 'master' into azure-matrix-1gpu
5ba0a44
@vofish
Remove pre-commit hook
38ec118
rsgowman
rsgowman reviewed Jan 9, 2026
View reviewed changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hubatish hubatish hubatish approved these changes

+1 more reviewer

@rsgowman rsgowman rsgowman left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ready to pull

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.