Skip to content

Scale-to-zero enforcement with e2e tests and HPA configuration - #585

Merged
ev-shindin merged 18 commits into
llm-d:mainfrom
ev-shindin:feature/scale-to-zero-enforcement
Jan 21, 2026
Merged

Scale-to-zero enforcement with e2e tests and HPA configuration#585
ev-shindin merged 18 commits into
llm-d:mainfrom
ev-shindin:feature/scale-to-zero-enforcement

Conversation

@ev-shindin

@ev-shindin ev-shindin commented Jan 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • Add scale-to-zero enforcer package (internal/scaletozero/) that handles retention period-based scaling decisions
  • Move scale-to-zero query registration from collector to dedicated package
  • Enable HPAScaleToZero feature gate configuration in Kind cluster setup
  • Make HPA minReplicas configurable via Helm values and install script
  • Add HPAScaleToZero feature gate detection for OpenShift clusters
  • Create comprehensive e2e tests for both Kind and OpenShift environments
  • Add separate make targets for running scale-to-zero tests without impacting other e2e tests
  • Enable scale-to-zero e2e tests in Kind CI workflow

Changes

New Package: internal/scaletozero/

  • enforcer.go: Core enforcement logic with retention period tracking
  • queries.go: Prometheus query registration (moved from collector)
  • Comprehensive unit tests

HPA Configuration

  • Helm chart: hpa.minReplicas is now configurable (default: 1 for safety)
  • Install script: HPA_MIN_REPLICAS environment variable support
  • Kind setup: Automatic HPAScaleToZero feature gate enablement via ENABLE_SCALE_TO_ZERO

E2E Tests

  • Kind: test/e2e-saturation-based/e2e_scale_to_zero_test.go
  • OpenShift: test/e2e-openshift/scale_to_zero_test.go
  • Both tests are isolated via Ginkgo focus patterns

Documentation

  • Updated docs/integrations/hpa-integration.md with OpenShift-specific instructions

Test plan

  • Run make test to verify unit tests pass
  • Run make test-e2e-scale-to-zero to test Kind scale-to-zero behavior
  • Run make test-e2e-openshift-scale-to-zero on OpenShift cluster with HPAScaleToZero enabled
  • Verify existing e2e tests (make test-e2e-saturation-based) still pass

Notes

  • HPAScaleToZero is an alpha Kubernetes feature requiring explicit enablement
  • OpenShift requires cluster admin to modify the FeatureGate CR
  • Tests gracefully skip if the feature gate is not enabled

Commits

  1. feat(scale-to-zero): add enforcer package and e2e tests
  2. feat(hpa): enable HPAScaleToZero feature gate and configurable minReplicas
  3. feat(openshift): add HPAScaleToZero feature gate detection and docs
  4. feat(ci): add separate make targets for scale-to-zero e2e tests
  5. ci: enable scale-to-zero e2e tests in Kind CI workflow

@ev-shindin
ev-shindin force-pushed the feature/scale-to-zero-enforcement branch from 5d51723 to d6b5a56 Compare January 14, 2026 13:52
@clubanderson

clubanderson commented Jan 14, 2026

Copy link
Copy Markdown
Contributor

/ok-to-test

@github-actions

github-actions Bot commented Jan 14, 2026

Copy link
Copy Markdown
Contributor

🚀 E2E tests triggered by /ok-to-test

View the OpenShift E2E workflow run

@ev-shindin
ev-shindin force-pushed the feature/scale-to-zero-enforcement branch 2 times, most recently from 3375c15 to 380466a Compare January 14, 2026 15:37
lionelvillard
lionelvillard reviewed Jan 15, 2026
View reviewed changes
Comment thread internal/engines/scaletozero/enforcer.go Outdated
lionelvillard
lionelvillard reviewed Jan 15, 2026
View reviewed changes
Comment thread internal/engines/saturation/engine.go Outdated
lionelvillard
lionelvillard reviewed Jan 15, 2026
View reviewed changes
Comment thread internal/scaletozero/queries.go Outdated
lionelvillard
lionelvillard reviewed Jan 15, 2026
View reviewed changes
Comment thread internal/scaletozero/enforcer.go Outdated
lionelvillard
lionelvillard reviewed Jan 15, 2026
View reviewed changes
Comment thread internal/engines/pipeline/enforcer.go
lionelvillard
lionelvillard reviewed Jan 15, 2026
View reviewed changes
Comment thread internal/engines/pipeline/enforcer.go
@ev-shindin
ev-shindin force-pushed the feature/scale-to-zero-enforcement branch 7 times, most recently from 77ce83a to ca362cb Compare January 19, 2026 14:24
@ev-shindin
feat(scale-to-zero): add enforcer for scale-to-zero policy enforcement
accc732 
Implement scale-to-zero enforcement logic:

- Add Enforcer in internal/engines/pipeline to enforce scale-to-zero
  policy based on ConfigMap settings and request count metrics
- Update CRD to allow numReplicas=0 in DesiredOptimizedAlloc status
- Add CollectModelRequestCount in collector to query request metrics
  for retention period checking
- Integrate enforcer into saturation engine pipeline
- Return errors from collector when metrics unavailable to prevent
  premature scale-to-zero decisions
@ev-shindin
test(unit): add unit tests for scale-to-zero enforcer and collector
026b014 
Add comprehensive unit tests for scale-to-zero functionality:

- Add tests for Enforcer in internal/engines/pipeline covering:
  - Scale-to-zero enabled/disabled scenarios
  - Retention period enforcement
  - Metrics unavailability handling
- Add tests for CollectModelRequestCount in collector/registration
  covering error cases when metrics are unavailable
@ev-shindin
ev-shindin force-pushed the feature/scale-to-zero-enforcement branch 3 times, most recently from 19fd48a to 8e30123 Compare January 19, 2026 14:57
@ev-shindin ev-shindin self-assigned this Jan 19, 2026
@ev-shindin
ev-shindin force-pushed the feature/scale-to-zero-enforcement branch from 8e30123 to fabfb0b Compare January 19, 2026 15:48
@ev-shindin
ev-shindin force-pushed the feature/scale-to-zero-enforcement branch 4 times, most recently from f801ef0 to 7da1f1d Compare January 20, 2026 10:23
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread Makefile Outdated
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread internal/collector/registration/scale_to_zero.go Outdated
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread internal/collector/registration/scale_to_zero.go Outdated
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread internal/collector/registration/scale_to_zero.go Outdated
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread internal/collector/registration/scale_to_zero.go Outdated
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread internal/collector/registration/scale_to_zero.go
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread internal/engines/pipeline/enforcer.go Outdated
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread internal/engines/pipeline/enforcer.go Outdated
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread test/e2e-openshift/e2e_suite_test.go Outdated
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread test/e2e-openshift/scale_to_zero_test.go
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread test/utils/e2eutils.go Outdated
@clubanderson clubanderson mentioned this pull request Jan 20, 2026
Merged
3 tasks
@ev-shindin
deploy: default BENCHMARK_MODE to false
19c4595 
Change the default value of BENCHMARK_MODE from "true" to "false" and
add clarification that istioBench env is required in helmfile when
enabling benchmark mode.

Also update the CI e2e OpenShift workflow to explicitly set
BENCHMARK_MODE=false since the istioBench environment is not available
in the llm-d helmfile.
lionelvillard
lionelvillard reviewed Jan 20, 2026
View reviewed changes
Comment thread .github/workflows/ci-e2e-openshift.yaml
@clubanderson clubanderson mentioned this pull request Jan 20, 2026
Merged
@ev-shindin
refactor: address review comments for scale-to-zero PR
5da44e3 
- Change logging levels from DEBUG to VERBOSE for error conditions
  in scale_to_zero.go (keep DEBUG for "no values" case)
- Rename RequestCountFunc to RequestCountFuncType for clarity
- Move isHPAScaleToZeroEnabled to dedicated feature_gates_test.go
- Extract DumpControllerLogs and DumpVAStatus to debug_helpers.go
- Use saturation.DefaultVariantCost instead of local constant in enforcer
@ev-shindin
ev-shindin force-pushed the feature/scale-to-zero-enforcement branch from a5fb1fa to 5da44e3 Compare January 20, 2026 19:02
@ev-shindin
test: run all e2e tests by default and skip scale-to-zero if feature …
63cf440 
…gate disabled

- Move IsHPAScaleToZeroEnabled to shared test/utils/feature_gates.go
- Update OpenShift tests to use shared function
- Skip deployment scale-to-zero test if HPAScaleToZero feature gate not enabled
- Remove default focus on "Saturation Mode" from test-e2e target to run all tests
@ev-shindin
ci: run e2e tests in parallel with scale-to-zero enabled and disabled
6a7f2e0 
Use matrix strategy to run e2e tests on two Kind clusters concurrently:
- One with HPAScaleToZero feature gate enabled
- One with HPAScaleToZero feature gate disabled

This ensures tests pass regardless of the feature gate setting and
validates behavior in both configurations.
@ev-shindin
test: use unique app label for scale-to-zero test
943a70f 
- Use distinct name 'llm-d-sim-stz' for scale-to-zero test resources
  to avoid conflicts with saturation test resources that use 'llm-d-sim'
- Skip scale-to-zero tests that require minReplicas=0 when HPAScaleToZero
  feature gate is not enabled
- Use minReplicas=1 in Scale-to-Zero Disabled test when feature gate
  is not available
@ev-shindin
test: fix Eventually assertions in TargetResolved tests
35f8ad9 
Use func(g Gomega) pattern with g.Expect() instead of func() with
Expect() inside Eventually blocks. The previous pattern caused tests
to fail immediately on first check instead of retrying.
@ev-shindin
test: increase CI timeout and skip retention period test when feature…
f800a88 
… gate disabled

- Increase e2e-tests job timeout from default 30m to 60m
- Skip retention period test when HPAScaleToZero feature gate is not
  enabled since it depends on scale-to-zero test completing first
@ev-shindin
test: improve HPAScaleToZero feature gate detection
6030084 
- Try multiple label selectors for kube-controller-manager pod
- Check both Command and Args for feature gate flag
- Fall back to finding pod by name prefix if label selectors fail
- This fixes false negatives when running on Kind clusters
@ev-shindin
test: increase e2e test timeout and remove redundant targets
e9cd65a
@ev-shindin
test: wait for load generator to be ready before stability check
b80a79c
@ev-shindin
test: enable KV cache metrics in llm-d-sim for e2e tests
c7009d7 
Without the --enable-kvcache flag, llm-d-sim doesn't emit
vllm:kv_cache_usage_perc metrics, causing the autoscaler to
default to 0% usage and trigger inappropriate scale-downs.

Enable KV cache with --enable-kvcache, --kv-cache-size=1024,
and --block-size=16 flags.
lionelvillard
lionelvillard approved these changes Jan 21, 2026
View reviewed changes

@lionelvillard lionelvillard left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a minor comment. LGTM
@thanks @ev-shindin !

Comment thread internal/engines/pipeline/enforcer.go
ctx context.Context,
modelID string,
namespace string,
saturationTargets map[string]int,

@lionelvillard lionelvillard Jan 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe create a type alias for map[string]int. Like this it's not clear what string and int represent.

@ev-shindin
ev-shindin merged commit 134dde5 into llm-d:main Jan 21, 2026
4 checks passed
This was referenced Feb 14, 2026
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lionelvillard lionelvillard lionelvillard approved these changes

Assignees

@ev-shindin ev-shindin

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ev-shindin @clubanderson @lionelvillard