fix: E2E improvements and pending-aware saturation scaling

[clubanderson]

Summary

This PR adds pending-aware scaling to prevent cascade scale-ups and fixes E2E test stability issues.

Changes

Pending-Aware Saturation Scaling

• Add PendingReplicas field to VariantState interface to track pods that are starting but not yet ready
• Detect pending replicas by comparing spec.replicas vs status.readyReplicas in the saturation engine
• Skip additional scale-up decisions when pending replicas exist, preventing cascade scaling that wastes resources
• Prevents the autoscaler from repeatedly triggering scale-ups while pods are still loading models

Namespace Collision Fix

• Fix model grouping in saturation engine to include namespace in the group key
• Prevents incorrect metric aggregation when the same model is deployed in multiple namespaces

E2E Test Stability

• Add deployment stability check before starting load generation tests
• Wait for spec.replicas == status.replicas == status.readyReplicas (no pods in transition)
• Remove workaround that dynamically adjusted baseline during tests
• Ensures tests measure scale-up from a truly stable baseline state

Kind Emulator Improvements

• Add utility functions for running kubectl commands and getting pod metrics
• Improve test infrastructure reliability

Test Plan

• Unit tests pass (make test)
• Local E2E tests pass (make test-e2e)
• OpenShift E2E tests pass (CI)
• Verified pending-aware logic triggers correctly under load
• Verified deployment stability check prevents race conditions

---

EPP Pod Readiness Analysis

Analysis of the EPP (Endpoint Picker) codebase to validate the pending-aware scaling heuristic.

Key Finding: EPP Only Routes to Ready Pods

1. Pod Readiness Check (pkg/epp/util/pod/pod.go)

func IsPodReady(pod *corev1.Pod) bool {
    if !pod.DeletionTimestamp.IsZero() {
        return false  // Being deleted = not ready
    }
    // Check for PodReady condition = True
    for _, condition := range pod.Status.Conditions {
        if condition.Type == corev1.PodReady {
            return condition.Status == corev1.ConditionTrue
        }
    }
    return false
}

2. Pod Reconciler (pkg/epp/controller/pod_reconciler.go:91-102)

func (c *PodReconciler) updateDatastore(logger logr.Logger, pod *corev1.Pod) {
    if !podutil.IsPodReady(pod) || !c.Datastore.PoolLabelsMatch(pod.Labels) {
        c.Datastore.PodDelete(pod.Name)  // NOT READY → REMOVE from routing pool
    } else {
        c.Datastore.PodUpdateOrAddIfNotExist(pod)  // READY → ADD to routing pool
    }
}

3. Pod Resync (pkg/epp/datastore/datastore.go:330-334)

for _, pod := range podList.Items {
    if !podutil.IsPodReady(&pod) {
        continue  // Skip non-ready pods entirely
    }
    // ... add to datastore
}

Implications for WVA Pending-Aware Scaling

EPP Behavior	WVA Impact
Non-ready pods are excluded from routing pool	Traffic only goes to ready pods
Pending pods receive zero traffic	Queue builds on ready pods only
Pod becomes ready → immediately added to pool	New capacity comes online instantly

Validation of WVA Heuristic

Our pending-aware scaling is correct because:

1. EPP won't route to pending pods - So when WVA sees queue buildup, it's only on ready pods
2. Scaling up while pending exists is wasteful - The pending pod will add capacity once ready, but EPP can't use it yet
3. Waiting for pending→ready is the right call - Once ready, EPP immediately routes traffic, reducing queue pressure

Potential edge case to consider:

• If a pending pod takes too long to become ready (model loading), the ready pods may become severely overloaded
• WVA could consider a timeout: "if pending for >X minutes, scale up anyway"

Bottom line: EPP's behavior validates our pending-aware heuristic. EPP only routes to ready pods, so waiting for pending pods to become ready before scaling further is correct.

[Copilot]

Pull request overview

This PR fixes a label mismatch between vLLM pods and the WVA ServiceMonitor selector by ensuring the llm-d.ai/model label is set to match the deployment name during E2E setup.

• Updates the deploy_llm_d_infrastructure() function to set the model label via yq before deploying llm-d components
• Addresses a default label value issue in the llm-d-modelservice chart that prevented proper ServiceMonitor selection

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[asm582]

Address review

[asm582]

Need design input

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

[asm582]

Please add PR link to codebase and let all E2Es pass.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.