🐛 Fix CI E2E: clean up orphaned cluster-scoped WVA resources by clubanderson · Pull Request #715 · llm-d/llm-d-autoscaling

clubanderson · 2026-02-13T04:02:18Z

Summary

Adds orphan detection for cluster-scoped WVA resources (ClusterRoles, ClusterRoleBindings) whose owning namespace no longer exists
These stale resources block fresh helm installs because Helm refuses to adopt resources owned by a different release
Same fix as llm-d/llm-d-infra#10 (nightly workflow), applied to the CI E2E workflow

Root Cause

When a WVA installation's namespace is deleted (e.g., failed CI run cleanup, manual deletion), cluster-scoped resources like ClusterRoles are left behind. The next CI run tries to create the same ClusterRole, but Helm rejects it:

ClusterRole "workload-variant-autoscaler-metrics-auth-role" exists and cannot be imported
into the current release: invalid ownership metadata; key "meta.helm.sh/release-namespace"
must equal "llm-d-nightly-wva": current value is "llm-d-autoscaler"

Changes

Pre-cleanup (before deploy):

Scans all WVA-labeled ClusterRoles/ClusterRoleBindings
For each, checks if the owning namespace (from meta.helm.sh/release-namespace) still exists
Deletes only orphans where the namespace is gone — active installations are never touched

Post-cleanup (after tests):

Existing: deletes cluster-scoped resources by app.kubernetes.io/instance label (matches WVA controller release)
New: also deletes cluster-scoped resources owned by this PR's namespaces (covers helmfile-created resources with different instance labels)

Test plan

CI E2E passes on this PR
Verify other WVA installations on the cluster are not affected
Verify orphaned resources from deleted namespaces are cleaned up

github-actions · 2026-02-13T04:04:44Z

GPU Pre-flight Check ✅

GPUs are available for e2e-openshift tests. Proceeding with deployment.

Resource	Total	Allocated	Available
GPUs	50	32	18

Cluster	Value
Nodes	16 (7 with GPUs)
Total CPU	993 cores
Total Memory	10383 Gi
GPUs required	4 (min) / 6 (recommended)

Copilot

Pull request overview

Updates the OpenShift CI E2E workflow to proactively remove stale, cluster-scoped Workload Variant Autoscaler (WVA) RBAC resources that can block subsequent Helm installs when their owning namespace has been deleted.

Changes:

Add a pre-deploy scan that deletes WVA-labeled ClusterRoles/ClusterRoleBindings whose meta.helm.sh/release-namespace no longer exists.
Extend post-test cleanup to also delete cluster-scoped WVA resources annotated as owned by this PR’s namespaces (in addition to the existing instance-label-based deletion).

Copilot · 2026-02-13T04:07:13Z

+          echo "Checking for orphaned cluster-scoped WVA resources..."
+          for kind in clusterrole clusterrolebinding; do
+            for resource in $(kubectl get "$kind" -l app.kubernetes.io/name=workload-variant-autoscaler -o jsonpath='{range .items[*]}{.metadata.name}={.metadata.annotations.meta\.helm\.sh/release-namespace}{"\n"}{end}' 2>/dev/null); do
+              name="${resource%%=*}"
+              ns="${resource##*=}"
+              if [ -n "$ns" ] && ! kubectl get namespace "$ns" &>/dev/null; then
+                echo "  Deleting orphaned $kind/$name (owning namespace '$ns' no longer exists)"
+                kubectl delete "$kind" "$name" --ignore-not-found || true
+              fi
+            done
+          done
+


This orphan/namespace-owned cluster-scope cleanup logic is duplicated (same kubectl get ... jsonpath + parsing) in both pre-cleanup and post-cleanup sections. To reduce drift and make future changes safer, consider factoring it into a small helper script or a reusable step/function within the workflow.

Suggested change

echo "Checking for orphaned cluster-scoped WVA resources..."

for kind in clusterrole clusterrolebinding; do

for resource in $(kubectl get "$kind" -l app.kubernetes.io/name=workload-variant-autoscaler -o jsonpath='{range .items[*]}{.metadata.name}={.metadata.annotations.meta\.helm\.sh/release-namespace}{"\n"}{end}' 2>/dev/null); do

name="${resource%%=*}"

ns="${resource##*=}"

if [ -n "$ns" ] && ! kubectl get namespace "$ns" &>/dev/null; then

echo " Deleting orphaned $kind/$name (owning namespace '$ns' no longer exists)"

kubectl delete "$kind" "$name" --ignore-not-found || true

fi

done

done

cleanup_orphaned_wva_cluster_resources() {

echo "Checking for orphaned cluster-scoped WVA resources..."

for kind in clusterrole clusterrolebinding; do

for resource in $(kubectl get "$kind" -l app.kubernetes.io/name=workload-variant-autoscaler -o jsonpath='{range .items[*]}{.items[*]}{.metadata.name}={.metadata.annotations.meta\.helm\.sh/release-namespace}{"\n"}{end}' 2>/dev/null); do

name="${resource%%=*}"

ns="${resource##*=}"

if [ -n "$ns" ] && ! kubectl get namespace "$ns" &>/dev/null; then

echo " Deleting orphaned $kind/$name (owning namespace '$ns' no longer exists)"

kubectl delete "$kind" "$name" --ignore-not-found || true

fi

done

done

}

cleanup_orphaned_wva_cluster_resources

Copilot · 2026-02-13T04:07:14Z

+          for kind in clusterrole clusterrolebinding; do
+            for resource in $(kubectl get "$kind" -l app.kubernetes.io/name=workload-variant-autoscaler -o jsonpath='{range .items[*]}{.metadata.name}={.metadata.annotations.meta\.helm\.sh/release-namespace}{"\n"}{end}' 2>/dev/null); do
+              name="${resource%%=*}"
+              ns="${resource##*=}"
+              if [ "$ns" = "$LLMD_NAMESPACE" ] || [ "$ns" = "$LLMD_NAMESPACE_B" ] || [ "$ns" = "$WVA_NAMESPACE" ]; then
+                echo "  Deleting $kind/$name (owned by PR namespace '$ns')"
+                kubectl delete "$kind" "$name" --ignore-not-found || true
+              fi


Here too, kubectl get stderr is fully suppressed (2>/dev/null), which can mask listing failures and lead to incomplete cleanup while still printing success messages. Consider preserving errors (or checking exit status) so troubleshooting doesn’t require re-running with extra logging.

Copilot · 2026-02-13T04:07:14Z

+          for kind in clusterrole clusterrolebinding; do
+            for resource in $(kubectl get "$kind" -l app.kubernetes.io/name=workload-variant-autoscaler -o jsonpath='{range .items[*]}{.metadata.name}={.metadata.annotations.meta\.helm\.sh/release-namespace}{"\n"}{end}' 2>/dev/null); do
+              name="${resource%%=*}"
+              ns="${resource##*=}"
+              if [ -n "$ns" ] && ! kubectl get namespace "$ns" &>/dev/null; then
+                echo "  Deleting orphaned $kind/$name (owning namespace '$ns' no longer exists)"
+                kubectl delete "$kind" "$name" --ignore-not-found || true
+              fi


Both loops silence all kubectl get stderr (2>/dev/null), which can hide real failures (e.g., RBAC/auth issues or API errors) and make the orphan-cleanup appear to run while doing nothing. Consider handling the empty-result case without suppressing errors, or explicitly checking the command exit code and logging a warning/error when the list operation fails.

github-actions · 2026-02-13T04:20:01Z

GPU Pre-flight Check ✅

GPUs are available for e2e-openshift tests. Proceeding with deployment.

Resource	Total	Allocated	Available
GPUs	50	32	18

Cluster	Value
Nodes	16 (7 with GPUs)
Total CPU	993 cores
Total Memory	10383 Gi
GPUs required	4 (min) / 6 (recommended)

github-actions · 2026-02-13T04:25:29Z

GPU Pre-flight Check ✅

GPUs are available for e2e-openshift tests. Proceeding with deployment.

Resource	Total	Allocated	Available
GPUs	50	32	18

Cluster	Value
Nodes	16 (7 with GPUs)
Total CPU	993 cores
Total Memory	10383 Gi
GPUs required	4 (min) / 6 (recommended)

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

Copilot · 2026-02-13T04:26:11Z

+            # Search by name pattern (not labels — helmfile deployments may use different labels)
+            # Use jq to reliably extract annotation keys containing dots/slashes
+            kubectl get "$kind" -o json 2>/dev/null | \
+              jq -r '.items[] | select(.metadata.name | contains("workload-variant-autoscaler")) | "\(.metadata.name)\t\(.metadata.annotations["meta.helm.sh/release-namespace"] // "")"' 2>/dev/null | \


The jq filter uses contains("workload-variant-autoscaler") which will match any resource name containing this substring, potentially including unrelated resources if they happen to have similar names. Consider using a more specific filter such as startswith("workload-variant-autoscaler") or matching against specific known prefixes to reduce the risk of false positives.

Suggested change

jq -r '.items[] | select(.metadata.name | contains("workload-variant-autoscaler")) | "\(.metadata.name)\t\(.metadata.annotations["meta.helm.sh/release-namespace"] // "")"' 2>/dev/null | \

jq -r '.items[] | select(.metadata.name | startswith("workload-variant-autoscaler")) | "\(.metadata.name)\t\(.metadata.annotations["meta.helm.sh/release-namespace"] // "")"' 2>/dev/null | \

Copilot · 2026-02-13T04:26:11Z

+          # (covers helmfile-created resources whose instance label differs from WVA_RELEASE_NAME)
+          for kind in clusterrole clusterrolebinding; do
+            kubectl get "$kind" -o json 2>/dev/null | \
+              jq -r '.items[] | select(.metadata.name | contains("workload-variant-autoscaler")) | "\(.metadata.name)\t\(.metadata.annotations["meta.helm.sh/release-namespace"] // "")"' 2>/dev/null | \


The jq filter uses contains("workload-variant-autoscaler") which will match any resource name containing this substring, potentially including unrelated resources if they happen to have similar names. Consider using a more specific filter such as startswith("workload-variant-autoscaler") or matching against specific known prefixes to reduce the risk of false positives.

Suggested change

jq -r '.items[] | select(.metadata.name | contains("workload-variant-autoscaler")) | "\(.metadata.name)\t\(.metadata.annotations["meta.helm.sh/release-namespace"] // "")"' 2>/dev/null | \

jq -r '.items[] | select(.metadata.name | startswith("workload-variant-autoscaler-")) | "\(.metadata.name)\t\(.metadata.annotations["meta.helm.sh/release-namespace"] // "")"' 2>/dev/null | \

Copilot · 2026-02-13T04:26:12Z

+            kubectl get "$kind" -o json 2>/dev/null | \
+              jq -r '.items[] | select(.metadata.name | contains("workload-variant-autoscaler")) | "\(.metadata.name)\t\(.metadata.annotations["meta.helm.sh/release-namespace"] // "")"' 2>/dev/null | \
+              while IFS=$'\t' read -r name ns; do
+                if [ "$ns" = "$LLMD_NAMESPACE" ] || [ "$ns" = "$LLMD_NAMESPACE_B" ] || [ "$ns" = "$WVA_NAMESPACE" ]; then
+                  echo "  Deleting $kind/$name (owned by PR namespace '$ns')"
+                  kubectl delete "$kind" "$name" --ignore-not-found || true
+                fi
+              done


The jq command and loop logic at lines 806-813 are identical to lines 571-578 except for the namespace checking condition. Consider extracting this into a shell function to reduce code duplication and improve maintainability. For example, define a function cleanup_wva_resources() that takes the kind and a namespace-matching function as parameters.

The helmfile uses a generic release name "workload-variant-autoscaler" which produces non-unique ClusterRole names. On shared clusters, these may be owned by another namespace's release, causing Helm ownership conflicts. Pre-cleanup: adopt shared resources by patching Helm release annotations to our namespace. Post-cleanup: delete resources owned by our namespaces. Uses jq (not jsonpath) to handle annotation keys with dots/slashes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Andrew Anderson <andy@clubanderson.com>

github-actions · 2026-02-13T04:40:49Z

GPU Pre-flight Check ✅

GPUs are available for e2e-openshift tests. Proceeding with deployment.

Resource	Total	Allocated	Available
GPUs	96	63	33

Cluster	Value
Nodes	18 (12 with GPUs)
Total CPU	2721 cores
Total Memory	24307 Gi
GPUs required	4 (min) / 6 (recommended)

clubanderson · 2026-02-13T05:10:20Z

@lionelvillard @ev-shindin — This PR adds orphan ClusterRole cleanup to the CI E2E workflow, preventing Helm ownership conflicts when running on shared OpenShift clusters. Would appreciate a review/merge to unblock CI. Thanks!

clubanderson · 2026-02-13T14:14:07Z

Consolidated into #719

@main

- Fix CI E2E: clean up orphaned cluster-scoped WVA resources - Fix Helm chart: make ClusterRole names unique per namespace - Fix broken reusable workflow references (@2b273d6 → @main) Consolidates #715, #716, and #718. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@main

- Fix CI E2E: clean up orphaned cluster-scoped WVA resources - Fix Helm chart: make ClusterRole names unique per namespace - Fix broken reusable workflow references (@2b273d6 → @main) Consolidates #715, #716, and #718. Signed-off-by: Andrew Anderson <andy@clubanderson.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

@main

…-d#719) - Fix CI E2E: clean up orphaned cluster-scoped WVA resources - Fix Helm chart: make ClusterRole names unique per namespace - Fix broken reusable workflow references (@2b273d6 → @main) Consolidates llm-d#715, llm-d#716, and llm-d#718. Signed-off-by: Andrew Anderson <andy@clubanderson.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings February 13, 2026 04:02

Copilot started reviewing on behalf of clubanderson February 13, 2026 04:02 View session

Copilot AI reviewed Feb 13, 2026

View reviewed changes

clubanderson force-pushed the fix/ci-orphan-clusterrole-cleanup branch from 2e022e0 to f25d301 Compare February 13, 2026 04:16

Copilot AI review requested due to automatic review settings February 13, 2026 04:21

clubanderson force-pushed the fix/ci-orphan-clusterrole-cleanup branch from f25d301 to 923aa70 Compare February 13, 2026 04:21

Copilot started reviewing on behalf of clubanderson February 13, 2026 04:22 View session

Copilot AI reviewed Feb 13, 2026

View reviewed changes

clubanderson force-pushed the fix/ci-orphan-clusterrole-cleanup branch from 923aa70 to e3edd2c Compare February 13, 2026 04:37

clubanderson mentioned this pull request Feb 13, 2026

🐛 Fix ClusterRole cleanup, unique names, and workflow references #719

Merged

3 tasks

clubanderson closed this Feb 13, 2026

lionelvillard deleted the fix/ci-orphan-clusterrole-cleanup branch May 27, 2026 16:47

	jq -r '.items[] \| select(.metadata.name \| contains("workload-variant-autoscaler")) \| "\(.metadata.name)\t\(.metadata.annotations["meta.helm.sh/release-namespace"] // "")"' 2>/dev/null \| \
	jq -r '.items[] \| select(.metadata.name \| startswith("workload-variant-autoscaler")) \| "\(.metadata.name)\t\(.metadata.annotations["meta.helm.sh/release-namespace"] // "")"' 2>/dev/null \| \

Uh oh!

Conversation

clubanderson commented Feb 13, 2026

Summary

Root Cause

Changes

Test plan

Uh oh!

github-actions Bot commented Feb 13, 2026

GPU Pre-flight Check ✅

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 13, 2026

GPU Pre-flight Check ✅

Uh oh!

github-actions Bot commented Feb 13, 2026

GPU Pre-flight Check ✅

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 13, 2026

GPU Pre-flight Check ✅

Uh oh!

clubanderson commented Feb 13, 2026

Uh oh!

clubanderson commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants