cdi-operator constantly updates the webhooks

[lennartmoeller]

What happened:

When we try to install the CDI on our AKS, the cdi-operator constantly updates the webhooks (about 10 times a second).

Sometimes it switches the ordering of the match expressions:

What you expected to happen:

Webhooks shoudn't get updated over and over again.

How to reproduce it (as minimally and precisely as possible):

We install the CDI ...

CDI_VERSION="1.62.0"
kubectl apply -f "https://github.com/kubevirt/containerized-data-importer/releases/download/v${CDI_VERSION}/cdi-operator.yaml"
kubectl apply -f "https://github.com/kubevirt/containerized-data-importer/releases/download/v${CDI_VERSION}/cdi-cr.yaml"

... and if we view the logs of the cdi-operator ...

OPERATOR_POD_NAME=$(kubectl get pods -n cdi --no-headers -o custom-columns=":metadata.name" | grep '^cdi-operator' | head -n 1)
kubectl logs -n cdi -f $OPERATOR_POD_NAME

... the following lines repeat over and over again:

{"level":"debug","ts":"2025-07-07T13:09:20Z","logger":"events","msg":"Successfully updated resource *v1.ValidatingWebhookConfiguration cdi-api-datavolume-validate","type":"Normal","object":{"kind":"CDI","name":"cdi","uid":"e3c61a64-19c5-4efb-b15e-293d4372d18c","apiVersion":"cdi.kubevirt.io/v1beta1","resourceVersion":"1130627"},"reason":"UpdateResourceSuccess"}
{"level":"info","ts":"2025-07-07T13:09:20Z","logger":"cdi-operator","msg":"DIFF","Request.Namespace":"","Request.Name":"cdi","obj":{"apiVersion":"admissionregistration.k8s.io/v1","kind":"MutatingWebhookConfiguration","name":"cdi-api-datavolume-mutate"},"patch":"[{\"op\":\"remove\",\"path\":\"/webhooks/0/namespaceSelector/matchExpressions\"}]"}
{"level":"info","ts":"2025-07-07T13:09:20Z","logger":"cdi-operator","msg":"Resource updated","Request.Namespace":"","Request.Name":"cdi","namespace":"","name":"cdi-api-datavolume-mutate","type":"*v1.MutatingWebhookConfiguration"}

We only have this issue on AKS. We can't reproduce this behavior on a local machine.

Additional context:

N/A

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml): v1.62.0
• Kubernetes version (use kubectl version): v1.32.4
• DV specification: N/A
• Cloud provider or hardware configuration: Azure, AKS
• OS (e.g. from /etc/os-release): N/A
• Kernel (e.g. uname -a): N/A
• Install tools: N/A
• Others: N/A

[lennartmoeller]

In the meantime we figured out why this is happening.

The AKS control plane runs a built-in “Admissions Enforcer” webhook that rewrites or removes parts of every third-party admission-webhook configuration. When the CDI operator writes its webhook objects the enforcer immediately rewrites the namespaceSelector.matchExpressions field; the operator’s reconciliation loop then sees the difference, rewrites the object, the enforcer rewrites it back, and so on.

See: https://learn.microsoft.com/en-us/azure/aks/faq#can-admission-controller-webhooks-affect-kube-system-and-internal-aks-namespaces-

We disabled it with adding the following annotation to the MutatingWebhookConfigurations and ValidatingWebhookConfigurations so that the admissions enforcer ignores it: "admissions.enforcer/disabled": true

Or as a complete command with label selector:

kubectl annotate mutatingwebhookconfiguration,validatingwebhookconfiguration -l cdi.kubevirt.io=cdi-api admissions.enforcer/disabled=true --overwrite

While it's not possible to configure this within the yaml files, we hacked our way around it with a Job. It would be great if there's an option in the yaml files in the future to set custom labels to MutatingWebhookConfigurations and ValidatingWebhookConfigurations created by the cdi-operator to avoid this problem.

[akalenyu]

This came up in the sig-storage meeting, we suspect that it wasn't our intention to opinionate this field to begin with

containerized-data-importer/pkg/operator/resources/cluster/apiserver.go

Line 462 in 8c03050

NamespaceSelector: &metav1.LabelSelector{},

But we'd have to double check the semantics. Does kubevirt/kubevirt work fine in this regard?

[akalenyu]

@lennartmoeller check #3841

[lennartmoeller]

Does kubevirt/kubevirt work fine in this regard?

I think so. kubevirt/kubevirt isn't creating any mutatingwebhookconfigurations/validatingwebhookconfigurations, so I don't see any issue there.

@lennartmoeller check #3841

Haven't build or tried that out, but if this stops setting the namespace selector, it may work.

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale