kubeadm upgrade node seems to ignore UpgradeConfiguration.node.skipPhases

[nadiamoe]

What keywords did you search in kubeadm issues before filing this one?

skipPhases (https://github.com/kubernetes/kubeadm/issues?q=is%3Aissue+skipPhases)

Versions

kubeadm version (use kubeadm version):

Environment:

• Kubernetes version (use kubectl version):
  • Client Version: v1.32.0
  • Kustomize Version: v5.5.0
  • Server Version: v1.32.0
• Cloud provider or hardware configuration: Bare metal
• OS (e.g. from /etc/os-release): Arch Linux

What happened?

I have a simple config like this:

apiVersion: kubeadm.k8s.io/v1beta4
kind: UpgradeConfiguration
node:
  skipPhases:
    - addon/kube-proxy

However, when I run kubeadm --config /tmp/kubeadm-upgrade.yaml upgrade node, the command does not seem to be picking up skipPhases.

# cat /tmp/kubeadm-upgrade.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: UpgradeConfiguration
node:
  skipPhases:
    - addon/kube-proxy
# kubeadm --config /tmp/kubeadm-upgrade.yaml upgrade node
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
W0101 20:10:50.281670 4124985 configset.go:78] Warning: No kubeproxy.config.k8s.io/v1alpha1 config is loaded. Continuing without it: configmaps "kube-proxy" not found
[upgrade/preflight] Running pre-flight checks
[upgrade/preflight] Pulling images required for setting up a Kubernetes cluster
[upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection
[upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[upgrade/control-plane] Upgrading your Static Pod-hosted control plane instance to version "v1.32.0"...
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests1012888789"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Current and new manifests of kube-apiserver are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Current and new manifests of kube-controller-manager are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Current and new manifests of kube-scheduler are equal, skipping upgrade
[upgrade/control-plane] The control plane instance for this node was successfully upgraded!
[upgrade/kubeconfig] The kubeconfig files for this node were successfully upgraded!
W0101 20:10:53.963472 4124985 postupgrade.go:117] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config3233041470 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config3233041470/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Running kubeadm --config /etc/kubernetes/kubeadm.yaml upgrade node --skip-phases addon/kube-proxy does work.

Adding a different option, like dryRun: true to the yaml also works, which seems to indicate this file is actually being read:

# cat /tmp/kubeadm-upgrade.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: UpgradeConfiguration
node:
  dryRun: true
  skipPhases:
    - addon/kube-proxy
# kubeadm --config /tmp/kubeadm-upgrade.yaml upgrade node | head -n 20
[dryrun] Creating a real client from "/etc/kubernetes/admin.conf"
...

I cannot help but pay attention at:

[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.

Which may hint that my UpgradeConfiguration needs to be in-cluster, kubeadm init phase upload-config all does not seem to like it:

# kubeadm --config /tmp/kubeadm-upgrade.yaml init phase upload-config all
W0101 20:27:47.313269 4127138 initconfiguration.go:361] [config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta4, Kind=UpgradeConfiguration
no InitConfiguration or ClusterConfiguration kind was found in the YAML file
To see the stack trace of this error execute with --v=5 or higher

What you expected to happen?

kubeadm upgrade node should have honored UpgradeConfiguration.node.skipPhases and not deployed kube-proxy.

How to reproduce it (as minimally and precisely as possible)?

See above.

Anything else we need to know?

[SataQiu]

If you specify the --skip-phases through the command line, the command line arguments will take precedence over the configuration file.

[neolit123]

Which may hint that my UpgradeConfiguration needs to be in-cluster, kubeadm init phase upload-config all does not seem to like it:

we don't have plans to store upgradeconfiguration in the cluster. only clusterconfiguration.

Running kubeadm --config /etc/kubernetes/kubeadm.yaml upgrade node --skip-phases addon/kube-proxy does work.

However, when I run kubeadm --config /tmp/kubeadm-upgrade.yaml upgrade node, the command does not seem to be picking up skipPhases.

@SataQiu could there be a bug that skipPhases from upgradeconfiguration doesn't get propagated correctly?

[SataQiu]

/assign
i will check it soon

[SataQiu]

This is a bug, I will submit a PR to fix it.
One workaround is to use --skip-phases:

# kubeadm --config /tmp/kubeadm-upgrade.yaml upgrade node --skip-phases addon/kube-proxy
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
...
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[addons] Applied essential addon: CoreDNS

[SataQiu]

will fixed by kubernetes/kubernetes#129452

PTAL @neolit123 @nadiamoe

[neolit123]

@SataQiu i can send the cherry pick prs.

[nadiamoe]

Huge thanks for the light-speed response folks!

[SataQiu]

@SataQiu i can send the cherry pick prs.

Welcome to submit the cherry pick prs if you have time @neolit123

[neolit123]

here they are:

• Automated cherry pick of #129452 kubeadm: fix a bug where the node.skipPhases in UpgradeNodeConfigurat… kubernetes#129455
• Automated cherry pick of #129452 kubeadm: fix a bug where the node.skipPhases in UpgradeNodeConfigurat… kubernetes#129454

[neolit123]

@SataQiu please LGTM kubernetes/kubernetes#129454 as well
(you LGTM-ed kubernetes/kubernetes#129455 twice)