Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubeadm upgrade ignores ClusterConfiguration.proxy.disabled field #3137

Closed
rkojedzinszky opened this issue Dec 27, 2024 · 9 comments · Fixed by kubernetes/kubernetes#129418
Closed

kubeadm upgrade ignores ClusterConfiguration.proxy.disabled field #3137

rkojedzinszky opened this issue Dec 27, 2024 · 9 comments · Fixed by kubernetes/kubernetes#129418
Assignees
@neolit123
Labels
area/addons kind/bug Categorizes issue or PR as related to a bug. priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@rkojedzinszky
Copy link

What happened?

Upgraded my cluster from 1.31 to 1.32. I have removed kube-proxy from my environment as kube-router handled service proxy functionality. When kubeadm upgrade finished, I've noticed that kube-proxy has been deployed. Unfortunately, it conflicted with kube-router, caused some service outage.

What did you expect to happen?

Expected that kube-proxy being not redeployed.

How can we reproduce it (as minimally and precisely as possible)?

Even on a fresh 1.32.0 install, edit configmap kube-system/kubeadm-config, set proxy.disabled as documented. To verify the configuration file to be used during upgrade, run:

$ kubeadm upgrade apply --print-config plan

It shows that proxy has been disabled:

# kubeadm upgrade apply --print-config plan
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
W1227 21:57:21.759438    5283 configset.go:78] Warning: No kubeproxy.config.k8s.io/v1alpha1 config is loaded. Continuing without it: configmaps "kube-proxy" not found
[upgrade/config] Configuration used:
	apiServer:
	  extraArgs:
	  - name: authorization-mode
	    value: Node,RBAC
	apiVersion: kubeadm.k8s.io/v1beta4
	caCertificateValidityPeriod: 87600h0m0s
	certificateValidityPeriod: 8760h0m0s
	certificatesDir: /etc/kubernetes/pki
	clusterName: kubernetes
	controllerManager: {}
	dns: {}
	encryptionAlgorithm: RSA-2048
	etcd:
	  local:
	    dataDir: /var/lib/etcd
	imageRepository: registry.k8s.io
	kind: ClusterConfiguration
	kubernetesVersion: plan
	networking:
	  dnsDomain: cluster.local
	  podSubnet: 10.112.0.0/12
	  serviceSubnet: 10.96.0.0/12
	proxy:
	  disabled: true
	scheduler: {}

Howewer, running

$ kubeadm upgrade apply --yes v1.32.0

Ends with:

...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
...

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.32.0

Cloud provider

bare metal

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ uname -a
Linux node-0 6.1.0-26-amd64 kubernetes/kubernetes#1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64 GNU/Linux

Install tools

kubeadm

Container runtime (CRI) and version (if applicable)

cri-o 1.31

Related plugins (CNI, CSI, ...) and versions (if applicable)

kube-router as network plugin and service proxy
@rkojedzinszky rkojedzinszky added the kind/bug Categorizes issue or PR as related to a bug. label Dec 27, 2024
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 27, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@rkojedzinszky
Copy link
Author

/sig cluster-lifecycle

@k8s-ci-robot k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 27, 2024
@neolit123
Copy link
Member

/transfer kubeadm

@k8s-ci-robot k8s-ci-robot transferred this issue from kubernetes/kubernetes Dec 28, 2024
@neolit123 neolit123 added priority/backlog Higher priority than priority/awaiting-more-evidence. area/addons and removed sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 28, 2024
@neolit123
Copy link
Member

i don't think we have logic to skip the addons on upgrade based on the new clusterconfig fields. so that seems like a bug.

@rkojedzinszky
Copy link
Author

I had my cluster running for years without kube-proxy. Pre 1.32 upgrades checked for kube-proxy configmap/daemonset, and as they were not present, the upgrade step simply skipped kube-proxy. Now this also has changed with 1.32. Perhaps, this was an undocumented/unintended behavior.

@neolit123
Copy link
Member

we added phase support for upgrade apply and upgrade node. i have a fix in the works.
it should be backported to 1.32.

/assign

@neolit123 neolit123 mentioned this issue Dec 28, 2024
Merged
@neolit123
Copy link
Member

neolit123 commented Dec 28, 2024 •
edited
Loading

fix for 1.33:

we have an e2e test for this:
https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-no-addons-latest

but it did not catch the issue, because it only validates if upgrade will pass if the addon config maps are missing. we need to update it to account for the ClusterConfiguration.{addon}.Disabled cases.

@neolit123 neolit123 mentioned this issue Dec 28, 2024
Open
@neolit123
Copy link
Member

e2e update:

@neolit123 neolit123 mentioned this issue Dec 30, 2024
Open
@neolit123
Copy link
Member

cherry pick for 1.32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@neolit123 neolit123

Labels
area/addons kind/bug Categorizes issue or PR as related to a bug. priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kubeadm: skip disabled addons in clusterconfig on upgrade neolit123/kubernetes
3 participants
@neolit123 @rkojedzinszky @k8s-ci-robot