Release Proposal v2.19

[oomichi]

Below is a proposal for release notes:

commits

• start-sha 0e98814
• end-sha 1f65e6d

Announcements

We are looking for maintainers, reach out in #5432.

Deprecation / Removal

• [metrics server] Remove addon-resizer from image list (no longer in use) (chore: remove addon-resizer #8566, @cyril-corbon)
• Add kubeadm option to etcd_deployment_type to replace the etcd_kubeadm_enabled variable (ADD NOTE etcd_kubeadm_enabled is deprecated. You can set etcd_deployment_type to kubeadm to get the same behaviour.") (Add kubeadm option to etcd_deployment_type to replace the etcd_kubeadm_enabled variable #8317, @necatican)
• Removes runc-arm64-1.0.3 hash value for non existing binaries (replace runc 1.0.3 arm64 hash with 0 #8391, @Payback159)
• Drop containerd 1.4 support (Drop containerd 1.4 support #8780, @oomichi)

Feature / Major changes

• Add hashes for Kubernetes 1.24.0, 1.24.1, 1.21.12, v1.21.13, 1.22.8, 1.22.9, v1.22.10, 1.21.11, 1.23.5, 1.23.6, v1.23.7 and make kubernetes v1.23.7 default ([kubernetes] update versions 1.23.5, 1.22.8 and 1.21.11 #8628, [kubernetes] add hashes for 1.22.9 #8746, [kubernetes] add hashes for 1.24.0 #8783, [kubernetes] add hashes for 1.24.1 and other versions. #8876, [kubernetes] add 1.23.6, 1.21.12 and make 1.23.6 the new default #8760, @mzaian, @cristicalin)
• Add youki runtime support to CRI-O (Add youki runtime support #8411, @electrocucaracha)
• [etcd] add 0 hash for arm v3.5.2 to prevent deployment failures ([etcd] add 0 hash for arm v3.5.2 to prevent deployment failures #8651, @cristicalin)
• [etcd] ensure etcd is properly upgraded when managed by kubeadm ([etcd] ensure etcd is properly upgraded when managed by kubeadm #8722, @cristicalin)
• [etcd] Add etcd_max_request_bytes option to set the request size limit of etcd ([etcd] Add support for setting the request size limit #8849, @necatican)
• [etcd] add v3.5.1 for kubernetes 1.22+ (etcd: add etcd v3.5.1 for kubernetes 1.22+ #8588, @mzaian)
• [etcd] Added node label to etcd metrics (Add node label to etcd metrics #8475, @fungusakafungus)
• [Cilium] Update Cilium manifests and the default version to v1.11.3 (Overhaul Cilium manifests to match the newer versions #8717, @necatican)
• [Cilium] Add identity_allocation_mode support (Add identity_allocation_mode support for Cilium #8430, @necatican)
• [Cilium] Change Cilium setting identity_allocation_mode to cilium_identity_allocation_mode (⚠️ ADD NOTE: If the Cilium setting identity_allocation_mode has been overridden locally, it needs to be changed to cilium_identity_allocation_mode.) (Change Cilium setting identity_allocation_mode to cilium_identity_allocation_mode #8519, @tomberget)
• [cilium] Add the cilium ip-masq-agent configuration support (Support cilium ip-masq-agent configuration #8893, @mahjonp)
• [docker] add support for cri-dockerd as a replacement for dockershim ([docker] add support for cri-dockerd as a replacement for dockershim #8623, @cristicalin)
• Add dual-stack support to kubelet --node-ip parameter, it works if set ip6 option host vars (add ip6 host vars to support Dual Stack node InternalIP #8542, @kakkotetsu)
• Add ppc64le support (Added ppc64le support #8505, @mgiessing)
• Add runc v1.1.0 hash values to support multi-arch installation. (arm64, amd64) (add hash-values for runc v1.1.0 #8447, @Payback159)
• Add support for EventRateLimit plugin configuration (add support for EventRateLimit plugin configuration #8711, @alegrey91)
• Add support for including annotations on aws-ebs-csi-controller (Add support for including annotations on aws-ebs-csi-controller #8779, @dlouks)
• Add support for kube-vip (Add support for kube-vip #8669, @sathieu)
• Add support for service-account-lookup parameter (using kube_apiserver_service_account_lookup) (add support for service-account-lookup parameter #8781, @alegrey91)
• [ansible] add support for ansible 5 (ansible-core 2.12) ([ansible] add support for ansible 5 (ansible-core 2.12) #8512, @cristicalin)
• [ansible] make ansible 5.x the new default version ([ansible] make ansible 5.x the new default version #8660, @cristicalin)
• [ansible] update ansible and cryptography requirements ([ansible] update ansible and cryptography requirements #8826, @cristicalin)
• [cert-manager] Update cert-manager to 1.6.1 (cert-manager update to 1.6.1 #8377, @electrocucaracha)
• [cert-manager] Update cert-manager to v1.7.2 (Update cert-manager to v1.7.2 #8648, @rtsp)
• [cert-manager] Upgrade to v1.8.0 ([cert-manager] Upgrade to v1.8.0 #8688, @rtsp)
• Add Ubuntu 22.04 support (Fixed systemd-networkd restart for ubuntu 22.04, when using reset.yml #8841, [kubelet] set correct resolv.conf for Ubuntu 22.04 #8795, Remove aufs-tools from Ubuntu requirement #8754, @u2216, @arno01, @oomichi)
• Add evictionHard parameter to kubelet config (variables: eviction_hard/eviction_hard_control_plane) (feat: add eviction hard to kubelet config #8421, @cyril-corbon)
• Add hcloud as external cloud provider (add external hcloud cloud controller manager #8440, @oujonny)
• Add kube_router_cluster_asn option to set ASN number of the cluster (Add kube router cluster asn #8837, @rosskusler)
• Add option to use UpCloud's preconfigured server plans, firewalls and managed load balancers (upgrade to 2.4.0 from 2.0.0) (UpCloud server plan, firewall, load balancer integration #8758, @Ajarmar)
• Add possibility to remove ippools from cni config (Possible remove ippools from cni config #8845, @tomcsi)
• Add the ability to set tolerations (cert_manager_tolerations), nodeselector (cert_manager_nodeselector) and affinity (cert_manager_affinity) in cert-manager templates (Add tolerations / nodeselector / affinity to cert-manager #8389, @cyril-corbon)
• Add the possibility to use UpCloud csi-driver
  Add the possibility to use ansbile_host as api ip for localhost kubeconfig (UpCloud integration #8653, @robinAwallace)
• Add Hardening setup guide (docs: add hardening guide #8868, @alegrey91)
• Add variables to manage kubelet parameters (kubelet_streaming_connection_idle_timeout / kubelet_make_iptables_util_chains) (feat: add variables to manage kubelet parameters #8796, @alegrey91)
• Added the optional prompt or delay before uncordoning nodes after upgrades (see variable upgrade_node_post_upgrade_confirm). (Allow pausing after upgrade but before uncordon #8530, @mac-chaffee)
• Allow installation of a cluster using external CAs (kubernetes-ca, etcd-ca, kubernetes-front-proxy-ca) (Kubernetes CAs managed externally #8620, @julienlefur)
• Allow the customization of snapshot controller namespace using snapshot_controller_namespace (parameterized snaphot controller namespaces #8305, @liupeng0518)
• Allow to change cert-manager leader election namespace for GKE Autopilot support (cert-manager: Allow to change leader election namespace for GKE Autopilot support #8424, @rtsp)
• Allow to choose image pull commands based on container manager or override them (Allow to choose image pull commands based on container manager or override them #8380, @sathieu)
• Allow to specify CA data for webhooks (using kube_webhook_token_auth_url_skip_tls_verify / kube_webhook_token_auth) (Add optional setting for ca data in auth webhook #8777, @dlouks)
• Assert that IP range is enough for the nodes (Assert that IP range is enough for the nodes #8720, @eakyildirim)
• Bastion support now works for remove-node.yml (Add bastion support to remove-node.yml #8504, @roedie)
• Bump upcloud csi-driver to v0.2.1 (csi: bump upcloud csi driver #8784, @robinAwallace)
• Change default kube_encryption_algorithm to "secretbox" (⚠️ ADD NOTE: If you are already using encrypting secret a rest and have not set the kube_encryption_algorithm flag, then you must set kube_encryption_algorithm to aescbc since the default value has changed to the more secure secretbox standard.) (Encrypting Secret Data at Rest #8574, @Payback159)
• Explicit container_manager variable for Etcd hosts (Configure Etcd container_manager explicitly #8521, @vi7)
• Improve first_kube_control_plane variable management to avoid installation failures due to variable overlapping (Improve first_kube_control_plane variable management to avoid install… #8388, @unai-ttxu)
• Improve offline script generate_list.sh using ansible (feat(offline): Improve generate_list.sh using ansible (#8537) #8538, @tmurakam)
• [ingress-nginx] upgrade to 1.2.1
• Ingress controllers and external provisioners (respectively deployed via ingress_controller and external_provisioner roles meta dependencies) are now upgraded in upgrade-cluster.yml (Have ingress_controller and external_provisioner in upgrade-cluster.yml #8640, @mirwan)
• Local volume provisioner tolerations removed by default. (local volume provisioner should not run on control plane nodes by def… #8805, @spaced)
• Replace CLB with NLB for kube-apiserver domain in Terraform AWS contrib code ([Terraform-AWS] Replace CLB with NLB for kube-apiserver domain #8578, @sophalHong)
• Split kube_feature_gates variable for different kubernetes components (/!\ ⚠️ ADD NOTE : This add some variables that could be defined by the user. It doesn't introduce a breaking change because the previous unique variable (kube_feature_gates) still works as expected. kube_apiserver_feature_gates/kube_controller_feature_gates/kube_scheduler_feature_gates/kube_proxy_feature_gates/kubelet_feature_gates) (split kube_feature_gates variable for different kubernetes components #8677, @alegrey91)
• Helm-apps role for installing helm charts (Helm-apps role for installing helm charts #8347, @VannTen)
• Upgrade azuredisk csi to v1.10.0 (feat: upgrade azuredisk csi to v1.10.0 #8432, @cyril-corbon)
• Upgrade metrics-server to v0.5.2 and remove NET_BIND_SERVICE capabilities [⚠️ ADD NOTE: Move port to 4443]) (feat: upgrade metrics server to v0.5.2 #8338, @cyril-corbon)
• Vagrant: new var $ansible_verbosiity was introduced for setting up ansible verbosity level (Vagrantfile: add var to set ansible verbosity level #8639, @maciejaszek)
• [CI] Move from CentOS 8 to AlmaLinux 8 for kubespray CI, therefore CentOS 8 is no longer tested (CI: Replace CentOS 8 with AlmaLinux 8 before CentOS 8 EOL end of 2021 #8297, @cristicalin)
• [CI] split molecule testes to run in parallel ([CI] split molecule testes to run in parallel #8756, @cristicalin)
• [container image] use focal (ubuntu 20.04) base image for our docker builds ([container image] use focal (ubuntu 20.04) base image for our docker builds #8631, @cristicalin)
• [coredns] Allow overriding the default CoreDNS zone's cache plugin configuration via the coredns_default_zone_cache_block variable (Allow overriding CoreDNS default zone cache block #8488, @Tristan971)
• [csi-snapshotter] upgraded to 5.0.0 ([csi-snapshotter] upgrade to 5.0 #8403, @cristicalin)
• [download] add capability to specify alternative download mirrors for files (implement download mirrors support #8474, @cristicalin)
• [mitogen] update to 0.3.2 ([mitogen] update to 0.3.2 #8470, @cristicalin)
• [reset] remove containerd storage during reset ([reset] remove containerd storage during reset #8469, @cristicalin)
• [sysctl] set fs.may_detach_mounts=1 to address pods stuck in Terminating state ([sysctl] set fs.may_detach_mounts=1 even when CRIs don't set it themselves #8635, @cristicalin)

Network

• [Calico] upgrade calico to 3.19.4, 3.20.4 and 3.21.4 (default) and add 3.22.0 experimental support ([calico] upgrade release checksums #8544, @cristicalin)
• [Calico] add 3.22.1 ([calico] add 3.22.1 #8612, @cristicalin)
• [Calico] Add calico apiserver (using calico_apiserver_enabled variable) ([calico] add calico apiserver #8690, @liupeng0518)
• [Calico] Add support for IP6_AUTODETECTION_METHOD using new variable calico_ip6_auto_method (add support for Calico IP6_AUTODETECTION_METHOD #8541, @kakkotetsu)
• [Calico] upgrade default calico version to v3.22.3 ([calico] add 3.22.3 and 3.23.1 #8897, @germetist)
• [Calico] Add configurable ipam strictaffinity (using calico_ipam_strictaffinity param) (feat(calico): add configurable ipam strictaffinity #8581, @eyenx)
• [Calico] Change the calico cni name from cni0 to k8s-pod-network by default (set default value of name to "k8s-pod-network" #8813, @cyclinder)
• [Calico] Fix Wireguard support for CentOS Stream 9/RHEL 9 Beta (Calico: Fix Wireguard support for CentOS Stream 9/RHEL 9 Beta #8625, @ThisIsQasim)
• [Calico] fix calico-kube-controllers verbs (Calico: fix calico-kube-controllers verbs #8847, @irizzant)
• [calico] Some commands only need to be run once ([calico] some commands only need run once #8833, @liupeng0518)
• [calico] call calico checks early on to prevent altering the cluster with bad settings and causing traffic outages ([calico] call calico checks early on to prevent altering the cluster #8707, @cristicalin)
• [calico] make calico 3.21.x the news default and drop 3.18.x ([calico] drop 3.18.x and make 3.21.x the new default #8426, @cristicalin)
• [calico] switch default iptables backend detection to Auto ([calico] switch default iptables backend detection to Auto #8429, @cristicalin)
• [calico] Use vxlan instead of ipip as the default calico encapsulation mode. This change impacts existing deployments that don't explicitly set the encapsulation mode and will need to set calico_ipip_mode: Always and calico_network_backend: bird to avoid the upgrade process breaking. ([calico] don't enable ipip encapsulation by default and use vxlan in CI #8434, @cristicalin)
• [calico] upgrade default calico version to v3.21.5 (Calico: add new versions & related checksums #8745, @mzaian)
• [calico] Use ipamconfig instead of calico ipam command ([calico]use ipamconfig instead of calico ipam command #8839, @liupeng0518)
• [calico] don't clobber calico options set by the user ([calico] don't clobber calico options set by the user  #8815, @cristicalin)
• [flannel] Use install-cni-plugin to fit upstream (Update flannel use install-cni-plugin to fit upstream #8714, @zhengtianbao)
• [kube-ovn] Sync some feature with upstream ([kube-ovn]: update kube-ovn version and sync some feature #8790, @liupeng0518)
• [kube-ovn] The network plug-in kube-ovn does not require a cluster to allocate podcidr (Adaptive Kube-ovn #8454, @chenhuazhong)

Applications

• Instance customization via cloud init for openstack VMs deployed by terraform is now available. [⚠️ SEE NOTES :To use it, adjust contrib/terraform/openstack/modules/compute/templates/cloudinit.yaml before deployment. (Currently it uses one cloud init for all instances.)] (contrib: add cloud-init support for terraform vms #8394, @moss2k13)
• [MetalLB] Configure PriorityClassName for deployment (Configure PriorityClassName for MetalLB deployment #8362, @unai-ttxu)
• [MetalLB] Improve validation conditions for BGP Peers (improve validation conditions for MetalLB BGP Peers #8568, @kakkotetsu)
• [MetalLB] Upgrade metallb to v0.11.0 and add liveness and readiness probe (feat: upgrade metallb to v0.11.0 #8420, @cyril-corbon)
• [MetalLB] Allow to put node selectors and source address for each metallb peers (Allow to specify a source address for metallb peerings, and target on… #8534, @hightoxicity)
• [MetalLB] Added MetalLB BGP peer password authentication option. (Support metallb peer password #8792, @Oogy)
• [MetalLB] Add images to downloads (metallb: Add images to downloads #8715, @sathieu)
• [MetalLB] Fix wrong port name in metallb.yml.j2 (metrics not monitoring) (Fix wrong port name in metallb.yml.j2 #8510, @binkoni)
• [OpenStack] Allow disabling port security in terraform contrib code ([contrib] terraform openstack: allow disabling port security #8410, @cristicalin)
• [OpenStack] Updated openstack cloud controller to version v1.22.0 (Updated openstack cloud controller version to v1.22.0 #8629, @Xartos)
• [OpenStack] Create master nodes with for_each for openstack. Makes it easier to switch out master nodes via terraform. ([Openstack] master foreach and fixes #8709, @robinAwallace)
• [OpenStack] Fixed cluster roles for openstack cloud controller (Fixed cluster roles for openstack cloud controller #8638, @Xartos)
• [OpenStack] Fix templating of ansible_ssh_common_args in no_floating.yml if used as TF module (terrform/openstack: Fix templating of ansible_ssh_common_args in no_floating.yml if used as TF module #8646, @frittentheke)
• [OpenStack] allow disabling port security at port level ([terraform][openstack] allow disabling port_security at port level #8455, @cristicalin)
• [vSphere] Terraform code will need var.vapp when a vapp is referenced (vsphere_hostname is also removed) (Proposing fixes for contrib/terraform/vsphere/ #8436 #8441, @ceesios)
• [vsphere_csi] update to 2.5.1 and make external_vsphere_version 7.0u1 the default ([vsphere_csi] update to 2.5.1 and make external_vsphere_version 7.0u1 the default #8676, @cristicalin)
• Terraform/gcp: Allow to change extra disk types (terraform/gcp: Allow to change extra disk types #8524, @sathieu)
• Terraform/gcp: Allow to use preemptible VM instances (using two new variable master_preemptible and worker_preemptible) (terraform/gcp: Allow to use preemptible VM instances #8480, @sathieu)
• Terraform/gcp: Do not create unused subnetworks
  terraform/gcp: Upgrade to latest google provider (terraform/gcp: Do not create unused subnetworks and Upgrade to latest google provider #8497, @sathieu)
• [Terraform AWS] Add tag to AWS VPC subnets for automatic subnet discovery ([Terraform AWS] Add tag to AWS VPC subnets for automatic subnet discovery #8705, @sophalHong)
• [terraform] use modern day equinix metal provider ([terraform] use modern day equinix metal provider #8748, @cristicalin)

Container-Managers

• Check & uninstall container engine if needed (when changing container engine defined) (feat: check & uninstall container engine if needed #8439, @cyril-corbon)
• [Docker] Add epoch to docker-ce and docker-ce-cli packages to ensure docker upgrade (on rhel based) (Add epoch to docker-ce and docker-ce-cli packages to ensure docker up… #8618, @unai-ttxu)
• [containerd] make containerd_insecure_registries into a dict similar to containerd_registries [⚠️ ADD NOTE containerd_insecure_registries needs to be updated or won't work anymore] (Update containerd_insecure_registries to match containerd_registries  #8340, @mircyb)
• [containerd] upgrade versions to fix CVE-2022-23648 ([containerd]: upgrade versions to fix CVE-2022-23648 #8597, @cristicalin)
• [containerd] Upgrade containerd to 1.6.0 and re-enable arm architecture with default options
  [runc] make 1.1.0 the default
  [nerdctl] upgrade to 0.17.0 ([containerd] Upgrade containerd to 1.6.0 and re-enable arm64 architecture with default options #8555, @cristicalin)
• [containerd] add hashes for 1.15.11 and 1.6.2 and make 1.6.2 the default ([containerd] upgrade versions to address CVE-2022-24769 #8671, @cristicalin)
• [containerd] Update containerd to 1.5.9 (containerd update to 1.5.9 #8402, @electrocucaracha)
• [containerd] Fix containerd image download bug ([download]fix containerd images download bugs #8894, @liupeng0518)
• [containerd] nerdctl insecure registry support (nerdctl insecure registry config #8339, @mircyb)
• [containerd] Ensure containerd service unmasking (Ensure containerd service unmasking #8726, @rickerc)
• [cri-o] Update configuration of registries in cri-o [⚠️ ADD NOTE: ] (Update configuration of registries in cri-o #7852, @bsloeserwij)
• [cri-o] add cri-0 1.23.x ([cri-o] add cri-0 1.23.x #8599, @cristicalin)
• [crun] update to 1.4 and drop pre-1.x versions (crun update to 1.4 #8330, @cristicalin)
• [crun] upgrade to 1.4.3 ([crun] upgrade to 1.4.3 #8598, @cristicalin)

Bug or Regression

• Add ETCD_EXPERIMENTAL_INITIAL_CORRUPT_CHECK flag to etcd config (default to true) (Add ETCD_EXPERIMENTAL_INITIAL_CORRUPT_CHECK flag to etcd config #8664, @floryut)
• Add with_networks variable to external_hcloud_cloud in ansible playbook and network_zone variable to Hetzner Cloud Terraform. (Fixes for Hetzner terraform and Hetzner Cloud  #8702, @Anthony-Bible)
• Allow replacement of address prefixes for all images (Allow replacement of address prefixes for all images #8764, @ErikJiang)
• CRI-O: fix unqualified-search registries (CRI-O: fix unqualified-search registries #8496, @krystianmlynek)
• Change libvirt default disk controller from IDE to SCSI (fix disk controller type in Vagrantfile #8656, @190ikp)
• Do not remove package in validate container engine role when FCOS (do not remove package in validate container engine role when FCOS #8626, @LuckySB)
• Enable Kubespray deployment on vagrant (Enable Kubespray deployment on vagrant #8697, @oomichi)
• Enable several read-only tasks in check mode (Enable several read-only tasks in check mode #8584, @tjanson)
• Ensure all Kubelet required kernel values are configured when enabling protectKernelDefaults (Ensure all Kubelet required kernel values are configured when enablin… #8692, @unai-ttxu)
• Ensure taint configuration for secondary control-plane acting both as control-plane and node (Ensure taint configuration for secondary control-plane acting both as control-plane and node #8363, @unai-ttxu)
• Error: error parsing jsonpath {, unclosed action (Single quotes are missing in jsonpath argument of kubectl get node #8683, @emiran-orange)
• Fix DNS configuration when using resolvconf_mode='host_resolvconf' during scale (Fix DNS configuration when using resolvconf_mode='host_resolvconf' during scale #8361, @unai-ttxu)
• Fix GCP PVC creation on k8s v1.22 (add fix for GCP CSI driver #8616, @lmercl)
• Fix 0090-etchosts file when setting override_system_hostname=false (fix 0090-etchosts #7634, @liupeng0518)
• Fix kube-dns service will no longer be deleted if not created by kubespray (fix: kube-dns service deletion #8565, @cyril-corbon)
• Fix an issue the kube-vip manifest with extra space. (Fix the invalid kube vip manifest #8831, @yankay)
• Fix an issue users cannot skip redhat registration by specifying -e rhel_enable_repos=False (Added |bool for rhel_enable_repos #8871, @gleb108)
• Fix an issue where offline script could not output URLs of both containerd and krew. (Move containerd_version to defaults/main.yml #8379, @oomichi)
• Fix condition on kata_containers_version/kube_version check when kata_containers_enabled is false (Fix condition on kata_containers_version/kube_version check when kata_containers_enabled is false #8804, @emiran-orange)
• Fix container engine still installed on dedicated etcd node even if etcd_deployment_type: host (Fix container engine still installed on dedicated etcd node even if etcd_deployment_type: host #8386, @rtsp)
• Fix cri-o packages install for Rocky 8 (Fix cri-o packages install for Rocky 8 #8594, @brankomijuskovic)
• Fix etcd certificates reference to support etcd_kubeadm_enabled: true (Fix etcd certificates reference to support etcd_kubeadm_enabled:true #7766, @forselli-stratio)
• Fix imageRepository path for CoreDNS (ensure coredns repository namespace is kept) (fix(coredns): make sure to keep coredns repository namespace #8572, @nicolas-goudry)
• Fix incorrect condition type (fix: incorrect condition type #8822, @cyclinder)
• Fix incorrect leader election namespace with cert-manager leading to insufficient permission (cert-manager: Fix incorrect leader election namespace lead to insufficient permission #8433, @rtsp)
• Fix issue when PodSecurityPolicy is enabled static pods are now mirrored earlier by kubelet. Problem when installing HA etcd via kubeadm. ([PodSecurityPolicy] Move the install of psp #8744, @robinAwallace)
• Fix kubectl call before installing it when setting first_kube_control_plane/joined_control_planes (Fix kubectl call before installing it when setting first_kube_control_plane/joined_control_planes #8412, @floryut)
• Fix kubelet_kubelet_cgroups_cgroupfs pointing incorrectly to slice (Fix kubelet_kubelet_cgroups_cgroupfs #8500, @fungusakafungus)
• Fix print_hostnames of inventory.py (Fix print_hostnames of inventory.py #8554, @oomichi)
• Fix remove-node.yaml playbook fails when host is unreachable (Skip gathering facts when reset_nodes is false #8843, @oomichi)
• Fix removing docker-ce.repo failed (fix removing docker-ce.repo failed #8856, @Thearas)
• Fix the condition of drain on pre-remove task (Fix the condition of drain on pre-remove task #8634, @oomichi)
• Fix typo and duplicated declaration of ingressclasses (fix typo and duplicated declaration of ingressclasses #8591, @spaced)
• Fix vagrant default value for parameters local_path_provisioner_enabled/multi_networking (fix vagrant parameter #8650, @liupeng0518)
• Fix wrong item in mitogen contrib (Fixed mitogen.yml #8508, @kdszoom)
• Fixed a bug where hosts with NetworkManager enabled were having their /etc/resolv.conf file edited directly instead of through NM.
  Fixed a bug where DNS lookup failures would cause reset.yml or scale.yml to error out when resolvconf_mode=host_resolvconf (Fix host DNS settings being edited too soon, and not working with NetworkManager #8575, @mac-chaffee)
• Fixed a bug where updated versions of etcd weren't being applied. Check your etcd instances to make sure their versions are what you expect. If not, restarting all etcd members should apply the update. (Restart etcd if the etcd version changes #8556, @mac-chaffee)
• Fixed a bug where upgrade-cluster.yaml would not apply updates to etcd-events (Fix etcd_events not getting upgraded in upgrade-cluster.yml #8550, @mac-chaffee)
• Fixes missing checksum for kata-containers 2.2.3 on arm architectures (adding 0 checksum for kata_containers_version on arm(64) #8383, @Payback159)
• Fixes the etcd node removal by pointing ETCDCTL_ENDPOINTS to localhost (127.0.0.1) (Fixes the etcd node removal by pointing ETCDCTL_ENDPOINTS to localhost (127.0.0.1) #8526, @roedie)
• Prevent removing etcd member when running in check mode (Prevent removing etcd member when running in check mode #8570, @fungusakafungus)
• Removal flow: Waiting until Volumes will be detached from the node (Removal flow: Waiting until Volumes will be detached from the node #8739, @rocko-n)
• Removed quotation at nerdctl_extra_flags (Removed quotation at nerdctl_extra_flags. #8695, @T-Eberle)
• Run 0100-dhclient-hooks only if dhcpclient is enabled (Run 0100-dhclient-hooks if dhcpclient is enabled #8658, @oomichi)
• Update verbs for volumeattachments resource (Update verbs for volumeattachments resource #8731, @moule3053)
• Use correct service name for coredns when cleanup (wrong service name for coredns #8811, @weizhoublue)
• [Terraform-AWS] Fix error when creating subnets more than AZ (Fix: [Terraform-AWS] Error when creating subnets more than AZ #8516, @sophalHong)
• [cert-manager] Fix missing RBAC rules for ClusterRole cert-manager-cainjector ([cert-manager] Fix missing RBAC rules for ClusterRole cert-manager-ca… #8444, @onock)
• [containerd] avoid cleanup of /usr/bin on ostree distributions ([containerd] avoid cleanup of /usr/bin on ostree distributions #8624, @cristicalin)
• [reset] fix task inclusion logic for network plugin ([reset] fix task inclusion logic for network plugin #8727, @cristicalin)
• [systemd-resolved] Fix DNS early and late stages (dns_early|dns_late) of cluster deployment
  [systemd-resolved] Add upstream_dns_servers to FallbackDNS
  [cluster-reset] Revert DNS configuration to early stage (for instance: only defined upstream nameservers) ([systemd-resolved] Fix DNS configuration according to docs/dns-stack.… #8561, @onock)
• [⚠️ ADD NOTE: For now Calico and K8S 1.23 might broke under some OS, careful !] (Fix Fedora CI following ipset version in kube-proxy for k8s 1.23 #8397, @floryut)

Other (Cleanup or Flake)

• Add IPv6 listen directive to nginx if enable_dual_stack_networks (add IPv6 listen directive to nginx if enable_dual_stack_networks #8596, @kakkotetsu)
• Cleanup crictl configuration file during reset (Cleanup crictl configuration file #8569, @jayonlau)
• Remove check_mode: no from gen_certs_script.yml to prevent changing files (Check all places with check_mode: no for side effects #8573, @fungusakafungus)

Component versions:

• Kubernetes v1.23.7
• Etcd v3.5.3
• Docker v20.10
• Containerd v1.6.4
• CRI-O v1.22
• CNI-plugins v1.1.1
• Calico v3.22.3
• Cilium v1.11.3
• Flannel v0.17.0
• Kube-ovn v1.9.2
• Kube-Router v1.4.0
• Multus v3.8
• Weave v2.8.1
• Ceph-provisioner v2.1.0-k8s1.11
• Cert-manager v1.8.0
• CoreDNS v1.8.6
• Nginx-ingress v1.2.1

Known issues

n/a

Notes

[rtsp]

IIRC, current cert-manager in Component versions section should be at least v1.8.0.

[cristicalin]

IIRC, current cert-manager in Component versions section should be at least v1.8.0.

That's correct but our README.md still references the old version, it should be updated before we cut the release.

[rtsp]

IIRC, current cert-manager in Component versions section should be at least v1.8.0.

That's correct but our README.md still references the old version, it should be updated before we cut the release.

Thanks, I made #8886 to update README.md for cert-manager and few components that also outdated too.

[oomichi]

IIRC, current cert-manager in Component versions section should be at least v1.8.0.

That's correct but our README.md still references the old version, it should be updated before we cut the release.

Thanks, I made #8886 to update README.md for cert-manager and few components that also outdated too.

Thanks for updating the README.md @rtsp
We have merged the pull request and I am going to include the commit into the release.

[floryut]

Thanks guys ❤️

[oomichi]

A final boarding call: https://kubernetes.slack.com/archives/CDQSC941H/p1654032149214819 🏃