Creating a cluster with Cilium CNI doesn't work

[radekg]

I'm using the following configuration file:

---
kubeconfig_path: "~/.hetzner/kube/gmbh"
cluster_name: gmbh-infrastructure
k3s_version: v1.30.3+k3s1

networking:
  ssh:
    port: 22
    use_agent: false
    public_key_path: "~/.hetzner/ssh/gmbh_rsa.pub"
    private_key_path: "~/.hetzner/ssh/gmbh_rsa"
  allowed_networks:
    ssh: 
    - 0.0.0.0/0
    api:
    - 0.0.0.0/0
  public_network:
    ipv4: true
    ipv6: false
  private_network:
    enabled: true
    subnet: 10.0.0.0/16
  cni:
    enabled: true
    encryption: false
    mode: cilium
    cilium:
      # Optional: specify a path to a custom values file for Cilium Helm chart
      # When specified, this file will be used instead of the default values
      helm_values_path: "/home/radek/dev/hetzner/gmbh/cilium-values.yaml"
      chart_version: "v1.18.2"
  cluster_cidr: 10.244.0.0/16 # optional: a custom IPv4/IPv6 network CIDR to use for pod IPs
  service_cidr: 10.43.0.0/16 # optional: a custom IPv4/IPv6 network CIDR to use for service IPs. Warning, if you change this, you should also change cluster_dns!
  cluster_dns: 10.43.0.10 # optional: IPv4 Cluster IP for coredns service. Needs to be an address from the service_cidr range

schedule_workloads_on_masters: false
protect_against_deletion: false

# image: rocky-9 # optional: default is ubuntu-24.04
# snapshot_os: microos # otional: specified the os type when using a custom snapshot
masters_pool:
  instance_type: cpx21
  instance_count: 1
  locations:
  - nbg1

worker_node_pools:
- name: small
  instance_type: cpx21
  instance_count: 3
  location: nbg1

The cluster never comes up successfully:

[Configuration] Validating configuration...
[Configuration] ...configuration seems valid.
[Private Network] Creating private network...
[Private Network] ...private network created
[SSH key] Creating SSH key...
[SSH key] ...SSH key created
[Instance gmbh-infrastructure-master1] Creating instance gmbh-infrastructure-master1 (attempt 1)...
[Instance gmbh-infrastructure-master1] Instance status: initializing
[Instance gmbh-infrastructure-master1] Powering on instance (attempt 1)
[Instance gmbh-infrastructure-master1] Waiting for instance to be powered on...
[Instance gmbh-infrastructure-master1] Instance status: running
[Instance gmbh-infrastructure-master1] ...instance gmbh-infrastructure-master1 created
[Firewall] Creating firewall...
[Firewall] ...firewall created
[Instance gmbh-infrastructure-pool-small-worker1] Creating instance gmbh-infrastructure-pool-small-worker1 (attempt 1)...
[Instance gmbh-infrastructure-pool-small-worker3] Creating instance gmbh-infrastructure-pool-small-worker3 (attempt 1)...
[Instance gmbh-infrastructure-pool-small-worker2] Creating instance gmbh-infrastructure-pool-small-worker2 (attempt 1)...
[Instance gmbh-infrastructure-master1] 🕒 Awaiting cloud config (may take a minute...)
[Instance gmbh-infrastructure-master1] .
[Instance gmbh-infrastructure-pool-small-worker3] Instance status: initializing
[Instance gmbh-infrastructure-pool-small-worker3] Powering on instance (attempt 1)
[Instance gmbh-infrastructure-pool-small-worker2] Instance status: initializing
[Instance gmbh-infrastructure-pool-small-worker2] Powering on instance (attempt 1)
[Instance gmbh-infrastructure-pool-small-worker3] Waiting for instance to be powered on...
[Instance gmbh-infrastructure-pool-small-worker2] Waiting for instance to be powered on...
[Instance gmbh-infrastructure-pool-small-worker1] Instance status: initializing
[Instance gmbh-infrastructure-pool-small-worker1] Powering on instance (attempt 1)
[Instance gmbh-infrastructure-pool-small-worker1] Waiting for instance to be powered on...
[Instance gmbh-infrastructure-master1] Cloud init finished: 24.67 - Thu, 30 Oct 2025 12:47:54 +0000 - v. 25.1.4-0ubuntu0~24.04.1
[Instance gmbh-infrastructure-master1] Private network interface enp7s0 found
[Instance gmbh-infrastructure-master1] Private network IP: 10.0.0.2
[Instance gmbh-infrastructure-master1] Installing k3s...
[Instance gmbh-infrastructure-master1] [INFO]  Using v1.30.3+k3s1 as release
[Instance gmbh-infrastructure-master1] [INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.30.3+k3s1/sha256sum-amd64.txt
[Instance gmbh-infrastructure-master1] [INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.30.3+k3s1/k3s
[Instance gmbh-infrastructure-master1] [INFO]  Verifying binary download
[Instance gmbh-infrastructure-master1] [INFO]  Installing k3s to /usr/local/bin/k3s
[Instance gmbh-infrastructure-master1] [INFO]  Skipping installation of SELinux RPM
[Instance gmbh-infrastructure-master1] [INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[Instance gmbh-infrastructure-master1] [INFO]  Creating /usr/local/bin/crictl symlink to k3s
[Instance gmbh-infrastructure-master1] [INFO]  Creating /usr/local/bin/ctr symlink to k3s
[Instance gmbh-infrastructure-master1] [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[Instance gmbh-infrastructure-master1] [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[Instance gmbh-infrastructure-master1] [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[Instance gmbh-infrastructure-master1] [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[Instance gmbh-infrastructure-master1] [INFO]  systemd: Enabling k3s unit
[Instance gmbh-infrastructure-master1] Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[Instance gmbh-infrastructure-master1] [INFO]  systemd: Starting k3s
[Instance gmbh-infrastructure-pool-small-worker3] Instance status: starting
[Instance gmbh-infrastructure-pool-small-worker3] Powering on instance (attempt 1)
[Instance gmbh-infrastructure-pool-small-worker2] Instance status: initializing
[Instance gmbh-infrastructure-pool-small-worker2] Powering on instance (attempt 1)
[Instance gmbh-infrastructure-pool-small-worker3] Waiting for instance to be powered on...
[Instance gmbh-infrastructure-pool-small-worker2] Waiting for instance to be powered on...
[Instance gmbh-infrastructure-pool-small-worker1] Instance status: initializing
[Instance gmbh-infrastructure-pool-small-worker1] Powering on instance (attempt 1)
[Instance gmbh-infrastructure-pool-small-worker1] Waiting for instance to be powered on...
[Instance gmbh-infrastructure-master1] k3s installation completed successfully
[Instance gmbh-infrastructure-master1] Waiting for the control plane to be ready...
[Instance gmbh-infrastructure-pool-small-worker3] Instance status: running
[Instance gmbh-infrastructure-pool-small-worker2] Instance status: running
[Instance gmbh-infrastructure-pool-small-worker1] Instance status: running
[Control plane] Generating the kubeconfig file to /home/radek/.hetzner/kube/gmbh...
[Control plane] Switched to context "gmbh-infrastructure-master1".
[Control plane] ...kubeconfig file generated as /home/radek/.hetzner/kube/gmbh.
[Instance gmbh-infrastructure-pool-small-worker3] ...instance gmbh-infrastructure-pool-small-worker3 created
[Instance gmbh-infrastructure-master1] Validating master setup...
[Master Validation] ✅ Master validation successful
[Control plane] Generating the kubeconfig file to /home/radek/.hetzner/kube/gmbh...
[Control plane] Switched to context "gmbh-infrastructure-master1".
[Control plane] ...kubeconfig file generated as /home/radek/.hetzner/kube/gmbh.
[CNI] Installing Cilium...
[CNI] "cilium" already exists with the same configuration, skipping
[CNI] Release "cilium" does not exist. Installing it now.
[Instance gmbh-infrastructure-pool-small-worker2] ...instance gmbh-infrastructure-pool-small-worker2 created
[CNI] NAME: cilium
[CNI] LAST DEPLOYED: Thu Oct 30 13:48:22 2025
[CNI] NAMESPACE: kube-system
[CNI] STATUS: deployed
[CNI] REVISION: 1
[CNI] TEST SUITE: None
[CNI] NOTES:
[CNI] You have successfully installed Cilium with Hubble Relay and Hubble UI.
[CNI] 
[CNI] Your release version is 1.18.2.
[CNI] 
[CNI] For any further help, visit https://docs.cilium.io/en/v1.18/gettinghelp
[CNI] Waiting for daemon set "cilium" rollout to finish: 0 of 1 updated pods are available...
[Instance gmbh-infrastructure-pool-small-worker1] ...instance gmbh-infrastructure-pool-small-worker1 created

Workers never join. Frankly, no kubelet is installed on any worker nodes... All I see is:

kubectl --kubeconfig=/home/radek/.hetzner/kube/gmbh get nodes -o wide
NAME                          STATUS     ROLES                       AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
gmbh-infrastructure-master1   NotReady   control-plane,etcd,master   69s   v1.30.3+k3s1   10.0.0.2      <none>        Ubuntu 24.04.3 LTS   6.8.0-71-generic   containerd://1.7.17-k3s1

This seems to be happening because the CNI installation step happens before workers are bootstrapped. And as there are no workers, there's nowhere for cilium to deploy its pods.

https://github.com/vitobotta/hetzner-k3s/blob/main/src/kubernetes/installer.cr#L62-L68

The software installation should be happening after workers join, no...?

Changing the cni.mode to flannel works. But I don't want flannel.

How does one create a cluster with Cilium instead of flannel?

[radekg]

This diff fixes it for me:

diff --git a/src/kubernetes/installer.cr b/src/kubernetes/installer.cr
index e13b2a2..7e7cf16 100644
--- a/src/kubernetes/installer.cr
+++ b/src/kubernetes/installer.cr
@@ -61,12 +61,12 @@ class Kubernetes::Installer
 
     @masters, @first_master_instance = @control_plane_setup.set_up_control_plane(masters_installation_queue_channel, master_count, load_balancer)
 
-    @software_installer.install_all(@first_master_instance, @masters, ssh, autoscaling_worker_node_pools)
-
     if worker_count > 0
       workers = @worker_setup.set_up_workers(workers_installation_queue_channel, worker_count, @masters, @first_master_instance)
     end
 
+    @software_installer.install_all(@first_master_instance, @masters, ssh, autoscaling_worker_node_pools)
+
     switch_to_context(default_context)
 
     completed_channel.send(nil)

[vitobotta]

I don't use Cilium myself so I haven't realized there was an issue with it after a big refactoring I did redcently, and from the few questions and issues I've seen so far, it seems like not many people using hetzner-k3s are using Cilium either so it went a bit under the radar. Most users probably just go with the default settings because it's simpler.

Could you open a PR with the change?

[deubert-it]

Maybe it depends on the content of the networking.cni.cilium.helm_values_path that @radekg uses (/home/radek/dev/hetzner/gmbh/cilium-values.yaml)?

We are using cilium on several clusters with defaults and it works fine.

  cni:
    enabled: true
    encryption: false
    mode: cilium

Never saw the reported issue so far.

[vitobotta]

Thanks for the info @deubert-it ! This helps. @radekg I don't have much time at the moment to experiment with Cilium and see what settings cause the issue, so if you have a chance to investigate and report that would help a lot.

[radekg]

@vitobotta thanks for coming back to me. There's nothing secret to my Cilium configuration, @deubert-it:

bgpControlPlane:
  enabled: true
bpf:
  masquerade: false
  lbExternalClusterIP: true
cluster:
  name: my-cluster
cni:
  # when using Istio with CNI
  # https://docs.cilium.io/en/latest/network/servicemesh/istio/
  exclusive: false
debug:
  enabled: false
envoy:
  enabled: true
  securityContext:
    capabilities:
      keepCapNetBindService: true
      envoy:
      # Add NET_BIND_SERVICE to the list (keep the others!)
      - NET_ADMIN
      - PERFMON
      - BPF
      - NET_BIND_SERVICE
  image:
    override: quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324 # set by the installer
envoyConfig:
  enabled: true
gatewayAPI:
  # Enabling Gateway API enables enable-envoy-config.
  # I enabled that option explicitly above.
  # Since there are no load balancers in the embedded cluster, 
  # I set the hostNetwork.enabled=true
  # https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/#host-network-mode
  enabled: true
  # Enable Application-Layer Protocol Negotiation (ALPN) which wil attempt HTTP/2, then HTTP 1.1.
  # Services that wish to use HTTP/2 must indicate that via their appProtocol (GEP-1911).
  enableAlpn: true
  gatewayClass:
    # Always create a GatewayClass for Cilium
    create: auto
hubble:
  relay:
    enabled: true
  ui:
    enabled: true
ingressController:
  enabled: true
  ingressLBAnnotationPrefixes:
  - load-balancer.hetzner.cloud
image:
  pullPolicy: IfNotPresent
ipam:
  mode: kubernetes
kubeProxyReplacement: "true"
k8sServiceHost: 10.0.0.2
k8sServicePort: 6443
operator:
  image:
    override: quay.io/cilium/operator-generic:v1.18.2 # set by the installer
  replicas: 1
  rollOutPods: true
securityContext:
  capabilities:
    # -- Capabilities for the `cilium-agent` container, tale default
    #    add NET_BIND_SERVICE
    ciliumAgent:
      - CHOWN
      - KILL
      - NET_ADMIN
      - NET_BIND_SERVICE
      - NET_RAW
      - IPC_LOCK
      - SYS_MODULE
      - SYS_ADMIN
      - SYS_RESOURCE
      - DAC_OVERRIDE
      - FOWNER
      - SETGID
      - SETUID
socketLB:
  # when using Istio with CNI
  # https://docs.cilium.io/en/latest/network/servicemesh/istio/
  hostNamespaceOnly: true

[radekg]

@deubert-it @vitobotta I wonder if this became an issue after your recent refactor about a month ago? There's certainly no way cilium can roll out without worker nodes.

[vitobotta]

@radekg yes, I mentioned it in my first reply

[radekg]

hey @vitobotta, I have opened the PR some time ago.

[vitobotta]

Thanks @radekg - I'll take a look one of these days. There was a reason why I moved the installation of the software to before, in particular the cluster autoscaler. With your change, if a cluster is configured to only have autoscaled nodepools there might be a problem. But I will test and merge it as soon as I have a chance.

[codepainters]

I just hit the same problem, but in a slightly different flavor.

If I use defaults, everything is fine, cilium starts as expected. However if I switch from default cilium v1.17.2 to v1.19.1 (latest), I hit the problem:

» kubectl -n kube-system get nodes -o wide       
NAME           STATUS     ROLES                AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
test-master1   NotReady   control-plane,etcd   64s   v1.35.1+k3s1   10.0.0.2      <none>        Ubuntu 24.04.3 LTS   6.8.0-90-generic   containerd://2.1.5-k3s1
test-master2   NotReady   control-plane,etcd   17s   v1.35.1+k3s1   10.0.0.4      <none>        Ubuntu 24.04.3 LTS   6.8.0-90-generic   containerd://2.1.5-k3s1
test-master3   NotReady   control-plane,etcd   12s   v1.35.1+k3s1   10.0.0.3      <none>        Ubuntu 24.04.3 LTS   6.8.0-90-generic   containerd://2.1.5-k3s1

Interestingly, external IP is not set for some reason (also I have to take it all down and start from scratch, otherwise hetnzer-k3s tries to connect using private IP).

It seems that cilium agent pods never get ready for some reason:

» kubectl -n kube-system get pods | grep cilium    
cilium-22scq                       0/1     Running   1 (15s ago)   5m34s
cilium-76d92                       0/1     Running   1 (16s ago)   5m34s
cilium-envoy-6gssk                 1/1     Running   0             5m34s
cilium-envoy-6m9ls                 1/1     Running   0             5m34s
cilium-envoy-rbw22                 1/1     Running   0             5m34s
cilium-flhqc                       0/1     Running   1 (16s ago)   5m34s
cilium-operator-77bdc66f5b-xgfrv   0/1     Pending   0             5m34s

Unfortunately I'm not able to debug the root cause at the moment.

[vitobotta]

@codepainters Have you tried using a custom yaml values file for the helm chart with your desired settings?

[codepainters]

I'm not sure, tbh. I tried v1.19 without success and decided to stick with v1.17 for now (my goal was just to run a few experiments, anyway). I ended up with custom Cilium values file, but I no longer remember whether that was before or after trying v1.19.

BTW, it would be really helpful to have an option to save the values file rendered from the template during hetzner-k3s create. It would make things much easier when only one or two values need tweaking.

[codepainters]

I didn't want to leave it like that, so I repeated my experiments, my findings in #744