env-ci-1 k3s Certificate Expired

[pau-hedgehog]

Error from server: Get "https://192.168.90.20:10250/containerLogs/run-hlab/hlab-pg4j9-runner-zw7rs/runner?follow=true": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-02-02T09:42:59Z is after 2026-02-01T21:27:44Z

Cluster status:

kubectl get pods -A

NAMESPACE     NAME                                      READY   STATUS      RESTARTS        AGE
arc           arc-gha-rs-controller-f9c6b7699-2nt7j     1/1     Running     0               250d
... <other pods omitted for brevity> ...
run-hlab      hlab-pg4j9-runner-zw7rs                   2/2     Running     0               14h

Certificate expiry errors also appeared in k3s system logs:

sudo systemctl status k3s

Feb 02 09:56:43 run k3s[1248392]: E0202 09:56:43.450210 1248392 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2026-02-02T09:56:43Z is after 2026-02-01T21:27:44Z"

Workaround

On run.l:

sudo k3s certificate rotate
sudo systemctl restart k3s

Output confirming rotation:

INFO[0000] Server detected, rotating server certificates 
...
INFO[0000] Successfully backed up certificates for all services to path /var/lib/rancher/k3s/server/tls-1770026072, please restart k3s server or agent to rotate certificates 

On env-ci-1:

sudo systemctl restart k3s-agent

After that:

kubectl get nodes
NAME         STATUS                        ROLES                       AGE     VERSION
env-ci-1.l   Ready                         <none>                      388d    v1.28.5+k3s1
run          Ready                         control-plane,etcd,master   2y17d   v1.28.5+k3s1
...

kubectl get pods -A
run-hlab      hlab-pg4j9-runner-d49mp                   2/2     Running     0               87s

[pau-hedgehog]

@Frostman, we have a year to think what we do about cert rotation :)

ubuntu@run:~$ sudo openssl x509   -in /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt   -noout -dates
notBefore=Jan 17 08:14:04 2024 GMT
notAfter=Feb  2 09:54:50 2027 GMT