[HELP] Rancher wouldn't start on k3d v4.0.0, k3s v1.19.7+k3s1

[bbhenry]

What did you do

• How was the cluster created?

k3d cluster create rancher --image rancher/k3s:v1.19.7-k3s1 --api-port 6550 --servers 1 --agents 3 --port 443:443@loadbalancer

• What did you do afterwards?

kubectl create namespace cert-manage
helm install cert-manager jetstack/cert-manager --namespace cert-manager --set installCRDs=true --wait
kubectl create namespace cattle-system
helm install rancher rancher-latest/rancher --namespace cattle-system --set hostname=rancher.k3d.localhost

What did you expect to happen

I expect latest Rancher to be deployed and all related pods running without issues.

Screenshots or terminal output

Pods you see in the snapshot in red color keeps failing or restarting. All I can see is the Liveness probe failed and readiness probe failed on the Rancher pod for connection refused response.

k3d-rancher-agent-1  Readiness probe failed: Get "http://10.42.2.6:80/healthz": dial tcp 10.42.2.6:80: connect: connection refused

Which OS & Architecture

MacOs Catalina (10.15.7)
2.3 GHz 8-Core Intel Core i9
16 GB DDR4 RAM

Which version of k3d

k3d version v4.0.0
k3s version v1.19.7+k3s1

Which version of docker

docker version

Client: Docker Engine - Community
 Cloud integration: 1.0.7
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:12:42 2020
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8891c58
  Built:            Mon Dec 28 16:15:28 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.5.0)

Server:
 Containers: 8
  Running: 5
  Paused: 0
  Stopped: 3
 Images: 7
 Server Version: 20.10.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.121-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 1.944GiB
 Name: docker-desktop
 ID: E6DQ:3EYY:NGTG:EGNI:RJKX:D7OW:MBTB:MZRG:HKFB:ITFT:OV4E:LQ7X
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 81
  Goroutines: 76
  System Time: 2021-02-15T09:15:16.251852777Z
  EventsListeners: 3
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[iwilltry42]

Hey @bbhenry , I moved this to a discussion, as I do not think that it's closely related to k3d itself (e.g. functionality wise).
However, it would be good to get the logs of the restarting rancher pods to see what's going on there.