failling to pull docker.io/rancher/k3s:v1.25.6-k3s1

[sgandon]

With a specific docker setup that I will describe later the command k3d cluster create fails to pull the k3s docker image. But if I pull the image manually right before then k3d install works.

here is the failing logs.

k3d cluster create
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-k3s-default'            
INFO[0000] Created image volume k3d-k3s-default-images  
INFO[0000] Starting new tools node...                   
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
INFO[0001] Pulling image 'ghcr.io/k3d-io/k3d-tools:5.4.7' 
ERRO[0002] Failed Cluster Creation: failed setup of server/agent node k3d-k3s-default-server-0: failed to create node: runtime failed to create node 'k3d-k3s-default-server-0': failed to create container for node 'k3d-k3s-default-server-0': docker failed to pull image 'docker.io/rancher/k3s:v1.25.6-k3s1': docker failed to pull the image 'docker.io/rancher/k3s:v1.25.6-k3s1': Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp 127.0.0.1:443: connect: connection refused 
ERRO[0002] Failed to create cluster >>> Rolling Back    
INFO[0002] Deleting cluster 'k3s-default'               
ERRO[0002] failed to get cluster: No nodes found for given cluster 
FATA[0002] Cluster creation FAILED, also FAILED to rollback changes! 

and here are the succeeding logs with the workaround.

runner@example-runnerdeploy-fzjmm-8wqnv:/$ docker pull rancher/k3s:v1.25.6-k3s1
v1.25.6-k3s1: Pulling from rancher/k3s
b6f3d4acf14e: Pull complete 
d3be551d59b5: Pull complete 
Digest: sha256:93e79fa028112e19360ac2f6042d5981f74f951e23e7fc1a8a57c3bbf7da17ee
Status: Downloaded newer image for rancher/k3s:v1.25.6-k3s1
docker.io/rancher/k3s:v1.25.6-k3s1

runner@example-runnerdeploy-fzjmm-8wqnv:/$ k3d cluster create
INFO[0000] Prep: Network                                
INFO[0000] Re-using existing network 'k3d-k3s-default' (747945eeb86a956ba17beeb2cfcff43a2737f8f715d90fa3b4cf3ddc07441c54) 
INFO[0000] Created image volume k3d-k3s-default-images  
INFO[0000] Starting new tools node...                   
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
INFO[0001] Pulling image 'ghcr.io/k3d-io/k3d-tools:5.4.7' 
INFO[0001] Creating LoadBalancer 'k3d-k3s-default-serverlb' 
INFO[0002] Pulling image 'ghcr.io/k3d-io/k3d-proxy:5.4.7' 
INFO[0003] Starting Node 'k3d-k3s-default-tools'        
INFO[0006] Using the k3d-tools node to gather environment information 
INFO[0006] HostIP: using network gateway 172.18.0.1 address 
INFO[0006] Starting cluster 'k3s-default'               
INFO[0006] Starting servers...                          
INFO[0006] Starting Node 'k3d-k3s-default-server-0'     
INFO[0011] All agents already running.                  
INFO[0011] Starting helpers...                          
INFO[0011] Starting Node 'k3d-k3s-default-serverlb'     
INFO[0018] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap... 
INFO[0020] Cluster 'k3s-default' created successfully!  
INFO[0020] You can now use it like this:                
kubectl cluster-info

It seems that the pulling of the other images did work fine and only the k3s is failing.

So now more about the local setup.
we need to use our docker cache registry and do not want call dockerhub registry directly.
we have setup docker for the mirror like this

 cat /etc/docker/daemon.json 
{
  "registry-mirrors": [
    "https://our.personal.docker.mirror.com"
  ]
}

This is an authenticated registry and we are hitting a long known docker bug.
So in order to workaround it we have setup 2 things :

1/ this workaround

cat .docker/config.json 
{"auths": {
    "https://index.docker.io/v1/": { "auth": "zekey" },
    "our.personal.docker.mirror.com":{ "auth": "zekey" }
    }
}

2/ and more safety we added this workaround
see the last line.

 cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.42.0.217     example-runnerdeploy-fzjmm-8wqnv

# Entries added by HostAliases.
127.0.0.1       index.docker.io registry-1.docker.io    docker.io