How to expose a (service in a) k3d cluster to the Internet (through a public IP)?

[imesimone]

I'm going crazy because I'm totally unable to expose a service in k3d cluster to the Internet.

360° Situation

1. I have a domain.

2. A "vpn" subdomain DNS A record points to my public and static IP.

3. A "dns" subdomain DNS A record points to the same IP.

4. All the relevant ports on the router are open (53, 80, 443, 51820, etc) and redirected to a (Debian) home server.

   1. Port 53 is open because I have a DNSMasq instance running on the home server (on the network interface eno1, which is my "eth0").

   2. Port 51820 is open because I have a Wireguard VPN running on the home server (on the network interface wg0).

5. I have successfully verified that the VPN connection and the DNS are working correctly.

   1. For instance, when the VPN tunnel is open, I'm able to SSH inside my home server through a domain that is only present on its /etc/hosts.

6. On the home server I'm running a k3d cluster with the configuration below.

   1. As you may know, k3d creates a bridge network (k3d-k3s-default) and constraints the k8s nodes (which are containers) inside its subnet.

   2. A Docker bridge network is reachable only by other containers.

   3. The only exposed container is the ServerLB one (basically nginx-proxy with the configuration below), which is listening on ports 80 on all the network interfaces (0.0.0.0).

   4. To attach the cluster to the host network I should be using only one node, but this solution is quite silly.

7. If I deploy an Ingress Route (as in the YAML below), it partially works on the Debian host.

   1. http://argocd.localhost works fine in any web browser (I can access the dashboard).

   2. curl, dig, wget and any other CLI tool do not (they do not resolve it): why?

8. My goal is to make this host reachable inside the VPN (when I'm not at home), but as you may imagine it doesn't work. What I'm missing?

   1. I've tried with LoadBalancer, but nothing.

   2. For instance, this is the load balancer of Traefik, but the ExternalIPs (bold) are in the bridge network subnet, so unreachable: kube-system       traefik                                   LoadBalancer   10.43.134.142   **172.21.0.2,172.21.0.3**   80:32223/TCP,443:30136/TCP            108m

   3. I've tried with MetalLB, but nothing.

   4. Honestly, I've tried so many unsuccessful things that I'd rather keep them private to preserve my honour. 🥹

   5. I'm obviously missing some notions.

   6. Help me, please.

K3d conf

apiVersion: k3d.io/v1alpha4
kind: Simple
servers: 1
agents: 1
kubeAPI:
  host: "server.net"
  hostIP: "192.168.1.100"
  hostPort: "6445"
ports:
  - port: 80:80
    nodeFilters:
      - loadbalancer
registries:
  create:
    name: k3d-registry.localhost
    host: "0.0.0.0"
    hostPort: "5000"

Nginx configuration

stream {

  upstream 6443_tcp {
    server k3d-k3s-default-server-0:6443 max_fails=1 fail_timeout=10s;
  }

  server {
    listen        6443;
    proxy_pass    6443_tcp;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }

  upstream 80_tcp {
    server k3d-k3s-default-agent-0:80 max_fails=1 fail_timeout=10s;
    server k3d-k3s-default-server-0:80 max_fails=1 fail_timeout=10s;
  }

  server {
    listen        80;
    proxy_pass    80_tcp;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }

}

ArgoCD IngressRoute

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: argocd
  namespace: argocd
spec:
  routes:
    - match: Host(`argocd.localhost`)
      kind: Rule
      services:
      - name: argocd-server
        port: 80