Help with k3d networking and how to expose two apps/services over unique ports?

[ericis]

This question has been cross-posted to StackOverflow here: https://stackoverflow.com/questions/68547804/how-to-expose-two-apps-services-over-unique-ports-with-k3d

I am attempting to expose two services over two ports. As an alternative, I'd also love to know how to expose them over the same port and use different routes. I've attempted a few articles and a lot of configurations. Let me know where I'm going wrong with the networking of k3d + k3s / kubernetes + traefik (+ klipper?)...

I posted an example:
https://github.com/ericis/k3d-networking

The goal:

• Reach "app-1" on host over port 8080
• Reach "app-2" on host over port 8091

Steps

See: files in repo

1. Configure k3d cluster and expose app ports to load balancer

   ports:
     # map localhost to loadbalancer
     - port: 8080:80
       nodeFilters:
         - loadbalancer
     # map localhost to loadbalancer
     - port: 8091:80
       nodeFilters:
         - loadbalancer
   

2. Deploy apps with "deployment.yaml" in Kubernetes and expose container ports

   ports:
     - containerPort: 80
   

3. Expose services within kubernetes. Here, I've tried two methods.

   • Using CLI

     $ kubectl create service clusterip app-1 --tcp=8080:80
     $ kubectl create service clusterip app-2 --tcp=8091:80
     

   • Using "service.yaml"

     spec:
       ports:
       - protocol: TCP
         # expose internally
         port: 8080
         # map to app
         targetPort: 80
       selector:
         run: app-1
     

4. Expose the services outside of kubernetes using "ingress.yaml"

   backend:
     service:
       name: app-1
       port:
         # expose from kubernetes
         number: 8080
   

[iwilltry42]

Hi @ericis , thanks for starting this discussion!
I just came back from vacation, so sorry for the late reply.

Have you checked out https://k3d.io/usage/guides/exposing_services/#1-via-ingress-recommended already?

The ingress way described in the docs has the following flow (roughly, including abstractions affecting the flow):

Client (e.g. curl or your browser) -> k3d loadbalancer (nginx, external to the cluster) -> k3d node (k3s container) -> klipper servicelb (uses hostPort to expose type: LoadBalancer service of traefik inside the cluster) -> traefik (ingress) -> Service -> Pod -> Container

Going over your steps:

1. configured docker to map the ports 8080 and 8091 of your machine to port 80 on the k3d loadbalancer, which in turn forwards this to port 80 on the server nodes of the k3d cluster (i.e. k3s containers)
   -> This means, that the traffic will reach the ingress controller (traefik) on port 80 (http)
2. this is actually not required (but good for clarity), leaving this out wouldn't change anything
3. create services listening on 8080 and 8091, directing the traffic to port 80 on the target pods
   -> port 8080 and 8091 could be anything here, as how you defined them in the k3d config file doesn't affect the cluster in any way (it's only configured outside of the cluster in docker)
4. create an ingress pointing to those services

How you'd access them now:

• curl localhost:8080/ or curl localhost:8091/, but you cannot know for sure which service you will reach, as both are using the same path prefix (/)

After all, you could set both up with the same ports and just different names/labels and then use different paths (https://github.com/ericis/k3d-networking/blob/main/k8s-ingress-2.yaml#L12) in the ingress.
Then access them e.g. via curl localhost:8080/app-1 and curl localhost:8080/app-2.

I didn't test your manifests, so I may have missed something, but that's the theoretical part so far 👍

[myusufid]

so actually we can't have 2 apps with different ports?