proxy grpc/http2

[ilusharulkov]

How route grpc/http2 traffic to pods

k3d version v4.2.0

Hello! Thx for k3d. It is amazing

I have some trouble with route http2 traffic to pod inside cluster.
As I understend all traffic income to container rancher/k3d-proxy. It exposes 3 ports:

80
443
6443 <- kube api

After thet traffic route to nodes (containers), as described in /etc/nginx/nginx.conf (upstream section):

error_log stderr notice;

worker_processes auto;
events {
  multi_accept on;
  use epoll;
  worker_connections 1027;
}

stream {
  upstream server_nodes_6443 {
    server k3d-dev-server-0:6443 max_fails=1 fail_timeout=10s;
  }

  server {

    listen        6443;
    proxy_pass    server_nodes_6443;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }
  upstream server_nodes_443 {
    server k3d-dev-server-0:443 max_fails=1 fail_timeout=10s;
  }

  server {
    listen        443 ssl;
    proxy_pass    server_nodes_443;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }
  upstream server_nodes_80 {
    server k3d-dev-server-0:80 max_fails=1 fail_timeout=10s;
  }

  server {
    listen        80;
    proxy_pass    server_nodes_80;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }
}

As I see there's no http2 parameter in 'listen' section. I think it is the reason on issue.
Error after I add http2 section to config:

the invalid "http2" parameter in /etc/nginx/nginx.conf:26

I think expose NodePort is the bad solution.

Any ideas of route http2 traffic to cluster via ingress controller ?

[iwilltry42]

Hi @ilusharulkov, thanks for opening this issue!
What's the error logs that you get when trying to access your application via grpc via Ingress?
Did you specify the grpc protocol in your Ingress definition?
The load balancer in front of the cluster is doing plain TCP proxying, so shouldn't need any extra configuration for grpc.
I'll give it a try myself 👍

[iwilltry42]

I just gave it a try and it worked without issues like this:

1. Create k3d cluster with port-forwarding on loadbalancer: k3d cluster create testgrpc --port 8443:443@loadbalancer --port 8080:80@loadbalancer
2. Deploy example grpc mock-server with the following manifests (based on https://github.com/ciena/docker-grpc-mock)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grpctest
  labels:
      name: grpctest
  annotations:
    ingress.kubernetes.io/protocol: h2c
spec:
  rules:
  - host: grpctest.localhost
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: grpctest
            port: 
              number: 50051
  tls:
    - hosts:
      - grpctest.localhost
---
apiVersion: v1
kind: Service
metadata:
  name: grpctest
spec:
  selector:
    app: grpctest
  ports:
  - port: 50051
    targetPort: 50051

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grpctest
spec:
  selector:
    matchLabels:
      app: grpctest
  template:
    metadata:
      labels:
        app: grpctest
    spec:
      containers:
      - name: grpctest
        image: iwilltry42/grpc-mock:20210330.example-1
        env:
          - name: GRPC_VERBOSITY
            value: DEBUG
          - name: GRPC_TRACE
            value: all
        command:
          - node
          - /proto/mock.js
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 50051

3. Test from localhost using grpcurl (https://github.com/fullstorydev/grpcurl): grpcurl -proto ./example/greeter.proto -plaintext -d '{"message": "Hello"}' grpctest.localhost:8080 greeter.Greeter/Hello (inside the https://github.com/ciena/docker-grpc-mock repository root directory)

$ grpcurl -proto ./example/greeter.proto -plaintext -d '{"message": "Hello"}' grpctest.localhost:8080 greeter.Greeter/Hello
{
  "message": "Hello"
}

[ilusharulkov]

Hello, @iwilltry42 !
I forgot one important information 😳...
I use k3d with --disable=traefik and my ingress conroller is nginx
When i make an grpc call to service in nginx's pod:

10.42.1.147 - - [30/Mar/2021:19:53:06 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - 216f90a9c599384cc6e1646625d9f8d4

My ingress conroller configuration:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: {{ .Chart.Name }}-search
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
spec:
  rules:
    - host: {{...}}
      http:
        paths:
          - backend:
              serviceName: {{ .Chart.Name }}-search
              servicePort: grpc
            path: /pro.Search

[iwilltry42]

To me this looks about right on first sight.
I assume you went through the respective documentation? https://kubernetes.github.io/ingress-nginx/examples/grpc/
Let's start from the beginning then. What error do you even see when trying to connect to your service from the outside via ingress?

[ilusharulkov]

In my client app (golang) I see error:

rpc error: code = Unavailable desc = connection closed

my test function:

func main() {
	conn, err := grpc.Dial(fmt.Sprintf("%s:%d", "hostname", 443), grpc.WithInsecure())
	if err != nil {
		panic(err)
	}
	client := search.NewSearchClient(conn)

	res, err := client.AutoAdd(context.TODO(), nil)
	if err != nil {
		logrus.WithError(err).Fatal("err")
	}
	logrus.Info(res)
}

at this moment in nginx log:

10.42.1.1 - - [31/Mar/2021:18:20:38 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - 81d9d30fe3620dbd31692d321d44faf5 

nginx ingress version:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v0.44.0
  Build:         f802554ccfadf828f7eb6d3f9a9333686706d613
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.6

-------------------------------------------------------------------------------

I have some thought about cerificates. May be I need sertificate for this request ?

[ilusharulkov]

Yes )
I've create self sign certificate, add

crt
key

file to ingress controller description, and crt file to client and it is work!
thx for your help!

[iwilltry42]

Okay, so to wrap this up: You needed to add the TLS/SSL certificate (self-signed) to the ingress controller and then add the tls section to your ingress definition, right?
Which makes sense given the fact that you're accessing it via 443 and thus need to have this there.
Glad that it's working now for you :)