ALB targetgroup healthcheck to Gateway HTTPRoute with hostnames field set returns 404

[remicode]

Description
alb healthcheck to internal Gateway returns 404 while browsing to the same url returns 200 when hostnames field is set on HTTPRoute.

The setup is a aws alb gateway with HTTPRoute(hostnames set) ==> istio gateway with HTTPRoute(hostnames set) ==> internal kubernetes service

visiting http://test.example.com/productpage returns 200 but health check to path /productpage returns 404.

Example manifests found here

Manifests here

apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: aws-alb-gateway-class
spec:
  controllerName: gateway.k8s.aws/alb
---
# my-nlb-gateway.yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: my-aws-http-gateway
spec:
  gatewayClassName: aws-alb-gateway-class
  infrastructure:
    parametersRef:
      group: gateway.k8s.aws
      kind: LoadBalancerConfiguration
      name: aws-alb-gateway-config # Must be in the same namespace as the Gateway
  listeners:
  - name: http-app
    hostname: "test.example.com"
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: Same
--- 
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: bookinfo-http
spec:
  hostnames:
    - "test.example.com"
  parentRefs:
    - name: my-aws-http-gateway
      sectionName: http-app # Refers to the specific listener on the Gateway
  rules:
  - backendRefs:
    - name: bookinfo-gateway-istio # Kubernetes Service
      port: 80    
---
apiVersion: gateway.k8s.aws/v1beta1
kind: LoadBalancerConfiguration
metadata:
  name: aws-alb-gateway-config
spec:
  scheme: internet-facing
---
apiVersion: gateway.k8s.aws/v1beta1
kind: TargetGroupConfiguration
metadata:
  name: bookinfo-gateway-istio-tg-config
spec:
  targetReference:
    name: bookinfo-gateway-istio
  defaultConfiguration:
    healthCheckConfig:
      healthCheckProtocol: http
      healthCheckPath: /productpage
      healthCheckInterval: 60
      healthyThresholdCount: 3
---
#Internal east-west gateway
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: bookinfo-gateway
  annotations:
    networking.istio.io/service-type: NodePort
  labels:
    app: bookinfo-gateway
    version: "v1"
spec:
  gatewayClassName: istio
  listeners:
    - name: http
      hostname: "*.example.com"
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: bookinfo
spec:
  hostnames:
    - "test.example.com"
  parentRefs:
    - name: bookinfo-gateway
  rules:
    - matches:
        - path:
            type: Exact
            value: /productpage
      backendRefs:
        - name: productpage
          port: 9080
---
apiVersion: v1
kind: Service
metadata:
  name: productpage
  labels:
    app: productpage
    service: productpage
    istio.io/ingress-use-waypoint: "true"
spec:
  ports:
    - port: 9080
      name: http
  selector:
    app: productpage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-v1
  labels:
    app: productpage
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v1
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9080"
        prometheus.io/path: "/metrics"
      labels:
        app: productpage
        version: v1
    spec:
      serviceAccountName: bookinfo-productpage
      containers:
        - name: productpage
          image: docker.io/istio/examples-bookinfo-productpage-v1:1.19.1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9080
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}

• Controller logs/error messages while reproducing the issue:
  Only thing in logs controller logs which see irrelevant:

,"reconcileID":"67dbe2c7-cd32-44cf-880b-70ccca943def","error":"targetgroup configuration [istio-bookinfo-ambient/bookinfo-gateway-istio-tg-config] is still in use"

Internal gateway log when page is loaded from browser

 "GET /productpage HTTP/1.1" 200 - via_upstream - "-" 0 5290 106 81

Internal gateway log for health check

"GET /productpage HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "ELB-HealthChecker/2.0" 

Expected Behavior
health check to path /productpage should return 200 same as visiting http://test.example.com/productpage from browser.

Actual Behavior
visiting http://test.example.com/productpage from browser loads page correctly and returns 200 but health check from load balancer to a path /productpage returns 404.

Environment

• AWS Load Balancer controller version: `v2.14.01
• Kubernetes version: v1.32.9-eks-113cf36

Additional Context
The traffic is being routed correctly for both the health check and browsing of the service but because the health check from the load balancer doesn't set the hostname field the second HTTPRoute in front of the internal service doesn't see any matching routes and returns a 404.

This is confirmed by removing the hostname and hostnames field from the internal gateway and HTTPRoute which results in a successful 200 for the health check from the load balancer. "GET /productpage HTTP/1.1" 200 - via_upstream - "ELB-HealthChecker/2.0"

Can you provide some guidance on if this scenario should work and and how one would successfully set a health check for a specific service in this case?

[zac-nixon]

I don't think it's possible to configure hostname passing for the ALB Healthcheck. One work around I can think of is setting up a healthcheck route that serves healthcheck data statically, e.g. without hostname based routing.

[remicode]

Thank you for responding so quickly.
What I am trying to achieve is to have a shared load balancer(potentially different hostnames) for multiple services and a shared ingress gateway(east-west) say per namespace.

In this multi backend service setup, the only way hostname-less routing would work would be to have a different health check path for each service. So you wouldn't be able to say have /health be the health check path for 2 different backend services.

Does that sound correct?
I guess i'm trying to figure out the right setup to share an alb for different hostname urls using the gateway api and the controller? Is there any sort of docs/guide on what that could look like?

[zac-nixon]

I think this question is more general, and we're constrained by what AWS ALB supports. This stack overflow is still relevant in terms of AWS ALB health check capabilities. https://stackoverflow.com/questions/50728484/health-check-route-organisation-in-microserviceish-setup-behind-aws-alb. It's exactly what you describe, multiple health check paths.

We're still working on our documentation for the Gateway API. I don't think the different hostname URLs setup is any different than setting up the infrastructure with the Ingress API. I'll think about how to document this use-case.

[remicode]

I was able to get multiple hostnames on a single alb working successfully using the manifests in the original post and adding additional HTTPRoutes with a hostname for each additional service. The only issue I ran into was the health check returning 404.

My initial thought was, why did this work successfully with ingress when there is a hosts field as well on an ingress?
I suspect the handoff to the second gateway is the issue. If I had the HTTPRoutes on just the the initial controller's gateway and hand off the request to the internal service directly then the health check might have worked as it did with ingress.

Thank you for taking the time to respond and for the controller as well. It's a great product to use.