Can't get health checks to work in ECS

[mig5]

On ECS, the loadbalancer/target group can perform health checks to whatever URL etc you want on the container.

I have mine set to target /health on the container.

I keep getting the errors below

time=2022-01-12T02:26:08Z level=info msg=started handling request http_request=map[headers:map[accept-encoding:gzip, compressed connection:close user-agent:ELB-HealthChecker/2.0] host:10.42.1.57:49312 method:GET path:/health query:<nil> remote:10.42.3.71:4444 scheme:http]

time=2022-01-12T02:26:08Z level=error msg=Could not serve http connection audience=application error=map[message:X-Forwarded-Proto header is missing] http_request=map[headers:map[accept-encoding:gzip, compressed connection:close user-agent:ELB-HealthChecker/2.0] host:10.42.1.57:49312 method:GET path:/health query:<nil> remote:10.42.3.71:4444 scheme:http] service_name=ORY Hydra service_version=v1.10.7

time=2022-01-12T02:26:08Z level=error msg=An error occurred while handling a request audience=application error=map[message:can not serve request over insecure http] http_request=map[headers:map[accept-encoding:gzip, compressed connection:close user-agent:ELB-HealthChecker/2.0] host:10.42.1.57:49312 method:GET path:/health query:<nil> remote:10.42.3.71:4444 scheme:http] http_response=map[status_code:502] service_name=ORY Hydra service_version=v1.10.7

time=2022-01-12T02:26:08Z level=info msg=completed handling request http_request=map[headers:map[accept-encoding:gzip, compressed connection:close user-agent:ELB-HealthChecker/2.0] host:10.42.1.57:49312 method:GET path:/health query:<nil> remote:10.42.3.71:4444 scheme:http] http_response=map[headers:map[content-type:application/json] size:68 status:502 text_status:Bad Gateway took:343.211µs]

Because of this, the container keeps cycling as the 502 is being thrown with {"error":"error","error_description":"The error is unrecognizable"}

This is a surprise to me, as #731 shows this has come up before, but apparently /health should ignore this check? Or is it a regression?

Note: the AWS loadbalancers don't send the Host header, they just use the IP of the host running container, is that a problem?

I have made sure to set all the relevant params as environment variables - I can confirm that the IPs used by the Loadbalancer should be allowed:

        {
          "name": "SERVE_ADMIN_TLS_ALLOW_TERMINATION_FROM",
          "value": "10.42.1.0/24,10.42.3.0/24"
        },
        {
          "name": "SERVE_PUBLIC_TLS_ALLOW_TERMINATION_FROM",
          "value": "10.42.1.0/24,10.42.3.0/24"
        },
        {
          "name": "SERVE_TLS_ALLOW_TERMINATION_FROM",
          "value": "10.42.1.0/24,10.42.3.0/24"
        },

Is it because the request is just http:// from the loadbalancer to port 4444?

[mig5]

My bad - the route is /health/alive (or /health/ready)! I was using / and getting the error, which led me to the PR/issue above, where it was /health, but which doesn't work by itself either. In the end, I used /health/ready as the best indication of a 'healthy' Hydra.

[vinckr]

Hey, no worries. Thanks for supplying your solution 🙌