[occm] Getting timeouts when creating loadbalancers on large cloud.

[mikeruu]

• openstack-cloud-controller-manager (occm)

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:
Getting timeouts when attempting to create service of type LoadBalancer, this is a large public cloud which seems to be taking longer that 1 minute to respond to the request.

Have tried installing multiple versions of the OCCM Helm chart and they all suffer the same issue, including the latest.

What you expected to happen:
I expect a configurable timeout to allow the request to go through and the Loadbalancer gets created.

How to reproduce it:
Similar to issue #1304
Deploy the standard ingress-nginx helm chart with all default values, it gets a timeout context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I1217 16:33:23.882410      10 leaderelection.go:281] successfully renewed lease kube-system/cloud-controller-manager
I1217 16:33:24.434207      10 controller.go:869] Finished syncing service "default/nginx-loadbalancer" (1m0.147072366s)
E1217 16:33:24.434252      10 controller.go:298] error processing service default/nginx-loadbalancer (retrying with exponential backoff): failed to ensure load balancer: error creating loadbalancer kube_service_eq-poc_default_nginx-loadbalancer: error creating loadbalancer {"name":"kube_service_eq-poc_default_nginx-loadbalancer","description":"Kubernetes external service default/nginx-loadbalancer from cluster eq-poc","vip_subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace","vip_network_id":"f508a314-ff2f-457f-94e3-4493de14045e","provider":"amphora","listeners":[{"protocol":"TCP","protocol_port":80,"name":"listener_0_kube_service_eq-poc_default_nginx-loadbalancer","default_pool":{"lb_algorithm":"ROUND_ROBIN","protocol":"TCP","name":"pool_0_kube_service_eq-poc_default_nginx-loadbalancer","members":[{"address":"172.20.8.201","protocol_port":30390,"name":"172.20.8.201","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.53","protocol_port":30390,"name":"172.20.8.53","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.80","protocol_port":30390,"name":"172.20.8.80","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.109","protocol_port":30390,"name":"172.20.8.109","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.118","protocol_port":30390,"name":"172.20.8.118","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.146","protocol_port":30390,"name":"172.20.8.146","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"}],"healthmonitor":{"type":"TCP","delay":5,"timeout":3,"max_retries":1,"max_retries_down":3,"name":"monitor_0_kube_service_eq-poc_default_nginx-loadbalancer"}},"connection_limit":-1,"timeout_client_data":50000,"timeout_member_data":50000,"timeout_member_connect":5000,"timeout_tcp_inspect":0,"allowed_cidrs":["0.0.0.0/0"],"tags":["kube_service_eq-poc_default_nginx-loadbalancer"]}],"tags":["kube_service_eq-poc_default_nginx-loadbalancer"]}: Post "https://octavia.api.sjc3.rackspacecloud.com/v2.0/lbaas/loadbalancers": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I1217 16:33:24.434932      10 event.go:307] "Event occurred" object="default/nginx-loadbalancer" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: error creating loadbalancer kube_service_eq-poc_default_nginx-loadbalancer: error creating loadbalancer {\"name\":\"kube_service_eq-poc_default_nginx-loadbalancer\",\"description\":\"Kubernetes external service default/nginx-loadbalancer from cluster eq-poc\",\"vip_subnet_id\":\"f52e22a9-7856-4caa-8a2a-80a4056d2ace\",\"vip_network_id\":\"f508a314-ff2f-457f-94e3-4493de14045e\",\"provider\":\"amphora\",\"listeners\":[{\"protocol\":\"TCP\",\"protocol_port\":80,\"name\":\"listener_0_kube_service_eq-poc_default_nginx-loadbalancer\",\"default_pool\":{\"lb_algorithm\":\"ROUND_ROBIN\",\"protocol\":\"TCP\",\"name\":\"pool_0_kube_service_eq-poc_default_nginx-loadbalancer\",\"members\":[{\"address\":\"172.20.8.201\",\"protocol_port\":30390,\"name\":\"172.20.8.201\",\"subnet_id\":\"f52e22a9-7856-4caa-8a2a-80a4056d2ace\"},{\"address\":\"172.20.8.53\",\"protocol_port\":30390,\"name\":\"172.20.8.53\",\"subnet_id\":\"f52e22a9-7856-4caa-8a2a-80a4056d2ace\"},{\"address\":\"172.20.8.80\",\"protocol_port\":30390,\"name\":\"172.20.8.80\",\"subnet_id\":\"f52e22a9-7856-4caa-8a2a-80a4056d2ace\"},{\"address\":\"172.20.8.109\",\"protocol_port\":30390,\"name\":\"172.20.8.109\",\"subnet_id\":\"f52e22a9-7856-4caa-8a2a-80a4056d2ace\"},{\"address\":\"172.20.8.118\",\"protocol_port\":30390,\"name\":\"172.20.8.118\",\"subnet_id\":\"f52e22a9-7856-4caa-8a2a-80a4056d2ace\"},{\"address\":\"172.20.8.146\",\"protocol_port\":30390,\"name\":\"172.20.8.146\",\"subnet_id\":\"f52e22a9-7856-4caa-8a2a-80a4056d2ace\"}],\"healthmonitor\":{\"type\":\"TCP\",\"delay\":5,\"timeout\":3,\"max_retries\":1,\"max_retries_down\":3,\"name\":\"monitor_0_kube_service_eq-poc_default_nginx-loadbalancer\"}},\"connection_limit\":-1,\"timeout_client_data\":50000,\"timeout_member_data\":50000,\"timeout_member_connect\":5000,\"timeout_tcp_inspect\":0,\"allowed_cidrs\":[\"0.0.0.0/0\"],\"tags\":[\"kube_service_eq-poc_default_nginx-loadbalancer\"]}],\"tags\":[\"kube_service_eq-poc_default_nginx-loadbalancer\"]}: Post \"https://octavia.api.sjc3.rackspacecloud.com/v2.0/lbaas/loadbalancers\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
I1217 16:33:25.894044      10 leaderelection.go:281] successfully renewed lease kube-system/cloud-controller-manager

Anything else we need to know?:

Interestingly when I grab the payload and send it using CURL, the LoadBalancer gets created successfully.

curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-Auth-Token: $OS_TOKEN" \
  "https://octavia.api.sjc3.rackspacecloud.com/v2/lbaas/loadbalancers" \
-d '{"loadbalancer":{"name":"kube_service_eq-poc_ingress-nginx_ingress-nginx-controller","description":"Kubernetes external service ingress-nginx/ingress-nginx-controller from cluster eq-poc","vip_subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace","provider":"amphora","listeners":[{"protocol":"TCP","protocol_port":80,"name":"listener_0_kube_service_eq-poc_ingress-nginx_ingress-nginx-controller","default_pool":{"lb_algorithm":"ROUND_ROBIN","protocol":"TCP","name":"pool_0_kube_service_eq-poc_ingress-nginx_ingress-nginx-controller","members":[{"address":"172.20.8.118","protocol_port":32333,"name":"172.20.8.118","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.146","protocol_port":32333,"name":"172.20.8.146","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.201","protocol_port":32333,"name":"172.20.8.201","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.53","protocol_port":32333,"name":"172.20.8.53","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.80","protocol_port":32333,"name":"172.20.8.80","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.109","protocol_port":32333,"name":"172.20.8.109","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"}],"healthmonitor":{"type":"TCP","delay":5,"timeout":3,"max_retries":1,"max_retries_down":3,"name":"monitor_0_kube_service_eq-poc_ingress-nginx_ingress-nginx-controller"}},"connection_limit":-1,"timeout_client_data":50000,"timeout_member_data":50000,"timeout_member_connect":5000,"timeout_tcp_inspect":0,"allowed_cidrs":["0.0.0.0/0"],"tags":["kube_service_eq-poc_ingress-nginx_ingress-nginx-controller"]},{"protocol":"TCP","protocol_port":443,"name":"listener_1_kube_service_eq-poc_ingress-nginx_ingress-nginx-controller","default_pool":{"lb_algorithm":"ROUND_ROBIN","protocol":"TCP","name":"pool_1_kube_service_eq-poc_ingress-nginx_ingress-nginx-controller","members":[{"address":"172.20.8.118","protocol_port":32451,"name":"172.20.8.118","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.146","protocol_port":32451,"name":"172.20.8.146","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.201","protocol_port":32451,"name":"172.20.8.201","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.53","protocol_port":32451,"name":"172.20.8.53","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.80","protocol_port":32451,"name":"172.20.8.80","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"},{"address":"172.20.8.109","protocol_port":32451,"name":"172.20.8.109","subnet_id":"f52e22a9-7856-4caa-8a2a-80a4056d2ace"}],"healthmonitor":{"type":"TCP","delay":5,"timeout":3,"max_retries":1,"max_retries_down":3,"name":"monitor_1_kube_service_eq-poc_ingress-nginx_ingress-nginx-controller"}},"connection_limit":-1,"timeout_client_data":50000,"timeout_member_data":50000,"timeout_member_connect":5000,"timeout_tcp_inspect":0,"allowed_cidrs":["0.0.0.0/0"],"tags":["kube_service_eq-poc_ingress-nginx_ingress-nginx-controller"]}],"tags":["kube_service_eq-poc_ingress-nginx_ingress-nginx-controller"]}}' | python -mjson.tool

Environment:

• openstack-cloud-controller-manager(or other related binary) version:
  openstack-cloud-controller-manager-2.28.5 v1.28.3
• OpenStack version: Openstack Helm
• Others: