bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused`

[vortegatorres]

Current Behavior

After scaling a backend Deployment up and then back down, APISIX continues to keep the removed Pod IP in the upstream/endpoints set and keeps trying to route traffic to it.

Observed timeline (UTC, sanitized IPs):

• ~11:20 UTC: scaled backend from 3 → 4 replicas for ~1 hour (new pod IP = X).
• ~12:20 UTC: scaled back down 4 → 3 (pod with IP X removed).
• After scale-down, APISIX still routed to X:8000 (stale) and produced spikes of:
  • 111: Connection refused
• Many successful traces/logs during the error window showed both stale and valid upstreams at the same time, e.g. (second request succeded after retried):
  -upstream_addr: "X:8000, Y:8000"
• Hours later (~04:00 UTC), after restarting the APISIX deployment, the stale IP stopped appearing in upstream_addr and the errors stopped.

Expected Behavior

After the Pod is deleted and Kubernetes Endpoints/EndpointSlices are updated, APISIX should stop routing to the removed Pod IP immediately (or within the expected controller sync/update window), and should not keep stale upstream targets.

Error Logs

For some requests, we first show the (111: Connection refused) error.

2026-01-29 04:05:10.638 error 2026/01/29 03:05:10 [error] 49#49: *1392456 connect() failed (111: Connection refused) while connecting to upstream, client: XXXXXX, server: _, request: "POST /api HTTP/1.1", upstream: "http://YYYYYY:8000/api", host: "url" 

And then the retried successfully request with the two IPs in the upstream_addr (staled IP + correct IP):

2026-01-29 04:05:11.430 {
  "ts": "2026-01-29T03:05:11+00:00",
  "service": "apisix",
  "resp_body_size": "0",
  "host": "url",
  "address": "XXXXX",
  "request_length": "595",
  "method": "POST",
  "uri": "/api",
  "status": "204",
  "user_agent": "Go-http-client/2.0",
  "resp_time": "0.512",
  "upstream_addr": "X:8000, Y:8000",
  "upstream_status": "502, 204",
  "traceparent": "00-0bc72f67dd2ee53ce6d7f03e4a4eb7d6-3a5171c169e8eb46-01",
  "trace_id": "0bc72f67dd2ee53ce6d7f03e4a4eb7d6",
  "span_id": "3a5171c169e8eb46",
  "org_slug": "",
  "matched_uri": ""
} 

Steps to Reproduce

This error is not deterministic; it doesn't always happen, but if you were to try to reproduce it, this would be the closest way to do so:

• Deploy APISIX using the APISIX Helm chart v2.12.6 in standalone mode (no etcd).
• Deploy a backend service behind APISIX (any Kubernetes Service/Deployment that APISIX routes to).
• Run steady traffic through APISIX to that backend.
• Temporarily scale the backend Deployment up (e.g., from 3 replicas to 4) and keep it running for ~1 hour.
• Scale the backend back down to the original replica count (e.g., 4 → 3), so one Pod is terminated and its IP is removed.

Environment

• APISIX Helm chart 2.12.6.
• APISIX Ingress controller version: 2.0.1
• Kubernetes cluster version: 1.32
• OS version if running APISIX Ingress controller in a bare-metal environment: Linux x86_64 GNU/Linux
• ACD: 0.23.1

[AlinsRan]

Thank you for reporting this issue and for the details you’ve provided.

We attempted to reproduce the problem in our local environment using the following steps:

1. deploy apisix and apisix-ingress-controller:

helm install apisix --version 2.12.6 \
  --namespace ingress-apisix \
  --create-namespace \
  --set apisix.deployment.role=traditional \
  --set apisix.deployment.role_traditional.config_provider=yaml \
  --set etcd.enabled=false \
  --set ingress-controller.enabled=true \
  --set ingress-controller.config.provider.type=apisix-standalone \
  --set ingress-controller.apisix.adminService.namespace=ingress-apisix \
  --set ingress-controller.gatewayProxy.createDefault=true \
  apisix/apisix

2. deploy httpbin upstream:

kubectl apply -f https://raw.githubusercontent.com/apache/apisix-ingress-controller/refs/heads/v2.0.0/examples/httpbin/deployment.yaml

3. apply route

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: getting-started-ip
spec:
  ingressClassName: apisix
  http:
    - name: getting-started-ip
      match:
        paths:
          - /ip
      backends:
        - serviceName: httpbin
          servicePort: 80

4. scale httpbin to 4 replicas:

kubectl scale deployment httpbin-deployment --replicas=4

Check the upstream configuration of the data plane:

curl localhost:9180/apisix/admin/configs -H "X-API-Key: edd1c9f034335f136f87ad84b625c8f1" -s | jq .upstreams
[
  {
    "id": "ad735c2f",
    "type": "roundrobin",
    "labels": {
      "managed-by": "apisix-ingress-controller"
    },
    "name": "default_getting-started-ip_0",
    "nodes": [
      {
        "port": 80,
        "host": "10.244.0.7",
        "weight": 100
      },
      {
        "port": 80,
        "host": "10.244.0.15",
        "weight": 100
      },
      {
        "port": 80,
        "host": "10.244.0.14",
        "weight": 100
      },
      {
        "port": 80,
        "host": "10.244.0.13",
        "weight": 100
      }
    ],
    "modifiedIndex": 1769998251570
  }
]

Check the logs after sending the request:

5. scale httpbin to 1 replicas:

kubectl scale deployment httpbin-deployment --replicas=1

Check the upstream configuration of the data plane:

curl localhost:9180/apisix/admin/configs -H "X-API-Key: edd1c9f034335f136f87ad84b625c8f1" -s | jq .upstreams
[
  {
    "id": "ad735c2f",
    "type": "roundrobin",
    "labels": {
      "managed-by": "apisix-ingress-controller"
    },
    "name": "default_getting-started-ip_0",
    "nodes": [
      {
        "port": 80,
        "host": "10.244.0.7",
        "weight": 100
      }
    ],
    "modifiedIndex": 1769998390864
  }
]

Check the logs after sending the request:

However, under these conditions, we were unable to reproduce the issue.

To help us investigate further, could you please provide some additional information?

Minimal Reproducible Example

• A simplified configuration or setup that can reliably reproduce the issue
• If applicable, a sample request along with the expected and actual behavior

Logs or Error Output

• Relevant error or warning logs
• Stack traces or additional diagnostic output, if available

Additional Context

• Whether the issue occurs consistently or intermittently
• Any specific conditions under which the issue is more likely to occur

In addition, please check whether there are any misconfigured routes or plugins.
In standalone mode, if any route or plugin configuration is invalid, the entire full configuration set may fail to be synchronized to the data plane, including upstream addrs.

With this information, we should be able to narrow down the root cause more effectively.
Thanks again for your report, and we appreciate your help in moving this forward.

[vortegatorres]

As mentioned earlier, the issue is intermittent and not deterministic. In almost all cases where we observed the replica count scaling up and down, everything worked as expected. Because of that, I can’t provide additional reproduction steps beyond what I already did and what you’ve already tried.

Regarding the logs: during the incident, APISIX first reported a (111: Connection refused) while connecting to the upstream:

2026-01-29 04:05:10.638 error 2026/01/29 03:05:10 [error] 49#49: *1392456 connect() failed (111: Connection refused) while connecting to upstream, client: XXXXXX, server: _, request: "POST /api HTTP/1.1", upstream: "http://YYYYYY:8000/api", host: "url"

Right after that, the request was retried and succeeded. The key detail is that upstream_addr contains two IPs: it first tried a stale IP, then retried using the IP that worked:

2026-01-29 04:05:11.430 {
  "ts": "2026-01-29T03:05:11+00:00",
  "service": "apisix",
  "resp_body_size": "0",
  "host": "url",
  "address": "XXXXX",
  "request_length": "595",
  "method": "POST",
  "uri": "/api",
  "status": "204",
  "user_agent": "Go-http-client/2.0",
  "resp_time": "0.512",
  "upstream_addr": "X:8000, Y:8000",
  "upstream_status": "502, 204",
  "traceparent": "00-0bc72f67dd2ee53ce6d7f03e4a4eb7d6-3a5171c169e8eb46-01",
  "trace_id": "0bc72f67dd2ee53ce6d7f03e4a4eb7d6",
  "span_id": "3a5171c169e8eb46",
  "org_slug": "",
  "matched_uri": ""
}

Sorry — I don’t have more relevant logs or additional steps to reproduce. This seems like a corner case and may not be easy to reproduce reliably.

[AlinsRan]

When the issue occurs, are there any error logs in the Ingress Controller?

If endpoint updates are delayed, this situation may occur temporarily.
Could you clarify whether the problem recovers automatically after some time, or if it persists for an extended period?

If it persists for a long time, does reducing the interval of scheduled full synchronization help mitigate the issue, or does the problem remain even after multiple full sync cycles?

[vortegatorres]

When the issue occurs, are there any error logs in the Ingress Controller?

Sorry, I missed this earlier. This isn't happening just during the incident, ingress-controller is consistently logging this error:

2026-02-03 23:26:15.434 error 2026-02-03T22:26:15.434Z	ERROR	provider	apisix/provider.go:282	failed to sync	{"error": "failed to sync 1 configs: GatewayProxy/k6-api-prod/apisix-config"} 

2026-02-03 23:26:15.432 error 2026-02-03T22:26:15.431Z	ERROR	provider.client	client/client.go:210	failed to sync resources	{"name": "GatewayProxy/k6-api-prod/apisix-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/k6-api-prod/apisix-config: [ServerAddr: http://<REDACTED_IP_1>:9180,http://<REDACTED_IP_2>:9180,http://<REDACTED_IP_3>:9180,http://<REDACTED_IP_4>:9180,http://<REDACTED_IP_5>:9180,http://<REDACTED_IP_6>:9180, Err: socket hang up]]"} 

2026-02-03 23:26:15.431 error 2026-02-03T22:26:15.431Z	ERROR	provider.client	client/client.go:269	failed to execute adc command	{"config": {"name":"GatewayProxy/k6-api-prod/apisix-config","serverAddrs":["http://<REDACTED_IP_1>:9180","http://<REDACTED_IP_2>:9180","http://<REDACTED_IP_3>:9180","http://<REDACTED_IP_4>:9180","http://<REDACTED_IP_5>:9180","http://<REDACTED_IP_6>:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/k6-api-prod/apisix-config: [ServerAddr: http://<REDACTED_IP_1>:9180,http://<REDACTED_IP_2>:9180,http://<REDACTED_IP_3>:9180,http://<REDACTED_IP_4>:9180,http://<REDACTED_IP_5>:9180,http://<REDACTED_IP_6>:9180, Err: socket hang up]"} 

2026-02-03 23:26:15.431 error 2026-02-03T22:26:15.431Z	ERROR	provider.executor	client/executor.go:142	failed to run http sync for server	{"server": "http://<REDACTED_IP_1>:9180,http://<REDACTED_IP_2>:9180,http://<REDACTED_IP_3>:9180,http://<REDACTED_IP_4>:9180,http://<REDACTED_IP_5>:9180,http://<REDACTED_IP_6>:9180", "error": "ServerAddr: http://<REDACTED_IP_1>:9180,http://<REDACTED_IP_2>:9180,http://<REDACTED_IP_3>:9180,http://<REDACTED_IP_4>:9180,http://<REDACTED_IP_5>:9180,http://<REDACTED_IP_6>:9180, Err: socket hang up"} 

2026-02-03 23:26:15.431 error 2026-02-03T22:26:15.431Z	ERROR	provider.executor	client/executor.go:328	ADC Server sync failed	{"result": {"status":"all_failed","total_resources":6,"success_count":0,"failed_count":6,"success":[],"failed":[{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket hang up","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket hang up","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket hang up","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket hang up","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket hang up","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket hang up","response":{"status":0,"headers":null}}]}, "error": "ADC Server sync failed: socket hang up"} 

I found a known fix described here: #2704. That said, I don’t think it’s the source of this issue in our case. When this happens, I still see plenty of log lines indicating the sync completed successfully, despite the errors, for example:

INFO	provider.client	client/client.go:177	syncing all resources 

If endpoint updates are delayed, this situation may occur temporarily. Could you clarify whether the problem recovers automatically after some time, or if it persists for an extended period?

It recovers after restarting the apisix deployment.

If it persists for a long time, does reducing the interval of scheduled full synchronization help mitigate the issue, or does the problem remain even after multiple full sync cycles?

How can I reduce the interval?

Also @AlinsRan, can the use of Upstream health-checks help with the problem: https://docs.api7.ai/apisix/how-to-guide/traffic-management/health-check/

[AlinsRan]

How can I reduce the interval?

Endpoint updates will also immediately trigger synchronization.

https://github.com/apache/apisix-helm-chart/blob/7e5b8dff478dcc86373d6bd9071ab5444c97602b/charts/apisix-ingress-controller/values.yaml#L96

INFO provider.client client/client.go:177 syncing all resources

It indicates that the synchronization is in progress, not that it has been completed successfully.
If the synchronization succeeds, APISIX should not retain IPs of Pods that have already been deleted.

use of Upstream health-checks

Using health check ups can alleviate this issue.

socket hang up

I think you can solve this problem first and then continue to observe.

It recovers after restarting the apisix deployment.

What I find confusing is why restarting APISIX resolves the issue, whereas restarting apisix-ingress-controller does not. This seems to suggest that the problem may not be solely on the controller side.

To better understand the situation, could you please share more details about your deployment?

• How many APISIX Pods are running?
• Are there any CPU or memory limits configured for APISIX or apisix-ingress-controller?
• Have you observed the CPU and memory usage of both APISIX and apisix-ingress-controller around the time when the issue occurs?

[vortegatorres]

Using health check ups can alleviate this issue.

I will try this then.

I think you can solve this problem first and then continue to observe.

I applied the fix defined in that issue, and the number of errors dropped drastically, but still it see few of these errors:

2026-02-05 11:29:27.240 error 2026-02-05T10:29:27.240Z	ERROR	provider	apisix/provider.go:282	failed to sync	{"error": "failed to sync 1 configs: GatewayProxy/<REDACTED>/apisix-config"}

2026-02-05 11:29:27.238 error 2026-02-05T10:29:27.238Z	ERROR	provider.client	client/client.go:210	failed to sync resources	{"name": "GatewayProxy/<REDACTED>/apisix-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/<REDACTED>/apisix-config: [ServerAddr: http://<SERVER_1>:9180,http://<SERVER_2>:9180,http://<SERVER_3>:9180,http://<SERVER_4>:9180,http://<SERVER_5>:9180, Err: connect ECONNREFUSED <SERVER_5>:9180]]"}

2026-02-05 11:29:27.238 error 2026-02-05T10:29:27.238Z	ERROR	provider.client	client/client.go:269	failed to execute adc command	{"config": {"name":"GatewayProxy/<REDACTED>/apisix-config","serverAddrs":["http://<SERVER_1>:9180","http://<SERVER_2>:9180","http://<SERVER_3>:9180","http://<SERVER_4>:9180","http://<SERVER_5>:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/<REDACTED>/apisix-config: [ServerAddr: http://<SERVER_1>:9180,http://<SERVER_2>:9180,http://<SERVER_3>:9180,http://<SERVER_4>:9180,http://<SERVER_5>:9180, Err: connect ECONNREFUSED <SERVER_5>:9180]"}

2026-02-05 11:29:27.238 error 2026-02-05T10:29:27.238Z	ERROR	provider.executor	client/executor.go:142	failed to run http sync for server	{"server": "http://<SERVER_1>:9180,http://<SERVER_2>:9180,http://<SERVER_3>:9180,http://<SERVER_4>:9180,http://<SERVER_5>:9180", "error": "ServerAddr: http://<SERVER_1>:9180,http://<SERVER_2>:9180,http://<SERVER_3>:9180,http://<SERVER_4>:9180,http://<SERVER_5>:9180, Err: connect ECONNREFUSED <SERVER_5>:9180"}

2026-02-05 11:29:27.238 error 2026-02-05T10:29:27.237Z	ERROR	provider.executor	client/executor.go:328	ADC Server sync failed	{"result": {"status":"partial_failure","total_resources":5,"success_count":4,"failed_count":1,"success":[{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"0001-01-01T00:00:00Z","synced_at":"2026-02-05T10:29:23Z","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"0001-01-01T00:00:00Z","synced_at":"2026-02-05T10:29:23Z","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"0001-01-01T00:00:00Z","synced_at":"2026-02-05T10:29:23Z","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"0001-01-01T00:00:00Z","synced_at":"2026-02-05T10:29:23Z","response":{"status":0,"headers":null}}],"failed":[{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-05T10:29:27.237Z","synced_at":"0001-01-01T00:00:00Z","reason":"connect ECONNREFUSED <SERVER_5>:9180","response":{"status":0,"headers":null}}]}, "error": "ADC Server sync failed: connect ECONNREFUSED <SERVER_5>:9180"}

How many APISIX Pods are running?

Between 4 and 15.

Are there any CPU or memory limits configured for APISIX or apisix-ingress-controller?
Have you observed the CPU and memory usage of both APISIX and apisix-ingress-controller around the time when the issue occurs?

Yes, we have limits, but during the incidents, the CPU and memory usages were very low.

[AlinsRan]

It recovers after restarting the apisix deployment.

I would like to confirm again whether restarting APISIX alone can resolve this issue, and if restarting ingress-apisix can achieve the same effect.

When the issue occurs, could you obtain the relevant data of the service before and after the restart via the standalone API and provide it for subsequent analysis?

Reference document for the standalone API: https://apisix.apache.org/zh/docs/apisix/deployment-modes/#example

[vortegatorres]

I would like to confirm again whether restarting APISIX alone can resolve this issue, and if restarting ingress-apisix can achieve the same effect.

Restarting APISIX definitely works; I didn't try restarting the ingress-controller because I didn't have the chance. Something new to add here is that I have observed that it can auto-recover if I don't restart APISIX for some time, sometimes hours.

When the issue occurs, could you obtain the relevant data of the service before and after the restart via the standalone API and provide it for subsequent analysis?

There is nothing relevant in the ingress-controller logs when it suddenly auto-recovered, just this message that is logged every minute:

2026-02-11 13:28:41.961 info {
  "level": "info",
  "message": "PUT /sync",
  "requestId": "594c8636-c0d2-43d3-ba37-689ce866ea4e",
  "timestamp": "2026-02-11T12:28:41.961Z"
} 

2026-02-11 13:28:41.955 info 2026-02-11T12:28:41.955Z	INFO	provider.client	client/client.go:201	syncing resources for config	{"service_number": 89} 

2026-02-11 13:28:41.954 info 2026-02-11T12:28:41.953Z	INFO	provider.client	client/client.go:177	syncing all resources 

2026-02-11 13:27:41.965 info {
  "level": "info",
  "message": "PUT /sync",
  "requestId": "e803b466-d422-469d-9444-ac347be15d61",
  "timestamp": "2026-02-11T12:27:41.965Z"
} 

2026-02-11 13:27:41.956 info 2026-02-11T12:27:41.956Z	INFO	provider.client	client/client.go:201	syncing resources for config	{"service_number": 89} 

2026-02-11 13:27:41.954 info 2026-02-11T12:27:41.953Z	INFO	provider.client	client/client.go:177	syncing all resources 

[AlinsRan]

Issue Investigation Checklist

1. Logs and Runtime Information

1.1 Complete Logs

Please provide the full logs during the time when the issue occurred.

Ingress Pod

• ingress container logs
• adc-server container logs
• If possible, please enable debug logging and reproduce the issue.

Debug configuration references:

• Ingress debug:
  https://github.com/apache/apisix-helm-chart/blob/9d5ad2a28f7e75490e05f9dd977cd29791f13629/charts/apisix-ingress-controller/values.yaml#L80
• ADC debug:
  https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix-ingress-controller/values.yaml#L77

APISIX Pod

• error.log
• access.log entries that include /apisix/admin/configs API calls

---

2. Restart Verification

Please confirm:

• Does restarting ingress-apisix resolve the issue?

This helps determine whether the problem originates from the ingress/ADC side or the APISIX data plane.

---

3. Standalone API Configuration Comparison

When the issue occurs:

• Call the standalone API
• Retrieve the data plane configuration before and after restart
• Compare the configuration differences

Standalone API reference:
https://apisix.apache.org/zh/docs/apisix/deployment-modes/#example

This step is critical for root cause analysis.

---

4. Resource Limits and Usage

Please provide:

• Resource requests/limits for ingress pods
• Resource requests/limits for APISIX pods
• CPU and memory usage when the issue occurs

Insufficient resources may cause sync failures or abnormal behavior.

---

2. Configuration Conflict Investigation

2.1 GatewayProxy Conflict

Please check whether multiple GatewayProxy resources are pointing to the same data plane.

You can run:

kubectl get gatewayproxy -A -o yaml

Multiple GatewayProxy resources sharing the same data plane may cause configuration conflicts.

---

2.2 Multiple Ingress Controllers

Please confirm:

• Whether multiple apisix-ingress-controller instances are deployed in the same Kubernetes cluster
• If yes, whether they are properly isolated (e.g., ingressClass, namespace scope, etc.)

Improper isolation may also lead to configuration conflicts.

---

3. Investigation Goals

The goal is to determine:

• Whether the issue is on the ingress/ADC side or the APISIX data plane side
• Whether there is a configuration conflict
• Whether improper resource limits are causing synchronization failures