Skip to content

fix(gateway,common-server): back off on unavailable token endpoints and reduce cache memory - #873

Merged
ron96g merged 3 commits into
mainfrom
fix/gateway-token-error-backoff
Nov 19, 2026
Merged

ron96g merged 3 commits into
mainfrom
fix/gateway-token-error-backoff

Conversation

@ron96g

@ron96g ron96g commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Problem

After the v0.48.1 rollout the gateway operator kept getting evicted on dedicated-non-prod-1 (node memory pressure). While investigating, the live log showed every Route and Consumer of the dtt-infra gateway being reconciled every ~3-4s indefinitely:

rate limit error (503): OAuth2 token endpoint error: oauth2: cannot fetch token: 503 Service Unavailable
Response: 404 page not found

673 such errors in ~2.5 minutes. The identity provider of that zone is unreachable. common-server/pkg/client.HandleError mapped 503 (like 429) to a fixed 3s retry delay, which bypasses the controller's exponential backoff, so the operator hammers the broken endpoint forever and keeps running full reconciles (secret-manager lookups, Kong client leases, logging) for those objects.

Changes

Retryable errors no longer carry a fixed delay: they are returned to controller-runtime, whose rate limiter owns the (exponential) backoff.

  • common-server: HandleError maps 429 and 503 to retryable errors without a fixed delay, like 409/500/502/504. The now unused RetryDelay var is removed. Affects all users of the shared mapping (secret-manager, file-manager, pubsub, rover-server linter).
  • gateway: Kong Admin API 429 responses (CheckStatusCode) no longer use a fixed 3s delay.
  • gateway: OAuth2 token endpoint errors (oauth2.RetrieveError) are classified via the shared client.HandleError instead of a gateway-specific classifier (400/403/405 block, everything else retries with backoff).
  • gateway: an oauth2.RetrieveError without Response is treated as a retryable error instead of panicking in the reconciler.
  • gateway: the manager cache drops managedFields (cache.TransformStripManagedFields()). Reconcilers never read them; this reduces the memory footprint of every cached Route/Consumer/ConsumeRoute/RouteListener/Gateway.

Testing

  • New common-server/pkg/client/errors_test.go (the mapping had no tests).
  • New specs in gateway/pkg/kong/client/error_test.go for token endpoint errors wrapped in *url.Error, including a RetrieveError without response; Kong 429 spec now expects no fixed delay.
  • make test passes for common-server and gateway; lint clean for changed files.

Not in scope

  • Identity keeps its own 429 handling (CheckHTTPStatus, fixed 3s delay); to be aligned separately.
  • The broken issuer of the dtt-infra gateway (environment config).
  • Possible cluster-wide ConfigMap informer started by CA bundle resolution via the cached client (gateway/cmd/main.go), only relevant if a Gateway sets spec.tls; to be evaluated separately.

… delay

A 503 means the server is unavailable, not that the caller is too fast.
Classify it like other server errors so controllers back off exponentially.
…ds from cache

An unreachable OAuth2 token endpoint (503/429/5xx) was retried every 3s
forever for every Route and Consumer of the affected gateway. Token
endpoint failures are now retryable without a fixed delay, so the
controller backs off exponentially.

The manager cache drops managedFields, which reconcilers never read,
to reduce the memory footprint.
@ron96g ron96g self-assigned this Nov 18, 2026
Rate-limited responses (429) no longer carry a fixed 3s retry delay, so
controller-runtime's rate limiter owns the backoff, like for 5xx.

- common-server: HandleError maps 429 to a retryable error without delay;
  the unused RetryDelay var is removed.
- gateway: Kong CheckStatusCode drops the fixed delay for 429.
- gateway: OAuth2 token endpoint errors are classified via the shared
  client.HandleError instead of a gateway-specific classifier.
- gateway: guard against an oauth2.RetrieveError without Response, which
  would otherwise panic in the reconciler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ron96g
ron96g merged commit 107e81b into main Nov 19, 2026
106 checks passed
@ron96g
ron96g deleted the fix/gateway-token-error-backoff branch November 19, 2026 11:46
dtit-semantic-release Bot pushed a commit that referenced this pull request Nov 19, 2026
## [0.48.4](v0.48.3...v0.48.4) (2026-11-19)

### Bug Fixes

* **gateway,common-server:** back off on unavailable token endpoints and reduce cache memory ([#873](#873)) ([107e81b](107e81b))
* **identity:** clean up keycloak rotation state when rotation is disabled ([#874](#874)) ([df6ce07](df6ce07))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants