Repository navigation
fix(gateway,common-server): back off on unavailable token endpoints and reduce cache memory - #873
Merged
Merged
Conversation
… delay A 503 means the server is unavailable, not that the caller is too fast. Classify it like other server errors so controllers back off exponentially.
…ds from cache An unreachable OAuth2 token endpoint (503/429/5xx) was retried every 3s forever for every Route and Consumer of the affected gateway. Token endpoint failures are now retryable without a fixed delay, so the controller backs off exponentially. The manager cache drops managedFields, which reconcilers never read, to reduce the memory footprint.
Rate-limited responses (429) no longer carry a fixed 3s retry delay, so controller-runtime's rate limiter owns the backoff, like for 5xx. - common-server: HandleError maps 429 to a retryable error without delay; the unused RetryDelay var is removed. - gateway: Kong CheckStatusCode drops the fixed delay for 429. - gateway: OAuth2 token endpoint errors are classified via the shared client.HandleError instead of a gateway-specific classifier. - gateway: guard against an oauth2.RetrieveError without Response, which would otherwise panic in the reconciler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
julius-malcovsky
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After the v0.48.1 rollout the gateway operator kept getting evicted on
dedicated-non-prod-1(node memory pressure). While investigating, the live log showed every Route and Consumer of thedtt-infragateway being reconciled every ~3-4s indefinitely:673 such errors in ~2.5 minutes. The identity provider of that zone is unreachable.
common-server/pkg/client.HandleErrormapped 503 (like 429) to a fixed 3s retry delay, which bypasses the controller's exponential backoff, so the operator hammers the broken endpoint forever and keeps running full reconciles (secret-manager lookups, Kong client leases, logging) for those objects.Changes
Retryable errors no longer carry a fixed delay: they are returned to controller-runtime, whose rate limiter owns the (exponential) backoff.
HandleErrormaps 429 and 503 to retryable errors without a fixed delay, like 409/500/502/504. The now unusedRetryDelayvar is removed. Affects all users of the shared mapping (secret-manager, file-manager, pubsub, rover-server linter).CheckStatusCode) no longer use a fixed 3s delay.oauth2.RetrieveError) are classified via the sharedclient.HandleErrorinstead of a gateway-specific classifier (400/403/405 block, everything else retries with backoff).oauth2.RetrieveErrorwithoutResponseis treated as a retryable error instead of panicking in the reconciler.managedFields(cache.TransformStripManagedFields()). Reconcilers never read them; this reduces the memory footprint of every cached Route/Consumer/ConsumeRoute/RouteListener/Gateway.Testing
common-server/pkg/client/errors_test.go(the mapping had no tests).gateway/pkg/kong/client/error_test.gofor token endpoint errors wrapped in*url.Error, including aRetrieveErrorwithout response; Kong 429 spec now expects no fixed delay.make testpasses forcommon-serverandgateway; lint clean for changed files.Not in scope
CheckHTTPStatus, fixed 3s delay); to be aligned separately.dtt-infragateway (environment config).gateway/cmd/main.go), only relevant if a Gateway setsspec.tls; to be evaluated separately.