Repository navigation
fix(api,common): avoid cache sync timeouts at operator startup - #863
Merged
Merged
Conversation
The api operator crash-looped with "timed out waiting for cache to be synced for kind source: *v1.Api". Since controller-runtime v0.23 a source only counts as synced once every initial-list event has passed through its event handler. The api controllers' map functions run an unindexed label-selector List per event, which made startup O(watched × primaries) and exceeded the 2m CacheSyncTimeout. - Skip the informer's initial-list creates on all Watches of the api, apiexposure and apisubscription controllers via cc.SkipInitialListPredicate. The For sources already enqueue every primary at startup, so the replay was redundant. - Make CacheSyncTimeout configurable (cache-sync-timeout / CACHE_SYNC_TIMEOUT, default 2m) in common config and apply it to every operator manager. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 16-skip-initial-list-on-watches: a Watches whose map function lists from the cache and only enqueues the controller's For type must add cc.SkipInitialListPredicate inside cc.Count, so the informer's initial-list replay cannot exceed CacheSyncTimeout at startup. Also flags the predicate on For sources, on watches enqueuing other types, and outside cc.Count. Selftest: 9/9 with 0% flake on the fast judge (3 fixtures x 3 samples). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ron96g
requested review from
BjoernKarma,
julius-malcovsky,
lukas016 and
stefan-ctrl
November 16, 2026 10:35
lukas016
approved these changes
Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The api operator crash-looped at startup with:
This PR fixes the cause, makes the timeout configurable everywhere, and adds an audit rule so the pattern doesn't come back.
Why it happened
Since controller-runtime v0.23 (we're on v0.25.2), a watch only counts as synced once every initial-list event has gone through its predicates and event handler. The api controllers'
EnqueueRequestsFromMapFunchandlers run a label-selectorr.List(...)per event. Label selectors aren't indexed in the cache, so each call is a full scan plus deep copies. At startup that's O(watched objects × primary objects), under a 500m CPU limit, and it exceeded the default 2-minuteCacheSyncTimeout.That work is redundant: the
For(...)source already enqueues every primary object from its own initial list.Changes
api/internal/controller/{api,apiexposure,apisubscription}_controller.go): all 11Watches(...)now includecc.SkipInitialListPredicate{}insidecc.Count(...), so skipped events are still counted asfiltered. Every map function involved only enqueues the controller'sFortype, so nothing is missed at startup. Events after startup (including creates) still pass.Owns(...)is unchanged because it doesn't list.cc.Count; it's now wrapped like the others.CacheSyncTimeout(common/pkg/config): newcache-sync-timeout/CACHE_SYNC_TIMEOUT, defaulting to2m(controller-runtime's default, so behaviour doesn't change). It's applied to all 17 controller-runtime managers, including projector, and documented indeveloper-journey/creating-an-operator.md.16-skip-initial-list-on-watches(HIGH, complexity 1): flags cache-listingWatchesthat enqueue theFortype without the skip, the predicate onForor on non-Forwatches, and the predicate outsidecc.Count. Selftest: 9/9 on thefastjudge with 0% flake; cost 44.6 questions per 100 in-scope files.Validation
make build: all 18 touched modules.make test:commonandapi, including the new predicate tests and the config env-override test.make lint: clean for every changed file. The common, rover, gateway, agentic and spectre lint failures are in untouched files and were already there.apisubscription_controller.go:176) sits on untouched code; thatListalready filters by environment label.Reviewer notes
CACHE_SYNC_TIMEOUTcan be raised without a release.Fortype. Rule 16 will flag them when they're next touched.