Skip to content

fix(api,common): avoid cache sync timeouts at operator startup - #863

Merged
ron96g merged 2 commits into
mainfrom
agents/fix-api-sync-timeout-error
Nov 16, 2026
Merged

ron96g merged 2 commits into
mainfrom
agents/fix-api-sync-timeout-error

Conversation

@ron96g

@ron96g ron96g commented Nov 16, 2026

Copy link
Copy Markdown
Member

What

The api operator crash-looped at startup with:

failed to wait for apisubscription caches to sync kind source: *v1.Api:
timed out waiting for cache to be synced for kind source: *v1.Api

This PR fixes the cause, makes the timeout configurable everywhere, and adds an audit rule so the pattern doesn't come back.

Why it happened

Since controller-runtime v0.23 (we're on v0.25.2), a watch only counts as synced once every initial-list event has gone through its predicates and event handler. The api controllers' EnqueueRequestsFromMapFunc handlers run a label-selector r.List(...) per event. Label selectors aren't indexed in the cache, so each call is a full scan plus deep copies. At startup that's O(watched objects × primary objects), under a 500m CPU limit, and it exceeded the default 2-minute CacheSyncTimeout.

That work is redundant: the For(...) source already enqueues every primary object from its own initial list.

Changes

  • Skip the startup replay (api/internal/controller/{api,apiexposure,apisubscription}_controller.go): all 11 Watches(...) now include cc.SkipInitialListPredicate{} inside cc.Count(...), so skipped events are still counted as filtered. Every map function involved only enqueues the controller's For type, so nothing is missed at startup. Events after startup (including creates) still pass. Owns(...) is unchanged because it doesn't list.
    • The apiexposure → Application watch had no cc.Count; it's now wrapped like the others.
    • The predicate already existed and gateway uses it. Its doc comment now explains when to use it and why.
  • Configurable CacheSyncTimeout (common/pkg/config): new cache-sync-timeout / CACHE_SYNC_TIMEOUT, defaulting to 2m (controller-runtime's default, so behaviour doesn't change). It's applied to all 17 controller-runtime managers, including projector, and documented in developer-journey/creating-an-operator.md.
  • Audit rule 16-skip-initial-list-on-watches (HIGH, complexity 1): flags cache-listing Watches that enqueue the For type without the skip, the predicate on For or on non-For watches, and the predicate outside cc.Count. Selftest: 9/9 on the fast judge with 0% flake; cost 44.6 questions per 100 in-scope files.

Validation

  • make build: all 18 touched modules.
  • make test: common and api, including the new predicate tests and the config env-override test.
  • make lint: clean for every changed file. The common, rover, gateway, agentic and spectre lint failures are in untouched files and were already there.
  • code-auditor branch audit: clean. One advisory (rule 13, apisubscription_controller.go:176) sits on untouched code; that List already filters by environment label.

Reviewer notes

  • Not yet verified on a cluster. If startup is still slow there, CACHE_SYNC_TIMEOUT can be raised without a release.
  • The same Watches + List-in-mapfunc pattern exists in event, agentic, file, rover, pubsub, gateway (routelistener), identity, notification, organization, admin, application, approval, sftp and spectre. They're left for follow-ups because each watch needs checking that it only enqueues its controller's For type. Rule 16 will flag them when they're next touched.

ron96g and others added 2 commits November 16, 2026 11:24
The api operator crash-looped with "timed out waiting for cache to be
synced for kind source: *v1.Api". Since controller-runtime v0.23 a source
only counts as synced once every initial-list event has passed through
its event handler. The api controllers' map functions run an unindexed
label-selector List per event, which made startup O(watched × primaries)
and exceeded the 2m CacheSyncTimeout.

- Skip the informer's initial-list creates on all Watches of the api,
  apiexposure and apisubscription controllers via
  cc.SkipInitialListPredicate. The For sources already enqueue every
  primary at startup, so the replay was redundant.
- Make CacheSyncTimeout configurable (cache-sync-timeout /
  CACHE_SYNC_TIMEOUT, default 2m) in common config and apply it to every
  operator manager.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 16-skip-initial-list-on-watches: a Watches whose map function lists
from the cache and only enqueues the controller's For type must add
cc.SkipInitialListPredicate inside cc.Count, so the informer's
initial-list replay cannot exceed CacheSyncTimeout at startup. Also flags
the predicate on For sources, on watches enqueuing other types, and
outside cc.Count.

Selftest: 9/9 with 0% flake on the fast judge (3 fixtures x 3 samples).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ron96g
ron96g merged commit 58fd1b2 into main Nov 16, 2026
258 of 268 checks passed
@ron96g
ron96g deleted the agents/fix-api-sync-timeout-error branch November 16, 2026 13:21
dtit-semantic-release Bot pushed a commit that referenced this pull request Nov 16, 2026
## [0.48.1](v0.48.0...v0.48.1) (2026-11-16)

### Bug Fixes

* **api,common:** avoid cache sync timeouts at operator startup ([#863](#863)) ([58fd1b2](58fd1b2))
* **gateway:** discover and reuse legacy Kong plugins ([#805](#805)) ([d82c122](d82c122))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants