Summary
TaskRunImagePullFailed (added to fail fast on ImagePullBackOff, config default-imagepullbackoff-timeout, default 0) does not distinguish transient, self-resolving image-pull errors from permanent ones. Under bursty, high-concurrency pod creation — e.g. an HA controller deployment with a high threads-per-controller value — many pods can request an image pull from the same node within a very short window, exceeding the container runtime's image-pull rate limit (CRI-O's default imagePullQPS ~5/sec/node). The resulting ErrImagePull: pull QPS exceeded is transient and normally resolves within a few seconds via Kubernetes' own backoff/retry, but with the default default-imagepullbackoff-timeout: 0, the controller kills the TaskRun on the first occurrence, before that retry can happen.
Reproduction
- OpenShift Pipelines 1.23.0 / Tekton Pipelines v1.12.0 (commit
7ce6b655b0a061a8bdca75d3d5e18b7140f90ea3), but the code path (taskrun/taskrun.go PullImageFailed detection → TaskRunImagePullFailed stop) is unchanged upstream.
- Controller config: 10 replicas (HA, Deployments + lease-based leader election),
threads-per-controller: 32, kube-api-qps: 50, kube-api-burst: 50.
- Workload: 1000 PipelineRuns, 4 parallel TaskRuns each, all using
registry.access.redhat.com/ubi8-micro:latest (so imagePullPolicy defaults to Always), 20 concurrent PipelineRuns across 5 namespaces.
- Cluster: 5 worker nodes, CRI-O default image-pull QPS.
Observed
-
34.5% PipelineRun failure rate at concurrency 20 (345/1000).
-
Controller logs: 2,093 ErrImagePull: pull QPS exceeded occurrences and 460 TaskRunImagePullFailed stops in a single 10-minute run.
-
Cross-configuration isolation test (identical scenario/image/cluster, only HA-replica-count and thread-count varied):
| Configuration |
Total reconciler threads |
Workqueue depth (mean) |
TaskRun→Pod lag (mean) |
PipelineRun failures (conc 20) |
pull QPS exceeded count |
| Standard (1 replica, 2 threads) |
2 |
525 |
69.7s |
0 |
0 |
| HA only (10 replicas, 2 threads) |
20 |
249 |
2.0s |
0 |
0 |
| QBT only (1 replica, 32 threads) |
32 |
196 |
6.6s |
0 |
0 |
| HA + QBT (10 replicas, 32 threads) |
320 |
9.5 |
0.18s |
334 |
2,093 |
Only the HA+QBT configuration drains the workqueue fast enough (320 concurrent reconcilers) to create pods in near-instantaneous bursts (~80 pods within ~0.5s) instead of being naturally paced by workqueue backlog. That burst is what exceeds CRI-O's per-node pull QPS. Controller CPU/memory, API server, etcd, and worker node resources are all well within limits in every configuration — this is not a resource-exhaustion issue.
-
Failed-PipelineRun duration is consistently ~18-19s, matching the timing of a single Kubernetes image-pull backoff cycle — i.e., these TaskRuns would very likely have succeeded on Kubernetes' own next retry had they not been terminated first.
-
Same behavior reproduced on Tekton v1.22 (192 failures at concurrency 20, before TaskRunImagePullFailed existed) purely from CRI-O QPS exhaustion without controller-side retries being cut short — confirming the burst/QPS mechanism is independent of this feature, but the feature makes it substantially worse (79.7% more failures on v1.23 at the same concurrency).
Expected
A single transient pull QPS exceeded (or other clearly-recoverable) image-pull error should not immediately terminate a TaskRun via TaskRunImagePullFailed. Either:
- Classify known-transient error substrings (
pull QPS exceeded, 502, 504, i/o timeout, TLS handshake timeout, connection refused) separately from permanent ones (invalid image name, auth denied) and only fail-fast on the latter, or
- Reconsider the default value of
default-imagepullbackoff-timeout (currently 0) so that some retry window is given by default rather than opt-in via config.
Related issues
Additional context
Full root-cause analysis and cross-configuration data available on request — happy to share the raw controller logs, workqueue depth metrics, and per-configuration failure counts referenced above.
Summary
TaskRunImagePullFailed(added to fail fast onImagePullBackOff, configdefault-imagepullbackoff-timeout, default0) does not distinguish transient, self-resolving image-pull errors from permanent ones. Under bursty, high-concurrency pod creation — e.g. an HA controller deployment with a highthreads-per-controllervalue — many pods can request an image pull from the same node within a very short window, exceeding the container runtime's image-pull rate limit (CRI-O's defaultimagePullQPS~5/sec/node). The resultingErrImagePull: pull QPS exceededis transient and normally resolves within a few seconds via Kubernetes' own backoff/retry, but with the defaultdefault-imagepullbackoff-timeout: 0, the controller kills the TaskRun on the first occurrence, before that retry can happen.Reproduction
7ce6b655b0a061a8bdca75d3d5e18b7140f90ea3), but the code path (taskrun/taskrun.goPullImageFaileddetection →TaskRunImagePullFailedstop) is unchanged upstream.threads-per-controller: 32,kube-api-qps: 50,kube-api-burst: 50.registry.access.redhat.com/ubi8-micro:latest(soimagePullPolicydefaults toAlways), 20 concurrent PipelineRuns across 5 namespaces.Observed
34.5% PipelineRun failure rate at concurrency 20 (345/1000).
Controller logs: 2,093
ErrImagePull: pull QPS exceededoccurrences and 460TaskRunImagePullFailedstops in a single 10-minute run.Cross-configuration isolation test (identical scenario/image/cluster, only HA-replica-count and thread-count varied):
pull QPS exceededcountOnly the HA+QBT configuration drains the workqueue fast enough (320 concurrent reconcilers) to create pods in near-instantaneous bursts (~80 pods within ~0.5s) instead of being naturally paced by workqueue backlog. That burst is what exceeds CRI-O's per-node pull QPS. Controller CPU/memory, API server, etcd, and worker node resources are all well within limits in every configuration — this is not a resource-exhaustion issue.
Failed-PipelineRun duration is consistently ~18-19s, matching the timing of a single Kubernetes image-pull backoff cycle — i.e., these TaskRuns would very likely have succeeded on Kubernetes' own next retry had they not been terminated first.
Same behavior reproduced on Tekton v1.22 (192 failures at concurrency 20, before
TaskRunImagePullFailedexisted) purely from CRI-O QPS exhaustion without controller-side retries being cut short — confirming the burst/QPS mechanism is independent of this feature, but the feature makes it substantially worse (79.7% more failures on v1.23 at the same concurrency).Expected
A single transient
pull QPS exceeded(or other clearly-recoverable) image-pull error should not immediately terminate a TaskRun viaTaskRunImagePullFailed. Either:pull QPS exceeded,502,504,i/o timeout,TLS handshake timeout,connection refused) separately from permanent ones (invalid image name, auth denied) and only fail-fast on the latter, ordefault-imagepullbackoff-timeout(currently0) so that some retry window is given by default rather than opt-in via config.Related issues
default-imagepullbackoff-timeoutconfig this feature relies on; that config exists but defaults to0(immediate fail) and doesn't distinguish error types.default-imagepullbackoff-timeoutbeing inconsistently honored depending on whether entrypoint inference is needed; different code path, same symptom family (configured grace period not protecting against a transient failure).Additional context
Full root-cause analysis and cross-configuration data available on request — happy to share the raw controller logs, workqueue depth metrics, and per-configuration failure counts referenced above.