kube-state-metrics Metrics Handler Failure with "context canceled" Error

[kittanb]

Description:
The metrics handler in kube-state-metrics fails to start, resulting in a "context canceled" error. Despite this error, the service does not crash and continues to serve the metrics collected at the time of the failure. The issue occurs during the auto-detection of sharding settings.

Environment:
Deployment method: Helm chart
Chart version: 5.19.0
Chart reference: prometheus-community/kube-state-metric

Steps to Reproduce:
Deploy kube-state-metrics using the specified Helm chart version 5.19.0.
Monitor the logs of the kube-state-metrics pod.
Observe the "context canceled" error in the logs.
Expected Behavior:
The metrics handler should start successfully and continue collecting and serving metrics without interruption.

Actual Behavior:
The metrics handler fails to start, and the following error is logged:
E1212 20:07:19.015489 1 discovery.go:253] "failed to run metrics handler" err="context canceled"
However, the service does not crash and continues to serve the metrics collected at the time of the failure.

Logs:
I1212 20:07:16.096580 1 config.go:84] "Using custom resource plural" resource="autoscaling.k8s.io_v1_VerticalPodAutoscaler" plural="verticalpodautoscalers"
I1212 20:07:16.096635 1 config.go:209] "reloaded factory" GVR="autoscaling.k8s.io/v1, Resource=verticalpodautoscalers"
I1212 20:07:16.296547 1 builder.go:208] "Updating store" GVR="autoscaling.k8s.io/v1, Resource=verticalpodautoscalers"
I1212 20:07:16.296612 1 discovery.go:274] "discovery finished, cache updated"
I1212 20:07:16.298253 1 metrics_handler.go:106] "Autosharding enabled with pod" pod="kube-prometheus/kube-state-metrics-0"
I1212 20:07:16.298283 1 metrics_handler.go:107] "Auto detecting sharding settings"
I1212 20:07:19.014737 1 config.go:84] "Using custom resource plural" resource="autoscaling.k8s.io_v1_VerticalPodAutoscaler" plural="verticalpodautoscalers"
I1212 20:07:19.014860 1 config.go:209] "reloaded factory" GVR="autoscaling.k8s.io/v1, Resource=verticalpodautoscalers"
I1212 20:07:19.015001 1 builder.go:208] "Updating store" GVR="autoscaling.k8s.io/v1, Resource=verticalpodautoscalers"
I1212 20:07:19.015052 1 discovery.go:274] "discovery finished, cache updated"
I1212 20:07:19.015088 1 metrics_handler.go:106] "Autosharding enabled with pod" pod="kube-prometheus/kube-state-metrics-0"
I1212 20:07:19.015118 1 metrics_handler.go:107] "Auto detecting sharding settings"
E1212 20:07:19.015489 1 discovery.go:253] "failed to run metrics handler" err="context canceled"
Impact:
The failure of the metrics handler prevents the collection and serving of new metrics. However, the service continues to serve the metrics collected before the failure occurred. This can lead to stale or outdated metrics being served, which may impact monitoring and alerting systems relying on kube-state-metrics.

Versions:
kube-state-metrics version: 2.12.0
Kubernetes version: v1.29.5
Cloud provider or hardware configuration: vCloud

[k8s-ci-robot]

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mrueg]

please test with the latest kube-state-metrics version which is v2.14.0