Add documentation for Kubernetes health checks

Author: rana

docs/img/resource-health-check/kubernetes-health-metrics-tctl.png

@@ -0,0 +1,283 @@
+---
+title: Teleport Kubernetes Health Checks
+sidebar_label: Health Checks
+description: How to configure Teleport Kubernetes health checks and view health.
+tags:
+- conceptual
+- zero-trust
+- infrastructure-identity
+---
+
+This documentation provides an overview of health checks to Kubernetes clusters with Teleport. Teleport Kubernetes Services periodically check the connectivity and permissions of enrolled Kubernetes clusters, and is available in Teleport version `18.4` and later.
+
+## Why check the health of a Kubernetes cluster?
+
+- **Observability**: Discover network and permission issues before users do. Unhealthy Kubernetes clusters are visible in a Teleport UI, command line tool, or Prometheus metrics.
+- **High Availability**: Automatically route and distribute connections to healthy Kubernetes clusters in a high-availability configuration.
+
+## What's checked?
+
+Kubernetes permissions and a Kubernetes health endpoint are checked to determine whether a Kubernetes cluster is both up and usable with Teleport.
+
+Four Kubernetes RBAC permissions are routinely checked with the Kubernetes [SelfSubjectAccessReview](https://kubernetes.io/docs/reference/kubernetes-api/authorization-resources/self-subject-access-review-v1/) API. The permissions are part of minimum requirements for Teleport to work with a Kubernetes cluster. The checked permissions are:
+- Impersonate users
+- Impersonate groups
+- Impersonate service accounts
+- Get pods
+
+If a permission can't be checked, the Kubernetes cluster's [/readyz](https://kubernetes.io/docs/reference/using-api/health-checks/) endpoint is called to further distinguish connection errors and Kubernetes component errors.
+
+## Health states
+
+A Kubernetes cluster is in a `healthy`, `unhealthy`, or `unknown` state.
+- `healthy` indicates a Kubernetes cluster's health was checked and is fine
+- `unhealthy` indicates a Kubernetes cluster's health was checked and is not usable for some reason
+- `unknown` indicates a Kubernetes cluster has been excluded from health checks, or the first health check is initializing
+
+<Admonition type="warning">
+
+`unknown` Kubernetes clusters may be unhealthy.
+
+`unknown` Kubernetes clusters are not checked for health due to:
+- Running a pre-`18.4` version of Teleport Kubernetes Service
+- Explicitly configuring `health_check_config` labels to exclude a Kubernetes cluster
+</Admonition>
+
+## Viewing health
+
+Kubernetes cluster health is viewed through the Teleport web UI, desktop Connect UI, `tctl` CLI tool, or Prometheus metrics.
+
+**Teleport Web & Connect UI**
+
+Click the refresh icon to get the latest health of Kubernetes clusters.
+
+![Kubernetes health warning in the UI](../../../img/resource-health-check/kubernetes-health-warning.png)
+
+**Teleport `tctl` CLI**
+
+Run `tctl get kube_server/<your-kube-server-name>` for an overview of Kubernetes cluster health for a specific Kubernetes service.
+```yaml
+kind: kube_server
+...
+status:
+  target_health:
+    address: 192.168.106.2:58458
+    message: 1 health check passed
+    protocol: http
+    status: healthy
+    transition_reason: threshold_reached
+    transition_timestamp: "2025-10-13T19:26:58.842855Z"
+version: v3
+```
+
+Teleport Prometheus metrics
+
+Health check metrics offer a high-level view of Kubernetes cluster health. The total number of Kubernetes clusters in a `healthy`, `unhealthy`, or `unknown` state are monitored with gauge metrics `teleport_resources_health_status_healthy`, `teleport_resources_health_status_unhealthy`, and `teleport_resources_health_status_unknown`.
+
+- `teleport_resources_health_status_healthy{type="kubernetes"}` is the total number of _healthy_ Kubernetes clusters
+- `teleport_resources_health_status_unhealthy{type="kubernetes"}` is the total number of _unhealthy_ Kubernetes clusters
+- `teleport_resources_health_status_unknown{type="kubernetes"}` is the total number of Kubernetes clusters in an _unknown_ state
+
+A [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) expression may be used to determine the total number of Kubernetes clusters.
+```promql
+teleport_resources_health_status_healthy{type="kubernetes"} +
+teleport_resources_health_status_unhealthy{type="kubernetes"} +
+teleport_resources_health_status_unknown{type="kubernetes"}
+```
+
+A PromQL expression may be used to detect the presence of unhealthy Kubernetes clusters.
+```promql
+teleport_resources_health_status_unhealthy{type="kubernetes"} > 0
+```
+
+<Admonition type="note">
+Notice that Prometheus metrics don't distinguish which cluster is unhealthy.
+
+Use a Teleport UI, or `tctl kube ls --query 'health.status == "unhealthy"'` to determine which cluster is unhealthy.
+</Admonition>
+
+The Teleport `tctl top` command displays Kubernetes health check metrics.
+
+Steps to view health check metrics with `tctl`:
+- Run `tctl top`
+- Navigate to the "Raw Metrics" tab by pressing the right arrow `→` key several times
+- Enter search mode by pressing the forward slash `/` key
+- Type or paste `teleport_resources_health_status`
+
+![Health metrics with the tctl top command](../../../img/resource-health-check/kubernetes-health-metrics-tctl.png)
+
+Health check metrics may also be viewed with the Teleport diagnostic endpoint `http://<diagnostic-address>/metrics`.
+```text
+# HELP teleport_resources_health_status_healthy Number of healthy resources
+# TYPE teleport_resources_health_status_healthy gauge
+teleport_resources_health_status_healthy{type="kubernetes"} 99972
+# HELP teleport_resources_health_status_unhealthy Number of unhealthy resources
+# TYPE teleport_resources_health_status_unhealthy gauge
+teleport_resources_health_status_unhealthy{type="kubernetes"} 3
+# HELP teleport_resources_health_status_unknown Number of resources in an unknown health state
+# TYPE teleport_resources_health_status_unknown gauge
+teleport_resources_health_status_unknown{type="kubernetes"} 25
+```
+
+## Configuring health checks
+
+The Teleport `tctl` CLI tool enables reading, adding, editing, and deleting `health_check_config` resources.
+
+`health_check_config` resources offer a way to configure and selectively apply health checks to Kubernetes clusters.
+
+An example `health_check_config`.
+```yaml
+kind: health_check_config
+version: v1
+metadata:
+  name: example
+  description: Example healthcheck configuration
+spec:
+  # interval is the time between each health check. Default 30s.
+  interval: 30s
+  # timeout is the health check connection establishment timeout. Default 5s.
+  timeout: 5s
+  # healthy_threshold is the number of consecutive passing health checks
+  # after which a target's health status becomes "healthy". Default 2.
+  healthy_threshold: 2
+  # unhealthy_threshold is the number of consecutive failing health checks
+  # after which a target's health status becomes "unhealthy". Default 1.
+  unhealthy_threshold: 1
+  # match is used to select Kubernetes clusters that apply these settings.
+  # Kubernetes clusters are matched by label selectors and at least one label selector
+  # must be set.
+  # If multiple `health_check_config` resources match the same Kubernetes cluster,
+  # the matching health check configs are sorted by name and only the first
+  # config applies.
+  match:
+    # kubernetes_labels matches Kubernetes cluster labels. An empty value is ignored.
+    # If kubernetes_labels_expression is also set, then the match result is the logical
+    # AND of both.
+    kubernetes_labels:
+      - name: env
+        values:
+          - dev
+          - staging
+    # kubernetes_labels_expression is a label predicate expression to match Kubernetes clusters.
+    # An empty value is ignored.
+    # If kubernetes_labels is also set, then the match result is the logical AND of both.
+    kubernetes_labels_expression: 'labels["owner"] == "platform-team"'
+```
+
+The default `health_check_config` enables all Kubernetes clusters to participate in health checks from version `18.4` onward.
+```yaml
+kind: health_check_config
+metadata:
+  description: Enables all health checks by default
+  labels:
+    teleport.internal/resource-type: preset
+  name: default
+  namespace: default
+spec:
+  match:
+    kubernetes_labels:
+    - name: '*'
+      values:
+      - '*'
+version: v1
+```
+
+Multiple different `health_check_config` resources may be created for different groups of Kubernetes clusters. When multiple `health_check_config` match the same Kubernetes cluster, configs are sorted in ascending order by name, and only the first config applies (e.g., the name "00-my-config" has greater precedence than "10-my-config").
+
+**`tctl` health check commands**
+
+Read the default health check config with `tctl get`:
+```bash
+$ tctl get health_check_config/default
+```
+
+Create a new health check config with `tctl create`:
+```bash
+$ tctl create health_check_config.yaml
+```
+
+Update an existing config interactively with `tctl edit`:
+```bash
+$ tctl edit health_check_config/default
+```
+
+Delete a health check config with `tctl rm`:
+```bash
+$ tctl rm health_check_config/example
+```
+
+Teleport Kubernetes Services are notified of changes to `health_check_config`, and reevaluate whether a Kubernetes cluster participates in health checks, applying any changes.
+
+## FAQ
+
+### How do I see unhealthy Kubernetes clusters?
+
+The Teleport web and Connect UI highlight unhealthy Kubernetes clusters.
+
+Clicking a highlighted Kubernetes cluster shows details of an unhealthy Kubernetes cluster.
+
+![Kubernetes health warning in the UI](../../../img/resource-health-check/kubernetes-health-warning)
+
+It may take approximately `5m` for a health change to be reported.
+
+Click the circular arrow refresh icon to get the latest health status. 
+
+The Teleport `tctl` CLI tool also searches and displays unhealthy Kubernetes clusters.
+```code
+tctl kube ls --query 'health.status == "unhealthy"' 
+```
+
+### What guidance is there for troubleshooting unhealthy Kubernetes clusters?
+
+See the [Kubernetes Service troubleshooting guide](./troubleshooting.mdx) for specific errors returned by health checks.
+
+### How do I disable Kubernetes health checks?
+
+Add a filtering label to exclude Kuberenetes clusters with the CLI command `tctl edit health_check_config/default`. If you have multiple `health_check_config`, such as `health_check_config/example-a`, `health_check_config/example-b`, each config would be adjusted.
+
+Adding any label to `health_check_config` tells Teleport to filter out Kubernetes clusters which don't match the label. 
+
+For example, adding `healthcheck` and `enabled` to `kubernetes_labels` in the following example would exclude Kubernetes clusters without the label.
+```yaml
+kind: health_check_config
+metadata:
+  description: Enables all health checks by default
+  labels:
+    teleport.internal/resource-type: preset
+  name: default
+  namespace: default
+  revision: 040e3839-c248-4b9d-898f-4ae88b1d4752
+spec:
+  match:
+    kubernetes_labels:
+      - name: 'healthcheck'
+        values:
+          - 'enabled'
+version: v1
+```
+
+The default is to match all enrolled Kubernetes clusters.
+
+```yaml
+spec:
+  match:
+    kubernetes_labels:
+      - name: '*'
+        values:
+          - '*'
+```
+
+### Do health check metrics show which Kubernetes cluster is unhealthy?
+
+No. A specific Kubernetes cluster's health cannot be determined from the `teleport_resources_health_status_*` health metrics.
+
+The quantity of unhealthy Kubernetes clusters are available from metrics.
+
+### How do I configure health checks for high-availability?
+
+No additional configuration is needed.
+
+Health-based connection routing is automatic when multiple Teleport Kubernetes Services are proxying to the same Kubernetes cluster.
+
+Configuring high-availability with multiple Teleport Kubernetes Services proxying the same Kubernetes cluster would be needed.
+