@@ -231,6 +231,173 @@ kube_pod_status_phase{phase="Pending"} * on(namespace, pod) group_left() (time()
231231
232232---
233233
234+ ## Service Mesh Detectors
235+
236+ ### LinkerdControlPlaneDetector
237+
238+ ** Purpose** : Detects linkerd control plane deployments with zero available replicas. When the control plane is down, proxy injection fails, mTLS breaks, and traffic routing stops for all meshed services.
239+
240+ ** Entity Type** : ` service_mesh_control_plane `
241+
242+ ** Query** :
243+ ``` promql
244+ kube_deployment_status_replicas_available{namespace="linkerd"} == 0
245+ ```
246+
247+ ** Severity** : ` FATAL `
248+
249+ ** Blast Radius** : 15 (affects all meshed services)
250+
251+ ** Hint** : "Check pod status: kubectl get pods -n linkerd"
252+
253+ ---
254+
255+ ### LinkerdProxyInjectionDetector
256+
257+ ** Purpose** : Detects linkerd pods in CrashLoopBackOff. Catches proxy-injector, identity, and destination service failures.
258+
259+ ** Entity Type** : ` service_mesh_control_plane `
260+
261+ ** Query** :
262+ ``` promql
263+ kube_pod_container_status_waiting_reason{namespace="linkerd",reason="CrashLoopBackOff"} > 0
264+ ```
265+
266+ ** Severity** : ` CRITICAL `
267+
268+ ** Blast Radius** : 10
269+
270+ ** Hint** : "Proxy injector or identity service failure"
271+
272+ ---
273+
274+ ### IstioControlPlaneDetector
275+
276+ ** Purpose** : Detects istiod with zero available replicas. When istiod is down, xDS config distribution stops and new deployments break.
277+
278+ ** Entity Type** : ` service_mesh_control_plane `
279+
280+ ** Query** :
281+ ``` promql
282+ kube_deployment_status_replicas_available{namespace="istio-system",deployment="istiod"} == 0
283+ ```
284+
285+ ** Severity** : ` FATAL `
286+
287+ ** Blast Radius** : 15
288+
289+ ** Hint** : "Check pod status: kubectl get pods -n istio-system"
290+
291+ ---
292+
293+ ### IstioSidecarInjectionDetector
294+
295+ ** Purpose** : Detects istio-system pods in CrashLoopBackOff. Catches istiod, gateway, and pilot failures.
296+
297+ ** Entity Type** : ` service_mesh_control_plane `
298+
299+ ** Query** :
300+ ``` promql
301+ kube_pod_container_status_waiting_reason{namespace="istio-system",reason="CrashLoopBackOff"} > 0
302+ ```
303+
304+ ** Severity** : ` CRITICAL `
305+
306+ ** Blast Radius** : 10
307+
308+ ** Hint** : "Sidecar injector or pilot failure"
309+
310+ ---
311+
312+ ## Service Mesh Certificate Detectors
313+
314+ ### LinkerdCertExpiryDetector
315+
316+ ** Purpose** : Detects linkerd identity certificates approaching expiry. Certificate expiry is the silent killer of service meshes — mTLS fails across all meshed services without warning.
317+
318+ ** Entity Type** : ` service_mesh_certificate `
319+
320+ ** Query** :
321+ ``` promql
322+ (identity_cert_expiry_timestamp - time()) < 604800
323+ ```
324+
325+ ** Severity** (tiered):
326+ - ` WARNING ` : < 7 days remaining
327+ - ` CRITICAL ` : < 48 hours remaining
328+ - ` FATAL ` : < 24 hours remaining or expired
329+
330+ ** Blast Radius** : 20 (highest — cert expiry kills the entire mesh)
331+
332+ ** Interval** : 60s
333+
334+ ** Hint** : "Rotate certs: linkerd check --proxy; Renew: linkerd upgrade | kubectl apply -f -"
335+
336+ ---
337+
338+ ### IstioCertExpiryDetector
339+
340+ ** Purpose** : Detects istio root certificate approaching expiry. When the Citadel root cert expires, all workload certificates become invalid.
341+
342+ ** Entity Type** : ` service_mesh_certificate `
343+
344+ ** Query** :
345+ ``` promql
346+ (citadel_server_root_cert_expiry_timestamp - time()) < 604800
347+ ```
348+
349+ ** Severity** (tiered):
350+ - ` WARNING ` : < 7 days remaining
351+ - ` CRITICAL ` : < 48 hours remaining
352+ - ` FATAL ` : < 24 hours remaining or expired
353+
354+ ** Blast Radius** : 20
355+
356+ ** Interval** : 60s
357+
358+ ** Hint** : "Check status: istioctl proxy-status; Rotate: istioctl create-remote-secret"
359+
360+ ---
361+
362+ ### Certificate Monitoring Setup
363+
364+ Service mesh certificate metrics are ** often missing from Prometheus** . This is the most common reason cert expiry goes undetected.
365+
366+ ** Why cert metrics are missing** :
367+ - Linkerd identity service not in Prometheus scrape targets
368+ - Istiod metrics endpoint not scraped
369+ - cert-manager not exporting metrics
370+ - ServiceMonitor or PodMonitor CRDs missing
371+
372+ ** How to verify** :
373+ ``` bash
374+ # Check if linkerd metrics are being scraped
375+ curl -s http://prometheus:9090/api/v1/targets | \
376+ jq ' .data.activeTargets[] | select(.labels.job | contains("linkerd"))'
377+
378+ # Verify cert metric exists
379+ curl -s http://prometheus:9090/api/v1/query? query=identity_cert_expiry_timestamp
380+ curl -s http://prometheus:9090/api/v1/query? query=citadel_server_root_cert_expiry_timestamp
381+ ```
382+
383+ ** Required scrape targets** :
384+ - ** Linkerd** : ` linkerd-identity ` (port 9990), ` linkerd-proxy-injector ` (port 9995)
385+ - ** Istio** : ` istiod ` (port 15014)
386+ - ** cert-manager** (optional): ` certmanager_certificate_expiration_timestamp_seconds `
387+
388+ ---
389+
390+ ### Service Mesh Metric Requirements
391+
392+ | Metric | Source | Detector |
393+ | --------| --------| ----------|
394+ | ` kube_deployment_status_replicas_available ` | kube-state-metrics | Control plane health |
395+ | ` kube_pod_container_status_waiting_reason ` | kube-state-metrics | Component crashes |
396+ | ` identity_cert_expiry_timestamp ` | linkerd-identity | Linkerd cert expiry |
397+ | ` citadel_server_root_cert_expiry_timestamp ` | istiod | Istio cert expiry |
398+
399+ ---
400+
234401## Adding Custom Detectors
235402
236403To add a custom detector:
@@ -373,7 +540,7 @@ Post-MVP detector candidates:
373540- ** DatabaseReplicationLagDetector** : PostgreSQL/MySQL replication lag
374541- ** KafkaUnderReplicatedPartitionsDetector** : Kafka partition health
375542- ** RedisMemoryPressureDetector** : Redis memory usage
376- - ** CertificateExpirationDetector** : TLS certificate expiration
543+ - ~~ ** CertificateExpirationDetector** : TLS certificate expiration~~ (implemented in v0.1.1 as LinkerdCertExpiry + IstioCertExpiry)
377544- ** PVCFullDetector** : PersistentVolumeClaim usage
378545- ** NodeNotReadyDetector** : Kubernetes node health
379546- ** DeploymentReplicaMismatchDetector** : Desired vs actual replicas
0 commit comments