Add monitoring rules to check the pods memory and CPU usage against limits

[bryanasdev000]

Overview

Add monitoring rules (Prometheus) to check current CPU and RAM usage against pod limits.

Use Case

It would be nice to have alerts when the resources of the containers are about to exceed the limit allocated to them, this would avoid an OOM from Kubernetes and an eventual failover loop.

Desired Behavior

When I reach the scenario of having a pod close to the limit of allowed resources, be warned by the alertmanager to take some action, updating with new limits or adding new replicas for example.

Environment

Tell us about your environment:

Please provide the following details:

• Platform: AKS
• Platform Version: v1.19.11 , 4.7.0
• PGO Image Tag: centos8-4.7.0
• Postgres Version13)
• Storage: azure-disk (default)
• Number of Postgres clusters: 2

Additional Information

NA.

[jkatz]

This information is available from the Pod metrics that are collected, so you can add your own alerts if needed.