Add consolidationPolicy: Underweight

[koreyGambill]

Description

What problem are you trying to solve?
We've created fallback on-demand NodePools with lower scheduling weight than our spot instance NodePools (AWS). When spot instances are hard to find, Karpenter schedules our (fallback) on-demand ec2, but it never consolidates back to the spot instances so it ends up being really expensive. I would love an official setting that allows Karpenter to consolidate based on weighted preferences rather than just utilization.

In this feature, if all the pods on a low-weight node are compatible with a higher-weight node, Karpenter should work to create the higher-weight node and re-schedule the pods. For us, it would help reduce costs, but in general it makes sense that users would care about using higher weighted nodes. I would expect this to still obey the consolidateAfter setting.

Something like this could work in the yaml

disruption:
    # Changed to a list type for the purpose of clearer yaml now that there are 3 options
    consolidationPolicy: 
      # If Underutilized and Underweight are set, Karpenter will re-schedule
      # the node if some pods can be put on a higher weight node, and the 
      # rest could fit on other existing nodes of the same weight
      - Empty
      - Underutilized
      - Underweight  # This would allow consolidating if all pods could be put on a higher weight node

How important is this feature to you?
Low-Medium - I have a workaround (setting the on-demand NodePool to expire after 4hrs), but it has a couple drawbacks.

1. We are waiting up to 4hrs to get back to our optimal state
2. We cannot re-use the NodePool for workloads that need a long lifespan. With this feature it would be possible since Karpenter wouldn't be able to reconsolidate the node if those pods were on it (due to taints/tolerations/affinities).

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

In general, this is the way that Karpenter should perform. Can you walk through the exact scenario that you are seeing? When Karpenter performs its simulations, it's going to consider the highest weight NodePool first (which should be your spot NodePool). Once it finds a scheduling decision, so long as the newer instance type is cheaper, it will consolidate it. All of this should be true in your scenario where you are moving back from on-demand to spot.

[jonathan-innis]

/triage needs-information

[koreyGambill]

The behavior we observed was that the lower weighted nodes (which were the on-demand nodes) would never be consolidated. Over time, we end up with more lower weighted nodes than higher weighted nodes since they don't experience spot terminations. So most of our nodes are actually the on-demand nodes which have a lower weight.

I wouldn't expect the consolidation to work as is, though. Our consolidationPolicy is set to "Empty". It would seem like a bug to me then, if Karpenter was consolidating those nodes for budgetary reasons. We cannot use the consolidationPolicy "Underutilized" because it results in a massive turnover of nodes which costs a ton through AWS Config costs.

I'm proposing a 3rd reason to consolidate, which would be if a higher weighted node would be available to schedule on. Which, seems distinctly different to me than "Underutilized" which should consolidate if the total CPU/Mem requested could be reduced by shuffling pods and shutting down nodes.

[github-actions]

This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.

[mariuskimmina]

@koreyGambill I feel we have been running into a similar situation, what seems to work for us at the moment is to use WhenEmpty on the nodepool that has only spot instances and WhenUnderutilized on the nodepool that serves as on-demand fallback. This way we avoid disruption if the workloads are running on spot but allow them if they are scheduled on-demand.

That said, we have also been feeling like a consolidation policy that captures the case of "I have both spot and on-demand instances in this nodepool and I want to only ever consolidate on-demand nodes back to spot instances" is missing.