feat: Add consolidation policies for WhenCheaper, WhenUnderutilizedOrCheaper

[cnmcavoy]

Fixes #1430

The motivation is to allow cluster operators to configure some nodepools for single-node consolidation (and emptiness), and configure some nodepools for multi-node consolidation, but not single-node consolidation. New enum values for the consolidationPolicy make this possible.

Description

Adds two new consolidation policy enum values:

• WhenCheaper
• WhenUnderutilizedOrCheaper

Changes the semantics of the existing consolidation policy enum value:

• WhenUnderutilized

WhenUnderutilizedOrCheaper becomes the new default. WhenUnderutilizedOrCheaper is the same as WhenUnderutilized today. It functions the same, it is just a new name.

WhenUnderutilized semantics changes to only allow emptiness or multi-node consolidation to occur for a nodepool.

WhenCheaper only allows emptiness or single-node consolidation to occur for a nodepool.

How was this change tested?

Tested in a kind cluster and with make presubmit.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cnmcavoy
Once this PR has been reviewed and has the lgtm label, please assign gjtempleton for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coveralls]

Pull Request Test Coverage Report for Build 9961312317

Details

• 32 of 48 (66.67%) changed or added relevant lines in 6 files are covered.
• 4 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 78.011%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controllers/disruption/multinodeconsolidation.go	1	5	20.0%
pkg/controllers/disruption/singlenodeconsolidation.go	1	5	20.0%
pkg/apis/v1/nodepool_conversion.go	24	32	75.0%

Files with Coverage Reduction	New Missed Lines	%
pkg/test/expectations/expectations.go	2	94.64%
pkg/controllers/disruption/drift.go	2	89.66%

Totals
Change from base Build 9947083879:	-0.1%
Covered Lines:	8809
Relevant Lines:	11292

---

💛 - Coveralls

[ellistarn]

Can you write an RFC: https://karpenter.sh/docs/contributing/design-guide/?

[cnmcavoy]

RFC: #1433

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[github-actions]

This PR has been inactive for 14 days. StaleBot will close this stale PR after 14 more days of inactivity.