WIP: ✨ Add MachineDrainRule "WaitCompleted" #11545
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
cncf-cla: yes
vincepri
force-pushed
the
pod-drain-waitcompleted
branch
from
631a543 to
bab1c1b
Compare
Signed-off-by: Vince Prignano <[email protected]>
vincepri
force-pushed
the
pod-drain-waitcompleted
branch
from
bab1c1b to
211839e
Compare
vincepri
added
the
area/machine
vincepri
added
priority/important-soon
sbueringer
reviewed
Dec 30, 2024
| // MachineDrainRuleDrainBehavior defines the drain behavior. Can be either "Drain" or "Skip". | ||
| // +kubebuilder:validation:Enum=Drain;Skip | ||
| // MachineDrainRuleDrainBehavior defines the drain behavior. Can be either "Drain", "Skip", or "WaitCompleted". | ||
| // +kubebuilder:validation:Enum=Drain;Skip;WaitCompleted |
There was a problem hiding this comment.
Please check if the proposal should be amended accordingly
| @@ -22,13 +22,13 @@ import ( | |||
|
|
|||
| const ( | |||
| // PodDrainLabel is the label that can be set on Pods in workload clusters to ensure a Pod is not drained. | |||
| // The only valid value is "skip". | |||
| // The only valid value is "skip" or "waitcompleted". | |||
There was a problem hiding this comment.
Suggested change
| // The only valid value is "skip" or "waitcompleted". | |
| // The only valid values are "skip" and "waitcompleted". |
| @@ -498,6 +507,10 @@ func (r EvictionResult) ConditionMessage(nodeDrainStartTime *metav1.Time) string | |||
| conditionMessage = fmt.Sprintf("%s\nAfter above Pods have been removed from the Node, the following Pods will be evicted: %s", | |||
| conditionMessage, PodListToString(r.PodsToTriggerEvictionLater, 3)) | |||
| } | |||
| if len(r.PodsToWaitCompleted) > 0 { | |||
| conditionMessage = fmt.Sprintf("%s\nWaiting for the following Pods to complete without drain: %s", | |||
There was a problem hiding this comment.
Suggested change
| conditionMessage = fmt.Sprintf("%s\nWaiting for the following Pods to complete without drain: %s", | |
| conditionMessage = fmt.Sprintf("%s\nWaiting for the following Pods to complete without eviction: %s", |
Maybe more precise
| @@ -275,6 +286,11 @@ func (d *Helper) drainLabelFilter(ctx context.Context, pod *corev1.Pod) PodDelet | |||
| log.V(4).Info(fmt.Sprintf("Skip evicting Pod, because Pod has %s label", clusterv1.PodDrainLabel)) | |||
| return MakePodDeleteStatusSkip() | |||
| } | |||
| if labelValue, found := pod.ObjectMeta.Labels[clusterv1.PodDrainLabel]; found && strings.EqualFold(labelValue, string(clusterv1.MachineDrainRuleDrainBehaviorWaitCompleted)) { | |||
| log := ctrl.LoggerFrom(ctx, "Pod", klog.KObj(pod)) | |||
| log.V(4).Info(fmt.Sprintf("Skip evicting Pod, because Pod has %s label", clusterv1.PodDrainLabel)) | |||
There was a problem hiding this comment.
Should we maybe start mentioning the label value (here and in l.286)
| @@ -300,6 +316,8 @@ func (d *Helper) machineDrainRulesFilter(machineDrainRules []*clusterv1.MachineD | |||
| log := ctrl.LoggerFrom(ctx, "Pod", klog.KObj(pod)) | |||
| log.V(4).Info(fmt.Sprintf("Skip evicting Pod, because MachineDrainRule %s with behavior %s applies to the Pod", mdr.Name, clusterv1.MachineDrainRuleDrainBehaviorSkip)) | |||
| return MakePodDeleteStatusSkip() | |||
| case clusterv1.MachineDrainRuleDrainBehaviorWaitCompleted: | |||
There was a problem hiding this comment.
Let's add the same log as in l.316+317
| @@ -281,6 +281,7 @@ func (d *Helper) EvictPods(ctx context.Context, podDeleteList *PodDeleteList) Ev | |||
| var podsToTriggerEvictionLater []PodDelete | |||
| var podsWithDeletionTimestamp []PodDelete | |||
There was a problem hiding this comment.
Please check the existing test coverage for the modified funcs and extend accordingly for the new case (we should have test coverage for everything, only new cases should be needed)
| @@ -498,6 +507,10 @@ func (r EvictionResult) ConditionMessage(nodeDrainStartTime *metav1.Time) string | |||
| conditionMessage = fmt.Sprintf("%s\nAfter above Pods have been removed from the Node, the following Pods will be evicted: %s", | |||
| conditionMessage, PodListToString(r.PodsToTriggerEvictionLater, 3)) | |||
| } | |||
| if len(r.PodsToWaitCompleted) > 0 { | |||
There was a problem hiding this comment.
@fabriziopandini Is this something that should be specifically handled on higher levels when the condition bubbles up? (like we handled PDB's etc.)
(could also be a follow-up PR)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This PR adds the ability for drain to wait for the completion of specific pods. This is useful in scenario where drain is either handled outside the context of
kubectl drainafter a Node is cordoned, or for long running batch Jobs that should be allowed to terminate on their own.Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #