Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

K8s-9372: Add modification for custom policy #57

Merged
merged 8 commits into from
Mar 28, 2024
16 changes: 11 additions & 5 deletions pkg/apis/cluster/v1alpha1/machineset_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ type MachineSetSpec struct {
MinReadySeconds int32 `json:"minReadySeconds,omitempty"`

// DeletePolicy defines the policy used to identify nodes to delete when downscaling.
// Defaults to "Random". Valid values are "Random, "Newest", "Oldest"
// Defaults to "Custom". Valid values are "Random, "Newest", "Oldest"
// +kubebuilder:validation:Enum=Random,Newest,Oldest
DeletePolicy string `json:"deletePolicy,omitempty"`

Expand All @@ -78,7 +78,7 @@ type MachineSetSpec struct {
}

// MachineSetDeletePolicy defines how priority is assigned to nodes to delete when
// downscaling a MachineSet. Defaults to "Random".
// downscaling a MachineSet. Defaults to "Custom".
squ94wk marked this conversation as resolved.
Show resolved Hide resolved
type MachineSetDeletePolicy string

const (
Expand All @@ -99,6 +99,12 @@ const (
// (Status.ErrorReason or Status.ErrorMessage are set to a non-empty value).
// It then prioritizes the oldest Machines for deletion based on the Machine's CreationTimestamp.
OldestMachineSetDeletePolicy MachineSetDeletePolicy = "Oldest"

// CustomMachineSetDeletePolicy prioritizes both Machines that have the annotation
// "cluster.k8s.io/delete-machine=yes" and Machines that are unhealthy
// (Status.ErrorReason or Status.ErrorMessage are set to a non-empty value).
// If then prioritizes the machine with Status different from Ready.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe say explicitly that it prioritizes new machines over old ones for deletion

CustomMachineSetDeletePolicy MachineSetDeletePolicy = "Custom"
)

/// [MachineSetSpec] // doxygen marker
Expand Down Expand Up @@ -203,9 +209,9 @@ func (m *MachineSet) Default() {
}

if m.Spec.DeletePolicy == "" {
randomPolicy := string(RandomMachineSetDeletePolicy)
log.Printf("Defaulting to %s\n", randomPolicy)
m.Spec.DeletePolicy = randomPolicy
customPolicy := string(CustomMachineSetDeletePolicy)
log.Printf("Defaulting to %s\n", customPolicy)
m.Spec.DeletePolicy = customPolicy
}
}

Expand Down
42 changes: 41 additions & 1 deletion pkg/controller/machineset/delete_policy.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ import (

"github.com/pkg/errors"

corev1 "k8s.io/api/core/v1"

"github.com/kubermatic/machine-controller/pkg/apis/cluster/v1alpha1"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Expand All @@ -48,15 +50,22 @@ const (

// maps the creation timestamp onto the 0-100 priority range.
func oldestDeletePriority(machine *v1alpha1.Machine) deletePriority {
// DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This
// field is set by the server when a graceful deletion is requested by the user, and is not
// directly settable by a client.
// If machine deletion was already set, machine OK to delete
if machine.DeletionTimestamp != nil && !machine.DeletionTimestamp.IsZero() {
return mustDelete
}
// If machine annotations is not empty and DeleteNodeAnnotation is set, machine OK to delete
if machine.ObjectMeta.Annotations != nil && machine.ObjectMeta.Annotations[DeleteNodeAnnotation] != "" {
return mustDelete
}
// If there are machine errors, delete the machine
if machine.Status.ErrorReason != nil || machine.Status.ErrorMessage != nil {
return mustDelete
}
// If machine is new, don't delete it "CreationTimestamp is a timestamp representing the server time when this object was created"
if machine.ObjectMeta.CreationTimestamp.Time.IsZero() {
return mustNotDelete
}
Expand All @@ -67,6 +76,34 @@ func oldestDeletePriority(machine *v1alpha1.Machine) deletePriority {
return deletePriority(float64(mustDelete) * (1.0 - math.Exp(-d.Seconds()/secondsPerTenDays)))
}


func customDeletePolicy(machine *v1alpha1.Machine) deletePriority {
squ94wk marked this conversation as resolved.
Show resolved Hide resolved
// Iterate through conditions to get if machine is not "Ready"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we always return mustDelete, there's no real order to the machines with that value.
That's something we should eventually improve; for now maybe add a comment.

However, maybe reorder these checks already:

  • deletion timestamp non zero
  • machine has no nodeRef -> still provisioning
  • deletion annotation
  • node status NotReady
  • error status

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I'll "split" the NodeReady/Conditions report to make machine selection more granular.

// Check types.go
for _, c := range machine.Status.Conditions {
if c.Type != corev1.NodeReady {
return mustDelete
squ94wk marked this conversation as resolved.
Show resolved Hide resolved
}
}
// Rest of conditions are extracted from the "Newest" policy, that is the one that
// matches the idea of deleting the newest machines (more possibilities to be WIP in the attachment to cluster)
// than the old ones.
if machine.DeletionTimestamp != nil && !machine.DeletionTimestamp.IsZero() {
squ94wk marked this conversation as resolved.
Show resolved Hide resolved
return mustDelete
}
// If annotations are set + DeleteAnnotation is present
if machine.ObjectMeta.Annotations != nil && machine.ObjectMeta.Annotations[DeleteNodeAnnotation] != "" {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unnecessary nil check:

Suggested change
if machine.ObjectMeta.Annotations != nil && machine.ObjectMeta.Annotations[DeleteNodeAnnotation] != "" {
if v, ok := machine.ObjectMeta.Annotations[DeleteNodeAnnotation]; ok && v != "" {

return mustDelete
}
// If there are errors on machine, delete it
if machine.Status.ErrorReason != nil || machine.Status.ErrorMessage != nil {
return mustDelete
}
// Thinking about this, maybe it's a good idea to delegate in oldestDeletePriority, like
// newestDeletePriority is doing if no match is done, or just return couldDelete
return mustDelete - oldestDeletePriority(machine)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense like this.

}

func newestDeletePriority(machine *v1alpha1.Machine) deletePriority {
if machine.DeletionTimestamp != nil && !machine.DeletionTimestamp.IsZero() {
return mustDelete
Expand Down Expand Up @@ -124,15 +161,18 @@ func getMachinesToDeletePrioritized(filteredMachines []*v1alpha1.Machine, diff i

func getDeletePriorityFunc(ms *v1alpha1.MachineSet) (deletePriorityFunc, error) {
// Map the Spec.DeletePolicy value to the appropriate delete priority function
// Defaults to customDeletePolicy if not specified
switch msdp := v1alpha1.MachineSetDeletePolicy(ms.Spec.DeletePolicy); msdp {
case v1alpha1.RandomMachineSetDeletePolicy:
return randomDeletePolicy, nil
case v1alpha1.NewestMachineSetDeletePolicy:
return newestDeletePriority, nil
case v1alpha1.OldestMachineSetDeletePolicy:
return oldestDeletePriority, nil
case v1alpha1.CustomMachineSetDeletePolicy:
return customDeletePolicy, nil
case "":
return randomDeletePolicy, nil
return customDeletePolicy, nil
default:
return nil, errors.Errorf("Unsupported delete policy %q. Must be one of 'Random', 'Newest', or 'Oldest'", msdp)
}
Expand Down
Loading