Skip to content

Latest commit

 

History

History
97 lines (74 loc) · 9.25 KB

handling-eviction-requests.md

File metadata and controls

97 lines (74 loc) · 9.25 KB

Handling Eviction Requests

Preface

The Kubernetes API supports API-initiated Eviction which allows to programmatically evict pods. The API is used for example by:

Since KubeVirt virtual machines are running inside virt-launcher pods - they are affected by Kubernetes' eviction mechanism. This requires special handling on KubeVirt's side, since virtual machines eviction is a bit more complex than the average pod. Evacuation is the term used by KubeVirt to describe the migration of a VMI as the result of virt-launcher pod eviction.

This document will describe how KubeVirt currently handles eviction requests.

Eviction Strategies

A VirtualMachineInstance can have one of four Eviction Strategies. The eviction strategy is defined in the VMI spec, with a fallback to a cluster-wide definition in the KubeVirt CustomResource.

The eviction strategy affects the way the VirtualMachineInstance will be evacuated:

Eviction Strategy Meaning
None No action will be taken, according to the specified 'RunStrategy' the VirtualMachine will be restarted or shutdown
LiveMigrate The VirtualMachine will be migrated instead of being shutdown
LiveMigrateIfPossible Same as LiveMigrate but only if the VirtualMachine is Live-Migratable, otherwise it will behave as None
External The VirtualMachine will be protected by a PDB and vmi.Status.EvacuationNodeName will be set on eviction. This is mainly useful for cluster-api-provider-kubevirt (capk) which needs a way for VMI’s to be blocked from eviction, yet signal capk that eviction has been called on the VMI so the capk controller can handle tearing the VMI down

Pod Eviction Webhook

virt-api serves a validating webhook which intercepts all eviction requests in the cluster:

kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io virt-api-validator -o yaml

The purpose of this webhook is to trigger VMI evacuation in cases where it is required. The way the webhook triggers the evacuation is by setting the VMI's Status.EvacuationNodeName field to the node name it is currently running on, so the evacuation controller will know it needs to migrate it to another node.

The webhook has the ability to:

  1. Approve the request - so it could be further processed
  2. Deny the request - the request will be declined without additional processing

The webhook admits eviction requests before kube-api checks them against Pod Distribution Budget objects.

In case the pod is not a virt-launcher pod - the eviction request is approved. Otherwise, depending on the VMI's eviction strategy and whether it is migratable - the webhook will potentially mark the VMI for evacuation and approve or deny the eviction request:

Eviction Strategy Is VMI migratable Is VMI marked for eviction Does Webhook approve eviction Webhook Response
None True/False False True 200 - Eviction granted
LiveMigrate True True True 200 - Eviction granted
LiveMigrate False False False 429 - Eviction denied
LiveMigrateIfPossible True True True 200 - Eviction granted
LiveMigrateIfPossible False False True 200 - Eviction granted
External True/False True True 200 - Eviction granted

Note
Since the webhook intercepts all eviction requests in the cluster, it is configured to be ignored in case kube-api fails to get a response from it. Ignored in this context means that the eviction request is considered to be approved by the webhook, and will be further checked against the PodDistributionBudget. Some virt-launcher pods should be protected from eviction even if the webhook fails, this is the reason PodDistributionBudget objects are required (described in the next section). In case the webhook is down, the virt-launcher pod will be protected from eviction by the PDB (if required), but the evacuation will not be triggered.

Pod Distribution Budget

In case the Pod Eviction Webhook approved the eviction, kube-api checks whether a PDB protects the virt-launcher pod. If there is a PDB protecting the virt-launcher pod - the eviction request is denied, otherwise it is approved and the pod is evicted.

In order for the evacuation of VMIs to happen in a controlled manner, KubeVirt protects part of the virt-launcher pods with a PDB which blocks eviction requests.

virt-controller has a Disruption Budget Controller which decides whether a virt-launcher pod should be protected based on the eviction strategy of its controlling VMI:

Eviction Strategy Is a PDB required
None False
LiveMigrate True
LiveMigrateIfPossible Only if the VMI is migratable
External True

Note
During a migration, the PDB that protects the source virt-launcher pod is expended by the migration controller to also protect the target pod.

Eviction Approval Summary

The eviction request's initiator will observe one of the following responses:

Eviction Strategy Is VMI migratable Is VMI marked for eviction Does Webhook approve eviction Does PDB allow eviction Final Response
None True/False False True True 200 - Eviction granted
LiveMigrate True True True False 429 - Eviction blocked by PDB
LiveMigrate False False False False 429 - Eviction denied by webhook
LiveMigrateIfPossible True True True False 429 - Eviction blocked by PDB
LiveMigrateIfPossible False False True True 200 - Eviction granted
External True/False True True False 429 - Eviction blocked by PDB

To summarize:

  1. The eviction request is granted only if both the webhook and the PDB allow them.
  2. The eviction request's initiator might get a 429 response, but the VMI will be migrated in the background.

Evacuation Controller

virt-controller has an evacuation controller which looks for potential VMIs to evict and tries to migrate them to another node.