A rules based pod killing container. Pod-Reaper was designed to kill pods that meet specific conditions. See the "Implemented Rules" section below for details on specific rules.
Pod-Reaper is configurable through environment variables. The pod-reaper specific environment variables are:
NAMESPACEthe kubernetes namespace where pod-reaper should look for podsGRACE_PERIODduration that pods should be given to shut down before hard killing the podSCHEDULEschedule for when pod-reaper should look for pods to reapRUN_DURATIONhow long pod-reaper should run before exitingEVICTtry to evict pods instead of deleting themEXCLUDE_LABEL_KEYpod metadata label (of key-value pair) that pod-reaper should excludeEXCLUDE_LABEL_VALUEScomma-separated list of metadata label values (of key-value pair) that pod-reaper should excludeREQUIRE_LABEL_KEYpod metadata label (of key-value pair) that pod-reaper should requireREQUIRE_LABEL_VALUEScomma-separated list of metadata label values (of key-value pair) that pod-reaper should requireREQUIRE_ANNOTATION_KEYpod metadata annotation (of key-value pair) that pod-reaper should requireREQUIRE_ANNOTATION_VALUEScomma-separated list of metadata annotation values (of key-value pair) that pod-reaper should requireDRY_RUNlog pod-reaper's actions but don't actually kill any podsMAX_PODSkill a maximum number of pods on each runPOD_SORTING_STRATEGYsorts pods before killing them (most useful when used with MAX_PODS)LOG_LEVELcontrol verbosity level of log messagesLOG_FORMATchoose between several formats of logging
Additionally, at least one rule must be enabled, or the pod-reaper will error and exit. See the Rules section below for configuring and enabling rules.
Example environment variables:
# pod-reaper configuration
NAMESPACE=test
SCHEDULE=@every 30s
RUN_DURATION=15m
EXCLUDE_LABEL_KEY=pod-reaper
EXCLUDE_LABEL_VALUES=disabled,false
# enable at least one rule
CHAOS_CHANCE=.001Default value: "" (which will look at ALL namespaces)
Controls which kubernetes namespace the pod-reaper is in scope for the pod-reaper. Note that the pod-reaper uses an InClusterConfig which makes use of the service account that kubernetes gives to its pods. Only pods (and namespaces) accessible to this service account will be visible to the pod-reaper.
Default value: nil (indicates to the use the default specified for pods)
Controls the grace period between a soft pod termination and a hard termination. This will determine the time between when the pod's containers are send a SIGTERM signal and when they are sent a SIGKILL signal. The format follows the go-lang time.duration format (example: "1h15m30s"). A duration of 0s can be considered a hard kill of the pod.
Default value: "@every 1m"
Controls how frequently pod-reaper queries kubernetes for pods. The format follows the upstream cron library https://godoc.org/github.com/robfig/cron. For most use cases, the interval format @every 1h2m3s is sufficient. But more complex use cases can make use of the * * * * * notation. The cron parser used can optionally support seconds if a sixth parameter is add. 12 * * * * * for example will run on the 12th second of every minute.
Default value: "0s" (which corresponds to running indefinitely)
Controls the minimum duration that pod-reaper will run before intentionally exiting. The value of "0s" (or anything equivalent such as the empty string) will be interpreted as an indefinite run duration. The format follows the go-lang time.duration format (example: "1h15m30s"). Pod-Reaper will not wait for reap-cycles to finishing waiting and will exit immediately (with exit code 0) after the duration has elapsed.
Warnings about RUN_DURATION
- pod-rescheduling: if the reaper completes, even successfully, it may be restarted depending on the pod-spec.
- self-reaping: the pod-reaper can reap itself if configured to do so, this can cause the reaper to not run for the expected duration.
Recommendations:
One time run:
- create a pod spec and apply it to kubernetes
- make the pod spec has
restartPolicy: Never - add an exclusion label and key using
EXCLUDE_LABEL_KEYandEXCLUDE_LABEL_VALUES - make the pod spec for the reaper match an excluded label and key to prevent it from reaping itself
Sustained running:
- do not use
RUN_DURATION - manage the pod reaper via a deployment
Use the Eviction API instead of pod deletion when reaping pods. The Eviction API will honor the disruption budget assigned to pods, and can for example be useful when reaping pods by duration to ensure that you don't reap all the pods of a specific deployment simultaneously, interrupting a published service. When a pod cannot be reaped due to a disruption budget, the reason will be logged as a warning.
These environment variables are used to build a label selector to exclude pods from reaping. The key must be a properly formed kubernetes label key. Values are a comma-separated (without whitespace) list of kubernetes label values. Setting exactly one of the key or values environment variables will result in an error.
A pod will be excluded from the pod-reaper if the pod has a metadata label has a key corresponding to the pod-reaper's exclude label key, and that same metadata label has a value in the pod-reaper's list of excluded label values. This means that exclusion requires both the pod-reaper and pod to be configured in a compatible way.
These environment variables build a label selector that pods must match in order to be reaped. Use them the same way as you would EXCLUDE_LABEL_KEY and EXCLUDE_LABEL_VALUES.
These environment variables build a annotation selector that pods must match in order to be reaped. Use them the same way as you would EXCLUDE_LABEL_KEY and EXCLUDE_LABEL_VALUES.
Default value: unset (which will behave as if it were set to "false")
Acceptable values are 1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False. Any other values will error. If the provided value is one of the "true" values then pod reaper will do select pods for reaper but will not actually kill any pods. Logging messages will reflect that a pod was selected for reaping and that pod was not killed because the reaper is in dry-run mode.
Default value: unset (which will behave as if it were set to "0")
Acceptable values are positive integers. Negative integers will evaluate to 0 and any other values will error. This can be useful to prevent too many pods being killed in one run. Logging messages will reflect that a pod was selected for reaping and that pod was not killed because too many pods were reaped already.
Default value: unset (which will use the pod ordering return without specification from the API server). Accepted values:
- (unset) - use the default ordering from the API server
random(case-sensitive) will randomly shuffle the list of pods before killingoldest-first(case-sensitive) will sort pods into oldest-first based on the pods start time. (!! warning below).youngest-first(case-sensitive) will sort pods into youngest-first based on the pods start time (!! warning below)pod-deletion-cost(case-sensitive) will sort pods based on the pod deletion cost annotation.
!! WARNINGS !!
Pod start time is not always defined. In these cases, sorting strategies based on age put pods without start times at the end of the list. From my experience, this usually happens during a race condition with the pod initially being scheduled, but there may be other cases hidden away.
Using pod-reaper against the kube-system namespace can have some surprising implications. For example, during testing I
found that the kube-schedule was owned by a master node (not a replicaset/daemon-set) and appeared to effectively ignore
delete actions. The age returned from kubectl was reset, but the actual pod start time was unaffected. As a result of
this, I found a looping scenario where the kube scheduler was effectively always the oldest pod.
In examples/pod-sorting-strategy.yml I mitigated this using by excluding on the label tier: control-plane
Pod reaper logs in JSON format using a logrus (https://github.com/sirupsen/logrus).
- rule load: customer messages for each rule are logged when the pod-reaper is starting
- reap cycle: a message is logged each time the reaper starts a cycle.
- pod reap: a message is logged (with a reason for each rule) when a pod is flag for reaping.
- exit: a message is logged when the reaper exits successfully (only is
RUN_DURATIONis specified)
Default value: Info
Messages this level and above will be logged. Available logging levels: Debug, Info, Warning, Error, Fatal and Panic
{"level":"info","msg":"loaded rule: chaos chance .3","time":"2017-10-18T17:09:25Z"}
{"level":"info","msg":"loaded rule: maximum run duration 2m","time":"2017-10-18T17:09:25Z"}
{"level":"info","msg":"executing reap cycle","time":"2017-10-18T17:09:55Z"}
{"level":"info","msg":"reaping pod","pod":"hello-cloud-deployment-3026746346-bj65k","reasons":["was flagged for chaos","has been running for 3m6.257891269s"],"time":"2017-10-18T17:09:55Z"}
{"level":"info","msg":"reaping pod","pod":"example-pod-deployment-125971999cgsws","reasons":["was flagged for chaos","has been running for 2m55.269615797s"],"time":"2017-10-18T17:09:55Z"}
{"level":"info","msg":"executing reap cycle","time":"2017-10-18T17:10:25Z"}
{"level":"info","msg":"reaping pod","pod":"hello-cloud-deployment-3026746346-grw12","reasons":["was flagged for chaos","has been running for 3m36.054164005s"],"time":"2017-10-18T17:10:25Z"}
{"level":"info","msg":"pod reaper is exiting","time":"2017-10-18T17:10:46Z"}Default value: Logrus
This environment variable modifies the structured log format for easy ingestion into different logging systems, including Stackdriver via the Fluentd format. Available formats: Logrus, Fluentd
Flags a pod for reaping based on a random number generator.
Enabled and configured by setting the environment variable CHAOS_CHANCE with a floating point value. A random number generator will generate a value in range [0,1) and if the the generated value is below the configured chaos chance, the pod will be flagged for reaping.
Example:
# every 30 seconds kill 1/100 pods found (based on random chance)
SCHEDULE=@every 30s
CHAOS_CHANCE=.01Remember that pods can be excluded from reaping if the pod has a label matching the pod-reaper's configuration. See the EXCLUDE_LABEL_KEY and EXCLUDE_LABEL_VALUES section above for more details.
Flags a pod for reaping based on a container within a pod having a specific container status.
Enabled and configured by setting the environment variable CONTAINER_STATUSES with a coma separated list (no whitespace) of statuses. If a pod is in either a waiting or terminated state with a status in the specified list of status, the pod will be flagged for reaping.
Example:
# every 10 minutes, kill all pods with a container with a status ImagePullBackOff, ErrImagePull, or Error
SCHEDULE=@every 10m
CONTAINER_STATUSES=ImagePullBackOff,ErrImagePull,ErrorNote that this will not catch statuses that are describing the entire pod like the Evicted status.
Flags a pod for reaping based on the pod status.
Enabled and configured by setting the environment variable POD_STATUSES with a comma separated list (no whitespace) of statuses. If the pod status in the specified list of status, the pod will be flagged for reaping.
Example:
# every 10 minutes, kill all pods with status ImagePullBackOff, ErrImagePull, or Error
SCHEDULE=@every 10m
POD_STATUSES=Evicted,UnknownNote that pod status is different than container statuses as it checks the status of the overall pod rather than the status of containers in the pod. The most obvious use case of this if dealing with Evicted pods.
Flags a pod for reaping based on it's pod.Status.Phase - i.e. Pending, Running, Succeeded, Failed, Unknown. See Kubernetes Go client docs on this type here.
Enabled and configured by setting the environment variable POD_PHASE_STATUSES with a comma-separated list (no whitespaces) of pod status phases. If a pod is in the status phase specified in the list, it will be flagged for reaping.
Example:
# every 10 minutes, kill all pods with status Pending or Failed
SCHEDULE=@every 10m
POD_PHASE_STATUSES=Pending,FailedFlags a pod for reaping based on the pods current run duration.
Enabled and configured by setting the environment variable MAX_DURATION with a valid go-lang time.duration format (example: "1h15m30s"). If a pod has been running longer than the specified duration, the pod will be flagged for reaping.
Flags a pod for reaping based on the time the pod has been unready.
Enabled and configured by setting the environment variable MAX_UNREADY with a valid go-lang time.duration format (example: "10m"). If a pod has been unready longer than the specified duration, the pod will be flagged for reaping.
Pod reaper uses the permissions of the pod's service account to list and delete pods. Unless specified, the service account used will be the default service account in the pod's namespace. By default, and in most cases, the default service account will not have the neccessary permissions to list and delete pods.
A pod will only be reaped if ALL rules flag the pod for reaping, but you can achieve reaping on OR logic by simply running another pod-reaper.
For example, in the same pod-reaper container:
CHAOS_CHANCE=.01
MAX_DURATION=2hMeans that 1/100 pods that also have a run duration of over 2 hours will be reaped. If you want 1/100 pods reaped regardless of duration and also want all pods with a run duration of over hours to be reaped, run two pod-reapers. one with: CHAOS_CHANCE=.01 and another with MAX_DURATION=2h.
Multiple pod-reapers can be easily managed and configured with kubernetes deployments. It is encouraged that if you are using deployments, that you leave the RUN_DURATION environment variable unset (or "0s") to let the reaper run forever, since the deployment will reschedule it anyway. Note that the pod-reaper can and will reap itself if it is not excluded.
You can run run pod-reaper as a one time, limited duration container by usable the RUN_DURATION environment variable. An example use case might be wanting to introduce a high degree of chaos into your kubernetes environment for a short duration:
# 30% chaos chance every 1 minute for 15 minutes
SCHEDULE=@every 1m
RUN_DURATION=15m
CHAOS_CHANCE=.3