Clearing Evicted Pods in a Kubernetes Cluster

Last updated on November 22, 2023

Kubernetes, a leading container orchestration platform, plays a pivotal role in deploying and managing containerised applications. However, various factors, such as resource constraints or node failures, can lead to pod evictions within a Kubernetes cluster. These evicted pods, if not managed effectively, can cause resource wastage and hinder the smooth functioning of the cluster. In this blog post, we will delve into the best practices for efficiently managing evicted pods in a Kubernetes cluster, irrespective of the namespace.

Understanding Evicted Pods in Kubernetes

When a pod gets evicted in Kubernetes, it is typically due to resource constraints or node failures, and Kubernetes retains information about these evicted pods, including their state and other relevant details. Clearing these evicted pods is crucial to maintain the optimal performance of the cluster and ensure efficient resource utilization.

1. Retrieving Information about Evicted Pods:

To retrieve information about evicted pods within the cluster, the following command can be used:

kubectl get pods --all-namespaces | grep Evicted

This command fetches a list of all pods in all namespaces and filters out the evicted ones, allowing for a quick overview of the affected pods.

2. Deleting Evicted Pods in One or All Namespaces:

To remove evicted pods, you can utilise the ‘kubectl’ command. For a specific namespace, use the following:

kubectl delete pods --field-selector 'status.phase==Failed' -n <namespace>

To clear evicted pods across all namespaces, execute:

kubectl delete pods --field-selector 'status.phase==Failed' --all-namespaces

These commands effectively delete all evicted pods, freeing up resources within the specified namespace or across the entire cluster.

3. Implementing a Cleanup Strategy:

To ensure a proactive approach to managing evicted pods, it is beneficial to set up a scheduled cleanup strategy. This can be achieved by creating a CronJob within Kubernetes that periodically executes the pod deletion command. Below is an example of a CronJob manifest:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: evicted-pod-cleanup
spec:
  schedule: "0 1 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cleanup-container
            image: bitnami/kubectl
            command: ["sh", "-c", "kubectl delete pods --all --field-selector 'status.phase==Failed' --all-namespaces"]
          restartPolicy: OnFailure

This CronJob is scheduled to run daily at 1:00 AM, executing the command to delete all evicted pods across all namespaces.

Efficient management of evicted pods is essential for maintaining the health and optimal performance of a Kubernetes cluster. By regularly monitoring and clearing evicted pods, you can ensure efficient resource utilization and a seamless operation of containerized applications. Implementing scheduled cleanup strategies further streamlines this process, allowing for an automated and proactive approach to managing evicted pods within the Kubernetes ecosystem.