Deleting a stuck Kubernetes namespace in terminating state

Last updated on November 22, 2023

Kubernetes is a powerful container orchestration platform that allows developers to manage and scale applications efficiently. However, sometimes things can go awry, and namespaces might get stuck in the “Terminating” state, preventing you from performing essential operations on your cluster. In this blog post, we will explore the common reasons behind this issue and provide a step-by-step guide on how to delete a namespace stuck in the Terminating state.

When you delete a Kubernetes namespace, the system initiates a graceful termination process, where it tries to terminate all resources (pods, services, deployments, etc.) associated with that namespace before removing it. During this process, the namespace enters the “Terminating” state.

If something goes wrong during the termination process, the namespace may get stuck in this state. This is often caused by one or more resources within the namespace failing to terminate properly.

Diagnosing the Issue

Before attempting to delete the stuck namespace, it’s essential to identify the root cause of the problem. Here are some steps you can follow to diagnose the issue:

1. List all namespaces:

kubectl get namespaces

2. Check the status of the namespace you want to delete:

kubectl get namespace <NAMESPACE_NAME> -o json | jq '.status.phase'

3. List all resources in the namespace:

kubectl get all -n <NAMESPACE_NAME>

4. Examine the events associated with the namespace:

kubectl describe namespace <NAMESPACE_NAME>

Common Causes of Stuck Terminating State

Several factors can contribute to a namespace getting stuck in the Terminating state:

1. Orphaned Finalisers: Finalisers are hooks that allow the Kubernetes API server to perform certain operations before deleting resources. If a resource has an orphaned finaliser, it can prevent the namespace from terminating.

2. Hanging Resources: Sometimes, resources (e.g., pods or PersistentVolumeClaims) can hang due to underlying issues, such as failing termination scripts or network problems.

3. Dependencies: Resources in the namespace may have dependencies on resources outside the namespace, causing termination delays.

Solutions

Now that we understand the possible causes, let’s go through the steps to resolve the issue and force delete the stuck namespace:

1. Manually Delete Resources: If you have identified specific resources causing the problem, try deleting them individually:

kubectl delete <RESOURCE_TYPE> <RESOURCE_NAME> -n <NAMESPACE_NAME>

2. Remove Finalizers: If orphaned finalizers are the issue, you can remove them using ‘kubectl edit’. Locate the resource with the orphaned finalizer and remove the finalizer entry.

3. Force Delete the Namespace: If none of the above methods work, you can try force deleting the namespace, which bypasses the graceful termination process. Be cautious while using this method, as it may leave behind dangling resources.

kubectl delete namespace <NAMESPACE_NAME> --grace-period=0 --force

4. Verify Namespace Deletion: After attempting to delete the namespace, check if it has been removed successfully:

kubectl get namespaces | grep <NAMESPACE_NAME>

Another method to terminate a Kubernetes namespace involves using the Kubernetes API directly, which is not exposed through the regular kubectl commands. This approach is possible if you have a more recent version of kubectl that supports the kubectl replace --raw functionality (introduced in a version higher than 1.14, though the specific version may vary).

With this method, you can avoid using kubectl proxy and dependencies on tools like curl, especially in environments where such tools might not be available, such as BusyBox. Here’s a concise explanation of how you can achieve this:

kubectl get namespace "<NAMESPACE_NAME>" -o json \
  | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" \
  | kubectl replace --raw /api/v1/namespaces/<NAMESPACE_NAME>/finalize -f -

Deleting a Kubernetes namespace stuck in the Terminating state can be a challenging task, but with a systematic approach and understanding of potential issues, you can resolve the problem efficiently. Always remember to proceed with caution and consider the impact of force deleting a namespace. In cases where you are unsure about the cause or unsure of the potential impact, it’s advisable to seek help from experienced Kubernetes administrators or the Kubernetes community.

Happy troubleshooting!