How to cleanup container registry blobs in Kubernetes with garbage collection

Last updated on November 22, 2023

Registry blobs in a Kubernetes environment can accumulate over time, leading to increased storage usage and potential performance issues. To ensure the efficient operation of your container registry, it’s important to regularly clean up unnecessary blobs. In this post, we’ll explore a script that utilizes Kubernetes tools to clean up registry blobs effectively.

If you get messages from garbage collection that many blobs found but nothing removed, you can still delete them.

743 blobs marked, 0 blobs and 0 manifests eligible for deletion

Step 1

Find the Registry Pod Before starting the cleanup process, we need to locate the registry pod in your Kubernetes cluster. You can use the following command:

kubectl get pods -n container-registry

This command retrieves the list of pods in the container-registry namespace, allowing you to identify the registry pod based on its name.

Step 2

Connect to the Registry Pod Once you have identified the registry pod, you can establish a connection to it using the kubectl exec command. The script begins with the following command:

kubectl exec -n container-registry -it <registry-pod-name> -- sh

Replace <registry-pod-name> with the actual name of the registry pod obtained in Step 1.

This command initiates an interactive shell session within the registry pod, enabling you to execute commands directly.

Step 3

Removing Outdated Manifest Tags The next step focuses on removing outdated manifest tags that have not been used for a specific period, such as 14 days. This is achieved using the find command and the following line:

find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/tags/* -type d -mtime +14 -maxdepth 1 -exec rm -rf {} \;

Or you can run this command from kubernetes:

kubectl exec -n container-registry -it <registry-pod-name> -- sh -c 'find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/tags/* -type d -mtime +14 -maxdepth 1 -exec rm -rf {} \;'

This command locates the directories corresponding to outdated manifest tags and deletes them using the rm -rf command. Adjust the -mtime value according to your requirements.

Step 4

Deleting Unreferenced Blob Revisions Next, we target unreferenced blob revisions using a similar approach. The command used in the script is as follows:

find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/revisions/sha256/* -type d -mtime +14 -maxdepth 1 -exec rm -rf {} \;

Or you can run this command from kubernetes:

kubectl exec -n container-registry -it <registry-pod-name> -- sh -c 'find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/revisions/sha256/* -type d -mtime +14 -maxdepth 1 -exec rm -rf {} \;'

Similarly to Step 3, this command locates the unreferenced blob revision directories and removes them with rm -rf. Again, you can modify the -mtime parameter to fit your specific needs.

Step 5

Registry Garbage Collection Finally, we perform registry garbage collection to clean up any remaining unused blobs. The script executes the command:

kubectl exec -n container-registry -it <registry-pod-name> -- /bin/registry garbage-collect -m /etc/docker/registry/config.yml

Replace <registry-pod-name> with the actual name of the registry pod. This command triggers the garbage collection process by utilizing the /bin/registry tool with the specified configuration file path.

Then you can create bash script to automate this job: blobs-cleanup.sh

#!/bin/bash

kubectl exec -n container-registry -it <registry-pod-name> -- sh -c 'find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/tags/* -type d -mtime +14 -maxdepth 1 -exec rm -rf {} \;'

kubectl exec -n container-registry -it <registry-pod-name> -- sh -c 'find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/revisions/sha256/* -type d -mtime +14 -maxdepth 1 -exec rm -rf {} \;'

kubectl exec -n container-registry -it <registry-pod-name> -- /bin/registry garbage-collect -m /etc/docker/registry/config.yml

Replace <registry-pod-name> with the actual name of the registry pod.
Make it executablechmod +x blobs-cleanup.sh

Now we can add cron expression for every 2 weeks on the 1st and the 15th of every month at 1:30 AM:

30 1 1,15 * * /home/username/blobs-cleanup.sh

By following this script, you can easily clean up registry blobs within a Kubernetes cluster using Kubernetes tools. Regularly performing this cleanup helps optimize storage usage and maintain the performance of your container registry. Remember to adjust the time parameters according to your specific requirements and consult the documentation for more advanced cleanup strategies.