When running Kubernetes (K8s), managing container images efficiently is key to maintaining node stability and performance. The kubelet, Kubernetes’ node agent, periodically triggers garbage collection (GC) to reclaim disk space by removing unused images. While this is great for resource management, what if you need to keep specific images-like jenkins/inbound-agent:3283.v92c105e0f819-8 – on your nodes indefinitely, even if no Pods are actively using them? Maybe it’s for an air-gapped environment, a critical CI/CD pipeline, or just to avoid the overhead of re-pulling large images. Enter image pinning with containerd, a powerful solution to lock down specific images and protect them from garbage collection.
In this post, I’ll walk you through why image garbage collection happens, how to pin images using containerd in a Kubernetes environment, and a step-by-step example with the Jenkins Inbound Agent image. Let’s dive in!
Why Does Kubernetes Garbage Collect Images?
Kubernetes relies on the kubelet to manage container images on each node. When disk usage hits a configurable threshold (e.g., 85% by default), the kubelet calls the underlying container runtime – often containerd-to prune images that aren’t referenced by running containers. This process considers factors like:
- Disk pressure: High and low thresholds (imageGCHighThresholdPercent and imageGCLowThresholdPercent).
- Image age: A minimum age before GC kicks in (imageMinimumGCAge).
- Usage: Only images not tied to active containers are eligible.
This works well for most cases, but it can be a headache if you need specific images to persist-say, for offline clusters or to ensure fast Pod startup times without re-pulling from a registry. Adjusting kubelet GC policies globally helps, but it’s a blunt tool. What if you just want to protect one image, like jenkins/inbound-agent:3283.v92c105e0f819-8?
Pinning Images with Containerd: A Precise Fix
Since Kubernetes 1.20, containerd has become the default container runtime for many clusters, replacing Docker. Starting with containerd 1.7, a handy feature lets you pin images, marking them as untouchable by garbage collection. When the kubelet triggers GC, containerd respects these pins, leaving your chosen images intact. No extra Pods, no cluster-wide hacks – just a runtime-level solution.
Here’s how it works and how to apply it.
Step-by-Step: Pinning jenkins/inbound-agent image
Let’s pin the jenkins/inbound-agent:3283.v92c105e0f819-8 image on a Kubernetes node. This assumes your cluster uses containerd (check with kubectl get nodes -o wide and look at the runtime version, or ctr –version on a node).
1. Access the Node
SSH into the Kubernetes node where the image is stored – or where you want it to reside. If you’re managing multiple nodes, you’ll need to repeat this process or automate it (more on that later).
ssh <node-ip>2. Check for the Image
First, see if the image is already present in containerd’s storage under the k8s.io namespace (used by Kubernetes):
sudo ctr -n k8s.io images ls | grep jenkins/inbound-agent:3283.v92c105e0f819-8If it’s not there, pull it manually from the registry (e.g., Docker Hub):
sudo ctr -n k8s.io images pull docker.io/jenkins/inbound-agent:3283.v92c105e0f819-8You should see output confirming the image is downloaded.
3. Pin the Image
Now, add the io.cri-containerd.pinned label to mark it as pinned:
sudo ctr -n k8s.io images label docker.io/jenkins/inbound-agent:3283.v92c105e0f819-8 io.cri-containerd.pinned=pinnedThis command modifies the image metadata. The value pinned is what containerd looks for – any other value won’t work.
4. Verify the Pin
Check that the label is applied:
sudo ctr -n k8s.io images ls | grep jenkins/inbound-agent:3283.v92c105e0f819-8You’ll see something like:
docker.io/jenkins/inbound-agent:3283.v92c105e0f819-8 ... io.cri-containerd.pinned=pinnedThat’s it! This image is now safe from garbage collection, even if no Pods reference it.
5. Unpinning the Image (If you change your mind)
What if you later decide jenkins/inbound-agent:3283.v92c105e0f819-8 doesn’t need to stick around forever? You can unpin it by removing the label, making it eligible for garbage collection again:
sudo ctr -n k8s.io images label docker.io/jenkins/inbound-agent:3283.v92c105e0f819-8 io.cri-containerd.pinned-Note the trailing – after io.cri-containerd.pinned – this tells containerd to remove the label entirely. Verify it’s gone:
sudo ctr -n k8s.io images ls | grep docker.io/jenkins/inbound-agent:3283.v92c105e0f819-8 The io.cri-containerd.pinned=pinned label should no longer appear. If no Pods are using the image, it’ll be fair game for the next GC cycle.
How Does It Work?
When the kubelet initiates image GC, it delegates the task to containerd. Containerd scans its image store, identifies unused images, and prunes them – unless they have the io.cri-containerd.pinned=pinned label. Pinned images are skipped, leaving them on disk until you manually remove them or update the label. It’s a clean, runtime-level control that doesn’t mess with your Kubernetes manifests.
Considerations and Tips
- Containerd Version: You need containerd 1.7 or later. Check with ctr –version. If you’re on an older version (or using Docker), this won’t work – consider upgrading or using a dummy Pod workaround instead.
- Multi-Node Clusters: Images are stored per node, so repeat the pinning process on each node where you need the image. For automation, a DaemonSet or script via a configuration management tool (e.g., Ansible) can help.
- Re-Pulls: If the image is deleted and re-pulled (e.g., after a manual prune), you’ll need to reapply the label. Watch for this in dynamic environments.
- Disk Space: Pinned images still take up space, so monitor node storage to avoid filling disks.
Why Not Alternatives?
You might wonder about other approaches, like running a dummy Pod or tweaking kubelet GC thresholds. Here’s why pinning stands out:
- Dummy Pods: Creating a Pod to keep jenkins/inbound-agent:3283.v92c105e0f819-8 “in use” works, but it adds cluster overhead and risks deletion if someone cleans up unused Pods.
- Kubelet Config: Raising GC thresholds (e.g., imageGCHighThresholdPercent) delays pruning but affects all images, not just the one you care about. Pinning is precise.
For offline setups or critical images, pinning is the gold standard as of 2025.
Wrapping Up
Pinning images with containerd gives you fine-grained control over Kubernetes’ garbage collection, ensuring key images like jenkins/inbound-agent:3283.v92c105e0f819-8 stay put without hacks or extra resources. It’s a simple, runtime-level trick that’s perfect for CI/CD agents, base images, or anything you don’t want to re-download. Next time you’re fighting image churn, give it a try- just make sure your containerd is up to date!