Last updated on January 22, 2024
Kubernetes continues to grow in popularity, leading to an increased deployment of clusters in production environments. Amidst this surge, security considerations sometimes take a backseat. One often overlooked security aspect within Kubernetes revolves around the image pull policy, particularly concerning multi-tenant clusters. In this blog post, we will delve into the significance of the AlwaysPullImages admission control in Kubernetes, explore its implementation, and shed light on potential security vulnerabilities when this control is absent.
Recently, I encountered a perplexing issue in my Kubernetes deployment that left me scratching my head. The problem revolves around a setup where a remote node, responsible for building and storing images, hosts a private registry. The actual deployment occurs on a separate node, where Kubernetes attempts to pull images from the remote private registry.
The peculiar scenario unfolds when I remove an image from the remote private registry and attempt to run a Pod with the same image on the deployment node, where the image has been previously pulled. Despite setting the imagePullPolicy to IfNotPresent and even experimenting with Never, Kubernetes seems determined to pull the image from the remote registry, resulting in a frustrating failure.
Pulling image "10.1.0.1:32000/service-name:stage-66"
Failed to pull image "10.1.0.1:32000/service-name:stage-66": rpc error: code = NotFound desc = failed to pull and unpack image "10.1.0.1:32000/service-name:stage-66": failed to resolve reference "10.1.0.1:32000/service-name:stage-66": 10.1.0.1:32000/service-name:stage-66: not found
Error: ErrImagePull Back-off pulling image "10.1.0.1:32000/service-name:stage-66"
Error: ImagePullBackOffHowever, when I manually check the local node using:
ctr i ls -q | grep service-name 10.1.0.1:32000/service-name:stage-66 10.1.0.1:32000/service-name@sha256:27bbdae5d6cc49d18f8ffed260faa21a88e15a2204a27e834a86568710ac8dbdI can see that the image indeed exists locally. This discrepancy between the local existence of the image and Kubernetes’ insistence on pulling it from the remote registry led me to investigate the root cause of this behavior.
Understanding Admission Controls:
In a Kubernetes cluster, admission controls serve as mechanisms to inject additional handling during the creation and management of resources. These controls are categorized as either validating or mutating, akin to “checks” and “changes.”
The focus of this post lies on the AlwaysPullImages admission control, a crucial element in securing containerized workloads.
Enabling/disabling the AlwaysPullImages Admission Control:
To activate the AlwaysPullImages admission control, administrators manipulate kube-apiserver parameters. In Kubernetes versions up to v1.9, the parameter is --admission-control, while v1.10 onwards uses --enable-admission-plugins or --disable-admission-plugins for disabling.. Order matters in the former, but not in the latter. This control is pivotal for enforcing image pull policies consistently across the cluster.
Kubernetes v1.9 and Below:
kube-apiserver --admission-control=...,AlwaysPullImages In versions up to v1.9, use the --admission-control flag with a comma-delimited list of admission controls. Ensure that AlwaysPullImages is included.
Kubernetes v1.10 Onwards:
kube-apiserver --enable-admission-plugins=…,AlwaysPullImageskube-apiserver --disable-admission-plugins=…,AlwaysPullImagesStarting from v1.10, use the --enable-admission-pluginsor--disable-admission-plugins flag. The order of plugins doesn’t matter in this case.
Functionality of AlwaysPullImages: An Illustration:
When the AlwaysPullImages admission control is enabled, it mandates the image pull policy to be set to “Always,” irrespective of how it is specified during resource creation. Consider a pod manifest for Nginx with an explicitly set imagePullPolicy:
apiVersion: v1 kind: Pod metadata: name: test-image-pull-policy spec: containers: - name: nginx image: nginx:1.14 imagePullPolicy: IfNotPresentDespite setting the imagePullPolicy to “IfNotPresent,” the AlwaysPullImages plugin forces it to “Always” during execution. This ensures that cached images cannot be used, contributing to a more secure deployment.
Security Implications in the Absence of AlwaysPullImages:
Without the AlwaysPullImages control, a potential security vulnerability emerges. Consider the following scenario:
- Admin1 Creates Pod1:
- Admin1 creates Pod1 using SuperSecretImage1, specifying ImagePullSecret1 for access to the SecureContainerRegistry.
- Kubernetes Pulls and Caches Image:
- Kubernetes pulls SuperSecretImage1, caches it on the node, and creates Pod1 accordingly.
- Admin2 Attempts Unauthorized Access:
- Admin2, lacking access to ImagePullSecret1, attempts to create Pod2 using SuperSecretImage1 with an
imagePullPolicyof IfNotPresent. Since the image is cached, this operation is successful, allowing unauthorized access.
- Admin2, lacking access to ImagePullSecret1, attempts to create Pod2 using SuperSecretImage1 with an
AlwaysPullImages solves this problem. In its presence, Admin2’s attempt to create Pod2 would be thwarted as the admission control would mutate the imagePullPolicy to “Always,” preventing unauthorized access to cached images.
Beyond Kubernetes: Considerations for Image Security:
While AlwaysPullImages enhances image security within Kubernetes, it’s essential to acknowledge that security is multi-layered. Even with this control enabled, unauthorized users with access to underlying cluster machines could potentially circumvent container registry security by directly saving images. Administrators must remain vigilant about user access to cluster machines to maintain a holistic security posture.
MicroK8s and AlwaysPullImages:
MicroK8s simplifies Kubernetes deployment, but certain configurations, like AlwaysPullImages, can catch users by surprise. The cis-hardening plugin, aimed at enhancing cluster security, enables AlwaysPullImages to ensure the latest images are used consistently.
If you find that this default behavior conflicts with your deployment requirements, fear not. MicroK8s provides a way to customize this setting. By editing the kube-apiserver configuration, you can adjust the admission plugins to include or exclude AlwaysPullImages.
Modifying MicroK8s AlwaysPullImages Behavior:
To disable AlwaysPullImages in MicroK8s, follow these steps:
Open the kube-apiserver configuration file:
sudo nano /var/snap/microk8s/current/args/kube-apiserverFind the lines specifying admission plugins. They might look like:
--enable-admission-plugins=EventRateLimit,NodeRestriction,AlwaysPullImagesModify the line to disable AlwaysPullImages:
--enable-admission-plugins=EventRateLimit,NodeRestriction
--disable-admission-plugins=AlwaysPullImagesSave the changes and restart the MicroK8s daemon:
systemctl restart snap.microk8s.daemon-kubeliteThis adjustment ensures that AlwaysPullImages is no longer enforced by MicroK8s, providing more control over image pull behaviors in your Kubernetes deployment.
Conclusion:
Understanding how Kubernetes handles container images is crucial for deploying applications securely in distributed environments. AlwaysPullImages is a valuable admission control, ensuring a consistent image pull policy and mitigating security vulnerabilities. Keeping security at the forefront is paramount to minimizing risks and ensuring that unauthorized users or processes are unable to access sensitive resources.