You might have encountered the situation where you’re working on an application, building an image, and trying to deploy it to Kubernetes, but you’re finding that your changes are nowhere to be seen. K8s is running an old version of your app. Why is this happening?
Why does a Pod run an old image?
The usual reason that Kubernetes is running a previous or old version of your application, is that it has already pulled and cached the image on a worker node. To change this behaviour, you need to use the
Take a quick look at this birds-eye view of Kubernetes.
When you tell Kubernetes to create a Pod, it figures out which worker node is going to run the Pod, and then the Kubelet pulls the image onto that node.
This is fine if the image doesn’t exist on the node.
But what happens if the image has already been pulled before?
If the desired image already exists on a node, Kubernetes won’t pull the image again.
This might cause you a problem if you’ve pushed a new image to the same tag (e.g.
Kubernetes doesn’t automatically detect that the image has changed, so it will just use the previous cached image. This means you could potentially be running an old version of the image.
So how do you fix it? With ImagePullPolicy
One of the great things about Kubernetes is that almost everything is configured with YAML (or JSON).
Sadly that’s also one of the annoying things about Kubernetes. You kind of need to know which knob to tweak, to fix something. And that usually means you need to know which API object needs to be changed, too. Is it in the Pod? A Service? Or where?
In this case, to solve this problem you need to force Kubernetes to re-pull the image. This means setting a flag on the Pod spec (or anything which creates Pods, like a Deployment) called
Set imagePullPolicy to Always
imagePullPolicy is a field on the
The default value of
imagePullPolicy, which is
IfNotPresent, means that Kubernetes will only pull the image if it doesn’t already exist.
But if you change this setting to
Always, Kubernetes will always pull the image, irrespective of whether it already exists on the node or not.
This basically ensures you always pull a fresh image. (ahhh fresh)
Here I’m setting
imagePullPolicy on a Deployment. I’m using a Deployment because you’re probably using one to create Pods, rather than creating Pod objects directly:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx imagePullPolicy: Always name: nginx ports: - containerPort: 8080
So why isn’t ‘Always’ the default?
If you think about it, setting
imagePullPolicy: Always is pretty inefficient. Kubernetes will always pull the image, even if nothing has changed.
This means you’ll be using more bandwidth, your Pods will take a few seconds longer to start, and and if you’re using a public registry like Docker Hub, you might start hitting your rate limits.
But, if you’re knee-deep in development, you’re pushing regularly to the same tag, like
latest, you might want your changes to be reflected when the next Pod starts.
Always will tell Kubernetes to do this.
For more info, see the related section in the Kubernetes docs.
Got any thoughts on what you've just read? Anything wrong, or no longer correct? Sign in with your GitHub account to leave a comment.
(All comments get added as Issues in our GitHub repo here, using the comments tool Utterances)