Old Pods’ Home: Why is Kubernetes using an old image?

You might have encountered the situation where you’re working on an application, building an image, and trying to deploy it to Kubernetes, but you’re finding that your changes are nowhere to be seen. K8s is running an old version of your app. Why is this happening?

Why does a Pod run an old image?

The usual reason that Kubernetes is running a previous or old version of your application, is that it has already pulled and cached the image on a worker node. To change this behaviour, you need to use the imagePullPolicy setting.

Take a quick look at this birds-eye view of Kubernetes.

When you tell Kubernetes to create a Pod, it figures out which worker node is going to run the Pod, and then the Kubelet pulls the image onto that node.

This is fine if the image doesn’t exist on the node.

But what happens if the image has already been pulled before?

If the desired image already exists on a node, Kubernetes won’t pull the image again.

This might cause you a problem if you’ve pushed a new image to the same tag (e.g. latest).

Kubernetes doesn’t automatically detect that the image has changed, so it will just use the previous cached image. This means you could potentially be running an old version of the image.

So how do you fix it? With ImagePullPolicy

In this case, to solve this problem you need to force Kubernetes to re-pull the image. This means setting a flag on the Pod spec (or anything which creates Pods, like a Deployment) called imagePullPolicy.

Set imagePullPolicy to Always

imagePullPolicy is a field on the containers object.

The default value of imagePullPolicy, which is IfNotPresent, means that Kubernetes will only pull the image if it doesn’t already exist.

But if you change this setting to Always, Kubernetes will always pull the image, irrespective of whether it already exists on the node or not.

This basically ensures you always pull a fresh image. (ahhh fresh)

Here I’m setting imagePullPolicy on a Deployment. I’m using a Deployment because you’re probably using one to create Pods, rather than creating Pod objects directly:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 8080

So why isn’t ‘Always’ the default?

If you think about it, setting imagePullPolicy: Always is pretty inefficient. Kubernetes will always pull the image, even if nothing has changed.

This means you’ll be using more bandwidth, your Pods will take a few seconds longer to start, and and if you’re using a public registry like Docker Hub, you might start hitting your rate limits.

But, if you’re knee-deep in development, you’re pushing regularly to the same tag, like develop or latest, you might want your changes to be reflected when the next Pod starts.

So setting imagePullPolicy to Always will tell Kubernetes to do this.

For more info, see the related section in the Kubernetes docs.