Old Pods’ Home: Why is Kubernetes using an old image?

You might have encountered the situation where you’re working on an application, building an image, and trying to deploy it to Kubernetes, but you’re finding that your changes are nowhere to be seen. K8s is running an old version of your app. Why is this happening?

Why does a Pod run an old image?

The usual reason that Kubernetes is running a previous or old version of your application, is that it has already pulled and cached the image on a worker node. To change this behaviour, you need to use the imagePullPolicy setting.

Take a quick look at this birds-eye view of Kubernetes.

Kubernetes architecture diagram

The key components in a Kubernetes cluster

Source: Tutorial Works

When you tell Kubernetes to create a Pod, it figures out which worker node is going to run the Pod, and then the Kubelet pulls the image onto that node.

This is fine if the image doesn’t exist on the node.

But what happens if the image has already been pulled before?

If the desired image already exists on a node, Kubernetes won’t pull the image again.

This might cause you a problem if you’ve pushed a new image to the same tag (e.g. latest).

Kubernetes doesn’t automatically detect that the image has changed, so it will just use the previous cached image. This means you could potentially be running an old version of the image.

So how do you fix it? With ImagePullPolicy

One of the great things about Kubernetes is that almost everything is configured with YAML (or JSON).

Sadly that’s also one of the annoying things about Kubernetes. You kind of need to know which knob to tweak, to fix something. And that usually means you need to know which API object needs to be changed, too. Is it in the Pod? A Service? Or where?

In this case, to solve this problem you need to force Kubernetes to re-pull the image. This means setting a flag on the Pod spec (or anything which creates Pods, like a Deployment) called imagePullPolicy.

Set imagePullPolicy to Always

imagePullPolicy is a field on the containers object.

The default value of imagePullPolicy, which is IfNotPresent, means that Kubernetes will only pull the image if it doesn’t already exist.

But if you change this setting to Always, Kubernetes will always pull the image, irrespective of whether it already exists on the node or not.

This basically ensures you always pull a fresh image. (ahhh fresh)

Here I’m setting imagePullPolicy on a Deployment. I’m using a Deployment because you’re probably using one to create Pods, rather than creating Pod objects directly:

apiVersion: apps/v1
kind: Deployment
  name: nginx
    app: nginx
      app: nginx
        app: nginx
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        - containerPort: 8080

So why isn’t ‘Always’ the default?

If you think about it, setting imagePullPolicy: Always is pretty inefficient. Kubernetes will always pull the image, even if nothing has changed.

This means you’ll be using more bandwidth, your Pods will take a few seconds longer to start, and and if you’re using a public registry like Docker Hub, you might start hitting your rate limits.

But, if you’re knee-deep in development, you’re pushing regularly to the same tag, like develop or latest, you might want your changes to be reflected when the next Pod starts.

So setting imagePullPolicy to Always will tell Kubernetes to do this.

For more info, see the related section in the Kubernetes docs.

Tom Donohue

By Tom Donohue, Editor | Twitter | LinkedIn

Tom is the founder of Tutorial Works. He’s an engineer and open source advocate. He uses the blog as a vehicle for sharing tutorials, writing about technology and talking about himself in the third person. His very first computer was an Acorn Electron.

Join the discussion

Got some thoughts on what you've just read? Want to know what other people think? Or is there anything technically wrong with the article? (We'd love to know so that we can correct it!) Join the conversation and leave a comment.

Comments are moderated.