Kubernetes ImagePullBackOff error: what you need to know

One day you can be enjoying plain sailing on the good ship Kubernetes.

Then, the next minute, you’re struggling to start your application and you’re getting the error:

ImagePullBackOff

Oh dear.

So what exactly does ImagePullBackOff actually mean?

The status ImagePullBackOff means that a Pod couldn’t start because Kubernetes could not pull a container image. The ‘BackOff’ part indicates that Kubernetes will keep trying to pull the image, with an increasing back-off delay.

So what causes this error, why does it happen, and how do you begin to fix it?

In Kubernetes, the kubelet agent on each node pulls the image

In a Kubernetes cluster, there’s an agent on each node called the kubelet which is responsible for running containers on that node.

The kubelet has responsibility for containers running on that node, and for reporting what’s happening back up to the central Kubernetes API.

If a container image doesn’t already exist on a Node, the kubelet will instruct the container runtime to pull it.

If the image can’t be pulled, the kubelet will report ImagePullBackOff

As Joe Beda writes in the book Kubernetes Up And Running:

Kubernetes relies on the fact that images described in a Pod manifest are available across every machine in the cluster.

So every node in the cluster needs to be able to get that image.

If something prevents the container runtime from pulling an image onto the node that Kubernetes has scheduled your Pod onto, the kubelet will report back ErrImagePull, then ImagePullBackOff, and keep trying.

This means that the error is a signal that something is blocking Kubernetes from being able to pull the image you want onto a specific node.

Causes

What are some of the possible causes? Usually:

  • The image or tag doesn’t exist

  • You’ve made a typo in the image name or tag

  • The image registry requires authentication

  • You’ve exceeded a rate or download limit on the registry

Let’s look at each of these in turn.

You’re referencing an image or tag that doesn’t exist

This is perhaps the cause of this error that I experience most often! I also like to call it “the butterfingers cause”, because it’s usually as a result of me making a typo.

If you’re trying to create a Pod which references an image name or tag that doesn’t exist, you’ll get ImagePullBackOff.

You can see this in action quite easily by creating a pod using kubectl run, and giving it a nonsense image name:

$ kubectl run mytest --image=n0nexistentimage/n0nexist:bla
pod/mytest created
$ kubectl get pod
NAME     READY   STATUS         RESTARTS   AGE
mytest   0/1     ErrImagePull   0          5s
$ kubectl get pod
NAME     READY   STATUS             RESTARTS   AGE
mytest   0/1     ImagePullBackOff   0          45s

If you’re sure that the name is correct, maybe the tag has been retired?

If you’re following an old tutorial or book which references a specific image tag, make sure the tag still exists. If it doesn’t, find out which replacement tag is good for you to use instead.

You’re using a private registry, but you’ve not supplied credentials

Most enterprises that use Kubernetes will use it with a private container image registry.

This is because companies generally don’t want to publish their private, internal apps to Docker Hub.

So if you need to pull an image from a private image registry, you need to make sure that you provide Kubernetes with the credentials it will need to pull the image.

You can do this by creating a Secret.

This is also true if you’re pulling from a publicly-accessible, but password-protected registry, like the Red Hat registry, or even private images on Google Container Registry, GitHub Container Registry, etc.

The registry is blocking you, or there’s some other registry issue

This is starting to become a more common reason since Docker introduced rate limits on Docker Hub.

Once you hit your maximum download limit on Docker Hub, you’ll be blocked and this might cause your ImagePullBackOff error.

You’ll either need to fork out for an account or find another place to get your image from. See the tiers explained here:

Docker Hub pricing tiers showing free account limit
You’re subject to rate limits on Docker Hub’s free tier

This error can also happen if your registry requires SSL/TLS authentication, but you don’t trust its certificate. Make sure you follow the instructions to set up TLS authentication.

How to troubleshoot it

We want to make sure that we can pull the image.

  • from our machine

  • from a node in the cluster

  • if you’re not using a registry, does the image already exist on the node?

  • are any other pods currently running on that node?

Use kubectl describe to get the root cause

The best place to start with troubleshooting this issue is with kubectl describe. This will show you the full error log of the Pod, so you can see what’s causing the issue.

I’ve created a Pod which references an image that doesn’t exist. Here’s the Events output from kubectl describe <pod name>:

kubectl describe command showing 'repository does not exist' error
kubectl describe will show you events and errors

The error reads:

Error response from daemon: pull access denied for n0nexistentimage/n0nexist, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied

So that’s a clear message that I’ve got the name wrong!

If it’s a private registry, check your Secret and Pod

If you’re pulling an image from a private registry, make sure that you’ve created a Secret containing the credentials you need to acceess.

Also make sure that you’ve added the Secret in the appropriate namespace!

How to create a Secret for a private registry

To create a Secret to pull images from a private registry, use kubectl create secret.

This example creates a secret to be able to pull images from Docker Hub (docker.io):

kubectl create secret docker-registry docker-hub \ 
    --docker-username=YOURUSERNAME \ 
    --docker-password=YOURPASSWORD \ 
    --docker-server=docker.io

You’ll also need to set the imagePullSecrets field on your Pod. This field tells Kubernetes which Secret it should use, when authenticating to the registry. Like this:

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
    - name: myapp
      image: registry.example.com/myteam/myapp:latest
      imagePullPolicy: Always
  imagePullSecrets:
  - name: the-secret

Increase log levels on the kubelet

If you have access to the nodes themselves, you can get more information by turning up the log levels on the kubelet.

You can do this by setting the --v parameter (this changes log level verbosity).

Check how widespread the problem is

  • Are there other Pods already running on that node? If the node is already running other Pods fine, then are they pulling from the same registry? What is different about this Pod?

  • Can you pull the image locally? Can you pull the image from your own workstation? Try docker pull or podman pull and see if you can fetch the image. This might give you a clue as to why it’s failing.

  • Can the node in the cluster pull the image? Try jumping on to the node itself, via SSH. Can you run a docker pull and get the image directly?

The info you get from this stages will help you figure out where exactly the problem is!

Recapping it

So, to recap:

  • We learned that an image must be accessible from every node in a Kubernetes cluster

  • The kubelet, the agent that runs on each node in the cluster, calls the container runtime and instructs it to pull the image. If it fails, it reports back this error.

  • It can be caused by typos, wrong tag names, missing registry secrets.

  • If no images can be pulled, there might be a problem with your network setup.

What’s next?

You might want to read more about the kubelet, or using images with Kubernetes.