Kubernetes ImagePullBackOff error: what you need to know

One day you can be enjoying plain sailing on the good ship Kubernetes. Then, the next minute, you’re struggling to start your application and you’re getting the error:

ImagePullBackOff

Oh dear.

So what exactly does ImagePullBackOff mean?

The status ImagePullBackOff means that a Pod couldn’t start, because Kubernetes couldn’t pull a container image. The ‘BackOff’ part means that Kubernetes will keep trying to pull the image, with an increasing delay (‘back-off’).

So what causes this error, why does it happen, and how do you begin to fix it?

In Kubernetes, the kubelet agent on each node pulls the image

In a Kubernetes cluster, there’s an agent on each node called the kubelet which is responsible for running containers on that node.

It’s time for an analogy! Imagine a bunch of warehouses, and in each warehouse, there’s one forklift, which does the lifting and running of containers.

The warehouses are your nodes. The forklifts are the kubelets.

The kubelet has responsibility for containers running on that node, and for reporting what’s happening back up to the central Kubernetes API.

If a container image doesn’t already exist on a Node, the kubelet will instruct the container runtime to pull it.

If the image can’t be pulled, the kubelet will report ImagePullBackOff

As Joe Beda writes in the book Kubernetes Up And Running:

Kubernetes relies on the fact that images described in a Pod manifest are available across every machine in the cluster.

So every node in the cluster needs to be able to get that image.

If something prevents the container runtime from pulling an image onto the node that Kubernetes has scheduled your Pod onto, the kubelet will report back ErrImagePull, then ImagePullBackOff, and keep trying.

So the ImagePullBackOff error is a sign that something is blocking Kubernetes from being able to pull the image you want onto a specific node.

Now let’s check out the root causes of this ImagePullBackOff error.

Causes

So let’s look at some of the possible causes for the error. They are usually:

  • The image or tag doesn’t exist

  • You’ve made a typo in the image name or tag

  • The image registry requires authentication

  • You’ve exceeded a rate or download limit on the registry

Now let’s look at each of the causes in turn.

You’re referencing an image or tag that doesn’t exist

This is perhaps the cause of this error that I experience most often! I also like to call it “the butterfingers cause”, because it’s usually the result of me making a typo.

If you’re trying to create a Pod which references an image name or tag that doesn’t exist, you’ll get ImagePullBackOff.

You can see this in action quite easily by creating a pod using kubectl run, and giving it an image name that’s complete nonsense:

$ kubectl run mytest --image=n0nexistentimage/n0nexist:bla
pod/mytest created
$ kubectl get pod
NAME     READY   STATUS         RESTARTS   AGE
mytest   0/1     ErrImagePull   0          5s
$ kubectl get pod
NAME     READY   STATUS             RESTARTS   AGE
mytest   0/1     ImagePullBackOff   0          45s

But what if you think the image should exist?

If you’re 100% sure that the name is correct, maybe the tag has been retired?

If you’re following an old tutorial or book which references a specific image tag, check that the tag still exists. If it doesn’t, find out which replacement tag is good for you to use instead.

You’re using a private registry, but you’ve not supplied credentials

Most enterprises that use Kubernetes tend to use it with a private container image registry.

This is because companies generally don’t want to publish their private, internal apps to Docker Hub.

So if you need to pull an image from a private image registry, you need to make sure that you provide Kubernetes with the credentials it will need to pull the image.

You can do this by creating a Secret.

This is also true if you’re pulling from a publicly-accessible, but password-protected registry, like the Red Hat registry, or even private images on Google Container Registry, GitHub Container Registry, etc.

The registry is blocking you, or there’s some other registry issue

This is a very common reason for ImagePullBackOff since Docker introduced rate limits on Docker Hub.

You might be trying to pull an image from Docker Hub without realising it. If your image field on your Pod just references a name, like nginx, it’s probably trying to download this image from Docker Hub.

This pod spec doesn’t reference a registry, so it’ll probably look in Docker Hub first:

apiVersion: v1
kind: Pod
metadata:
  name: mytest
  namespace: default
spec:
  containers:
  - image: nginx
    name: nginx

Once you hit your maximum download limit on Docker Hub, you’ll be blocked and this might cause your ImagePullBackOff error.

You’ll either need to sign in with an account, or find another place to get your image from. See the tiers explained here:

Docker Hub pricing tiers showing free account limit

You’re subject to rate limits on Docker Hub’s free tier

This error can also happen if your registry requires SSL/TLS authentication, but you don’t trust its certificate. Make sure you follow the instructions to set up TLS authentication.

How to troubleshoot it

We want to make sure that we can pull the image. You can try pulling the image from a few different places:

  • from our machine

  • from a node in the cluster

  • if you’re not using a registry, does the image already exist on the node?

  • are any other pods currently running on that node?

Use kubectl describe to get the root cause

The best place to start with troubleshooting this issue is with kubectl describe. This will show you the full error log of the Pod, so you can see what’s causing the issue.

I’ve created a Pod which references an image that doesn’t exist. Here’s the Events output from kubectl describe <pod name>:

kubectl describe command showing 'repository does not exist' error

kubectl describe will show you events and errors

The error reads:

Error response from daemon: pull access denied for n0nexistentimage/n0nexist, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied

So that’s a clear message that I’ve got the name wrong!

If it’s a private registry, check your Secret and Pod

If you’re pulling an image from a private registry, make sure that you’ve created a Secret containing the credentials you need to acceess.

Also make sure that you’ve added the Secret in the appropriate namespace!

How to create a Secret for a private registry

To create a Secret to pull images from a private registry, use kubectl create secret.

This example creates a secret to be able to pull images from Docker Hub (docker.io):

kubectl create secret docker-registry docker-hub \ 
    --docker-username=YOURUSERNAME \ 
    --docker-password=YOURPASSWORD \ 
    --docker-server=docker.io

You’ll also need to set the imagePullSecrets field on your Pod. This field tells Kubernetes which Secret it should use, when authenticating to the registry. Like this:

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
    - name: myapp
      image: registry.example.com/myteam/myapp:latest
      imagePullPolicy: Always
  imagePullSecrets:
  - name: the-secret

Increase log levels on the kubelet

If you have access to the nodes themselves, you can get more information by turning up the log levels on the kubelet.

You can do this by setting the --v parameter (this changes log level verbosity).

Check how widespread the problem is

  • Are there other Pods already running on that node? If the node is already running other Pods fine, then are they pulling from the same registry? What is different about this Pod?

  • Can you pull the image locally? Can you pull the image from your own workstation? Try docker pull or podman pull and see if you can fetch the image. This might give you a clue as to why it’s failing.

  • Can the node in the cluster pull the image? Try jumping on to the node itself, via SSH. Can you run a docker pull and get the image directly?

The info you get from this stages will help you figure out where exactly the problem is!

Recapping it

So, to recap:

  • We learned that an image must be accessible from every node in a Kubernetes cluster

  • The kubelet, the agent that runs on each node in the cluster, calls the container runtime and instructs it to pull the image. If it fails, it reports back this error.

  • ImagePullBackOff` can be caused by typos, wrong tag names, missing registry secrets.

  • If no images can be pulled, there might be a problem with your network setup.

Read more about the kubelet or using images with Kubernetes.

Want to be more confident with Kubernetes, and know how to troubleshoot errors? Check out Linux Foundation’s Kubernetes administration course. Or, if you’re an app developer, check out Kubernetes for Developers.

Tom Donohue

By Tom Donohue, Editor | Twitter | LinkedIn

Tom is the founder of Tutorial Works. He’s an engineer and open source advocate. He uses the blog as a vehicle for sharing tutorials, writing about technology and talking about himself in the third person. His very first computer was an Acorn Electron.

Join the discussion

Got some thoughts on what you've just read? Want to know what other people think? Or is there anything technically wrong with the article? (We'd love to know so that we can correct it!) Join the conversation and leave a comment.

Comments are moderated.