One day you can be enjoying plain sailing on the good ship Kubernetes. Then, the next minute, you’re struggling to start your application and you’re getting the error:
So what exactly does
ImagePullBackOff actually mean?
ImagePullBackOff means that a Pod couldn’t start because Kubernetes could not pull a container image. The ‘BackOff’ part indicates that Kubernetes will keep trying to pull the image, with an increasing back-off delay.
So what causes this error, why does it happen, and how do you begin to fix it?
In Kubernetes, the kubelet agent on each node pulls the image
In a Kubernetes cluster, there’s an agent on each node called the kubelet which is responsible for running containers on that node.
It’s time for an analogy! Imagine a bunch of warehouses, and in each warehouse, there’s one forklift, which does the lifting and running of containers.
The warehouses are your nodes. The forklifts are the kubelets.
The kubelet has responsibility for containers running on that node, and for reporting what’s happening back up to the central Kubernetes API.
If a container image doesn’t already exist on a Node, the kubelet will instruct the container runtime to pull it.
If the image can’t be pulled, the kubelet will report ImagePullBackOff
As Joe Beda writes in the book Kubernetes Up And Running:
Kubernetes relies on the fact that images described in a Pod manifest are available across every machine in the cluster.
So every node in the cluster needs to be able to get that image.
If something prevents the container runtime from pulling an image onto the node that Kubernetes has scheduled your Pod onto, the kubelet will report back
ImagePullBackOff, and keep trying.
This means that the error is a signal that something is blocking Kubernetes from being able to pull the image you want onto a specific node.
What are some of the possible causes? Usually:
The image or tag doesn’t exist
You’ve made a typo in the image name or tag
The image registry requires authentication
You’ve exceeded a rate or download limit on the registry
Let’s look at each of these in turn.
You’re referencing an image or tag that doesn’t exist
This is perhaps the cause of this error that I experience most often! I also like to call it “the butterfingers cause”, because it’s usually as a result of me making a typo.
If you’re trying to create a Pod which references an image name or tag that doesn’t exist, you’ll get
You can see this in action quite easily by creating a pod using
kubectl run, and giving it a nonsense image name:
$ kubectl run mytest --image=n0nexistentimage/n0nexist:bla pod/mytest created $ kubectl get pod NAME READY STATUS RESTARTS AGE mytest 0/1 ErrImagePull 0 5s $ kubectl get pod NAME READY STATUS RESTARTS AGE mytest 0/1 ImagePullBackOff 0 45s
If you’re sure that the name is correct, maybe the tag has been retired?
If you’re following an old tutorial or book which references a specific image tag, make sure the tag still exists. If it doesn’t, find out which replacement tag is good for you to use instead.
You’re using a private registry, but you’ve not supplied credentials
Most enterprises that use Kubernetes will use it with a private container image registry.
This is because companies generally don’t want to publish their private, internal apps to Docker Hub.
So if you need to pull an image from a private image registry, you need to make sure that you provide Kubernetes with the credentials it will need to pull the image.
You can do this by creating a Secret.
This is also true if you’re pulling from a publicly-accessible, but password-protected registry, like the Red Hat registry, or even private images on Google Container Registry, GitHub Container Registry, etc.
The registry is blocking you, or there’s some other registry issue
This is starting to become a more common reason since Docker introduced rate limits on Docker Hub.
You might be trying to pull an image from Docker Hub without realising it. If your
image field on your Pod just references a name, like
nginx, it’s probably trying to download this image from Docker Hub.
This pod spec doesn’t reference a registry, so it’ll probably look in Docker Hub first:
apiVersion: v1 kind: Pod metadata: name: mytest namespace: default spec: containers: - image: nginx name: nginx
Once you hit your maximum download limit on Docker Hub, you’ll be blocked and this might cause your
You’ll either need to fork out for an account or find another place to get your image from. See the tiers explained here:
This error can also happen if your registry requires SSL/TLS authentication, but you don’t trust its certificate. Make sure you follow the instructions to set up TLS authentication.
How to troubleshoot it
We want to make sure that we can pull the image.
from our machine
from a node in the cluster
if you’re not using a registry, does the image already exist on the node?
are any other pods currently running on that node?
kubectl describe to get the root cause
The best place to start with troubleshooting this issue is with
kubectl describe. This will show you the full error log of the Pod, so you can see what’s causing the issue.
I’ve created a Pod which references an image that doesn’t exist. Here’s the Events output from
kubectl describe <pod name>:
The error reads:
Error response from daemon: pull access denied for n0nexistentimage/n0nexist, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied
So that’s a clear message that I’ve got the name wrong!
If it’s a private registry, check your Secret and Pod
If you’re pulling an image from a private registry, make sure that you’ve created a Secret containing the credentials you need to acceess.
Also make sure that you’ve added the Secret in the appropriate namespace!
How to create a Secret for a private registry
To create a Secret to pull images from a private registry, use
kubectl create secret.
This example creates a secret to be able to pull images from Docker Hub (
kubectl create secret docker-registry docker-hub \ --docker-username=YOURUSERNAME \ --docker-password=YOURPASSWORD \ --docker-server=docker.io
You’ll also need to set the
imagePullSecrets field on your Pod. This field tells Kubernetes which Secret it should use, when authenticating to the registry. Like this:
apiVersion: v1 kind: Pod metadata: name: myapp spec: containers: - name: myapp image: registry.example.com/myteam/myapp:latest imagePullPolicy: Always imagePullSecrets: - name: the-secret
Increase log levels on the kubelet
If you have access to the nodes themselves, you can get more information by turning up the log levels on the kubelet.
You can do this by setting the
--v parameter (this changes log level verbosity).
Check how widespread the problem is
Are there other Pods already running on that node? If the node is already running other Pods fine, then are they pulling from the same registry? What is different about this Pod?
Can you pull the image locally? Can you pull the image from your own workstation? Try
podman pulland see if you can fetch the image. This might give you a clue as to why it’s failing.
Can the node in the cluster pull the image? Try jumping on to the node itself, via SSH. Can you run a
docker pulland get the image directly?
The info you get from this stages will help you figure out where exactly the problem is!
So, to recap:
We learned that an image must be accessible from every node in a Kubernetes cluster
The kubelet, the agent that runs on each node in the cluster, calls the container runtime and instructs it to pull the image. If it fails, it reports back this error.
It can be caused by typos, wrong tag names, missing registry secrets.
If no images can be pulled, there might be a problem with your network setup.
You might want to read more about the kubelet, or using images with Kubernetes.
Got any thoughts on what you've just read? Anything wrong, or no longer correct? Sign in with your GitHub account to leave a comment.
(All comments get added as Issues in our GitHub repo here, using the comments tool Utterances)