Kubernetes ImagePullBackOff: Troubleshooting with Examples

Currently we are seeing how to troubleshoot the Kubernetes issue and related topics. Part of that, today let see another important error, you may have experienced sometime this error when you worked on the Kubernetes. This error could frustrate if you are unfamiliar with it, as this doesn’t have single reason to fail, there is varies possible reason beyond this error. Like that we have picked on the error which is nothing but “Kubernetes ImagePullBackOff error”, in this post, we are going to see what could be a reason for this error and how to troubleshoot it.

What Does an ImagePullBackOff Error Mean?

The ImagePull part of the ImagePullBackOff error primarily relates to your Kubernetes container runtime unable to pull the image from a private or public container registry. The Backoff part indicates that Kubernetes will continuously pull the image with an increasing backoff delay. Kubernetes will keep on increasing the delay with each attempt until it reaches the limit of five minutes.

Here are some of the possible causes behind your pod getting stuck in the ImagePullBackOff state:

• Image is not defined properly
• Tag may changed/missing/incorrect
• Image name/path not correct
• Image missing/incorrect
• Image may private, and there is a auth failure
• Check pull secret defined or not
• Secret name Is correct?
• Secrets are correct?
• Network issue
• Container registry Rate Limits

How we can Troubleshoot ImagePullBackOff?

Live Demo:

Let’s check one by one reason listed above, and how to fid the issue.

Image is not defined properly

In most cases, the error could be either from a typo or the reference is not correct, and you’re referring to an image with different path. Let’s try to replicate this by creating a pod with a fake image name.

# kubectl run demoapp --image=foxutech/foxutechimage:latest
deployment.apps/demoapp created

As you can see, the pod is stuck in an ImagePullBackOff because the image doesn’t exist and we cannot pull the image.

# kubectl get pod
NAME                       READY   STATUS             RESTARTS   AGE
demoapp-6fbd57ff7c-78ms8   0/1     ImagePullBackOff   0          28s

For better understand the issue and find more details about this error, use the kubectl describe command, if there is details, you can use kubectl get events command.  With this error we could see what is the main reason the pod is stuck. In the describe, you can check on events section which will contains detailed explanations.

Issue with Tag

There could be cases where the image tag you’re trying to pull is retired, or you entered the wrong tag name. In those cases, your pod will again get stuck in the ImagePullBackOff state, as seen in the following code snippet.

Let’s try to declare some wrong tag and see, how it looks like.

# kubectl run nginx --image=nginx:foxutech
deployment.apps/nginx created

# kubectl get pod

In the following output, the message indicates that tag foxutech doesn’t exist for image nginx.

Failed to pull image "nginx:foxutech": rpc error: code = Unknown desc = Error response from daemon: manifest for nginx:foxutech not found: manifest unknown: manifest unknown

Hence the image pull is unsuccessful.

# kubectl describe pod nginx

Image Missing

Another reason, if we automated any image build and push, we expect it should complete automatically, but some unexpected case, the image push got failed or other issue, but without knowing that, if we try to deploy a pod, it will stick imagepullbackoff error.

You can check this image is in the registry and try again. Let’s try something,

# kubectl run nginx --image=imagename/imagename:v1 
deployment.apps/imagename created

As you can check the imagename is not valid, it will fail for sure.

# kubectl get po
NAME                         READY   STATUS             RESTARTS   AGE 
imagename-8665fffb48-7zz8q   0/1     ImagePullBackOff   0          28s

Private Image Registry and Wrong Credentials or not defined/Provided

In enterprise world we are suggested or proposed to use the internal/private registry to store the image or some vendor may distribute via, their private registry. This could be possible if the team/organization decide not to use public registry like dockerhub due to security or other internal reason, in this case, we should pass the authentication credential, otherwise the imagepull will fail with same error

In the following example, we’re trying to replicate this issue by spinning up a pod that uses an image from a private registry.

# cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: demoapp
  namespace: default
spec:
  containers:
    - name: nginx
      image: foxutech.azurecr.io/nginx
      imagePullPolicy: IfNotPresent
  imagePullSecrets:
    - name: acr-secretes

We have neither added a secret to Kubernetes nor reference of the secret in pod definition. The pod will again get stuck in the ImagePullBackOff status and the message confirms that access is denied to pull an image from the registry:

# kubectl describe pod mypod

To resolve this error, create a secret using the following kubectl command. The following kubectl command creates a secret for a private Docker registry.

# kubectl create secret docker-registry acr-secrets \
    --namespace argocd-motoskia \
    --docker-server=foxutech.azurecr.io \
    --docker-username=foxutech \
    --docker-password=8owM7r+c0KGGxymAJ8291poPm0Wzx3BN

Add your secret to your pod definition, as explained in the following snippet.

Network Issue

There could be a widespread network issue on all the nodes of your Kubernetes cluster, and the container runtime will not be able to pull the image from the container registry. Let’s try to replicate that scenario.

# kubectl run nginx --image=nginx:latest 

pod/nginx created

# kubectl describe pod nginx

Failed to pull image "nginx:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:latest" : failed to resolve reference "docker.io/library/nginx:latest" : failed to do request: Head https://registry-1.docker.io/v2/library/nginx/manifests/latest: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: server misbehaving

Container Registry Rate Limits

Most container registries have implemented some rate limits (i.e., number of images you can pull) to protect their infrastructure. For example, with Docker Hub, anonymous and free Docker Hub users can only request 100 and 200 container image pull requests per six hours. If you exceed your maximum download limit, you’ll be blocked, resulting in ImagePullBackOff error.

To resolve this for Docker Hub, you would need to upgrade to a Pro or Team account. Many other popular container image registries like GCR or ECR also has same limitations.

You can follow us on social media, to get some regular updates

• Facebook: https://www.facebook.com/foxutech/
• Instagram: https://www.instagram.com/foxutech/
• YouTube: Foxutech
• Twitter: https://twitter.com/foxutech
• Medium: FoxuTech – Medium