How to Troubleshoot Kubernetes Pod in Pending State

By
motoskia
-
0
2993
How to Troubleshoot Kubernetes Pod in Pending State

Part of our Kubernetes troubleshooting series, in this article will see how to troubleshoot, pod on pending state and reasons.

Pod in Pending Status

In K8s, the “Pending” status for a Pod means that the Pod has been accepted by the K8s system, but it hasn’t been scheduled for running on a node yet, or it has been scheduled but the necessary preparations are not complete.

Pod scheduling flow

For example:

# kubectl get pods --field-selector=status.phase=Pending

NAME                         READY   STATUS    RESTARTS   AGE
testing-pod                  0/1     Pending   0          5m11s

Common Reasons

Pod pending state reasons
  • Insufficient resources: If your cluster doesn’t have enough resources (CPU, memory, etc.) to meet the Pod’s requirements, the Pod will remain in the Pending state.
  • Taints and Tolerations: If a node is “tainted”, it’s marked to repel certain Pods. A Pod with a matching “toleration” can be scheduled on such a node. If no node with a suitable toleration for the Pod’s taints exists, the Pod will stay in Pending status.
  • Node Selector/Affinity rules: If you’ve set node selector or node/pod affinity/anti-affinity rules that the scheduler can’t satisfy, the Pod will remain in Pending status.
  • Persistent Volume Claims (PVCs): If a Pod is dependent on a PVC and that PVC is not ready or cannot be allocated, the Pod will stay in Pending state.
  • Image Pull Issues: If there are errors pulling the Docker image (for example, due to a typo in the image name, problems with the image registry, or ImagePullBackOff errors), the Pod will remain in Pending status.
  • Init Containers: If the Pod has Init Containers and they haven’t completed successfully, the Pod will remain in Pending status.
  • Dependency: The kubelet tries to verify all dependencies with other Kubernetes components before the pod starts. The pod will remain in a pending state until the dependencies are met.

How to Troubleshoot

Pod pending to running

Here are some steps you can take to troubleshoot Pods stuck in the Pending state:

  • Describe the Pod: The first step in troubleshooting a Pod stuck in Pending state is to get more information about the Pod. Run following commend to check it.
# kubectl describe pod <pod-name> 

This command will provide a lot of useful information, such as the current state of the Pod, recent events, and so on.

Check for Events: At the bottom of the kubectl describe pod <pod-name> output, you’ll see an “Events” section. This section can often provide clues about why the Pod is not starting. Look for events like “FailedScheduling”, which can indicate issues like insufficient resources on your nodes or or some PaaS cluster, wont in-cooperate events with describe command, you may need to run,

# kubectl get events | grep <pod-name>

  • Check the Scheduler: The K8s scheduler is responsible for assigning Pods to nodes. If there’s a problem with the scheduler, it could prevent Pods from being scheduled. You can check the scheduler’s logs with the 
# kubectl logs <scheduler-pod-name> -n kube-system 
# kubectl -n kube-system logs $(kubectl -n kube-system get pods|grep scheduler|awk '{print $1}')
  • Check Node Resources: If your nodes don’t have enough CPU or memory resources to meet the Pod’s requirements, the Pod will not be scheduled. You can check your nodes’ resources with the 
# kubectl describe node <node-name> 

Or Filter only the CPU details of all the node available on the cluster using

# kubectl describe nodes | grep 'Name:|Allocated' -A 5 | grep 'Name|cpu'

Look for the “Allocatable” and “Capacity” sections to see how much CPU and memory is available on the node.

  • Check the Image: If the Pod’s Docker image cannot be pulled, the Pod will not start. This could be due to a typo in the image name, a problem with the image registry, or a problem with the image itself. You can check the image pull status in the “Events” section of the kubectl describe pod <pod-name> output or check on kubectl get events.
  • Check for Taints and Tolerations: K8s nodes can be “tainted” to repel certain Pods, and Pods can have “tolerations” to be scheduled on tainted nodes. If your Pod doesn’t have the necessary tolerations to be scheduled on your tainted nodes, it will not be scheduled. You can check for taints on your nodes with the kubectl describe node <node-name> command, and you can check for tolerations in your Pod specification.
  • Check for Affinity and Anti-Affinity Rules: These are rules that influence where Pods get scheduled based on labels. If you’ve defined any of these rules, they could be preventing your Pod from being scheduled.
  • Check Persistent Volume Claims (PVCs): If the Pod is waiting on a PVC that’s not ready or cannot be fulfilled, the Pod will be in Pending state.
  • Check Dependency: Find more details with kubectl describe po <pod-name> in message column. Sometime, there being no configuration map or secret, or the name given is incorrect. Because it hasn’t been released by another node yet, a volume cannot be mounted in the node. This occurs particularly when updating a statefulset because the volume installed must match the old pod. By trying to fix these issues, we can recover the pod stuck in a pending state due to dependency issues.
Pod Dependency

watch our troubleshooting series,

Google search engine