If you are working on kubernetes or heard about that with someone working on kubernetes, says Kubernetes is complex and hard to manage or troubleshoot. Same time you may see in kubernetes cluster either one, goes wrong sometimes. One of the key reasons for this could be it contains many components and variables, that may make it complicate to understand what the issue is.
Always remember the fundamental rules of troubleshooting in any scenarios.
- Use historical data, such as logs, and observation to identify the root cause of a problem.
- One of the key rules for the troubleshooting When you are ascertaining the cause or trying a fix, change only one variable at a time.
- Before trusting a fix, confirm that it works under different conditions.
As like basic of the troubleshooting, you should understand the component or architecture of the tool, like that in Kubernetes understand the components and administrative commands. This is critical to execute the first rule successfully and to debug Kubernetes application and service deployments.
Check the Events
Kubectl is the primary administrative tool for Kubernetes clusters and includes other commands. The command “kubectl get” reveals basic information about a particular resource. For example, “kubectl get pods” lists the available pods and their status, while “kubectl get services” lists the applications running as a service on a set of pods.
However, a more detailed option exists for troubleshooting: “kubectl describes pods”, also used as “kubectl describes pod (TYPE/NAME)”, provides detail about container or pod labels, resource requirements, state, readiness, restart count and events.
For example, an admin finds that “kubectl get pods” for a Nginx application shows a particular pod isn’t running. Using “kubectl describe pod nginx-wcre” shows the admin that the pod couldn’t be scheduled, due to inadequate available resources. The following is a subset of returned output that demonstrates the problem:
Name: nginx-wcre
Namespace: default
Node: /
Labels: app=nginx,pod-template-hash=fut
Status: Pending
...
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
2m 56s 10 {default-scheduler} Warning
FailedScheduling pod (nginx-deployment-wcre-as2aq) failed to fit in any node
fit failure on node (kubernetes-node-xxx): Node didn't have enough resource: CPU, requested: 250, used: 800, capacity: 1000
In this case resources are only one possible reason a pod might not work. Others include one or more downed nodes or connectivity problems with the internal or external load balancers.
Check the service status
In Kubernetes, externally accessible applications are exposed as a service to define a logical set of pods and access controls. Services are specified by a name and the targetPort attribute. The Kubernetes controller assigns the service a cluster IP address accessible by Ingress proxies. Administrators can encounter a problem when deploying a Kubernetes application by creating the pod but not the service. Verify the service status with the following commands:
# wget -O- hostnames # kubectl get svc hostnames
Either wget or kubectl could return an error like the following:
Resolving hostnames (hostnames)... failed: Name or service not known.
Or they could return this:
No resources found.
Error from server (NotFound): services "hostnames" not found
These responses indicate that Kubernetes has not created the service. To fix this problem, create and validate the service with the following commands. In this example, the service port number is 8080.
# kubectl expose deployment hostnames --port=80 --target-port=8080 > service/hostnames exposed # kubectl get svc hostname
Application deployments in Kubernetes
The way in which Kubernetes exposes applications as outside services can confound troubleshooting. There are several abstraction layers separated by network load balancers, such as the following:
- Nodes host the containers used within a pod. As detailed by the Kubernetes documentation, containers within a pod communicate via the node’s loopback interface — 127.0.0.1 — while pods in a cluster communicate via Kubernetes’ internal cluster networking.
- Services are network resources that expose an application running in pods to users and other services outside the cluster.
- Ingress provides application layer mapping via HTTPS to services for external users. Ingress controllers segregate traffic to different services using name-based hosting. Open-source Kubernetes supports AWS Application Load Balancer, the Ingress controller for Google Cloud and Nginx Ingress controllers. Third parties support the Envoy proxy and Istio service mesh.
Network and service configurations are often why a Kubernetes application is unreachable. This could be due to the service interface not pointing to a pod and port. To create a service, use the kubectl expose command. For example, to create and check a service configuration for a Nginx application, use the following:
# kubectl expose deployment/nginx-app # kubectl describe svc nginx-app
The output of the describe command in this example is the following:
Name: nginx-app
Namespace: default
Labels: run=nginx-app
Annotations: <none>
Selector: run=nginx-app
Type: ClusterIP
IP: 10.1.0.218
Port: <unset> 80/TCP
Endpoints: 10.2.2.5:80,10.2.3.4:80
Session Affinity: None
Events: <none>A common error is not matching a service’s targetPort attribute with the port that the container uses in the pod, specified as containerPort. A similar problem occurs if the service port isn’t configured properly in the Ingress controller.
Review event logs
Kubernetes components, such as kube-apiserver, kube-scheduler, kube-proxy and kubelet, as well as the containers, nodes and applications that run on Kubernetes, each generate logs. Log information should be archived for review for use in troubleshooting.
There is various log monitoring software’s in market for kubernetes, pick on the best and use it. As many tools collect information on and review Kubernetes events, such as kubewatch, Eventrouter and Event Exporter. These Kubernetes watchers can work in concert with log analysis software, like Grafana or Kibana.
Sloop monitors Kubernetes, recording histories of events and resource state changes and providing visualizations to aid in debugging past events.
Key features:
- Allows you to find and inspect resources that no longer exist (example: discover what host the pod from the previous deployment was using).
- Provides timeline displays that show rollouts of related resources in updates to Deployments, ReplicaSets, and StatefulSets.
- Helps debug transient and intermittent errors.
- Allows you to see changes over time in a Kubernetes application.
- Is a self-contained service with no dependencies on distributed storage.
Debugging Kubernetes deployments — with their many abstraction layers and configuration parameters — requires a considered approach. Start with the basics, and work up the software stack:
- Ensure that all pods are running. Check if any fail to deploy due to resource shortage.
- Analyze the internal traffic flow to verify that service requests make it to the correct pod.
- Test external traffic flow through a network load balancer and Ingress proxy to validate the network configuration.
- Review the event history with a visualization tool, like Sloop or Grafana, to spot anomalies and correct unexpected events. These steps help administrators to ascertain the root cause of a service failure.
