How to Troubleshoot Kubernetes Service 502 error(Bad Gateway)

What is Kubernetes 502 Bad Gateway?

A 502 Bad Gateway error is an 5xx server error that indicates a server received an invalid response from a proxy or gateway server. In Kubernetes, this can happen when a client attempts to access an application deployed within a pod, but one of the servers responsible for relaying the request the Ingress, the Service, or the pod itself – is not available/accessible or not properly configured.

In our last post we have seen how to troubleshoot 503 service unavailable error, in this will see how to troubleshoot 502 bad gateway messages. Even 502 error also difficult to diagnose and resolve in Kubernetes, because they can involve one or more moving parts in your Kubernetes cluster. In this post, let’s check the steps that can help you debug the issue and identify the most common causes. Please be noted, as like all the issues, it will be depending on the complexity of your setup and the components failing or misconfigured. For best troubleshooting any issue, it is important that, you should have good understanding on your environment.

How to Troubleshoot 502 Bad Gateway in Kubernetes

Consider a scenario in which you map a Service to a container within a pod, and the client is attempting to access an application running on that container. This creates several points of failure:

• The pod
• The container
• Network ports exposed on the container
• The Service
• The Ingress

Here are the basic steps to debugging a 502 error in a Kubernetes pod, which aims to identify a problem in one or more of these components.

1. Check if the Pod and Containers is Running

If the pod or one of its containers did not start, this could result in a 502 error to clients accessing an application running in the pod.

To identify if this is the case, run this command:

# kubectl get pods

• If the entire pod or the required containers are not running – restart the pod or force Kubernetes to reschedule it.
• If they are running – proceed to the next step.

2. Check if Containers are Listening on the Required Port

Identify what address and port the Service is attempting to access. Run the following command and examine the output to see whether the container running the application has an open port and is listening on the expected address:

# kubectl describe pod [pod-name]

• If you see the container is not listening on the port – check the pod specification. If the pod specification does not specify the port in the spec:containers:ports field, add it. If it does specify the port, but it was not opened for some reason, restart the pod.
• If the container is listening on the required post – proceed to the next step.

3. Check if the Service Is Active

If the pod and containers are running and listening on the correct port, the next step is to identify if the Service accessed by the client is active. Note there might be different Services mapped to different containers on the pod.

# kubectl get svc

• If you don’t see the required Service in the list – create it using the kubectl expose command.
• If you see it in the list – proceed to the next step.

4. Check if the Service is Mapped Correctly

A common issue is that the Service is not mapped to the pod exposed by your container. You confirmed previously that a container on your pod exposes a certain port. Now check if the Service maps to this same port.

# kubectl describe svc [service-name]

A healthy service should produce output like this, showing the port it is mapped to;

• If the Service is mapped to a different port – shut down using the command kubectl stop -f [service-name], change the service specification to map to the correct port, and recreate it using kubectl expose.
• If the Service is already mapped to the correct port – proceed to the next step.

5. Check if Ingress Exists

If the Service is healthy, the problem might be in the Ingress. Run this command:

# kubectl get ing

Check the list to see that an Ingress is active specifying the required external address and port.

• If there is no Ingress specifying the address and port – create one. Define an Ingress specification and run it using kubectl apply -f [ingress-config].yaml.
• If the Ingress exists – proceed to the next step.

6. Check Ingress Rules and Backends

An Ingress contains a list of rules matched against incoming HTTP(S) requests. Each path is matched with a backend service, defined with a service.name and either port name or number to access the service.

Run the following command to see the rules and backends defined in the Ingress:

# kubectl describe ingress [ingress-name]

There are two important things to check:

• The host and path accessed by the client is mapped to the IP and address on the Service.
• The backend associated with the Service is healthy.

A backend could be unhealthy because its pod does not pass a health check or fails to return a 200 response, due to an application issue. If the backend is unhealthy, you might see a message like this:

ingress.kubernetes.io/backends:

{"k8s-be-xxxxx--yyZ":"UNHEALTHY","k8s-be-xxxxx--zzY":"HEALTHY","k8s-be-xxxxx--aaB":"HEALTHY"}

• If the Ingress is not correctly mapped or unhealthy – fix the Ingress specification and deploy it using kubectl apply -f [ingress-config].yaml.
• If you still cannot find any issue – the problem is probably with your application. Look for application logs or messages that might indicate an error. Bash into your container and identify if the application is working.

This procedure will help you discover the most basic issues that can result in a 502 bad gateway error.

If you didn’t find the root cause, you will need a more in-depth investigation across multiple components in the Kubernetes deployment, like service, configMap, pod, etc. This required details knowledge about the application. If something related to networking you can check out previous post about networking troubleshooting, this could help.

Also check our troubleshooting series below, could provide more idea how to work on Kubernetes and troubleshooting quicker.

Other troubleshooting articles