What are Kubernetes & Containers Exit Codes – How to Fix it

When we work on the Linux system earlier our carrier, we may experienced some of our shell scripts get failed or some services get fails, best way to troubleshoot those issues are checking the exit code or how the service or script got stopped. This legacy continues, and even in modern container or microservices world, this continues. Lets check more about container exit codes and will understand more about the each exit codes. This should help to troubleshoot the issues better way.

Container Exit Codes

In case if the container got exited/terminated unexpectedly, container engines reports why the container got terminated with exit codes. The exit codes help to understand the issue and fix it. You may experience the same when you worked on Kubernetes, as container failures are one of the most commons caused on pod exceptions.

Related Article: K8s Troubleshooting Archives – FoxuTech

To troubleshoot the issues, we may need to check the logs and should try to get the exit codes, so it will help to fix the issue sooner sometimes.

The most common exit codes used by containers are:

CODE #	Reason	details
Exit Code 0	Purposely stopped	Used by developers/devops/sre to indicate that the container was automatically stopped
Exit Code 1	Application error	Container was stopped due to application error or incorrect reference in the image/configuration specification
Exit Code 125	Container failed to run error	The docker run command did not execute successfully
Exit Code 126	Command invoke error	A command specified in the image specification could not be invoked
Exit Code 127	File or directory not found	File or directory specified was not found
Exit Code 128	Invalid argument used on exit	Exit was triggered with an invalid exit code
Exit Code 134	Abnormal termination (SIGABRT)	The container aborted itself using the abort () function.
Exit Code 137	Immediate termination (SIGKILL)	Container was immediately terminated by the operating system via SIGKILL signal
Exit Code 139	Segmentation fault (SIGSEGV)	Container attempted to access memory that was not assigned to it and was terminated
Exit Code 143	Graceful termination (SIGTERM)	Container received warning that it was about to be terminated, then terminated
Exit Code 255	Exit Status Out of Range	Container exited, returning an exit code outside the acceptable range, meaning the cause of the error is not known

Let’s check in-details each exit code and how to troubleshoot each, to check the failed containers.

Container Lifecycle

To better understand the causes of container failure, let’s discuss the lifecycle of a container first. Taking Docker as an example – at any given time, a Docker container can be in one of several states:

• Created – the Docker container is created but not started yet (this is the status after running docker create, but before actually running the container)
• Up – the Docker container is currently running. This means the operating system process managed by the container is running. This happens when you use the commands docker start or docker run can happen using docker start or docker run.
• Paused – the container process was running, but Docker purposely paused the container. Typically, this happens when you run the Docker pause command.
• Exited – the Docker container has been terminated, usually because the container’s process was killed.

When a container reaches the Exited status, Docker will report an exit code in the logs, to inform you what happened to the container that caused it to shut down.

Understanding Container Exit Codes

Below we cover each of the exit codes in more detail.

Exit Code 0: Purposely Stopped

Exit Code 0 is triggered by developers/SRE/DevOps team when they purposely stop their container after a task completes. Technically, Exit Code 0 means that the foreground process is not attached to a specific container. This one as we are aware, we no need to worry much about it.

Things to while Exit Code 0?

1. Check the container logs to identify what caused the issue and why the container got exit.
2. Review the code and identify why it triggered Exit Code 0, and whether it is functioning correctly.

Exit Code 1: Application Error

Exit Code 1 indicates that the container was stopped due to one of the following:

• An application error – this could be a programming error in code run on the container, such as “divide by zero”, or advanced errors related to the runtime environment.
• An invalid reference – this means the image spec refers to a file that does not exist in the container image.

Things to while Exit Code 1?

1. Check the container log to see if one of the files listed in the image spec could not be found. If this is the issue, correct the image spec to point to the correct path and filename.
2. If you cannot find an incorrect file reference, check the container logs for an application error, and debug the library that caused the error.

Exit Code 125: Container Failed to Run

Exit Code 125 means that the command is used to run the container. For example, docker run was invoked in the system shell but did not execute successfully. Here are common reasons this might happen:

• An undefined flag was used in the command, for example docker run --abcd
• The user-defined in the image spec does not have sufficient permissions on the machine.
• Incompatibility between the container engine and the host operating system or hardware

Things to while Exit Code 125?

1. Check if the command used to run the container uses the proper syntax.
2. Check if the user running the container, or the context in which the command is executed in the image spec, has sufficient permissions to create containers on the host.
3. If your container engine provides other options for running a container, try them. For example, in Docker, try docker start instead of docker run
4. Test if you can run other containers on the host using the same username or context. If not, reinstall the container engine, or resolve the underlying compatibility issue between the container engine and the host setup.

Exit Code 126: Command Invoke Error

Exit Code 126 means that a command used in the container spec could not be invoked. This is often the cause of a missing dependency or an error in a continuous integration script used to run the container.

Things to while Exit Code 126?

1. Check the container logs to see which command could not be invoked.
2. Try running the container spec without the command to ensure you isolate the problem.
3. Troubleshoot the command to ensure you are using the correct syntax and all dependencies are available.
4. Correct the container spec and retry running the container.

Exit Code 127: File or Directory Not Found

Exit Code 127 means a command specified in the container specification refers to a non-existent file or directory.

Things to while Exit Code 127?

Same as Exit Code 126, identify the failing command and make sure you reference a valid filename and file path available within the container image.

Exit Code 128: Invalid Argument Used on Exit

Exit Code 128 means that code within the container triggered an exit command but did not provide a valid exit code. The Linux exit command only allows integers between 0-255, so if the process was exited with, for example, exit code 3.5, the logs will report Exit Code 128.

Things to while Exit Code 128?

1. Check the container logs to identify which library caused the container to exit.
2. Identify where the offending library uses the exit command and correct it to provide a valid exit code.

Exit Code 134: Abnormal Termination (SIGABRT)

Exit Code 134 means that the container abnormally terminated itself, closed the process and flushed open streams. This operation is irreversible, like SIGKILL (see Exit Code 137 below). A process can trigger SIGABRT by doing one of the following:

• Calling the abort() function in the libc library
• Calling the assert() macro, used for debugging. The process is then aborted if the assertion is false.

Things to while Exit Code 134?

1. Check container logs to see which library triggered the SIGABRT signal.
2. Check if process abortion was planned (for example because the library was in debug mode), and if not, troubleshoot the library and modify it to avoid aborting the container.

Exit Code 137: Immediate Termination (SIGKILL)

Exit Code 137 means that the container has received a SIGKILL signal from the host operating system. This signal instructs a process to terminate immediately, with no grace period. This can be either:

• Triggered when a container is killed via the container engine, for example when using the docker kill command.
• Triggered by a Linux user sending a kill -9 command to the process.
• Triggered by Kubernetes after attempting to terminate a container and waiting for a grace period of 30 seconds (by default)
• Triggered automatically by the host, usually due to running out of memory. In this case, the docker inspect command will indicate an OOMKilled error.

Things to while Exit Code 137?

1. Check logs on the host to see what happened prior to the container terminating, and whether it previously received a SIGTERM signal (graceful termination) before receiving SIGKILL.
2. If there was a prior SIGTERM signal, check if your container process handles SIGTERM and can gracefully terminate.
3. If there was no SIGTERM and the container reported an OOMKilled error, troubleshoot memory issues on the host.

Exit Code 139: Segmentation Fault (SIGSEGV)

Exit Code 139 means that the container received a SIGSEGV signal from the operating system. This indicates a segmentation error – a memory violation, caused by a container trying to access a memory location to which it does not have access. There are three common causes of SIGSEGV errors:

1. Coding error—container process did not initialize properly, or it tried to access memory through a pointer to previously freed memory.
2. Incompatibility between binaries and libraries—container process runs a binary file that is not compatible with a shared library, and thus may try to access inappropriate memory addresses.
3. Hardware incompatibility or misconfiguration—if you see multiple segmentation errors across multiple libraries, there may be a problem with memory subsystems on the host or a system configuration issue.

Things to while Exit Code 139?

1. Check if the container process handles SIGSEGV. On both Linux and Windows, you can handle a container’s response to segmentation violations. For example, the container can collect and report a stack trace.
2. If you need to further troubleshoot SIGSEGV, you may need to set the operating system to allow programs to run even after a segmentation fault occurs, to allow for investigation and debugging. Then, try to intentionally cause a segmentation violation and debug the library causing the issue.
3. If you cannot replicate the issue, check memory subsystems on the host and troubleshoot memory configuration.

Exit Code 143: Graceful Termination (SIGTERM)

Exit Code 143 means that the container received a SIGTERM signal from the operating system, which asks the container to gracefully terminate, and the container succeeded in gracefully terminating (otherwise you will see Exit Code 137). This exit code can be:

• Triggered by the container engine stopping the container, for example when using the docker stop or docker-compose down commands.
• Triggered by Kubernetes setting a pod to Terminating status and giving containers a 30 second period to gracefully shut down.

Things to while Exit Code 143?

Check host logs to see the context in which the operating system sent the SIGTERM signal. If you are using Kubernetes, check the kubelet logs to see when the pod was shut down.

In general, Exit Code 143 does not require troubleshooting. It means the container was properly shut down after being instructed to do so by the host.

Exit Code 255: Exit Status Out Of Range

When you see exit code 255, it implies the main entrypoint of a container stopped with that status. It means that the container stopped, but it is not known for what reason.

Things to whileExit Code 255?

1. If the container is running in a virtual machine, first try removing overlay networks configured on the virtual machine and recreating them.
2. If this does not solve the problem, try deleting and recreating the virtual machine, then rerunning the container on it.
3. Failing the above, bash into the container and examine logs or other clues about the entrypoint process and why it is failing.

How to check Container exit code on Kubernetes?

Whenever containers fail within a pod, or Kubernetes instructs a pod to terminate for any reason, containers will shut down with exit codes. Identifying the exit code can help you understand the underlying cause of a pod exception.

You can use the following command to view pod errors: 

# kubectl describe pod [name]

The result will look something like this:

Containers:
kubedns:
Container ID: ...
Image: ...
Image ID: ...
Ports: ...
Host Ports: ...
Args: ...
State: Running
   Started: ...
Last State: Terminated
   Reason: Error
   Exit Code: 255
   Started: ...
   Finished: ...
Ready: True
Restart Count: 1

Use the Exit Code provided by kubectl to troubleshoot the issue:

• If the Exit Code is 0 – the container exited normally, no troubleshooting is required.
• If the Exit Code is between1-128 – the container terminated due to an internal error, such as a missing or invalid command in the image spec..
• If the Exit Code is between 129-255 – the container was stopped as the result of an operating signal, such as SIGKILL or SIGINT
• If the Exit Code was exit(-1) or another value outside the 0-255 range, kubectl translates it to a value within the 0-255 range.

Standard Linux Interruption Signals

Linux programs send an exit code when they are interrupted by external signals. The value of the exit code is the value of the interrupt signal plus 128. For example, the value of SIGKILL is 9, so the program exit code is 9 + 128 = 137. For more standard interrupt signals, see the following table:

Signal	Status Code	Action	Description
SIGHUP	1	Term	Hangup detected on controlling terminal or death of controlling process
SIGINT	2	Term	Interrupt from keyboard
SIGQUIT	3	Core	Quit from keyboard
SIGILL	4	Core	Illegal Instruction
SIGABRT	6	Core	Abort signal from abort(3)
SIGFPE	8	Core	Floating-point exception
SIGKILL	9	Term	Kill signal
SIGSEGV	11	Core	Invalid memory reference
SIGPIPE	13	Term	Broken pipe: write to pipe with no readers; see pipe(7)
SIGALRM	14	Term	Timer signal from alarm(2)
SIGTERM	15	Term	Termination signal
SIGUSR1	30,10,16	Term	User-defined signal 1
SIGUSR2	31,12,17	Term	User-defined signal 2
SIGCHLD	20,17,18	Ign	Child stopped or terminated
SIGCONT	19,18,25	Cont	Continue if stopped
SIGSTOP	17,19,23	Stop	Stop process
SIGTSTP	18,20,24	Stop	Stop typed at terminal
SIGTTIN	21,21,26	Stop	Terminal input for background process
SIGTTOU	22,22,27	Stop	Terminal output for background process

Check our Kubernetes Troubleshooting Videos on FoxuTech – YouTube