We have seen recently about evicted state, one of the key reason for it, resource. In this post will see about the Resource limit and requests configuration in Kubernetes is the most neglected configuration. But if you want to manage your Kubernetes cluster in production or high load environment, you must spend time in understanding and observing the memory and CPU requirements of your applications running inside containers in different pods. This always a basic expectation of troubleshooting.
Before we see about resource limit on pods, we should understand about the Kubernetes nodes in a Kubernetes cluster, nodes contain fixed number of CPU core, a fixed amount of memory, some OS installed (mostly Linux), Kernel version, architecture, etc. Pods are nothing but as like application running on the host/node it will be running on the node.
Understanding Kubernetes Cluster
Also, we should understand few more on Kubernetes here high-level pictorial explanation for your reference.
Here we taken example of 3 nodes in this cluster nothing but 3 virtual machines. All the nodes are on a same network but will have different IP addresses. When we deploy new pod inside a namespace in Kubernetes, Kubernetes scheduler, deploys the pods/app to the node based on the resource availabilities. If there are multiple replicas, it distributed equally or based on resource availability, it deploys.
For example, if App1 represents an application, then it is running inside a container, which is inside a pod. And a Pod may have multiple containers running inside for some cases. Please be noted “it is recommended that each pod runs a single container”.
Hope this helped to visualize the node and pods in Kubernetes, now let’s see how we can manage resource allocation like CPU and memory for application and services running in Kubernetes pods.
Requests and Limits
Request and Limit are helps Kubernetes to manage the CPU and memory resource of the Pods. There are few other resources are there, still these two are the main resource to maintain the pods.
Requests is used to specify the amount of resource that should be allocated to any pod for utilization, which means the pod is assigned this much resource when it starts. Whereas the Limit is used to specify the maximum amount of resource available for the pod, it makes sure a pod never goes above a certain value.
When you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on. Each node has a maximum capacity for each of the resource types: the amount of CPU and memory it can provide for Pods. The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled containers is less than the capacity of the node. Note that although actual memory or CPU resource usage on nodes is very low, the scheduler still refuses to place a Pod on a node if the capacity check fails. This protects against a resource shortage on a node when resource usage later increases, for example, during a daily peak in request rate. In case there is no requirement met the pod will be on pending state.
If you have automatic scaling enabled in the Kubernetes cluster then a new node is added to the cluster in case of lack of resource. In case if you are not defining any limits on the manifest, Kubernetes considers that, all the resource available on the node can be use. Please note this may differ for some scenarios. In case, if there is lack of resources, it can lead to your pods getting evicted frequently due to insufficient memory or CPU. You can use the kubectl describe command to find the reason for an Evicted pod in your Kubernetes cluster. Here is the command:
kubectl describe pod POD_NAME -n NAMESPACE_NAMEIn the output if you see something like this then it is a resource issue.
CPU Resource
CPU resource refers to the number of CPU core required by the pod to start or to run smoothly. This is measured in CPU units which is generally millicore or millicpu in most cloud platforms. So, if you mention 2 as the value that means that your container requires 2 CPU cores to start. The value 2 is equivalent to 2000m in terms of millicore or millicpu.
We can specify value in fraction too, like 0.5 which means half CPU core is required, or 250m which means 1/4th CPU core is required.
One important point to note here is that if you set a value larger than the core count of your biggest node, your pod will never be scheduled, it will always stay in Pending state. For example, if you have a pod that needs 4 cores, but your Kubernetes cluster has only 2 core VMs – in that time your pod will never be scheduled!
Below we have shown the part of YAML which specifies the CPU limit and requests values:
resources:
requests:
cpu: “500m“
memory: “1024Mi"
limits:
cpu: “1000m"
memory: “2048Mi"The above configuration means the container will be assigned 1/2 CPU core, while its cannot utilize more than 1 CPU core.
Memory Resource
Just like CPU, we can specify the memory requirements of a container. Memory resources are defined in bytes. But we can use a simple Integer value or a value with suffix like P, T, G, M, K or two letter suffixes like Pi, Ti, Gi, Mi, Ki, etc. Usually, we give value is mebibyte which is same as megabyte.
Just like CPU, if you put in a memory request that is larger than the amount of memory on your nodes, the pod will never be scheduled.
resources:
requests:
cpu: “500m“
memory: “1024Mi"
limits:
cpu: “1000m"
memory: “2048Mi"Example:
With hope you have understand what is request and limits, this will help you to troubleshoot the pods get evicted or pending state or consuming more resources or not. With this you can enable some tunning to make your resource consumption responsible. Always enable the monitoring to check the consumption in production or equal, this will help to make sure high availability of the application.
You can follow us on social media, to get some short knowledges regularly.
- Facebook: https://www.facebook.com/foxutech/
- Instagram: https://www.instagram.com/foxutech/
- YouTube: Foxutech
- Twitter: https://twitter.com/foxutech
- Medium: FoxuTech – Medium
