Kubernetes Namespace Resource Quota and Limits

motoskia

2 years ago

Kubernetes Namespace Resource Quota and Limits

In this post will see what is Namespace resource Quota and limits for pods, in recent post we have seen about requests and limits, in that we have mentioned to keep the requests and limits in pods manifests. Also, we have explained how it is important. As mostly we just recommend to respective team, but after that we cannot be sure, are they implemented or they may forget when there is some critical work assigned or some time it is hard to follow the process or we cannot monitor all the time. What is cause? Container(pod) will take too much resources which can lead to other containers getting no resource to start or container may not even start, if the value for resources in more that total amount available in a single node.

All this cause big issue on your cluster or environment. How we can avoid? Well, if you are SRE/DevOps/k8s admin/whoever, you would allow everyone to create their own namespace and do whatever they want within their namespaces. Same time you should make sure the rule has been set with ResourceQuota and LimitRange.

The resource quota is the total allocated resources for a particular namespace, while limit range also used to assign limits for objects like containers (Pods) running inside the namespace.

This is one of the best practice recommended. Not only resources you can limit, you can limit number objects also, here is the table lists, what are you can limit it.

Resources:

Resource	Description
requests.cpu	The sum of the CPU requests of all pods cannot exceed this value
requests.memory	The sum of the Memory requests of all pods cannot exceed this value
requests.storage	The sum of the Storage requests of all persistent volume claims cannot exceed this value
limits.cpu	The sum of the CPU Limits of all pods cannot exceed this value
limits.memory	The sum of the Memory Limits of all pods cannot exceed this value

Objects:

Resource	Description
Configmaps	Total no of configmaps that can exists in a namespace
Persistentvolumeclaims	Total no of Persistentvolumeclaims that can exists in a namespace
Pods	Total no of Pods that can exists in a namespace
Replication controllers	Total no of replicationcontrollers that can exists in a namespace
Services	Total no of Services that can exists in a namespace
Service.loadbalancer	Total no of loadbalancers that can exists in a namespace
Service.nodeports	Total no of nodeports that can exists in a namespace
Secrets	Total no of secrets that can exists in a namespace

ResourceQuotas

After creating Namespaces, we can use the ResourceQuota object to limit down the total amount of resource used by the namespace. We can use ResourceQuota to set limits for different object types that can be created within a namespace along with setting quotas for resources like CPU and memory.

Resource Quota support mostly enabled in most of the cloud providers like GCE, EKS, etc. It will be enabled when the apiserver --enable-admission-plugins= flag has ResourceQuota as one of its arguments.

A ResourceQuota for setting quota on resources looks like this:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: quota
spec:
  hard:
    limits.cpu: "5"
    limits.memory: 2Gi
    requests.cpu: "3"
    requests.memory: 1Gi
    pods: "10"

Following are the resource quotas set:

limits.cpu is the maximum CPU cores limit for all the containers (Pods) in the Namespace, i.e. the entire namespace.
limits.memory is the maximum Memory limit for all containers(Pods) in the Namespace, i.e. the entire namespace.
requests.cpu is the maximum CPU millicores allocated for all the containers (Pods) in the Namespace. As per the above YAML, we can have 5 containers with 600m requests, 10 containers with 300m requests, etc. The total requested CPU in the Namespace should be less than 3000m.
requests.memory is the maximum Memory allocated for all the containers(Pods) in the Namespace. As per the above YAML, we can have 5 containers with 200MiB requests, 10 containers with 100MiB CPU requests, or 1 container with a 1000MiB or 1Gi request.

pods are the maximum number of pods allowed in the namespace.

LimitRange

We can also create a LimitRange object in our Namespace which can be used to set limits on resources on containers (Pods) running within the namespace. This is used to provide default limit values for Pods which do not specify this value themselves to equally distribute resources within a namespace.

A LimitRange provides constraints that can:

Apply minimum and maximum cpu resources usage limit per Pod or Container in a namespace.
Apply minimum and maximum memory request limit per PersistentVolumeClaim in a namespace.
Set a ratio between request and limit for a resource in a namespace.
Set default request/limit for resources within a namespace and then automatically set the limits to Containers at runtime

Here is a sample LimitRange YAML file:

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "resource-limits"
spec:
  limits:
    - type: "Pod"
      max:
        cpu: "2"
        memory: "1Gi"
      min:
        cpu: "200m"
        memory: "6Mi"
    - type: "Container"
      max:
        cpu: "2"
        memory: "1Gi"
      min:
        cpu: "100m"
        memory: "4Mi"
      default:
        cpu: "300m"
        memory: "200Mi"
      defaultRequest:
        cpu: "200m"
        memory: "100Mi"

The above YAML file has 4 sections, max, min, default, and defaultRequest.

The default section will set up the default limits for a container in a pod. Any container with no limits defined will get these values assigned as default.
The defaultRequest section will set up the default requests for a container in a pod. Any container with no requests defined will get these values assigned as default.
The max section will set up the maximum limits that a container in a Pod can set. The value specified in the default section cannot be higher than this value. Also, limits set on a container cannot be higher than this value. One important point to note here is that if max value is set and the default section is not set, then any containers that don’t explicitly set these values themselves will get the max values assigned to it as the limit.
The min section will set up the minimum Requests that a container in a Pod can set. The value specified in the defaultRequest section cannot be lower than this value. Similarly, requests set on a container cannot be lower than this value. One important point to note here is that if min value is set and the defaultRequest section is not set, then the min value becomes the default Request value too.