Resource Quota

When several users or teams share a cluster with a fixed number of nodes, there is a concern that one team could use more than its fair share of resources Resource quotas are a tool for administrators to address this concern.

A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. It can limit the quantity of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.

Compute Resource Quotas

These limits control how much CPU and memory can be allocated to pods in a namespace.

Let's create file resource_quota.yaml and put this following configuration.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: limited
spec:
  hard:
    requests.cpu: "1"
    requests.memory: "2Gi"
    limits.cpu: "2"
    limits.memory: "4Gi"

• request.cpu: Maximum total CPU requested across all pods in this namespace cannot exceed more than 1 core.

• request.memory: Maximum total memory requested across all pods in this namespace cannot exceed more than 2GB.

• limits.cpu: Maximum total CPU usage of all pods combined in this namespace cannot exceed more than 2 cores.

• limits.memory: Maximum total memory usage of all pods combined in this namespace cannot exceed more than 4GB.

Apply and Validate

Let's apply the configuration file using kubectl apply command.

➜ kubectl apply -f resource_quota.yaml 
resourcequota/compute-quota created

We can validate it using kubectl describe quota in the specific namespace like this.

➜ kubectl -n limited describe quota
Name:            compute-quota
Namespace:       limited
Resource         Used  Hard
--------         ----  ----
limits.cpu       0     2
limits.memory    0     4Gi
requests.cpu     0     1
requests.memory  0     2Gi

Storage Resource Quotas

These quotas limit how much storage (Persistent Volumes) can be created in a namespace.

Add few more line to our configuration file before in the spec.hard section to define the storage limit.

requests.storage: "64Gi"
persistentvolumeclaims: "10"

• requests.storage: The total storage requested by PVC in this namespace cannot exceed 64GB.

• persistentvolumeclaims: The total number of PVC in this namespace cannot exceed 10.

Apply and Validate

Let's apply our configuration file and validate.

➜ kubectl apply -f resource_quota.yaml
resourcequota/compute-quota configured

➜ kubectl -n limited describe quota   
Name:                   compute-quota
Namespace:              limited
Resource                Used  Hard
--------                ----  ----
limits.cpu              0     2
limits.memory           0     4Gi
persistentvolumeclaims  0     10
requests.cpu            0     1
requests.memory         0     2Gi
requests.storage        0     64Gi

Object Count Quotas

These quotas limit the number of objects (Pods, Services, ConfigMaps, etc.) a namespace can have.

Add few more line to our configuration file before in the spec.hard section to define pods, services, secrets, and configmaps limit.

pods: "5"
services: "2"
secrets: "2"
configmaps: "2"

• pods: The total number of Pods in a non-terminal state in this namespace cannot exceed 5. A pod is in a terminal state if .status.phase in (Failed, Succeeded) is true.

• services: The total number of services in this namespace cannot exceed 2.

• secrets: The total number of secrets in this namespace cannot exceed 2.

• configmaps: The total number of configmaps in this namespace cannot exceed 2.

Apply and Validate

Again, lets apply our configuration and validate.

➜ kubectl apply -f resource_quota.yaml

➜ kubectl -n limited describe quota   
Name:                   compute-quota
Namespace:              limited
Resource                Used  Hard
--------                ----  ----
configmaps              1     2
limits.cpu              0     2
limits.memory           0     4Gi
persistentvolumeclaims  0     10
pods                    0     5
requests.cpu            0     1
requests.memory         0     2Gi
requests.storage        0     64Gi
secrets                 0     2
services                0     2

We can see all of the resource limit that we just created. This will ensure this namespace resource will not exceed our defined limit.

What Happened if Limit Exceeded?

• If creating or updating a resource violates a quota constraint, the request will fail with HTTP status code 403 FORBIDDEN with a message explaining the constraint that would have been violated.

• If quotas are enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation.

• We can use LimitRange to enforce default request and limit for compute resource.

References

• https://kubernetes.io/docs/concepts/policy/resource-quotas/