Kubernetes DaemonSet – Explained

As we are seeing about Kubernetes resources and how to troubleshoot the Kubernetes issue with live demo. Part of that, in this article, we will have a look at the DaemonSet on Kubernetes.

What is a Kubernetes DaemonSet?

The DaemonSet feature is used to ensure that some or all of your pods are scheduled and running on every single available node in the Kubernetes cluster. This essentially runs a copy of the desired pod across all nodes.

Whenever there is new node has been added to a Kubernetes cluster, a new pod will be added to that new node. Similarly, when a node got removed, the DaemonSet controller ensures that the pod associated with that node is garbage collected.

DaemonSets are an integral part of the Kubernetes cluster facilitating administrators to easily configure services (pods) across all or a subset of nodes.

Where we can use DaemonSets?

As daemonSets helps to improve the performance of a Kubernetes cluster by distributing maintenance tasks and support services by deploying the pods across all the nodes. Even you can use this for the job which needs to be run on all the nodes, like metric monitoring, log collector, etc. Following is some examples use cases of DaemonSets:

• To run a daemon for logs collection on each node, such as Fluentd and logstash.
• To run a daemon for cluster storage like glusterd and ceph on each node.
• To run a daemon which does logs rotation and cleaning log files.
• To run a daemon that runs on each node, detects node problems and reports them to the api-server, like node-problem-detector.
• To run a daemon for node monitoring on every note, such as Prometheus Node Exporter, collectd.
• Even you can use for storing the cache across all the node, like if there is large file need to be accessible by the users, you can mount some cloud volume and expose it.

Depending on the requirement, you can set up one or multiple DaemonSets for a single type of daemon, with different flags, memory, CPU, etc. that supports multiple configurations and hardware types.

How to create a DaemonSet

Here is the sample daemonSet,

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      containers:
      - name: fluentd-elasticsearch
        image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2

As you can notice in the above structure, the apiVersion, kind, and metadata are required fields in every Kubernetes manifest. The DaemonSet specific fields come under the spec section—these fields are both mandatory.

• template. this is a required field that specifies a pod template for the DaemonSet to use. Along with the required fields for containers, this template requires appropriate labels (.spec.template.metadata.labels). You should also remember that a pod template of your DaemonSet must have a RestartPolicy equal to Always, which defaults to Always if not specified.
• selector. The selector for the pods managed by the DaemonSet. This value must be a label specified in the pod template. This value is fixed and cannot be changed after the initial creation of the DaemonSet. Changing this value will cause pods created via that DaemonSet to be orphaned. Kubernetes offers two ways to match matchLabels and matchExpressions for creating complex selectors.

Other optional fields

• template.spec.nodeSelector – This can be used to specify a subset of nodes that will create the Pod matching the specified selector.
• template.spec.affinity – This field can be configured to set the affinity that would run the pod only on nodes that match the configured affinity.

Creating a DaemonSet

Now let’s go ahead with creating a sample DaemonSet. Here, we will be using a “fluentd-elasticsearch” image that will run on every node in a Kubernetes cluster. Each pod would then collect logs and send the data to ElasticSearch.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:elasticsearch
        env:
        - name:  FLUENT_ELASTICSEARCH_HOST
          value: "xxxxxxx.foxutech.com"
        - name:  FLUENT_ELASTICSEARCH_PORT
          value: "30216"
        - name: FLUENT_ELASTICSEARCH_SCHEME
          value: "https"
        - name: FLUENT_UID
          value: "0"
        - name: FLUENT_ELASTICSEARCH_USER
          value: "xxxxxxxxxxx"
        - name: FLUENT_ELASTICSEARCH_PASSWORD
          value: "xxxxxxxxxx"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

First, let’s create the DaemonSet using the kubectl create command and retrieve the DaemonSet and pod information as follows:

# kubectl create -f daemonset-example.yaml
# kubectl get daemonset
# kubectl get pod -o wide

As you can see from the above output, our DaemonSet has been successfully deployed.

Depending on the nodes available on the cluster, it will scale automatically to match the number of nodes or a subset of nodes on the configuration.

Defining Tolerations for Pods

Taints and tolerations work together to ensure that pods are not scheduled into wrong nodes. Taints prohibit certain nodes from scheduling pods on them whereas tolerations allow (but not require) certain pods to be scheduled on nodes with matching taints.

Before tolerations can be applied in a DeamonSet, you should have an correct taint added to a node. A taint can be added to a node using kubectl taint command. For example:

# kubectl taint nodes nodeX taintKey=taintValue:NoSchedule

adds a taint to nodeX. The taint has a key taintKey and a value taintValue, and taint effect NoSchedule. This means that no pod will be able to schedule onto nodeX unless it has a matching toleration.

Once the taint is applied, you can use tolerations in a pod template of the DeamonSet in the.spec.template.spec.tolerations field. In our case, we use the tolerations with the key:node-role.kubernetes.io/master and the effect NoSchedule to allow Daemon pods to be scheduled on the master of our Kubernetes cluster.

Tolerations are useful when you want to run DaemonSet pods on nodes that prohibit scheduling. However, what about running DaemonSet pods on specific nodes matching certain criteria? In this scenario, we can use the.spec.template.spec.nodeSelector field that tells the DaemonSet controller to create pods only on those nodes which match the node selector (ex: “app: alphaOne”). For a given pod to run on a node, the latter must have each of the indicated key-value pairs as labels. In our example, the node must have a “app: alphaOne” label. Correspondingly, this label should be applied to a node/s so that the DaemonSet is able to schedule pods on this node/s. Make sure you have set the labels to the nodes, you can use kubectl command for that, like.

# kubectl label nodes <node-name> <label-key>=<label-value>

You can also achieve a more fine-grained control over how nodes are selected using a .spec.template.spec.affinity field. In that case, the DaemonSet controller will create Pods on nodes which match that node affinity. Now, as you understand the basics, we can create the DeamonSet by saving the spec above in a file (e.g fluentd-es.yml) and running:

# kubectl create -f fluentd-es.yml

Updating DaemonSets

When it comes to updating DaemonSets, if a node label is changed, DaemonSet will automatically add new pods to matching nodes while deleting pods from non-matching nodes. We can use the “kubectl apply” command to update a DaemonSet, as shown below.

# kubectl apply -f daemonset.yaml

There are two strategies that can be followed when updating DaemonSets:

• The default strategy in Kubernetes, this will delete old DaemonSet pods and automatically create new pods when a DaemonSet template is updated.
• When using this option, new DaemonSet pods are created only after a user manually deletes old DaemonSet pods.

These strategies can be configured using the spec.updateStrategy.type option.

Performing a Rolling Update on a DaemonSet

Since Kubernetes version 1.6, you can perform a rolling update on a DaemonSet. RollingUpdate is the default update strategy for DaemonSets. If it’s enabled, after you update a DaemonSet template, old DaemonSet pods will be removed, and new DaemonSet pods will be created automatically.

Let’s use a simple DaemonSet for Apache HTTP server to illustrate this.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: logging
spec:
  selector:
    matchLabels:
      app: httpd-logging
  template:
    metadata:
      labels:
        app: httpd-logging
    spec:
      containers:
        - name: webserver
          image: httpd
          ports:
          - containerPort: 80

As you see, we did not explicitly specify the RollingUpdate strategy (it’s default). However, you could set .spec.updateStrategy.type to RollingUpdate to achieve the same result. You may also want to set .spec.updateStrategy.rollingUpdate.maxUnavailable (defaults to 1) and .spec.minReadySeconds (defaults to 0) as well. Here, the first feature specifies the maximum number of Pods that can be unavailable during the update process, whereas the second feature specifies the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available.

# kubectl create -f httpd-dset.yml

After the DaemonSet is created, any updates to a RollingUpdate DaemonSet .spec.template will trigger a rolling update. For example, we can trigger the RollingUpdate by changing the container image used by the DaemonSet:

# kubectl create -f httpd-dset.yml

After setting a new image, you can watch the rollout status of the latest DaemonSet rolling update:

# kubectl rollout status ds/logging

Communicating with pods created by DaemonSet

There are multiple methods to communicate with pods created by DaemonSets. Here are some available options:

• The Push pattern: pods do not receive traffic. Instead, they push data to other services like ElasticSearch, for example.
• NodeIP and known port pattern: in this design, pods use the hostPort to acquire the node’s IP address. Clients can use the node IP and the known port (for example, port 80 if the DaemonSet has a web server) to connect to the pod.
• DNS pattern: create a Headless Service that selects the DaemonSet pods. Use Endpoints to discover DaemonSet pods.
• Service pattern: create a traditional service that selects the DaemonSet pods. Use NodePort to expose the pods using a random port. The drawback of this approach is that there is no way to choose a specific pod.

Why is kube-dns a deployment and kube-proxy a daemonset?

The reason behind that is that kube-proxy is needed on every node in the cluster to run IP tables, so that every node can access every pod no matter on which node it resides. Hence, when we make kube-proxy a daemonset and another node is added to the cluster at a later time, kube-proxy is automatically spawned on that node.

Kube-dns responsibility is to discover a service IP using its name and only one replica of kube-dns is enough to resolve the service name to its IP. Hence, we make kube-dns a deployment, because we don’t need kube-dns on every node.

Deleting DaemonSets

Deleting a DaemonSet is a simple task. To do that, simply run the kubectl delete command with the DaemonSet. This would delete the DaemonSet with all the underlying pods it has created.

# kubectl delete -f daemonset.yaml

We can use the cascade=false flag in the kubectl delete command to only delete the DaemonSet without deleting the pods. Deleting a DaemonSet will clean up all the pods that DaemonSet has created.