Managing Kubernetes Resources with Kustomize

In our last post we have seen about kustomize and also, we have seen where Kustomize will be used. Before we start, here will see some of its features and limitations are:

• It is already integrated into the kubectl CLI but the integrated version is normally outdated. Therefore, the standalone binary must be used to benefit from the latest features.
• It manages variants of resources with overlaying and merging yaml files in a structured way.
• It provides convenient built-in features to generate common resources like ConfigMaps and Secrets
• It has built-in transformers to modify resources.
• It can be extended via a plug-in mechanism.
• It is possible to dynamically change resources but this is restricted to particular fields of a resource.
• It only manages the yaml files and does not actively manage the resources in the cluster.

Philosophy

Let’s consider, your team using a Helm chart from an external organization or provider. As it is external one, we should modify as per our organization needs. How we can do? We need to either fork their repo and need to make all our changes and should apply to our cluster.

After a sometime you need to upgrade to latest version or any reason, like new version or security patching or some interesting new feature adaptation. In this case, to leverage those requirements, you have to fork the new Helm chart and re-apply your configuration changes. This may add more complexity to manage the changes and also adds risk in your application stability.

Here Kustomize comes to picture, it enables you to do that by creating a file that ties everything together, or optionally includes “overrides” for individual parameters. With that you can keep the changes in single files which helps to maintain it with versioning in GIT.

Installation

For a standalone kustomize installation (aka kustomize cli), use the following to set it up.

# curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash
# mv kustomize /usr/local/bin

Once you moved, you can run following command to test the command,

# kustomize -h

To check with kubectl,

# kubectl kustomize -h

Bases and overlays

Kustomize’s configuration helps to change the approach to leverage the use of kustomization layers so that the same base configuration files can be reused across multiple configurations. It achieves this with the concepts of bases and overlays.

• A base is a directory containing a file called kustomization.yaml, which can enumerate some set of resources with some customizations that will be applied to them. A base should be declared in the resources field of a kustomization file.
• An overlay is a directory that refers to another kustomization directory as its, or one of its, bases.

Overlays are what help us to accomplish our goal by producing variants without templating.

To start with Kustomize, you need to have your original yaml files describing any resources you want to deploy into your cluster. Let’s store the files in ./kustomize/base folder.

As we seen in last post, Kustomize not going to make any change original yaml files, all the changes will just apply customization above them to create new resources definitions.

Note: You can build base templates (e.g. for dev environment) at any point in time using the command kubectl apply -f ./kustomize/base/.

You can find all the example on kubernetes/kustomize at main · foxutech/kubernetes (github.com)

In this example, we will work with following service and a deployment resources, please this 2 files inside ./kustomize/base

# cat service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kustom-demo-app
spec:
  ports:
    - name: http
      port: 8080
  selector:
    app: kustom-demo-app

# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustom-demo-app
spec:
  selector:
    matchLabels:
      app: kustom-demo-app
  template:
    metadata:
      labels:
        app: kustom-demo-app
    spec:
      containers:
      - name: app
        image: nginx:latest
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP

Now let’s create new file inside the same folder, named kustomization.yaml :

# cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - service.yaml
  - deployment.yaml

This file will be reference point of your base and it describes the resources you use. Those resources are the path to the files relatively to the current file.

Note: This kustomization.yaml file could lead to errors when running kubectl apply -f ./kustomize/base/, you can either run it with the parameter –validate=false or simply not running the command against the whole folder, you can run the command on base directory, so you can avoid the error.

To apply your base template to your cluster, you just have to execute the following command:

# kubectl apply -k kustomize/base
or
# kustomize build

To see what will be applied in your cluster, we will mainly use in this article the command kustomize build instead of kubectl apply -k.

The result of kustomize build kustomize/base command will be the following, this nothing but combination of the resource we have in base directory.

apiVersion: v1
kind: Service
metadata:
  name: kustom-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: kustom-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustom-demo-app
spec:
  selector:
    matchLabels:
      app: kustom-demo-app
  template:
    metadata:
      labels:
        app: kustom-demo-app
    spec:
      containers:
      - image: nginx:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

Let’s Start

Now, lets start kustomize our app for some specific cases, and also we will see how the kustomation file extended with some one by one example.

We cannot cover all the functionalities in this post, but we tried to cover all the general example, which should help you to understand how the kustomize works.

First step, as we are going to take prod as example environment, for that lets create a required folder structure. for that create kustomize/overlays/prod with a kustomization.yaml inside it.

The kustomize/overlays/prod/kustomization.yaml has the following content:

# cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

Now you can build and check, this time we should see same result what we seen before with kustomize build

# kustomize build kustomize/overlays/prod/
apiVersion: v1
kind: Service
metadata:
  name: kustom-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: kustom-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustom-demo-app
spec:
  selector:
    matchLabels:
      app: kustom-demo-app
  template:
    metadata:
      labels:
        app: kustom-demo-app
    spec:
      containers:
      - image: nginx:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

Now we are set to proceed and experiment some examples, let check one by one.

Deployment Environment Variable

As major function or use of kustomize, we are not going to make any changes in base directories. Same applies to environment variables, we are not going to store anything or change in base directory, we are going to create a separate yaml in overlay folder and should reference it in kustomization.yaml.

We are going to create a environment file, This file name custom-env.yaml will contain env variables like below,

# cat kustomize/overlays/prod/custom-env.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustom-demo-app
spec:
  template:
    spec:
      containers:
        - name: app # (1)
          env:
            - name: CUSTOM_ENV_VARIABLE
              value: My First Value defined by Kustomize ❤️

Note: The name (1) key here is very important and allow Kustomize to find the right container which need to be modified.

You can see this yaml file isn’t valid by itself but it describes only the addition we would like to do on our previous base. We just have to add this file to a specific entry in the kustomize/overlays/prod/kustomization.yaml

apiVersion: kustomize.config.kustomize.io/v1beta1
kind: Kustomization

bases:
- ../../base

patchesStrategicMerge:
- custom-env.yaml

If we build this one, we will have the following result:

# kustomize build kustomize/overlays/prod
apiVersion: v1
kind: Service
metadata:
  name: kustom-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: kustom-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustom-demo-app
spec:
  selector:
    matchLabels:
      app: kustom-demo-app
  template:
    metadata:
      labels:
        app: kustom-demo-app
    spec:
      containers:
      - env:
        - name: CUSTOM_ENV_VARIABLE
          value: My First Value defined by Kustomize ❤️
        image: nginx:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

You can see our env block has been applied above our base and now the CUSTOM_ENV_VARIABLE (1) will be defined inside our deployment.yaml.

Add Replicas

Like in our previous example, let extend our base to define variables not already defined

Here, we would like to add information about the number of replicas. Like before, a chunk or yaml with just the extra info needed for defining replica will be enough:

# vim replica-and-rollout.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustom-demo-app
spec:
  replicas: 3
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate

And like before, add it to the list of patchesStrategicMerge in the kustomization.yaml:

# vim kustomization.yaml
apiVersion: kustomize.config.kustomize.io/v1beta1
kind: Kustomization

bases:
- ../../base

patchesStrategicMerge:
- custom-env.yaml
- replica-and-rollout.yaml

The result of the command kustomize build kustomize/overlays/prod give us the following result

# kustomize build kustomize/overlays/prod/
apiVersion: v1
kind: Service
metadata:
  name: kustom-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: kustom-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustom-demo-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kustom-demo-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kustom-demo-app
    spec:
      containers:
      - env:
        - name: CUSTOM_ENV_VARIABLE
          value: My First Value defined by Kustomize ❤️
        image: nginx:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

And you can see the replica number and rollingUpdate strategy have been applied above our base.

Secrets

Another example here let see how to create the secret, one of the important change we may perform often. You can do this from anywhere, the main purpose here is to define Kubernetes Secret without putting them inside Git.

To do so, kustomize has a sub-command to edit a kustomization.yaml and create a secret for you. You just have to use it in your deployment like if it already exists.

# kustomize/overlays/prod
# kustomize edit add secret kustom-demo-app --from-literal=db-password=ohmypassword

These commands will modify your kustomization.yaml and add a SecretGenerator inside it.

# kustomize build kustomize/overlays/prod/
apiVersion: v1
data:
  db-password: b2hteXBhc3N3b3Jk
kind: Secret
metadata:
  name: kustom-demo-app-62dhmggmh9
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  name: kustom-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: kustom-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustom-demo-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kustom-demo-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kustom-demo-app
    spec:
      containers:
      - env:
        - name: CUSTOM_ENV_VARIABLE
          value: My First Value defined by Kustomize ❤️
        image: nginx:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

In this example, we have used to direct password from command line, You can also use secret coming from properties file (with –from-file=file/path) or from env file (with –from-env-file=env/path.env)

If we want to use this secret from our deployment, we just have, like before, to add a new layer definition which uses the secret.

For example, this file will mount the db-password value as environment variables

# cat db-secret.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustom-demo-app
spec:
  template:
    spec:
      containers:
      - name: app
        env:
        - name: "DB_PASSWORD"
          valueFrom:
            secretKeyRef:
              name: kustom-demo-app
              key: db.password

And, like before, we add this to the kustomize/overlays/prod/kustomization.yaml

# kustomize build kustomize/overlays/prod/
apiVersion: v1
data:
  db-password: b2hteXBhc3N3b3Jk
kind: Secret
metadata:
  name: kustom-demo-app-62dhmggmh9
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  name: kustom-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: kustom-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustom-demo-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kustom-demo-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kustom-demo-app
    spec:
      containers:
      - env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              key: db.password
              name: kustom-demo-app-62dhmggmh9
        - name: CUSTOM_ENV_VARIABLE
          value: My First Value defined by Kustomize ❤️
        image: nginx:latest
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

You can see the secretKeyRef.name used is automatically modified to follow the name defined by Kustomize (1)

Note: Don’t forget, the command to put the secret inside the kustomization.yaml file should be made only from safe env and should not be committed.

Change the image of a deployment

Like secret, there is a custom directive to allow changing of image or tag directly from the command line. This is very useful if you need to deploy the image previously tagged by your continuous build system.

To do that, you can use the following command:

# cd kustomize/overlays/prod

# TAG=1.21.6 # (1)

# kustomize edit add secret kustom-demo-app --from-literal=db-password=ohmypassword   # To create my required secret

# kustomize edit set image nginx=nginx:$TAG

Here TAG used to shown for example in your CI/CD, as we mostly refer it from your pipeline.

The kustomize/overlays/prod/kustomization.yaml will be modified with those values:

# kustomize build kustomize/overlays/prod/
apiVersion: v1
data:
  db-password: b2hteXBhc3N3b3Jk
kind: Secret
metadata:
  name: kustom-demo-app-62dhmggmh9
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  name: kustom-demo-app
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: kustom-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustom-demo-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kustom-demo-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kustom-demo-app
    spec:
      containers:
      - env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              key: db.password
              name: kustom-demo-app-62dhmggmh9
        - name: CUSTOM_ENV_VARIABLE
          value: My First Value defined by Kustomize ❤️
        image: nginx:1.21.6
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

You see the first container.image of the deployment have been modified to be run with the version 1.21.6 (1).

That’s it for now, we will see more examples in coming post with GitOps integration or other use cases. If you like you can buymeacoffee.

Learn Kubernetes on Udemy, now Deal extended: Courses Up To 85% Off