Bitnami Sealed Secrets – Kubernetes Secret Management

Nowadays we are using GitOps for application deployment and for that we tend to put all the application’s information and configuration on Git, but same time can we do the same with Kubernetes Secrets YAML file on Git? The answer is definitely NO, as the Secret’s file contains just the base64 encoded value of our sensitive information; which can be passwords, keys, certificates, tokens etc.

Following is a sample YAML file for a Secret: –

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: d2VsY29tZSB0bw==
  password: Zm94dXRlY2g=

And we can easily decode the sensitive information with base64 command like following:

# echo -n Zm94dXRlY2g= | base64 -d

So, definitely we should not store Kubernetes Secrets on Git/any visible systems.  There are many different ways to externalize k8s secrets like Hashicorp’s Vault, Azure key vault, external secret operator, Helm Secrets, Secrets Store CSI Driver, Bitnami’s SealedSecret etc. In this blog we are going to explore about Bitnami’s Sealed Secret.

Sealed Secrets

Sealed Secrets is an open-source Kubernetes controller and a client-side CLI tool from Bitnami that aims to solve the “storing secrets in Git” part of the problem, using asymmetric crypto encryption. Sealed Secrets with an RBAC configuration preventing non-admins from reading secrets is an excellent solution for the entire problem.

How it works

It works as below;

• Encrypt the secret on the developer machine using a public key and the kubeseal CLI. This encodes the encrypted secret into a Kubernetes Custom Resource Definition (CRD)
• Deploy the CRD to the target cluster
• The Sealed Secret controller decrypts the secret using a private key on the target cluster to produce a standard Kubernetes secret.

The private key is only available to the Sealed Secrets controller on the cluster, and the public key is available to the developers. This way, only the cluster can decrypt the secrets, and the developers can only encrypt them.

Install

Client

# wget https://github.com/bitnami-labs/sealed-secrets/releases/download/<release-tag>/kubeseal-<version>-linux-amd64.tar.gz
Example: 
# wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.19.4/kubeseal-0.19.4-linux-amd64.tar.gz
# tar -xvzf kubeseal-<version>-linux-amd64.tar.gz kubeseal
# install -m 755 kubeseal /usr/local/bin/kubeseal 

Server

Sealed Secrets Controller

Current deployment process can be done manual helm install command or kubectl on targeted cluster:

Installing via helm chart

# helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
# helm dependency update sealed-secrets
# helm install sealed-secrets sealed-secrets/sealed-secrets --namespace kube-system --version 2.7.4 

Installing via Kubectl

# wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.1/controller.yaml
# kubectl apply -f controller.yaml

You can ensure that the relevant Pod is running as expected by executing the following command:

# kubectl get pods -n kube-system | grep sealed-secrets-controller

Sealing the Secrets

Create Secret

The code for Kubernetes Secret is given below. Save the code in a file name secrets.yaml.

apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: null
  name: my-secret
  namespace: test-ns
data:
  password: dXNlcm5hbWU=        # base64 encoded username
  username: cGFzc3dvcmQ=        # base64 encoded password

Here the username and passwords are base64 encoded. Change with appropriate secret for your use. You can retrieve the generated public key certificate using kubeseal and store it on your local disk:

# kubeseal --fetch-cert > public-key-cert.pem

kubeseal encrypts the Secret using the public key that it fetches at runtime from the controller running in the Kubernetes cluster. If a user does not have direct access to the cluster, then a cluster administrator may retrieve the public key from the controller logs and make it accessible to the user.

A SealedSecret CRD is then created using kubeseal as follows using the public key file:

# kubeseal --format=yaml --cert=public-key-cert.pem < secret.yaml > sealed-secret.yaml

Note that the keys in the original Secret—namely, username and password—are not encrypted in the SealedSecret, only their values are. You may change the names of these keys, if necessary, in the SealedSecret YAML file and still be able to deploy it successfully to the cluster. However, you cannot change the name and namespace of the SealedSecret. Doing so will invalidate the SealedSecret because the name and namespace of the original Secret are used as input parameters in the encryption process.

The YAML manifest that pertains to the Secret is no longer needed and may be deleted. The SealedSecret is the only resource that will be deployed to the cluster as follows:

# kubectl apply -f sealed-secret.yaml

Once the SealedSecret CRD is created in the cluster, the controller knows and unseals the underlying Secret using the private key and deploys it to the same namespace. This is seen by looking at the logs from the controller.

Managing the Sealing Key

Without the private key that is managed by the controller, there is no way to decrypt the encrypted data within a SealedSecret. Run the following command in order to retrieve the private key from the cluster:

# kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > master.yaml

Now, let’s first delete the installation of the controller, the Secret that it created which contains the private key, the SealedSecret resource named my-secret as well as the Secret that was unsealed from it.

# kubectl delete secret mysecret
# kubectl delete SealedSecret mysecret
# kubectl delete secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
# kubectl delete -f controller.yaml

Now, put the Secret containing the private key back into the cluster using the master.yaml file and redeploy the SealedSecret CRD, controller and RBAC artifacts on your EKS.

# kubectl apply -f master.yaml 
# kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
# kubectl apply -f controller.yaml

View the logs of the newly launched controller pod. Note that the name of the controller pod will be different in your cluster. As you can see from the logs, the controller was able to find the existing Secret sealed-secrets-xxxxxx in the kube-system namespace and therefore does not create a new key pair.

# kubectl logs -f sealed-secrets-controller-xxxxxxx-xxxx -n kube-system

Now, let’s redeploy the SealedSecret and verify that the controller is able to successfully unseal it.

# kubectl create -f sealed-secret.yaml 
# kubectl logs -f sealed-secrets-controller-xxxxxxx-xxxx -n kube-system

Backup and Restore

If you do want to make a backup of the encryption private keys, it’s easy to do from an account with suitable access:

# kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >main.key
# kubectl get secret -n kube-system sealed-secrets-key -o yaml >>main.key

NOTE: You need the second statement only if you ever installed sealed-secrets older than version 0.9.x on your cluster.

NOTE: This file will contain the controller’s public + private keys and should be kept safe!

To restore from a backup after some disaster, just put those secrets back before starting the controller – or if the controller was already started, replace the newly-created secrets and restart the controller:

# kubectl apply -f main.key
# kubectl delete pod -n kube-system -l name=sealed-secrets-controller

Can I decrypt my secrets offline with a backup key?

While treating sealed-secrets as long-term storage system for secrets is not the recommended use case, some people do have a legitimate requirement for being able to recover secrets when the k8s cluster is down and restoring a backup into a new SealedSecret controller deployment is not practical.

If you have backed up one or more of your private keys (see previous section), you can use the kubeseal --recovery-unseal --recovery-private-key file1.key, file2.key, command to decrypt a sealed secrets file.

How to use kubeseal if the controller is not running within the kube-system namespace?

If you installed the controller in a different namespace than the default kube-system, you need to provide this namespace to the kubeseal command line tool. There are two options:

You can specify the namespace via the command line option --controller-namespace <namespace>:

# kubeseal --controller-namespace sealed-secrets <mysecret.json >mysealedsecret.json

Via the environment variable SEALED_SECRETS_CONTROLLER_NAMESPACE:

# export SEALED_SECRETS_CONTROLLER_NAMESPACE=sealed-secrets
# kubeseal <mysecret.json >mysealedsecret.json

Useful Commands

Get the base64 encoded secret:

# kubectl -n sealed-secrets get secret sealed-secrets-xxxxxxx -o yaml

Get the public key:

# kubeseal --fetch-cert --controller-name=sealed-secrets --controller-namespace=sealed-secrets

Decode either base64 encoded key through kubectl:

# kubectl -n misc get secrets sealed-secrets-xxxxxx -o 'go-template={{index .data "tls.crt"}}' | base64 --decode
# kubectl -n misc get secrets sealed-secrets-xxxxxx -o 'go-template={{index .data "tls.key"}}' | base64 –decode

Sealed Secrets Scopes

K8s secrets are namespaced objects and when we create the SealedSecret object we need to somehow preserve that information. It should not happen that some un-authorized/malicious user renames it or apply it another namespace.  But sometimes we may be need to rename or deploy in a different namespace as well.

To accommodate above requirements, SealedSecrets support following scopes.

Strict

This is the default type, in which secret is sealed with exactly the same name and namespace. These attributes become part of the encrypted data and cannot changed while decrypting, to create the secret object.

Namespace-wide

We can rename the secret only within the given namespace.

Cluster-wide

We can unseal the in any namespace and can be given any name of our choice.

We can define the scope using –scope flag in kubeseal, like following: –

# kubeseal --scope cluster-wide <secret.yaml >sealed-secret.yaml

We can also use annotations within the k8s Secret to apply scopes before passing the configuration to kubeseal

sealedsecrets.bitnami.com/namespace-wide: "true" -> for namespace-wide
sealedsecrets.bitnami.com/cluster-wide: "true" -> for cluster-wide

If we don’t specify the scope flag, then kubeseal will automatically use the Strict Scope by default.

Secret Renewal and Rotation

As we install the SealedSecret Controller on the Kubernetes cluster, it creates a public/private key combination to encrypt and decrypt the actual user’s secrets. For security reason, this key combination gets renewed every 30 days and it can be customized as well. Please take a note that this is not a rotation, in which we replace the old ones with the new ones.

SealedSecret Controller keeps track of the older private keys to decrypt any users’ secrets created earlier. Only the latest public key would be used to encrypt users’ secrets.

For actual users secrets it is always a good practice to rotate them to create new SealedSecrets.

Advantages

• Supports template definition so that metadata can be added to the unsealed secrets. For example, you can add labels and annotations to the unsealed secrets using the template definition.
• The unsealed secrets will be owned by the sealed secret CRD and updated when the sealed secrets are updated.
• Certificates are rotated every 30 days by default, and this can be customized.
• Secrets are encrypted using unique keys for each cluster, namespace, and secret combination (private key + namespace name + secret name), preventing any loopholes in decryption. This behavior is configurable using scopes strict, namespace-wide, and cluster-wide during the sealing process.
• Can be used to manage existing secrets in the cluster.
• Has a VSCode extension to make it easier to use.

Disadvantages

• Since it unseals the sealed secrets into regular secrets, you can still decode them if you have access to the cluster and namespace.
• Requires resealing for each cluster environment, as the key pair will be unique for each cluster.