In this article, we are going to encrypt the Etcd database with a locally managed key.

It is very important to encrypt Etcd since all data including secrets of your Kubernetes cluster are stored by Etcd. Not encrypting Etcd database adds the risk of exposing Kubernetes secrets in the eventuality of it being accessed or stolen by an attacker.

By default, Kubernetes does not encrypt its Etcd database...

Declare a secret configuration file in "kube-apiserver"

Create a unique key and copy it:

head -c 32 /dev/urandom | base64
                    

Create a configuration file with the secret key:

sudo vi /etc/kubernetes/pki/etcd-encryption.yaml
                    

Contents to add:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: [Created base64 unique key with /dev/urandom]
    - identity: {}
                    

Once saved, only allow root user to read it. This way if there is a host compromise, a non root attacker cannot read it:

sudo chmod 600 /etc/kubernetes/pki/etcd-encryption.yaml
                    

Declare that configuration file in kube-apiserver's config file:

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
                    

Inside this file, add the following as the last option under the field "command:":

- --encryption-provider-config=/etc/kubernetes/pki/etcd-encryption.yaml
                    

For example:

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=...
    ...
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --encryption-provider-config=/etc/kubernetes/pki/etcd-encryption.yaml
    image: k8s.gcr.io/kube-apiserver:v1.24.0
                    

Wait for "kube-apiserver" to restart. No need to restart it manually, it should pick up the new configuration automatically. Check with the command below, the AGE column which should show a recent age for that container:

kubectl get pods -n kube-system | grep kube-apiserver
                    

The configuration file that we have created is saved in the folder “/etc/kubernetes/pki” because "kube-apiserver" is a container in Kubernetes which mounts specific folders (see below) and therefore, it can only access files inside those folders.
See below the list of mounted folders from kube-apiserver's config file "/etc/kubernetes/manifests/kube-apiserver.yaml":

    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
                    

Encrypt all existing secrets in Kubernetes

The command below reads all secrets in Kubernetes and updates them to apply server our new encryption configuration:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -
                    

From that point, all secrets in Kubernetes are encrypted.

Test new secrets will be encrypted

Create a new secret:

kubectl create secret generic secret1 -n default --from-literal=mykey=mysecretdata
                    

We will check the created secret data by directly accessing Etcd database. Run the commands below:

sudo apt-get install etcd-client

sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/secrets/default/secret1
                    

Verify that the output shows the stored secret with prefix "k8s:enc:aescbc:v1:". That confirms "aescbc" provider has encrypted the resulting data

Example output:

k8s:enc:aescbc:v1:key1:%8p6dQfa8HZe_u...>
                    

Verify the secret is correctly decrypted when retrieved via the Kubernetes API. The below command should output: "mysecretdata":

kubectl get secret secret1 -o jsonpath="{.data.mykey}" | base64 --decode
                    

Remove the secret:

kubectl delete secret secret1
                    

We successfully checked that Etcd encrypts its database.

Few notes

Locally managed key does not protect about a host comprising with root access. However, it is a better approach than the default settings of Kubernetes which does not encrypt Etcd database. Ideally we would like to use a KMS provider.

Kubernetes official recommendations:
"Encrypting secrets with a locally managed key protects against an etcd compromise, but it fails to protect against a host compromise. Since the encryption keys are stored on the host in the EncryptionConfig YAML file, a skilled attacker can access that file and extract the encryption keys. Therefore, Storing the raw encryption key in the EncryptionConfig only moderately improves your security posture, compared to no encryption. Please use a KMS provider for additional security."

Kubernetes doc about encryption with key stored locally

Kubernetes doc about encryption with key stored by a KMS provider

KMS providers

Equinix Smart Key Service

Hasicorp Vault