Kubernetes Secrets Management: A Practical Guide

If you've worked with Kubernetes, you've probably heard of or used Kubernetes secrets. They are one of many Kubernetes resources. As the name suggests, they're meant to be used with secrets in your cluster. Imagine that your application running in a pod on a Kubernetes cluster needs some credentials. 

Using a Kubernetes secret is the most straightforward way to provide these credentials to your application. But are they actually secure? What's the best way to use them? Should you use some other secret management solutions for your Kubernetes cluster? Read on to learn everything about Kubernetes secrets management. 

Why Is Secret Management Important?

Before we dive into the do's and don'ts of Kubernetes secret management, let's take a moment to discuss why it's important in the first place. You see, Kubernetes secrets are a nice built-in semi-secret management solution, but they are not entirely secret (we'll get to that later), and they don't create a complete secret management solution. The typical problem that quickly arises when you use Kubernetes secrets is how to create and store them securely before they end up in a Kubernetes cluster. Kubernetes doesn't come with any integration to secret vaults out of the box. Therefore, they need a bit more engineering effort beyond simple creation to be secure. 

Are Kubernetes Secrets Actually Secure?

As we mentioned, a critical aspect of Kubernetes secret management is the fact that Kubernetes secrets are not actually that secret. You may be surprised to hear that, but Kubernetes secrets are not encrypted and can be easily read by anyone with access to the cluster. Kubernetes secrets are only encoded using basic base64 format. Let me show you. I'll apply the following YAML definition file of my Kubernetes secret to the cluster using the kubectl apply command: 

$ cat  | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: example-secret
type: Opaque
stringData:
  username: admin
  password: super_secret_password
EOF
secret/example-secret created

Now that we've created a secret, you'd expect it to be difficult to get the plain text values again from the cluster. If I execute kubectl describe on our secret value, Kubernetes won't show you the values by default: 

kubectl describe secret example-secret
Name:         example-secret
Namespace:    default
Labels:       
Annotations:  
Type:         Opaque

Data
====
password:  21 bytes
username:  5 bytes

However, you can force it to show the values as follows: 

$ kubectl get secret example-secret -o jsonpath='{.data}'
{password:c3VwZXJfc2VjcmV0X3Bhc3N3b3Jk username:YWRtaW4=}

Now, we can see the values, but as you would expect, they're not in plain text. However, as we mentioned before, the values are in base64, which is very easy to decode using base64 binary that comes installed on all modern operating systems. You only need to pipe the above output to a base64 --decode command: 

$ kubectl get secret example-secret -o jsonpath='{.data.username}' | base64 --decode
admin
$ kubectl get secret example-secret -o jsonpath='{.data.password}' | base64 --decode
super_secret_password

As you can see, I didn't need to specify any encryption key or certificate. Anyone who has access to my cluster could do the same. 

Is This a Problem?

Is this a Kubernetes bug or vulnerability? No, not really. Kubernetes is simply not a secret management tool. It allows you to use Kubernetes secrets out of the box to get you started, but if you really want to stay secure, you'd use an external secret management solution. Another aspect of this is that it's possible to make Kubernetes secrets a bit more secure by applying RBAC rules to your cluster.

Also, in non-multi-tenant clusters, it's not that big of an issue since access to the cluster is limited to one team anyway. Everyone who has access to the cluster can probably access the secrets too. So, the fact that Kubernetes secrets are not that secret isn't automatically bad. It simply depends on the use case. 

For customers who use Release in AWS, we automatically assign a KMS key at cluster creation so that their secrets are actually encrypted at rest. If you are managing your own EKS cluster, you can find out how to do that by following these instructions.

Let's get into how to actually manage secrets in Kubernetes. 

Secrets vs. GitOps

One of the most common issues regarding secrets in Kubernetes is that you can't simply commit secret YAML definition files to your Git repository. This is because your secret would be there in plain text (or base64-encoded values if you use data instead of stringData—but as we just showed, base64 is easy to decode). And since manually applying secrets would be slow and not scalable, you need to find a way to store your YAML secrets definition securely. There are two popular approaches to doing so. Let's discuss both. 

External Secrets

The first option is to use the External Secrets tool. The idea behind it is quite clever. You store your secret values in a safe secret vault and only commit to Git repository YAML definition files that, instead of having the actual values, hold the reference to them. Then, you install External Secrets Operator on your cluster. And once you apply this reference YAML file, the ESO will go to your secrets vault, grab the true secret value, and create your ordinary Kubernetes secret on your cluster for you. Here's an example external secret resource definition: 

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: secretstore-sample
    kind: SecretStore
  target:
    name: secret-to-be-created
    creationPolicy: Owner
  data:
  - secretKey: secret-key-to-be-managed
    remoteRef:
      key: provider-key
      version: provider-key-version
      property: provider-key-property
  dataFrom:
  - extract:
      key: remote-key-in-the-provider

As you can see, there are no actual secret values here, just pointers to where that value is. So, if that file were exposed and read by someone that shouldn't read it, they still wouldn't know your actual secrets without getting access to your secrets vault. 

SealedSecrets

Another alternative is to use SealedSecrets project. It works differently but achieves the same result. SealedSecrets lets you encrypt the content of your Kubernetes secret YAML definition file. After encrypting, the file can be safely committed to the Git repository. It could even be exposed to the internet because only the SealedSecrets controller running in your cluster will be able to decrypt it. Here's what it looks like: 

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
  namespace: mynamespace
spec:
  encryptedData:
    foo: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....

After the controller decrypts it, SealedSecrets will create an ordinary Kubernetes secret for you. Therefore, you won't need to adjust your application code. 

Bypassing Kubernetes Secrets

The tools mentioned above have one thing in common: at the end of the day, they still create ordinary Kubernetes secrets. In highly regulated environments with strict security rules, it may be necessary to avoid Kubernetes secrets completely because they're only base64 encoded. In such cases, you need to find another solution for passing secrets to your pods. 

Kubernetes Secrets Store CSI Driver

One option is to use Kubernetes's own new Secrets Store CSI Driver. With this option, you store your secrets in the external secrets store. Then Secrets Store CSI Driver will load them from there and mount them directly to your pods as volumes. Therefore, you'll bypass Kubernetes secrets resources completely. 

However, there are two downsides to this approach. First is the fact that you'll need to adjust your application to load secrets from files instead of from environment variables like with normal Kubernetes secrets. Second, Secrets Store CSI Driver currently has alpha functionality, so it may not be fully stable. 

Hashicorp Vault Injector

Another option is to use Hashicorp Vault together with their Secret Injection option. The concept is similar to the Kubernetes Secrets Store CSI Driver. You store your secrets in Hashicorp Vault. Then, Hashicorp Vault Agent Injector will get the secrets for you and load them directly to the pod, bypassing Kubernetes secrets. And similarly to CSI Driver, Hashicorp Injector will load your secrets as volumes. However, in the case of Hashicorp, it will be shared memory volume instead of standard inline volume. 

Keep Your Secrets Safe

As you can see, Kubernetes secret management isn't as straightforward as one may think. It's not as simple as creating Kubernetes secret resources for your pods. Besides the fact that these secrets are not so secret, you must also consider the whole secret life cycle. Even if base64 encoding is enough in your case, you still need to figure out how to store your Kubernetes secret YAML definitions without exposing them. 

In this post, you learned a few ways to do that—and how to bypass Kubernetes secrets completely and pass your credentials directly to your pods. Your choice of option will depend on your use case and company specifics. 

If you want to learn more about Kubernetes or Security, look at our blog for more articles.