Skip to main content

Sealed Secrets

A plain Kubernetes Secret is base64-encoded, not encrypted. You can't safely commit one to git. Sealed Secrets solves this with a controller that runs in the cluster and is the only thing that can decrypt the values. You commit the encrypted SealedSecret to the repo and, on apply, the controller produces a real Secret from it.

Prerequisites​

  • Deploys from CI to staging and prod work (Part 7).
  • kubeseal is installed on your workstation (Part 1).
  • doctl is authenticated. You'll use DigitalOcean Spaces for off-cluster backup of the controller's signing key.

The threat model in 60 seconds​

A standard Secret:

kubectl create secret generic foo --from-literal=KEY=swordfish -o yaml --dry-run=client
# data:
#   KEY: c3dvcmRmaXNo
echo c3dvcmRmaXNo | base64 -d
# swordfish

Anyone who can read the manifest can read the value. Committing this to git makes the password public. Sealed Secrets fixes this by encrypting the value with the controller's public key (kubeseal does this on your workstation) and storing the result in a SealedSecret. Only the private key, held by the controller inside the cluster, can decrypt. The repo never sees the plaintext.

Step 1 — Install the controller (cluster-bootstrap)​

Install once per cluster as part of cluster-bootstrap. Apply the controller before any SealedSecret lands in the app overlays; otherwise the CRD doesn't exist and kubectl apply -k fails.

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm repo update

for ctx in do-staging do-prod; do
  helm --kube-context "$ctx" upgrade --install sealed-secrets sealed-secrets/sealed-secrets \
    --version 2.16.1 \
    --namespace kube-system \
    --set fullnameOverride=sealed-secrets-controller
done

kubectl --context do-staging -n kube-system rollout status deploy/sealed-secrets-controller
kubectl --context do-prod    -n kube-system rollout status deploy/sealed-secrets-controller

Verify the CRD exists:

kubectl --context do-staging get crd sealedsecrets.bitnami.com
kubectl --context do-prod    get crd sealedsecrets.bitnami.com

Step 2 — Fetch each cluster's public key​

Save each cluster's controller public key so you can encrypt without cluster access:

mkdir -p ~/code/expressapp/k8s/keys
for ctx in do-staging do-prod; do
  kubeseal --context "$ctx" --fetch-cert > ~/code/expressapp/k8s/keys/${ctx}.pem
done

Open one of them. It's a real PEM, safe to commit if you want. Treat it like a public key, because it is one.

Step 3 — Encrypt a value for each environment​

The same secret may need different values in staging and prod (e.g. a Stripe test key vs live key). Encrypt per environment using that environment's public key.

# staging: a fake token
echo -n "stg_fake_token_value" | \
  kubeseal --raw \
    --name expressapp-secrets --namespace expressapp-staging \
    --cert ~/code/expressapp/k8s/keys/do-staging.pem
# prints a base64 ciphertext, e.g. AgB+f4...

Compose a full SealedSecret:

kubectl --context do-staging -n expressapp-staging create secret generic expressapp-secrets \
  --from-literal=API_TOKEN="stg_fake_token_value" \
  --dry-run=client -o yaml | \
kubeseal --cert ~/code/expressapp/k8s/keys/do-staging.pem \
  --format yaml > ~/code/expressapp/k8s/overlays/staging/sealedsecret.yaml

Repeat for prod with the prod cert and (presumably) the real value:

kubectl --context do-prod -n expressapp-prod create secret generic expressapp-secrets \
  --from-literal=API_TOKEN="REAL_PROD_TOKEN_HERE" \
  --dry-run=client -o yaml | \
kubeseal --cert ~/code/expressapp/k8s/keys/do-prod.pem \
  --format yaml > ~/code/expressapp/k8s/overlays/prod/sealedsecret.yaml

Inspect one. The encryptedData block is opaque ciphertext. Commit both files. Never commit the plaintext.

head ~/code/expressapp/k8s/overlays/staging/sealedsecret.yaml
# apiVersion: bitnami.com/v1alpha1
# kind: SealedSecret
# metadata:
#   name: expressapp-secrets
#   namespace: expressapp-staging
# spec:
#   encryptedData:
#     API_TOKEN: AgBxX9...

Step 4 — Add the SealedSecret to each overlay​

Edit k8s/overlays/staging/kustomization.yaml and k8s/overlays/prod/kustomization.yaml:

resources:
  - ../../base
  - sealedsecret.yaml

Render to confirm it goes into the right namespace:

kustomize build k8s/overlays/staging | grep -A4 'kind: SealedSecret'
kustomize build k8s/overlays/prod    | grep -A4 'kind: SealedSecret'

Step 5 — Reference the resulting Secret from the Deployment​

Patch the Deployment so the app picks up the value as an env var. Edit k8s/base/deployment.yaml:

        envFrom:
          - configMapRef: { name: expressapp }
          - secretRef:    { name: expressapp-secrets }   # <-- new

The SealedSecret's metadata.name and the secretRef.name must match. The controller writes the materialized Secret with that name.

Push to master and watch CI roll out. After the deploy, the secret should exist as a regular Secret in the namespace:

kubectl --context do-staging -n expressapp-staging get secret expressapp-secrets
# NAME                 TYPE     DATA   AGE
# expressapp-secrets   Opaque   1      30s

Reading it back in plaintext (only available if you have RBAC):

kubectl --context do-staging -n expressapp-staging get secret expressapp-secrets \
  -o jsonpath='{.data.API_TOKEN}' | base64 -d
# stg_fake_token_value

Step 6 — Back up the controller signing key (required)​

This is the part most tutorials skip and most teams forget. The controller's private key lives in a Secret (or set of Secrets, one per key) in kube-system labelled sealedsecrets.bitnami.com/sealed-secrets-key=active. Without that key, every SealedSecret in git is undecryptable. Rebuilding the cluster from scratch becomes a re-encryption fire drill.

Create a DO Space for backups​

DigitalOcean Spaces is DO's object storage product, an S3-compatible bucket service. The course uses it as the off-cluster destination for both the sealed-secrets signing key (here) and etcd snapshots (Part 11), so a destroyed cluster can still be rebuilt with its data intact.

Once, for the whole course:

SPACE_NAME="k8slearn-backups-${USER}"
doctl spaces bucket create "$SPACE_NAME" --region fra1

# Spaces access key (also used by Part 11 etcd backups)
doctl auth init  # if you haven't configured a Spaces token
# In the DO control panel: API → Spaces Keys → Generate New Key
# Save them as env vars on your machine:
export SPACES_ACCESS_KEY=...
export SPACES_SECRET_KEY=...

Install the s3cmd or aws CLI to talk to Spaces (Spaces is S3-compatible):

pip install --user awscli   # or: brew install awscli

Configure a profile for Spaces:

mkdir -p ~/.aws
cat <<EOF >> ~/.aws/credentials
[spaces]
aws_access_key_id     = $SPACES_ACCESS_KEY
aws_secret_access_key = $SPACES_SECRET_KEY
EOF

Export and upload the key for each cluster​

export ENDPOINT="https://fra1.digitaloceanspaces.com"
export BUCKET="k8slearn-backups-${USER}"

for ctx in do-staging do-prod; do
  ENV=${ctx#do-}
  kubectl --context "$ctx" -n kube-system get secrets \
    -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
    -o yaml > /tmp/sskey-${ENV}.yaml

  aws --profile spaces --endpoint-url "$ENDPOINT" \
    s3 cp /tmp/sskey-${ENV}.yaml \
    s3://${BUCKET}/sealed-secrets/${ENV}/sskey-$(date +%Y%m%dT%H%M%S).yaml

  # encrypt at rest with age (recommended); or rely on Space being private + scoped keys
  rm -f /tmp/sskey-${ENV}.yaml
done

List what's in the Space:

aws --profile spaces --endpoint-url "$ENDPOINT" \
  s3 ls "s3://${BUCKET}/sealed-secrets/staging/"

In Part 11 you'll schedule this alongside the etcd snapshot so a (etcd, sealed-secrets-key) pair is captured every hour.

Step 7 — Verify the restore actually works (staging)​

A backup that hasn't been restored isn't a backup. On staging only:

# 1. Snapshot the current key (paranoia)
kubectl --context do-staging -n kube-system get secrets \
  -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
  -o yaml > /tmp/sskey-pre-test.yaml

# 2. Delete the controller's signing key Secret(s)
kubectl --context do-staging -n kube-system delete secrets \
  -l sealedsecrets.bitnami.com/sealed-secrets-key=active

# 3. Restart the controller — it will generate a fresh key, but cannot decrypt the old SealedSecrets
kubectl --context do-staging -n kube-system rollout restart deploy/sealed-secrets-controller

# 4. Try to decrypt an existing SealedSecret — it should fail
kubectl --context do-staging -n kube-system logs deploy/sealed-secrets-controller | tail
# "no key could decrypt secret API_TOKEN"

Now restore from the backup:

LATEST=$(aws --profile spaces --endpoint-url "$ENDPOINT" \
  s3 ls "s3://${BUCKET}/sealed-secrets/staging/" | sort | tail -n1 | awk '{print $4}')
aws --profile spaces --endpoint-url "$ENDPOINT" \
  s3 cp "s3://${BUCKET}/sealed-secrets/staging/${LATEST}" /tmp/sskey-restore.yaml

# Apply the restored key
kubectl --context do-staging apply -f /tmp/sskey-restore.yaml
kubectl --context do-staging -n kube-system rollout restart deploy/sealed-secrets-controller

# 5. Verify the SealedSecret decrypts again
kubectl --context do-staging -n expressapp-staging delete secret expressapp-secrets
kubectl --context do-staging apply -k k8s/overlays/staging
kubectl --context do-staging -n expressapp-staging get secret expressapp-secrets -o jsonpath='{.data.API_TOKEN}' | base64 -d
# stg_fake_token_value     <-- success: the restored key produced the right plaintext

Clean up:

rm -f /tmp/sskey-pre-test.yaml /tmp/sskey-restore.yaml

Don't run this on prod. A failed restore there is a production outage. The point of the drill is that you know the procedure works because you've done it.

Step 8 — One SealedSecret per overlay, never shared​

If the same value differs between environments, you keep two files: overlays/staging/sealedsecret.yaml and overlays/prod/sealedsecret.yaml, each encrypted with that cluster's public key. They aren't interchangeable. A SealedSecret encrypted for staging can't be decrypted by the prod controller, by design.

If the same value is the same in both environments, you still keep two files. Consistency beats DRY here; the cost of a single file accidentally being applied to the wrong cluster is much higher than the cost of duplicating ciphertext.

On key rotation​

Sealed Secrets supports key rotation by adding a new key alongside the old one; new encryptions use the new key, decryption still works with the old key until you re-encrypt. The course covers backup, not rotation. See the orientation chapter's Production Gaps.

Completion signal​

  • k8s/overlays/staging/sealedsecret.yaml and k8s/overlays/prod/sealedsecret.yaml are committed in the app repo. Each is encrypted with its cluster's controller key.
  • A kubectl apply -k k8s/overlays/staging produces a real Secret named expressapp-secrets in expressapp-staging, and kubectl get secret expressapp-secrets -o jsonpath='{.data.API_TOKEN}' | base64 -d returns the expected plaintext.
  • The same is true on prod with a different plaintext.
  • The controller's signing key for each cluster is uploaded to s3://k8slearn-backups-${USER}/sealed-secrets/<env>/....
  • The full restore drill above runs on staging in under five minutes and the final step prints the expected plaintext.

You're ready for 09. Observability.