Cluster Health and Backups

The Express app is stateless, but the cluster itself isn't. Two things must survive a destroyed control-plane droplet:

etcd: the state of every Kubernetes object you've applied.
The sealed-secrets controller's signing key. Without it, every SealedSecret you've committed to git becomes undecryptable.

A backup of one without the other is useless for rebuilding. This chapter watches the control plane and the nodes for trouble, takes regular snapshots of both, and rehearses a restore on staging to prove the procedure works end to end.

Prerequisites

Hardening is in place.
You can SSH to the control-plane droplets of both clusters.
The DigitalOcean Space and Spaces access keys from Part 8 exist.

Step 1 — kubelet healthz and component checks

Each control-plane component exposes a /healthz. Walk through them once so you know where to look:

ssh root@${CP_IP}
curl -sk https://localhost:6443/healthz                # API server
curl -sk https://localhost:10257/healthz               # kube-controller-manager
curl -sk https://localhost:10259/healthz               # kube-scheduler
curl -sk https://localhost:10250/healthz               # kubelet
# etcd healthz is gated by the same client certs the API server uses:
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

Each should return ok (or, for etcd, healthy: successfully committed proposal).

Step 2 — node-problem-detector

node-problem-detector (NPD) is a small agent that watches a node's kernel logs and other host signals, then writes Kubernetes Events and node conditions when it sees known failure modes (disk pressure, kernel deadlocks, oom-killer activity, etc.). It runs as a DaemonSet — the workload kind that automatically schedules one pod per node, the same shape Cilium and the DO CSI driver use. Whenever you see a new node in the cluster, every DaemonSet shows up there too without any manual intervention.

Install per cluster from cluster-bootstrap:

helm repo add node-problem-detector https://charts.deliveryhero.io/
helm repo update

for CTX in do-staging do-prod; do
  helm --kube-context "$CTX" upgrade --install node-problem-detector \
    deliveryhero/node-problem-detector \
    --version 2.3.13 \
    --namespace kube-system \
    --set metrics.enabled=true \
    --set settings.log_monitors[0]=/config/kernel-monitor.json \
    --set settings.log_monitors[1]=/config/docker-monitor.json
done

Confirm the DaemonSet:

kubectl --context do-staging -n kube-system get ds node-problem-detector
# DESIRED=3 CURRENT=3 READY=3

NPD events show up as Kubernetes events:

kubectl --context do-staging get events -A --field-selector source=node-problem-detector

Add a Prometheus scrape (NPD exposes :20257/metrics). This is already automatic if metrics.enabled=true because the chart creates a ServiceMonitor.

Step 3 — Cluster-health dashboard and alerts

Most rules you want are already in the kube-prometheus-stack defaults installed in Part 9. Walk through kubectl -n monitoring get prometheusrules -o name and skim the names. The notable ones for cluster health are:

Alert	What it tells you
`KubeNodeNotReady`	A node has been `NotReady` > 15m
`KubeAPIErrorBudgetBurn`	Multi-window SLO burn on API errors
`KubeletDown`	A kubelet has not scraped in 15m
`etcdMembersDown`	etcd quorum is at risk
`etcdHighNumberOfLeaderChanges`	leader churn
`KubePodCrashLooping`	pod restart loop

A control-plane dashboard JSON is shipped by the chart (grafana-dashboards-kubernetes, k8s-views-global, and friends). Open Grafana, then Kubernetes / API server, and confirm latencies and error rates are graphed.

Step 4 — Build the combined backup script

Both halves of the backup must land in the same scheduled run so they share a single timestamp.

cluster-bootstrap/backup/backup.sh (runs on the control plane droplet):

#!/usr/bin/env bash
set -euo pipefail

ENV="${ENV:?ENV is required (staging|prod)}"
BUCKET="${BUCKET:?Spaces bucket name}"
ENDPOINT="${ENDPOINT:-https://fra1.digitaloceanspaces.com}"
KCFG="/etc/kubernetes/admin.conf"

STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
WORKDIR="$(mktemp -d)"
trap 'rm -rf "$WORKDIR"' EXIT

# 1. etcd snapshot
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save "$WORKDIR/etcd-${STAMP}.db"

ETCDCTL_API=3 etcdctl \
  snapshot status --write-out=json "$WORKDIR/etcd-${STAMP}.db" \
  > "$WORKDIR/etcd-${STAMP}.meta.json"

# 2. sealed-secrets controller signing key(s)
kubectl --kubeconfig="$KCFG" -n kube-system get secrets \
  -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
  -o yaml > "$WORKDIR/sskey-${STAMP}.yaml"

# 3. ship to Spaces — both files for the same STAMP
aws --profile spaces --endpoint-url "$ENDPOINT" \
  s3 cp "$WORKDIR/etcd-${STAMP}.db"        "s3://${BUCKET}/${ENV}/etcd/etcd-${STAMP}.db"
aws --profile spaces --endpoint-url "$ENDPOINT" \
  s3 cp "$WORKDIR/etcd-${STAMP}.meta.json" "s3://${BUCKET}/${ENV}/etcd/etcd-${STAMP}.meta.json"
aws --profile spaces --endpoint-url "$ENDPOINT" \
  s3 cp "$WORKDIR/sskey-${STAMP}.yaml"     "s3://${BUCKET}/${ENV}/sealed-secrets/sskey-${STAMP}.yaml"

echo "OK ${STAMP}"

Install etcdctl and kubectl are already present on a kubeadm control plane. Install awscli:

ssh root@${CP_IP} 'apt-get update && apt-get install -y awscli'

Configure the Spaces profile on the control plane (read the values from your doctl and Spaces tokens):

ssh root@${CP_IP} 'mkdir -p ~/.aws && cat > ~/.aws/credentials' <<EOF
[spaces]
aws_access_key_id     = ${SPACES_ACCESS_KEY}
aws_secret_access_key = ${SPACES_SECRET_KEY}
EOF
ssh root@${CP_IP} 'chmod 600 ~/.aws/credentials'

Copy backup.sh:

scp cluster-bootstrap/backup/backup.sh root@${CP_IP}:/usr/local/sbin/backup.sh
ssh root@${CP_IP} 'chmod 700 /usr/local/sbin/backup.sh'

Test it once manually:

ssh root@${CP_IP} 'ENV=staging BUCKET=k8slearn-backups-yourname /usr/local/sbin/backup.sh'

Confirm both files arrived in Spaces with the same timestamp:

aws --profile spaces --endpoint-url https://fra1.digitaloceanspaces.com \
  s3 ls "s3://k8slearn-backups-yourname/staging/etcd/" | tail
aws --profile spaces --endpoint-url https://fra1.digitaloceanspaces.com \
  s3 ls "s3://k8slearn-backups-yourname/staging/sealed-secrets/" | tail

Step 5 — Schedule the backup

Two options, both fine. Use a systemd timer on the control-plane droplet, since it lives on the same host that has etcd's client certs.

/etc/systemd/system/k8s-backup.service:

[Unit]
Description=k8slearn backup (etcd snapshot + sealed-secrets key)

[Service]
Type=oneshot
Environment=ENV=staging
Environment=BUCKET=k8slearn-backups-yourname
Environment=ENDPOINT=https://fra1.digitaloceanspaces.com
ExecStart=/usr/local/sbin/backup.sh

/etc/systemd/system/k8s-backup.timer:

[Unit]
Description=Hourly k8slearn backup

[Timer]
OnCalendar=hourly
RandomizedDelaySec=180
Persistent=true

[Install]
WantedBy=timers.target

Enable:

ssh root@${CP_IP} '
  systemctl daemon-reload
  systemctl enable --now k8s-backup.timer
  systemctl list-timers k8s-backup.timer
'

Repeat on the prod control plane with Environment=ENV=prod.

Verify after the first scheduled run:

ssh root@${CP_IP} 'journalctl -u k8s-backup.service -n 50 --no-pager'

Step 6 — Retention policy

Spaces has no built-in lifecycle UI as of writing, but the S3 lifecycle API works. Apply a policy keeping 7 days of hourly snapshots, then daily for 30 days:

cluster-bootstrap/backup/lifecycle.json:

{
  "Rules": [
    {
      "ID": "expire-old-etcd",
      "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "Expiration": { "Days": 7 }
    }
  ]
}

aws --profile spaces --endpoint-url https://fra1.digitaloceanspaces.com \
  s3api put-bucket-lifecycle-configuration \
  --bucket k8slearn-backups-yourname \
  --lifecycle-configuration file://cluster-bootstrap/backup/lifecycle.json

(This is a simple policy. A real retention tier would use object tags to keep the 00:00 snapshot for longer.)

Step 7 — Restore drill (staging only)

This is the proof that the backups are useful. Do this on the staging cluster, never on prod.

Pick a snapshot

LATEST_ETCD=$(aws --profile spaces --endpoint-url https://fra1.digitaloceanspaces.com \
  s3 ls "s3://k8slearn-backups-yourname/staging/etcd/" | grep '\.db$' | sort | tail -n1 | awk '{print $4}')
LATEST_SS=$(aws --profile spaces --endpoint-url https://fra1.digitaloceanspaces.com \
  s3 ls "s3://k8slearn-backups-yourname/staging/sealed-secrets/" | sort | tail -n1 | awk '{print $4}')
echo "etcd: $LATEST_ETCD"
echo "key : $LATEST_SS"

Break the cluster

On the staging control plane. A quick aside: static pods are pods defined as YAML files in /etc/kubernetes/manifests/ and started directly by the kubelet — not by the API server, not by a Deployment. kubeadm uses them for the control plane (kube-apiserver, etcd, kube-controller-manager, kube-scheduler), which is why moving that directory out of the way is how you cleanly stop them.

ssh root@${STAGING_CP_IP}
# 1. Stop static-pod components so they don't fight you
mv /etc/kubernetes/manifests /etc/kubernetes/manifests.bak
crictl ps     # wait until kube-apiserver / etcd containers exit (~30s)

# 2. Wreck etcd's data
systemctl stop kubelet
mv /var/lib/etcd /var/lib/etcd.broken

# 3. Delete the sealed-secrets keypair on the control plane. Kubelet is stopped,
#    so do this against a fresh local etcd later, or just confirm Spaces has the key.

Restore etcd

# 4. Pull the snapshot to the control plane
aws --profile spaces --endpoint-url https://fra1.digitaloceanspaces.com \
  s3 cp "s3://k8slearn-backups-yourname/staging/etcd/${LATEST_ETCD}" /var/lib/etcd-snapshot.db

# 5. Restore into a fresh etcd data directory
ETCDCTL_API=3 etcdctl snapshot restore /var/lib/etcd-snapshot.db \
  --name "$(hostname)" \
  --initial-cluster "$(hostname)=https://$(hostname -I | awk '{print $1}'):2380" \
  --initial-cluster-token kubeadm-etcd-token \
  --initial-advertise-peer-urls "https://$(hostname -I | awk '{print $1}'):2380" \
  --data-dir /var/lib/etcd

chown -R 0:0 /var/lib/etcd

# 6. Restart the static-pod components
mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests
systemctl start kubelet

Wait ~60s and verify:

kubectl --context do-staging get nodes
# 3 Ready (eventually)
kubectl --context do-staging -n expressapp-staging get deploy
# expressapp present

Restore the sealed-secrets key

# 7. Apply the saved keypair Secret(s)
aws --profile spaces --endpoint-url https://fra1.digitaloceanspaces.com \
  s3 cp "s3://k8slearn-backups-yourname/staging/sealed-secrets/${LATEST_SS}" /tmp/sskey-restore.yaml
kubectl --context do-staging apply -f /tmp/sskey-restore.yaml
kubectl --context do-staging -n kube-system rollout restart deploy/sealed-secrets-controller

Critical proof step

# 8. Confirm a previously-committed SealedSecret still produces the right plaintext.
kubectl --context do-staging -n expressapp-staging delete secret expressapp-secrets
kubectl --context do-staging apply -k k8s/overlays/staging
kubectl --context do-staging -n expressapp-staging get secret expressapp-secrets \
  -o jsonpath='{.data.API_TOKEN}' | base64 -d
# stg_fake_token_value     <-- success: cluster is back, secrets decrypt with the restored key

If this last command prints the wrong value or the controller logs no key could decrypt secret, the restore is not done. Fall back to the pre-test snapshot and investigate.

Step 8 — Runbook entry

Commit a runbook page to cluster-bootstrap/docs/runbook-restore.md summarising:

which Space, bucket layout
the exact sequence above
typical timings (you measured them in the drill)
the critical proof step

This is the document the on-call operator reads at 03:00 when the prod control plane is broken. Treat it like code.

Completion signal

journalctl -u k8s-backup.service on each control plane shows hourly successful runs.
s3://k8slearn-backups-${USER}/staging/etcd/ and s3://k8slearn-backups-${USER}/staging/sealed-secrets/ each contain at least 24 snapshots from the last 24 hours.
You've walked through the restore on staging end to end, including the critical proof step where a committed SealedSecret decrypts to the expected plaintext after the controller key is restored.
cluster-bootstrap/docs/runbook-restore.md is checked in.

You're ready for 12. Production Polish.

Prerequisites​

Step 1 — kubelet healthz and component checks​

Step 2 — node-problem-detector​

Step 3 — Cluster-health dashboard and alerts​

Step 4 — Build the combined backup script​

Step 5 — Schedule the backup​

Step 6 — Retention policy​

Step 7 — Restore drill (staging only)​

Pick a snapshot​

Break the cluster​

Restore etcd​

Restore the sealed-secrets key​

Critical proof step​

Step 8 — Runbook entry​

Completion signal​