Production Polish

The cluster is safe and observed. This chapter takes the rough edges off the workload itself so real traffic, real node failures, and real bad deploys are uneventful.

Prerequisites

• Cluster health monitoring is in place.
• Observability (Part 9) shows real metrics for the Express app.

Step 1 — Right-size requests and limits with real data

Grafana → Workload overview for expressapp over the last 7 days. Read off:

• p95 CPU per pod
• p95 memory per pod
• peak CPU per pod during a load spike

A reasonable starting point for a small Node service like ours:

resources:
  requests:
    cpu:    100m       # p95 was ~30m; 100m leaves headroom
    memory: 128Mi      # p95 was ~80Mi
  limits:
    memory: 256Mi      # avoid OOM kills under transient spike
    # CPU limits intentionally omitted: throttling-in-place is worse than
    # being "guilty" of bursty consumption. See QoS class discussion below.

This matches the course's Production Polish policy in spec/initial.md, section 3.11: CPU and memory requests on every container, memory limits on every container, CPU limits optional. If you want CPU limits anyway (e.g. for noisy-neighbour isolation in a multi-tenant cluster), add them, set them generously above observed p99, and note why in a comment.

QoS class

Kubernetes picks the QoS class from your resources:

• Guaranteed: every container has requests == limits on both CPU and memory. Last to be evicted.
• Burstable: has requests but no limits, or requests < limits. Middle priority.
• BestEffort: no requests, no limits. First to be evicted.

The Express app is Burstable with the values above (no CPU limit, memory request below limit). That's the course's default per spec section 3.11. It trades a small eviction risk for the ability to burst on CPU during traffic spikes. A latency-sensitive prod service that can't tolerate eviction would go Guaranteed. State your choice in the values and a one-line comment in the deployment patch.

Apply through the existing prod overlay:

k8s/overlays/prod/deployment-patch.yaml:

apiVersion: apps/v1
kind: Deployment
metadata: { name: expressapp }
spec:
  template:
    spec:
      containers:
        - name: app
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits:   {            memory: 256Mi }   # CPU intentionally unbounded (Burstable)

Step 2 — PodDisruptionBudget

The PDB keeps the Express app available during voluntary disruptions like node drains and rolling updates.

Add k8s/base/pdb.yaml:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: expressapp
spec:
  minAvailable: 50%
  selector:
    matchLabels: { app.kubernetes.io/name: expressapp }

Reference it in k8s/base/kustomization.yaml:

resources:
  - serviceaccount.yaml
  - configmap.yaml
  - deployment.yaml
  - service.yaml
  - ingress.yaml
  - hpa.yaml
  - servicemonitor.yaml
  - pdb.yaml                # <-- new in Part 12

With replicas: 3 in prod and minAvailable: 50%, the cluster won't let more than one prod pod be evicted at a time.

Step 3 — Graceful SIGTERM

The Express app from Part 2 already handles SIGTERM. Re-check it:

function shutdown(signal) {
  console.log(`${signal} received, draining`);
  server.close(() => process.exit(0));     // stop accepting, drain in-flight
  setTimeout(() => process.exit(1), 10_000).unref();   // hard exit fallback
}
process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown('SIGINT'));

Kubernetes' shutdown sequence:

1. The pod's endpoint is removed from the Service (the kube-proxy reconciles iptables / Cilium reconciles eBPF maps).
2. The container receives SIGTERM.
3. After terminationGracePeriodSeconds (default 30s), the container receives SIGKILL.

A common bug: SIGTERM arrives before the endpoint removal is visible to upstream proxies, so traffic still routes to the dying pod. The fix is a preStop sleep that gives the cluster time to converge.

k8s/base/deployment.yaml:

        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "sleep 5"]
      terminationGracePeriodSeconds: 30

5 seconds of preStop sleep + 25 seconds of actual drain is plenty for an Express service.

Step 4 — Probe tuning

Bad probes are the most common source of false outages.

Probe	Failure cost	Tuning posture
Liveness	Pod restart (real outage if too tight)	Loose: failureThreshold ≥ 3, only on a cheap endpoint (/healthz that returns ok without dependencies)
Readiness	Pod taken out of the Service (no outage)	Tighter: marks a cold-starting or upstream-down pod as not ready
Startup	Replaces liveness while the app is starting	Generous total budget for cold starts

Update the base:

          startupProbe:
            httpGet: { path: /readyz, port: http }
            periodSeconds: 1
            failureThreshold: 30        # gives 30s of cold start before liveness takes over
          readinessProbe:
            httpGet: { path: /readyz, port: http }
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 10
            failureThreshold: 3

Rule of thumb: /healthz shouldn't depend on the database, cache, or any other moving part. /readyz is the right place to express "I'm healthy and my dependencies are reachable".

Step 5 — Rolling update strategy

Set RollingUpdate parameters explicitly so a deploy never drops the service below the PDB:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1            # 1 extra pod above replicas during rollout
      maxUnavailable: 0      # never go below desired count

With prod replicas: 3, maxSurge: 1 means at most 4 pods exist briefly. maxUnavailable: 0 means no pod is taken down until its replacement is Ready. Combined with the PDB, voluntary disruptions and rollouts cost zero serving capacity.

Step 6 — topologySpreadConstraints

A topologySpreadConstraint tells the scheduler to fan pods out across some failure domain — hostnames, availability zones, regions. Without it, the scheduler is free to stack all three Express pods on one node, and losing that node is an outage. With it, the scheduler prefers (or insists on) one pod per node.

Make sure replicas land on different nodes:

k8s/base/deployment.yaml (under spec.template.spec):

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app.kubernetes.io/name: expressapp }

ScheduleAnyway because the prod cluster has only two workers. DoNotSchedule would leave the 3rd replica Pending if both workers already have one. Switch to DoNotSchedule once you have ≥3 workers.

If you ever add zones (e.g. multi-DC), add a second constraint with topologyKey: topology.kubernetes.io/zone.

Step 7 — Drain a node, observe no impact

This is the proof. Pick a worker on the prod cluster (do-prod) and drain it. The PDB + maxUnavailable + topology spread should make this transparent to traffic.

In a separate terminal, run a continuous probe:

while true; do
  printf '%s ' "$(date +%H:%M:%S)"
  curl -s -o /dev/null -w '%{http_code}\n' "https://app.example.com/" -m 3 || echo "FAIL"
  sleep 1
done

In the main terminal:

NODE=$(kubectl --context do-prod get nodes -l '!node-role.kubernetes.io/control-plane' -o jsonpath='{.items[0].metadata.name}')
kubectl --context do-prod drain "$NODE" --ignore-daemonsets --delete-emptydir-data
# pods are evicted one at a time, replacements come up on the other worker
kubectl --context do-prod -n expressapp-prod rollout status deploy/expressapp
kubectl --context do-prod uncordon "$NODE"

The probe log should be a steady stream of 200s. If you see a FAIL or a non-200, dig in (probe tuning, preStop sleep, Traefik upstream/retry settings). Don't move on until a drain is clean.

Step 8 — Rollback drill

The rollback story has two ingredients:

1. Image immutability: production references the SHA-tagged image, so rolling back is a matter of pointing at an earlier SHA.
2. kubectl rollout undo restores the previous ReplicaSet's image and pod spec in one command.

Simulate a bad deploy on staging:

# Push an obviously broken commit and let CI deploy it.
echo 'process.exit(1)' >> app/server.js
git commit -am "deliberately broken"
git push origin master
gh run watch
# the rollout fails because /readyz never returns 200; CI 'rollout status' times out

Recover:

kubectl --context do-staging -n expressapp-staging rollout undo deploy/expressapp
kubectl --context do-staging -n expressapp-staging rollout status deploy/expressapp
# back to the previous revision in <60s

Revert the bad commit in git:

git revert HEAD --no-edit
git push origin master

Document the runbook in cluster-bootstrap/docs/runbook-rollback.md:

1. Identify revision: kubectl -n <ns> rollout history deploy/expressapp
2. Roll back:        kubectl -n <ns> rollout undo deploy/expressapp [--to-revision=N]
3. Watch:            kubectl -n <ns> rollout status deploy/expressapp
4. Revert in git:    git revert <bad-sha> && git push

The git revert step matters. The cluster is back to good, but if you skip the revert, the next CI run re-deploys the same bad code.

Step 9 — Production polish checklist

Before declaring the chapter done, confirm each line:

• requests and limits set on every container, justified by Grafana
• QoS class explicitly chosen and commented
• PodDisruptionBudget keeps ≥50% available
• SIGTERM handled in the app, preStop sleep added, terminationGracePeriodSeconds matches
• Startup / readiness / liveness probes tuned for cold starts and dependency outages
• RollingUpdate maxSurge: 1, maxUnavailable: 0
• topologySpreadConstraints spread replicas across nodes
• A drained node produced zero failed requests on the live probe
• Rollback runbook checked in; a deliberate bad deploy rolled back in under two minutes on staging

Completion signal

• kubectl --context do-prod drain <a worker> produced zero non-200 responses on the live request probe.
• A deliberately broken deploy to staging was rolled back from the documented procedure in under two minutes.
• kubectl -n expressapp-prod get pdb expressapp -o jsonpath='{.status.currentHealthy}/{.status.desiredHealthy}' returns 3/3.

You're ready for 13. GitOps with ArgoCD.