Hardening: RBAC, NetworkPolicies, Pod Security, Image Scanning

The cluster works. This chapter makes it safe to leave on. You'll shrink permissions to the minimum, block traffic no workload needs, enforce a strict Pod Security Standard on the app namespaces, and make image scans a merge-blocking check instead of a best-effort one.

Prerequisites

Observability is live so you can see the effect of policy changes (especially Hubble flows).

Step 1 — RBAC: shrink the CI deploy account

The CI deploy account was already created in Part 7 with a namespace-scoped Role. Verify it still can't escape its namespace:

for CTX in do-staging do-prod; do
  ENV=${CTX#do-}
  echo "=== $CTX ==="
  kubectl --context "$CTX" --as=system:serviceaccount:expressapp-${ENV}:ci-deployer auth can-i create namespaces                              # no
  kubectl --context "$CTX" --as=system:serviceaccount:expressapp-${ENV}:ci-deployer auth can-i get   secrets -n kube-system                  # no
  kubectl --context "$CTX" --as=system:serviceaccount:expressapp-${ENV}:ci-deployer auth can-i apply deployments -n expressapp-${ENV}        # yes
  kubectl --context "$CTX" --as=system:serviceaccount:expressapp-${ENV}:ci-deployer auth can-i delete deployments -n expressapp-${ENV}       # ? — see below
done

delete wasn't granted in Part 7. Add it deliberately if you want CI to remove resources during a kustomize resource drop; leave it out if you prefer manual cleanup.

Audit any ClusterRoleBindings that grant cluster-admin to a ServiceAccount you don't recognise (Helm charts sometimes do this in defaults):

for CTX in do-staging do-prod; do
  echo "=== $CTX ==="
  kubectl --context "$CTX" get clusterrolebindings -o json \
    | jq -r '.items[] | select(.roleRef.name=="cluster-admin") | "\(.metadata.name) -> \(.subjects[]|"\(.kind)/\(.namespace // "-")/\(.name)")"'
done

Anything you didn't intentionally grant should be investigated and trimmed.

Step 2 — App ServiceAccount with no permissions

The Express pod doesn't call the Kubernetes API; it shouldn't have a token at all. The ServiceAccount you wrote in k8s/base/serviceaccount.yaml already sets automountServiceAccountToken: false. Confirm at runtime:

kubectl --context do-staging -n expressapp-staging get pod -l app.kubernetes.io/name=expressapp -o jsonpath='{.items[0].spec.automountServiceAccountToken}'
# false
kubectl --context do-staging -n expressapp-staging exec deploy/expressapp -- ls /var/run/secrets/kubernetes.io/serviceaccount 2>&1 | head
# No such file or directory

If a token still mounts, the value also needs to be set at the pod spec level. Kustomize merges, but a null field in the deployment patch can re-enable mounting. Belt and suspenders:

spec:
  template:
    spec:
      automountServiceAccountToken: false

Step 3 — Cilium default-deny NetworkPolicies

A NetworkPolicy is a firewall rule expressed as a Kubernetes object: which pods can talk to which other pods (or external endpoints), on which ports. Without any policy, every pod can reach every other pod, which is fine for learning and terrible for production. Kubernetes ships a stock NetworkPolicy kind; Cilium adds a richer CiliumNetworkPolicy that can also match on DNS names, L7 fields, and namespace labels.

The pattern is "default-deny + named allows": once a namespace has a deny-everything policy, traffic only flows if another policy explicitly permits it. That's the only policy that actually changes behaviour. Apply it per app namespace:

cluster-bootstrap/network-policy/default-deny.yaml:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny
spec:
  endpointSelector: {}                # everything in the namespace
  ingress: []                         # deny all ingress not explicitly allowed
  egress: []                          # deny all egress not explicitly allowed

The matching allow policy, also applied per namespace:

cluster-bootstrap/network-policy/expressapp-allow.yaml:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: expressapp-allow
spec:
  endpointSelector:
    matchLabels: { app.kubernetes.io/name: expressapp }

  ingress:
    # 1. Traefik talks to the app
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: traefik
      toPorts:
        - ports: [ { port: "3000", protocol: TCP } ]

    # 2. Prometheus scrapes /metrics
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: monitoring
            app.kubernetes.io/name: prometheus
      toPorts:
        - ports: [ { port: "3000", protocol: TCP } ]

  egress:
    # cluster DNS
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports: [ { port: "53", protocol: UDP }, { port: "53", protocol: TCP } ]
    # uncomment if the app needs outbound HTTPS:
    # - toFQDNs:
    #     - matchPattern: "*.example-upstream.com"
    #   toPorts:
    #     - ports: [ { port: "443", protocol: TCP } ]

Apply per environment:

for CTX in do-staging do-prod; do
  ENV=${CTX#do-}
  kubectl --context "$CTX" -n expressapp-${ENV} apply -f cluster-bootstrap/network-policy/default-deny.yaml
  kubectl --context "$CTX" -n expressapp-${ENV} apply -f cluster-bootstrap/network-policy/expressapp-allow.yaml
done

Step 4 — Observe denials with Hubble

Hubble is Cilium's flow-observability layer (you enabled it back in Part 3). It captures every L3/L4/L7 connection the CNI handles and gives you a tcpdump-style stream of "who tried to talk to whom and was it allowed."

Without observability, NetworkPolicies are dangerous. You can lock yourself out and not understand why. Hubble shows the flows.

cilium hubble port-forward&
sleep 2
hubble observe --namespace expressapp-staging --verdict DROPPED --follow

Hit https://app.staging.example.com/ from your laptop. You should see one ingress flow from the Traefik pod allowed. Try kubectl exec from the app pod to curl https://example.com. You should see a DROPPED egress flow because external HTTPS isn't in the allow list.

If something legitimate is dropping, add an egress rule with the narrowest selector that fixes it. Adding 0.0.0.0/0 to egress and calling it done would defeat the point.

Step 5 — Pod Security Standards: `restricted`

Pod Security Standards (PSS) are three named profiles — privileged, baseline, restricted — that the upstream community defined as portable shorthand for "what a pod is allowed to do." Pod Security Admission (PSA) is the in-cluster controller that enforces them: you label a namespace, and any pod that violates the profile is rejected before it ever runs. restricted is the strict one — no root, no privilege escalation, no host networking, must drop all capabilities, must use a non-default seccomp profile.

The corresponding pod field is securityContext, which appears at both the pod level (settings like runAsNonRoot, fsGroup) and the container level (readOnlyRootFilesystem, capabilities.drop).

Label both app namespaces:

for CTX in do-staging do-prod; do
  ENV=${CTX#do-}
  kubectl --context "$CTX" label --overwrite namespace expressapp-${ENV} \
    pod-security.kubernetes.io/enforce=restricted \
    pod-security.kubernetes.io/audit=restricted \
    pod-security.kubernetes.io/warn=restricted
done

The Deployment must satisfy restricted to be admitted. Add a securityContext to k8s/base/deployment.yaml:

      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: [ "ALL" ]
          # ...
          volumeMounts:
            - { name: tmp,    mountPath: /tmp }
            - { name: home,   mountPath: /home/app }
      volumes:
        - { name: tmp,    emptyDir: {} }
        - { name: home,   emptyDir: {} }

readOnlyRootFilesystem: true is what most apps trip on. Node/Express writes to /tmp during operation, and the app user's home dir is sometimes needed. Mount emptyDirs for both.

Build a new image (the Dockerfile from Part 2 already creates and uses an app user). Push and deploy. Verify the pod admits cleanly:

kubectl --context do-staging -n expressapp-staging get pods -l app.kubernetes.io/name=expressapp
# Running, no admission warnings

Confirm a deliberately bad pod is rejected:

kubectl --context do-staging -n expressapp-staging run badpod --image=busybox --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"badpod","image":"busybox","securityContext":{"runAsUser":0}}]}}'
# Error from server (Forbidden): violates PodSecurity "restricted:latest":
#   runAsNonRoot != true, runAsUser = 0

Step 6 — Promote Trivy from "best-effort" to "required"

The trivy job is already in pr.yml from Part 6. Branch protection already requires it. Make the policy explicit and document exceptions in a checked-in file.

app/.trivyignore:

# Each line is a CVE ID we have decided to accept. Include who decided, when, and why.
# Example:
# CVE-2024-12345  — accepted 2026-03-01 by @alice — only affects optional libcurl plugin we do not enable.

Treat this file like a brownout list: every entry has an owner, a date, and a reason. Re-review quarterly.

The course's Trivy job runs:

severity: HIGH,CRITICAL
ignore-unfixed: true
The .trivyignore is honored automatically when the file is present at the workspace root.

Tighten the run by adding the same scan to the image build workflow so a broken master is caught even if a PR slipped through:

# in .github/workflows/build.yml, after the Kaniko step:
      - name: Trivy on built image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          image-ref: ${{ steps.tags.outputs.image }}:${{ steps.tags.outputs.sha }}
          severity: HIGH,CRITICAL
          ignore-unfixed: true
          exit-code: '1'

Step 7 — Document the Kaniko build hardening

The build pipeline you already wrote is hardened by construction:

Runs as a normal container action on a GitHub-hosted runner. No privileged mode, no Docker socket mounted.
Uses Chainguard's Kaniko image (cgr.dev/chainguard/kaniko), which is minimal, frequently rebuilt, and signed.
Builds the runtime image as a non-root app user (Dockerfile from Part 2).
Caches via DOCR (--cache-repo=...) rather than CI-runner local cache, so cache poisoning attempts are visible.

Make sure no future PR weakens this. Add a CODEOWNERS rule for the workflow file:

# .github/CODEOWNERS
/.github/workflows/build.yml  @your-github-handle

Step 8 — Smoke test the whole hardened pipeline

# (a) An apply that needs to escalate should fail.
kubectl --context do-staging --as=system:serviceaccount:expressapp-staging:ci-deployer \
  -n kube-system get secret
# error: forbidden

# (b) A pod from outside the traefik namespace cannot talk to the app on :3000.
kubectl --context do-staging -n default run probe --rm -it --restart=Never --image=alpine -- sh -c \
  'apk add --no-cache curl >/dev/null; curl -s -m 3 http://expressapp.expressapp-staging.svc.cluster.local/ || echo "DENIED"'
# DENIED

# (c) Traefik still reaches the app — the live site still loads.
curl -sI https://app.staging.example.com/ | head -n1
# HTTP/2 200

# (d) A "restricted" violation is rejected at admission.
kubectl --context do-staging -n expressapp-staging run badpod --image=busybox --restart=Never \
  --overrides='{"spec":{"securityContext":{"runAsUser":0}}}' 2>&1 | grep -i forbidden
# Forbidden: ...

# (e) A PR introducing a HIGH-severity image is blocked at merge.
#     (Add an old known-vulnerable dep to package.json in a throwaway branch and watch trivy fail.)

Completion signal

The CI deploy account passes (a) and can't read or modify anything outside its namespace.
A pod in any namespace other than traefik and monitoring can't reach the Express app on port 3000 (b).
The Express app still serves traffic through the ingress (c).
A restricted-violating pod is rejected at admission (d).
A PR introducing a HIGH or CRITICAL fixable CVE is blocked at merge (e), and the .trivyignore is the only path to accept one (with a documented justification).
Hubble shows zero DROPPED flows for the normal request path through traefik → expressapp.

You're ready for 11. Cluster Health and Backups.

Prerequisites​

Step 1 — RBAC: shrink the CI deploy account​

Step 2 — App ServiceAccount with no permissions​

Step 3 — Cilium default-deny NetworkPolicies​

Step 4 — Observe denials with Hubble​

Step 5 — Pod Security Standards: restricted​

Step 6 — Promote Trivy from "best-effort" to "required"​

Step 7 — Document the Kaniko build hardening​

Step 8 — Smoke test the whole hardened pipeline​

Completion signal​