Observability: Prometheus, VictoriaMetrics, Grafana

You can't operate a production cluster you can't see. This chapter installs a full metrics stack, plugs the Express app into it, and gives you the dashboards and alerts you'll actually look at when something is wrong. All four platform components (Prometheus, VictoriaMetrics, Grafana, Alertmanager) run on do-block-storage PVCs. This is the deliberate platform-state exception from the orientation: the application stays stateless; platform components must persist.

Prerequisites

• Sealed Secrets are working in both clusters (Part 8).
• The Express app exposes /metrics via prom-client (already done in Part 2).

Stack overview

Component	What it does
Prometheus Operator (from kube-prometheus-stack)	Provides ServiceMonitor, PodMonitor, Alertmanager, PrometheusRule CRDs. The app's ServiceMonitor works only because this is installed first.
Prometheus	Scrapes targets, evaluates alert rules, retains a short window (we use 15d).
VictoriaMetrics	Long-term store via Prometheus remoteWrite. We keep 12 months.
Alertmanager	Routes firing alerts to email/Slack/webhook.
Grafana	Dashboards over both Prometheus (recent) and VictoriaMetrics (history).
kube-state-metrics	Per-object state (kube_deployment_status_replicas etc.).
node-exporter	Per-node hardware/OS metrics.

Step 1 — Install kube-prometheus-stack

Once per cluster, from cluster-bootstrap. The chart installs the Operator and all the components above.

Create a values file cluster-bootstrap/observability/values-${ENV}.yaml:

fullnameOverride: kps

prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: "8GB"
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: do-block-storage
          accessModes: [ ReadWriteOnce ]
          resources: { requests: { storage: 10Gi } }
    # We add remoteWrite to VictoriaMetrics in step 2.
    remoteWrite: []
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    resources:
      requests: { cpu: 100m, memory: 512Mi }
      limits:   { cpu: 1,    memory: 2Gi }

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: do-block-storage
          accessModes: [ ReadWriteOnce ]
          resources: { requests: { storage: 1Gi } }
  config:
    route:
      receiver: 'default'
      group_by: ['alertname','namespace']
      repeat_interval: 12h
    receivers:
      - name: 'default'
        # swap in your own webhook later; default-off email is fine for the course
        webhook_configs: []

grafana:
  admin:
    existingSecret: grafana-admin       # we will seal this in step 4
  persistence:
    enabled: true
    storageClassName: do-block-storage
    size: 5Gi
  sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard
      searchNamespace: ALL
  service:
    type: ClusterIP
  ingress:
    enabled: true
    ingressClassName: traefik
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    hosts: [ "grafana.${HOST_BASE}" ]    # rendered with envsubst when applied
    tls:
      - hosts: [ "grafana.${HOST_BASE}" ]
        secretName: grafana-tls

(With ${HOST_BASE} set to staging.example.com or example.com per environment.)

Install:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

for ENV in staging prod; do
  CTX=do-${ENV}
  HOST_BASE=$( [ "$ENV" = "staging" ] && echo "staging.example.com" || echo "example.com" )
  envsubst < cluster-bootstrap/observability/values-${ENV}.yaml > /tmp/values.yaml
  helm --kube-context "$CTX" upgrade --install kps prometheus-community/kube-prometheus-stack \
    --version 65.5.0 \
    --namespace monitoring --create-namespace \
    -f /tmp/values.yaml
done

Verify the CRDs and pods:

for CTX in do-staging do-prod; do
  echo "=== $CTX ==="
  kubectl --context "$CTX" get crd servicemonitors.monitoring.coreos.com prometheuses.monitoring.coreos.com alertmanagers.monitoring.coreos.com prometheusrules.monitoring.coreos.com
  kubectl --context "$CTX" -n monitoring get pods
done

Expected: all Running. The prometheus-kps-prometheus-0 pod takes ~2 minutes on first start while it claims its PVC.

Step 2 — Install VictoriaMetrics (single-node)

A single-node VictoriaMetrics is plenty for the course's data volume. The HA/cluster variant exists for production at scale.

helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update

cat > /tmp/vm-values.yaml <<'EOF'
server:
  retentionPeriod: "12"        # months
  persistentVolume:
    enabled: true
    storageClass: do-block-storage
    size: 20Gi
  resources:
    requests: { cpu: 100m, memory: 512Mi }
    limits:   { cpu: 1,    memory: 2Gi }
EOF

for CTX in do-staging do-prod; do
  helm --kube-context "$CTX" upgrade --install vm vm/victoria-metrics-single \
    --version 0.12.0 \
    --namespace monitoring \
    -f /tmp/vm-values.yaml
done

Wire Prometheus' remoteWrite at the VM service URL. Re-render the kube-prometheus-stack values with this addition:

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: http://vm-victoria-metrics-single-server:8428/api/v1/write
        queueConfig:
          maxShards: 5
          minShards: 1
          capacity: 10000

Re-run helm upgrade kps .... Check that data is flowing in:

kubectl --context do-staging -n monitoring port-forward svc/vm-victoria-metrics-single-server 8428 &
sleep 1
curl -s 'http://localhost:8428/api/v1/query?query=up' | jq '.data.result | length'
# >0 within a minute
kill %1

Step 3 — Add a ServiceMonitor for the Express app

This is the first CRD-typed resource the app repo references. It only works because the Prometheus Operator from step 1 is installed.

Create k8s/base/servicemonitor.yaml:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: expressapp
  labels:
    release: kps                       # matches the Prometheus' serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: expressapp
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

Append it to k8s/base/kustomization.yaml:

resources:
  - serviceaccount.yaml
  - configmap.yaml
  - deployment.yaml
  - service.yaml
  - ingress.yaml
  - hpa.yaml
  - servicemonitor.yaml     # <-- new in Part 9

Push to master; staging deploys automatically. After the rollout, confirm Prometheus is scraping:

kubectl --context do-staging -n monitoring port-forward svc/kps-prometheus 9090 &
sleep 1
curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[] | select(.labels.job=="expressapp") | {endpoint:.scrapeUrl, health:.health}'
kill %1

Expected: every endpoint shows "health":"up". If you see "down", the Service port name must be http (matching ServiceMonitor.endpoints.port).

Tag a release to roll it to prod:

git tag v0.3.0 && git push origin v0.3.0

Step 4 — Seal the Grafana admin password

You referenced grafana-admin in the values file. Seal it in each cluster:

for CTX in do-staging do-prod; do
  ENV=${CTX#do-}
  PW=$(openssl rand -base64 18)
  echo "$ENV admin password: $PW"
  kubectl --context "$CTX" -n monitoring create secret generic grafana-admin \
    --from-literal=admin-user=admin \
    --from-literal=admin-password="$PW" \
    --dry-run=client -o yaml | \
  kubeseal --cert ~/code/expressapp/k8s/keys/${CTX}.pem --format yaml \
    > cluster-bootstrap/observability/grafana-admin-${ENV}.yaml
  kubectl --context "$CTX" apply -f cluster-bootstrap/observability/grafana-admin-${ENV}.yaml
done

Restart Grafana to pick up the Secret:

kubectl --context do-staging -n monitoring rollout restart deploy/kps-grafana
kubectl --context do-prod    -n monitoring rollout restart deploy/kps-grafana

Confirm the password by curling the Grafana ingress:

curl -s -u admin:"$PW" https://grafana.staging.example.com/api/health | jq .
# { "commit": "...", "database": "ok", "version": "..." }

Step 5 — Dashboards as ConfigMaps in git

The kube-prometheus-stack sidecar imports any ConfigMap in any namespace labelled grafana_dashboard=1. Save your dashboards as files in cluster-bootstrap/observability/dashboards/ and apply them.

A minimum set:

Dashboard	Source
Kubernetes cluster overview	the chart ships this, already imported
Workload overview	the chart ships this
Express RED (rate/errors/duration)	written below

cluster-bootstrap/observability/dashboards/express-red.json (skeleton; the full JSON is in the repo, queries are what matters):

PromQL queries:
  rate     : sum by (route) (rate(http_requests_total{job="expressapp"}[1m]))
  errors%  : sum by (route) (rate(http_requests_total{job="expressapp",status=~"5.."}[5m]))
             / sum by (route) (rate(http_requests_total{job="expressapp"}[5m]))
  latency  : histogram_quantile(0.95,
               sum by (le, route) (rate(http_request_duration_seconds_bucket{job="expressapp"}[5m])))

Wrap it in a ConfigMap:

kubectl --context do-staging -n monitoring create configmap dash-express-red \
  --from-file=express-red.json=cluster-bootstrap/observability/dashboards/express-red.json \
  --dry-run=client -o yaml | \
yq '.metadata.labels.grafana_dashboard = "1"' - | \
kubectl --context do-staging apply -f -

(Replace yq with kubectl label post-create if you don't have yq.)

Repeat for prod. The sidecar picks the dashboard up within a minute; refresh Grafana and find it under Dashboards → General.

Step 6 — Alertmanager rules

The chart ships dozens of useful rules out of the box (Watchdog, KubeNodeNotReady, KubePodCrashLooping, …). Add a small app-specific set as a PrometheusRule:

cluster-bootstrap/observability/expressapp-rules.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: expressapp
  namespace: monitoring
  labels:
    release: kps
spec:
  groups:
    - name: expressapp
      rules:
        - alert: ExpressErrorRateHigh
          expr: |
            (sum by (namespace) (rate(http_requests_total{job="expressapp",status=~"5.."}[5m]))
             / sum by (namespace) (rate(http_requests_total{job="expressapp"}[5m]))) > 0.05
          for: 5m
          labels: { severity: critical }
          annotations:
            summary: "Express 5xx rate > 5% in {{ $labels.namespace }}"
        - alert: ExpressHPAAtMax
          expr: kube_horizontalpodautoscaler_status_current_replicas{horizontalpodautoscaler="expressapp"}
                == kube_horizontalpodautoscaler_spec_max_replicas{horizontalpodautoscaler="expressapp"}
          for: 10m
          labels: { severity: warning }
          annotations:
            summary: "HPA pinned at max in {{ $labels.namespace }} for 10m"
        - alert: CertExpiringSoon
          expr: (avg by (namespace, name) (certmanager_certificate_expiration_timestamp_seconds) - time()) < 7*24*3600
          for: 15m
          labels: { severity: warning }
          annotations:
            summary: "Cert {{ $labels.name }} expires in <7 days"

Apply once per cluster:

for CTX in do-staging do-prod; do
  kubectl --context "$CTX" apply -f cluster-bootstrap/observability/expressapp-rules.yaml
done

Wire Alertmanager to your channel of choice. Slack example: store the webhook URL as a SealedSecret called alertmanager-slack-webhook, then patch the Alertmanager config:

alertmanager:
  config:
    route:
      receiver: 'slack'
    receivers:
      - name: 'slack'
        slack_configs:
          - api_url_file: /etc/alertmanager/secrets/alertmanager-slack-webhook/url
            channel: '#alerts'
            send_resolved: true
    secrets: [ alertmanager-slack-webhook ]

Re-helm upgrade kps.

Step 7 — Test the alert path end-to-end

Verify by triggering a crashloop and waiting for the canned KubePodCrashLooping alert:

kubectl --context do-staging -n expressapp-staging patch deploy expressapp --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/command","value":["sh","-c","exit 1"]}
]'

# Watch for the alert in the Alertmanager UI:
kubectl --context do-staging -n monitoring port-forward svc/kps-alertmanager 9093 &
sleep 1
xdg-open http://localhost:9093/  2>/dev/null || open http://localhost:9093/

After ~5 minutes the alert is firing. Revert:

kubectl --context do-staging -n expressapp-staging patch deploy expressapp --type=json -p='[
  {"op":"remove","path":"/spec/template/spec/containers/0/command"}
]'

Step 8 — Confirm VictoriaMetrics has the history

After data has flowed for a while, query VictoriaMetrics directly to prove the long-term path works:

kubectl --context do-staging -n monitoring port-forward svc/vm-victoria-metrics-single-server 8428 &
sleep 1
# 6 hours ago in epoch seconds, portable across GNU date and BSD/macOS date:
NOW=$(date +%s); SIX_HOURS=$((NOW - 6 * 3600))
curl -s "http://localhost:8428/api/v1/query_range?query=sum(rate(http_requests_total[5m]))&start=${SIX_HOURS}&end=${NOW}&step=300" | jq '.data.result[0].values | length'
# > 0
kill %1

Add VictoriaMetrics as a second Grafana datasource (Configuration → Data Sources → Add → Prometheus, URL http://vm-victoria-metrics-single-server:8428). Dashboards can query the short window from kps-prometheus for low latency and historical windows from VM.

Completion signal

• kubectl --context do-staging -n monitoring get pods shows kps-prometheus-*, kps-alertmanager-*, kps-grafana-*, vm-victoria-metrics-single-server-*, all Running, all on PVCs.
• The Express RED dashboard in Grafana reflects real traffic against https://app.staging.example.com/.
• Killing an Express pod produces a KubePodCrashLooping alert in Alertmanager.
• Metrics older than the in-cluster Prometheus retention window are still queryable via the VictoriaMetrics datasource in Grafana.
• ServiceMonitor expressapp is committed in k8s/base/ and present on both clusters.

You're ready for 10. Hardening.