GitOps with ArgoCD

So far CI has pushed changes into the cluster with kubectl apply -k. GitOps inverts that: the cluster pulls its own desired state from a git repository, continuously, and reconciles drift. This chapter sets up ArgoCD, moves the Express app onto a GitOps deploy, and contrasts the two models so you know when each one is appropriate.

Prerequisites

• Push-based deploys to staging and prod are working (Part 7).
• Production polish is in place: requests/limits, PDBs, rollback drill.

Why GitOps

Concern	Push (current)	Pull (ArgoCD)
Where credentials live	CI has kubeconfig secrets	Cluster pulls from git; CI does not touch cluster
Drift correction	None; a manual kubectl edit survives until the next CI deploy	Automatic; ArgoCD reverts drift within seconds
Multi-cluster fan-out	Each cluster needs its own kubeconfig in CI	One ArgoCD instance can manage many clusters
Mental model	"Apply on merge"	"Cluster matches the repo"
Failure mode	A bad CI run fails to deploy and stays unnoticed unless someone watches	Sync status is a continuous signal in the ArgoCD UI

The course teaches both because push is simpler for a small team and a single workload. Pull pays off as you accumulate clusters or want drift detection. You should be able to defend either choice.

Step 1 — Create the manifests repository

Split the k8s/ tree out of expressapp into its own repo expressapp-manifests:

mkdir -p ~/code/expressapp-manifests
cp -r ~/code/expressapp/k8s ~/code/expressapp-manifests/
cd ~/code/expressapp-manifests
git init -b master
gh repo create expressapp-manifests --source=. --private --remote=origin --push

Branch protection on master of the manifests repo is just as important as on the app repo. The course uses a PR-based promotion model so the protection rule is symmetric for staging and prod, and neither path needs a bypass token:

gh api -X PUT "repos/<OWNER>/expressapp-manifests/branches/master/protection" \
  -F required_pull_request_reviews.required_approving_review_count=1 \
  -F enforce_admins=true \
  -F required_linear_history=true \
  -F allow_force_pushes=false \
  -F allow_deletions=false \
  -F restrictions=

Staging promotions open a PR and auto-merge: the workflow opens the PR with --admin / gh pr merge --auto, so once required checks pass the PR merges itself. Prod promotions open a PR and wait for review. That way the same branch-protection rule covers both, and no bot or PAT needs to be on a bypass list.

Step 2 — Install ArgoCD in each cluster

Once per cluster, from cluster-bootstrap:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

cat > /tmp/argocd-values.yaml <<EOF
global:
  domain: argocd.${HOST_BASE}                # set per env when rendering

server:
  ingress:
    enabled: true
    ingressClassName: traefik
    hostname: argocd.${HOST_BASE}
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      traefik.ingress.kubernetes.io/router.middlewares: traefik-security-headers@kubernetescrd
    tls: true

configs:
  params:
    # Run argocd-server in HTTP inside the cluster; Traefik terminates TLS at the edge.
    # This avoids needing a Traefik ServersTransport to skip verifying ArgoCD's self-signed
    # backend cert. The public path is still HTTPS via cert-manager + Let's Encrypt.
    server.insecure: "true"
EOF

for ENV in staging prod; do
  CTX=do-${ENV}
  HOST_BASE=$( [ "$ENV" = "staging" ] && echo "staging.example.com" || echo "example.com" )
  envsubst < /tmp/argocd-values.yaml > /tmp/argocd-values-rendered.yaml
  helm --kube-context "$CTX" upgrade --install argocd argo/argo-cd \
    --version 7.4.0 \
    --namespace argocd --create-namespace \
    -f /tmp/argocd-values-rendered.yaml
done

Wait for the components:

kubectl --context do-staging -n argocd rollout status deploy/argocd-server

Reset the initial admin password to something you control. Grab the auto-generated one once, log in, then change it.

ADMIN_PW=$(kubectl --context do-staging -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d)
echo "initial admin password (staging): $ADMIN_PW"

Log in:

argocd login argocd.staging.example.com --username admin --password "$ADMIN_PW"
argocd account update-password --account admin --current-password "$ADMIN_PW" --new-password "$(openssl rand -base64 18)"
kubectl --context do-staging -n argocd delete secret argocd-initial-admin-secret    # the bootstrap secret should not persist

Repeat for prod.

Step 3 — Connect ArgoCD to the manifests repo

If the manifests repo is private (it should be), give ArgoCD a deploy key.

# generate a per-cluster deploy key
ssh-keygen -t ed25519 -N "" -f /tmp/argocd-staging.key -C "argocd-staging"
gh repo deploy-key add /tmp/argocd-staging.key.pub --title "argocd-staging" --repo <OWNER>/expressapp-manifests

argocd repo add git@github.com:<OWNER>/expressapp-manifests.git \
  --ssh-private-key-path /tmp/argocd-staging.key
rm -f /tmp/argocd-staging.key /tmp/argocd-staging.key.pub

argocd repo list should show the repo with status Successful.

Step 4 — Define the Applications

Each environment is one Application resource pointing ArgoCD at the right overlay.

cluster-bootstrap/argocd/app-staging.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: expressapp
  namespace: argocd
  finalizers: [ resources-finalizer.argocd.argoproj.io ]
spec:
  project: default
  source:
    repoURL: git@github.com:<OWNER>/expressapp-manifests.git
    targetRevision: HEAD
    path: k8s/overlays/staging
  destination:
    server: https://kubernetes.default.svc      # the staging cluster itself
    namespace: expressapp-staging
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions: [ ApplyOutOfSyncOnly=true ]

The prod application is the same with expressapp-prod, k8s/overlays/prod, and (because prod should not auto-sync image bumps without approval) automated removed:

  syncPolicy:
    syncOptions: [ ApplyOutOfSyncOnly=true ]

Apply once per cluster:

kubectl --context do-staging apply -f cluster-bootstrap/argocd/app-staging.yaml
kubectl --context do-prod    apply -f cluster-bootstrap/argocd/app-prod.yaml

Watch the sync:

argocd app get expressapp --grpc-web --server argocd.staging.example.com
# STATUS    HEALTH    REVISION
# Synced    Healthy   <sha>

In the ArgoCD UI, the Application now shows every object (Deployment, Service, Ingress, HPA, ServiceMonitor, PDB, SealedSecret) in a tree.

Step 5 — Change CI to commit, not apply

The app repo's deploy workflows go away. In their place, two tiny workflows open PRs against the manifests repo. Both workflows respect manifests-repo branch protection: staging's PR auto-merges, prod's PR waits for a reviewer.

MANIFESTS_REPO_TOKEN is a fine-grained PAT with Contents: write and Pull requests: write on the manifests repo only.

expressapp/.github/workflows/promote-staging.yml

name: promote-staging
on:
  push:
    branches: [ master ]

permissions:
  contents: read

jobs:
  promote:
    runs-on: ubuntu-latest
    env:
      DOCR_REGISTRY: ${{ vars.DOCR_REGISTRY }}
      GH_TOKEN: ${{ secrets.MANIFESTS_REPO_TOKEN }}
    steps:
      - uses: actions/checkout@v4
        with:
          repository: <OWNER>/expressapp-manifests
          token: ${{ secrets.MANIFESTS_REPO_TOKEN }}
          path: manifests
      - name: kustomize
        run: |
          curl -sL "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
          sudo mv kustomize /usr/local/bin/
      - name: Pin staging to the new SHA on a branch
        working-directory: manifests
        run: |
          git config user.email "ci@expressapp"
          git config user.name  "expressapp CI"
          BRANCH="promote/staging-${GITHUB_SHA}"
          git switch -c "$BRANCH"
          (cd k8s/overlays/staging && kustomize edit set image \
            REGISTRY_PLACEHOLDER/expressapp="${DOCR_REGISTRY}/expressapp:${GITHUB_SHA}")
          git add k8s/overlays/staging/kustomization.yaml
          git diff --cached --quiet && exit 0
          git commit -m "staging: ${GITHUB_SHA}"
          git push -u origin "$BRANCH"
          gh pr create --repo <OWNER>/expressapp-manifests \
            --base master --head "$BRANCH" \
            --title "staging: ${GITHUB_SHA}" \
            --body  "Automated staging promotion. Auto-merging once checks pass."
          gh pr merge "$BRANCH" --repo <OWNER>/expressapp-manifests --squash --auto --delete-branch

expressapp/.github/workflows/promote-prod.yml

name: promote-prod
on:
  push:
    tags: [ 'v*' ]

permissions:
  contents: read

jobs:
  provenance-check:
    # Same as in Part 6; refuse tags not reachable from master.
    runs-on: ubuntu-latest
    outputs:
      tag-sha: ${{ steps.resolve.outputs.tag-sha }}
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0, ref: ${{ github.ref }} }
      - id: resolve
        run: |
          TAG_SHA="$(git rev-list -n 1 "${GITHUB_REF##refs/tags/}")"
          echo "tag-sha=$TAG_SHA" >> "$GITHUB_OUTPUT"
      - run: |
          git fetch origin master:refs/remotes/origin/master
          git merge-base --is-ancestor "${{ steps.resolve.outputs.tag-sha }}" origin/master

  promote:
    needs: provenance-check
    runs-on: ubuntu-latest
    env:
      DOCR_REGISTRY: ${{ vars.DOCR_REGISTRY }}
      GH_TOKEN: ${{ secrets.MANIFESTS_REPO_TOKEN }}
      TAG_SHA: ${{ needs.provenance-check.outputs.tag-sha }}
    steps:
      - uses: actions/checkout@v4
        with:
          repository: <OWNER>/expressapp-manifests
          token: ${{ secrets.MANIFESTS_REPO_TOKEN }}
          path: manifests
      - run: |
          curl -sL "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
          sudo mv kustomize /usr/local/bin/
      - name: Open a review PR for prod
        working-directory: manifests
        run: |
          git config user.email "ci@expressapp"
          git config user.name  "expressapp CI"
          BRANCH="promote/prod-${GITHUB_REF_NAME}"
          git switch -c "$BRANCH"
          (cd k8s/overlays/prod && kustomize edit set image \
            REGISTRY_PLACEHOLDER/expressapp="${DOCR_REGISTRY}/expressapp:${TAG_SHA}")
          git add k8s/overlays/prod/kustomization.yaml
          git diff --cached --quiet && exit 0
          git commit -m "prod: ${GITHUB_REF_NAME} (${TAG_SHA})"
          git push -u origin "$BRANCH"
          gh pr create --repo <OWNER>/expressapp-manifests \
            --base master --head "$BRANCH" \
            --title "prod: ${GITHUB_REF_NAME}" \
            --body  "Promotion for tag ${GITHUB_REF_NAME} (commit ${TAG_SHA}). Requires review."
          # No --auto: a human reviewer merges.

After the PR merges (auto for staging, manual for prod) ArgoCD picks the change up. Prod's Application isn't automated, so the final sync still requires argocd app sync expressapp-prod (or a click in the UI). That gives you two gates on the prod path: PR review on the manifests repo, then a sync click in ArgoCD. Drop one of them later if it feels redundant.

Drop the old deploy-staging.yml and deploy-prod.yml from the app repo once you've verified the GitOps path works end to end. They're no longer needed and they still hold cluster credentials.

Step 6 — Demonstrate drift correction

In staging, manually edit something ArgoCD owns and watch it revert:

kubectl --context do-staging -n expressapp-staging scale deploy/expressapp --replicas=10
sleep 3
kubectl --context do-staging -n expressapp-staging get deploy expressapp -o jsonpath='{.spec.replicas}'
# 10
sleep 15
kubectl --context do-staging -n expressapp-staging get deploy expressapp -o jsonpath='{.spec.replicas}'
# 2     <-- selfHeal: true reverted it

In the ArgoCD UI, the Sync Status flashed OutOfSync and then back to Synced. The history shows who/what changed it.

Try the same on prod. Because automated is off, ArgoCD detects drift but doesn't revert until you click Sync. That's the desired posture for prod: ArgoCD shows you what's wrong, you decide whether to sync.

Step 7 — Tag-driven prod promote

On a release:

# in expressapp:
git tag v0.4.0
git push origin v0.4.0
# promote-prod.yml runs:
#   1. provenance-check (refusing tags not on master)
#   2. kustomize edit set image ... on the prod overlay in expressapp-manifests
#   3. opens a PR against expressapp-manifests/master — waits for a human reviewer
# the PR is reviewed and merged
# in ArgoCD UI: prod app is OutOfSync; click Sync (or `argocd app sync expressapp-prod`)

Both staging and prod open PRs against the manifests repo; the difference is that staging's PR auto-merges and prod's waits for review. Branch protection covers both paths uniformly: no bypass token, no special branch rules.

Step 8 — Compare push and pull

Re-read the table at the top of this chapter with the experience you now have. Specific situations:

• Single small team, one cluster, fast iteration: push is simpler. Stay with it.
• Multiple clusters / many environments / regulated audit: pull, because the cluster is the source of truth for what's running and the git log is the audit trail.
• Drift is dangerous (e.g. policy controllers, kustomize-managed CRDs): pull, because nothing survives a selfHeal reconcile loop.
• CI must never touch the cluster: pull. CI commits to git, ArgoCD reads from git. Cluster credentials never leave the cluster.

Step 9 — What ArgoCD costs

Operationally:

• Two extra pods per cluster (argocd-application-controller, argocd-server, plus repo-server and Redis).
• An exposed Ingress that must be locked down (the course uses letsencrypt-prod + admin password; in real use add SSO).
• A repo permission model to maintain. Who can edit which path in expressapp-manifests?

In Grafana, add the ArgoCD dashboards (argocd-application-controller exposes Prometheus metrics) so you can see sync latency and reconcile health.

Completion signal

• A commit to master of expressapp opens a PR against expressapp-manifests, which auto-merges once required checks pass, and ArgoCD then syncs the change to staging. No kubectl invocation by CI.
• A v* tag opens a non-auto-merging PR against expressapp-manifests; once a reviewer merges, the prod Application reports OutOfSync and a final argocd app sync expressapp-prod (or click in the UI) deploys it.
• Manually editing a Deployment field in staging reverts within ~15 seconds.
• Manifests-repo branch protection is the same for both flows; no bot identity is on a bypass list.
• Old deploy-staging.yml / deploy-prod.yml are deleted from the app repo; the only cluster-touching credential in CI is MANIFESTS_REPO_TOKEN, which writes only to git, not to a cluster.

You're ready for 14. Ship the Course Site.