Production GitOps with Argo CD on Amazon EKS

Why GitOps

Every team I have worked with that deploys Kubernetes manually hits the same wall: environments drift, rollbacks are painful, and nobody can tell you what is actually running in production right now.

GitOps fixes this by making Git the single source of truth. You declare what you want in a repo, and a controller running inside the cluster pulls those declarations and applies them continuously. No kubectl apply from laptops. No Jenkins jobs pushing manifests into clusters. The cluster converges to what Git says, always.

Argo CD is the tool that does this well. It watches your Git repos, compares what is declared with what is running, and reconciles the difference. If someone changes something directly in the cluster, Argo CD flags it as out-of-sync and can auto-correct it.

What This Post Covers

We will set up Argo CD on an EKS cluster and deploy a real application using the app-of-apps pattern. By the end you will have:

• Argo CD installed and accessible via an ALB
• An app-of-apps repo structure that scales to dozens of services
• Automated sync with self-heal and pruning
• Slack notifications on deploy success/failure
• RBAC so developers can view but not force-sync production

Step 1: Install Argo CD

Helm is the cleanest option:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

helm install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --version 7.8.0 \
  --set server.service.type=ClusterIP \
  --set configs.params."server\.insecure"=true

The server.insecure=true flag disables TLS at the Argo CD server level because we will terminate TLS at the ALB.

Get the initial admin password:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

Step 2: Expose via ALB Ingress

Create an Ingress resource that provisions an AWS ALB with ACM certificate:

# argocd-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-southeast-2:123456789012:certificate/abc-123
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
    alb.ingress.kubernetes.io/backend-protocol: HTTP
spec:
  rules:
    - host: argocd.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 80

kubectl apply -f argocd-ingress.yaml

This requires the AWS Load Balancer Controller to be installed on your cluster. If you do not have it, install it first - there are good docs on the AWS side.

Step 3: Repo Structure - App of Apps

The app-of-apps pattern is how you scale GitOps beyond a handful of services. Instead of creating one Argo CD Application per service manually, you have a root application that points to a directory of application definitions.

Here is the repo structure:

gitops-config/
  apps/                    # Root app-of-apps directory
    api-gateway.yaml
    user-service.yaml
    order-service.yaml
    payment-service.yaml
  base/                    # Shared Helm values / Kustomize bases
    values-common.yaml
  envs/
    dev/
      api-gateway.yaml     # Dev-specific overrides
      user-service.yaml
    staging/
      api-gateway.yaml
      user-service.yaml
    prod/
      api-gateway.yaml
      user-service.yaml

Each file in apps/ is a standard Argo CD Application manifest:

# apps/user-service.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: user-service
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/gitops-config
    targetRevision: main
    path: envs/prod/user-service
    helm:
      valueFiles:
        - ../../base/values-common.yaml
        - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: user-service
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true

The root application watches the apps/ directory:

# root-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/gitops-config
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Now when you add a new service, you just add a YAML file to apps/ and push. Argo CD picks it up automatically.

Step 4: Automated Sync with Self-Heal

The two flags that matter:

• prune: true - Argo CD deletes resources that are in the cluster but not in Git. If you remove a deployment from your manifests, Argo CD removes it from the cluster.
• selfHeal: true - If someone runs kubectl edit and changes a deployment directly, Argo CD reverts it back to what Git says within 3 minutes (the default sync interval).

These two settings together mean Git is always the truth. No exceptions.

You can tune the sync interval if 3 minutes is too slow:

# In the Argo CD ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  timeout.reconciliation: "60s"

This checks every 60 seconds instead of every 180.

Step 5: Slack Notifications

Argo CD has a built-in notification controller. Configure it to send deploy alerts to Slack:

# argocd-notifications-cm
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token
  trigger.on-sync-succeeded: |
    - when: app.status.sync.status == 'Synced'
      send: [app-sync-succeeded]
  trigger.on-sync-failed: |
    - when: app.status.sync.status == 'OutOfSync'
      oncePer: app.status.sync.revision
      send: [app-sync-failed]
  template.app-sync-succeeded: |
    slack:
      attachments: |
        [{
          "color": "#18be52",
          "title": "{{.app.metadata.name}} synced",
          "text": "Revision: {{.app.status.sync.revision}}\nStatus: Healthy",
          "footer": "Argo CD"
        }]
  template.app-sync-failed: |
    slack:
      attachments: |
        [{
          "color": "#e8112d",
          "title": "{{.app.metadata.name}} sync failed",
          "text": "Revision: {{.app.status.sync.revision}}\nCheck Argo CD for details.",
          "footer": "Argo CD"
        }]

Store the Slack token in a secret:

kubectl -n argocd create secret generic argocd-notifications-secret \
  --from-literal=slack-token=xoxb-your-slack-bot-token

Annotate your applications to subscribe:

kubectl -n argocd annotate application user-service \
  notifications.argoproj.io/subscribe.on-sync-succeeded.slack=deploys \
  notifications.argoproj.io/subscribe.on-sync-failed.slack=deploys

Now every deploy success or failure posts to the #deploys channel.

Step 6: RBAC Lockdown

In production, you do not want every developer force-syncing or deleting applications. Argo CD supports RBAC through its ConfigMap:

# argocd-rbac-cm
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # Admins - full access
    p, role:admin, applications, *, */*, allow
    p, role:admin, clusters, *, *, allow
    p, role:admin, repositories, *, *, allow

    # Developers - read-only + sync on non-prod
    p, role:developer, applications, get, */*, allow
    p, role:developer, applications, sync, dev/*, allow
    p, role:developer, applications, sync, staging/*, allow

    # Map SSO groups to roles
    g, platform-team, role:admin
    g, dev-team, role:developer

This means:

• Platform team can do anything
• Developers can view all apps but can only trigger syncs in dev and staging
• Nobody can sync production except the platform team (or the automated sync policy)

Common Pitfalls

1. Not using app-of-apps from day one. You start with 3 services and think you will manage. Then you have 25 and each one is a manually created Application. Start with app-of-apps from the beginning.

2. Leaving selfHeal off. If someone can kubectl edit a production deployment and that change sticks, you do not have GitOps. You have Git-sometimes.

3. Putting secrets in Git. Argo CD syncs whatever is in Git. Use Sealed Secrets, External Secrets Operator, or Vault sidecar injection. Never commit plain secrets to the GitOps repo.

4. One giant repo for everything. Monorepos work up to a point. Once you have multiple teams, split config repos per team or domain. Argo CD handles multiple source repos cleanly.

5. Skipping notifications. Deploys that nobody sees are deploys nobody trusts. Wire up Slack or Teams notifications from day one. It takes 10 minutes and saves hours of "did that deploy go out?" conversations.

Summary

GitOps with Argo CD is not complicated to set up. The hard part is discipline: keeping Git as the only way changes get to production, enforcing self-heal so manual changes do not stick, and structuring your repos so adding a new service takes minutes, not hours.

The app-of-apps pattern, automated sync with pruning and self-heal, Slack notifications, and RBAC lockdown give you a production-grade foundation. Start here, and build your promotion gates and environment workflows on top.