Gitops Principles

GitOps Principles Skill

Complete guide for implementing GitOps methodology in Kubernetes environments - the operational framework where Git is the single source of truth for declarative infrastructure and applications.

What is GitOps?

GitOps is a set of practices that uses Git repositories as the source of truth for defining the desired state of infrastructure and applications. An automated process ensures the production environment matches the state described in the repository.

The OpenGitOps Definition (CNCF)

GitOps is defined by four core principles established by the OpenGitOps project (part of CNCF):

Core Concepts Quick Reference

Git as Single Source of Truth

┌─────────────────────────────────────────────────────────────────┐
│                        GIT REPOSITORY                           │
│  (Single Source of Truth for Desired State)                    │
├─────────────────────────────────────────────────────────────────┤
│  manifests/                                                     │
│  ├── base/                    # Base configurations             │
│  │   ├── deployment.yaml                                        │
│  │   ├── service.yaml                                           │
│  │   └── kustomization.yaml                                     │
│  └── overlays/                # Environment-specific            │
│      ├── dev/                                                   │
│      ├── staging/                                               │
│      └── production/                                            │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼ Pull (not Push)
┌─────────────────────────────────────────────────────────────────┐
│                      GITOPS CONTROLLER                          │
│  (ArgoCD / Flux / Kargo)                                       │
│  - Continuously watches Git repository                          │
│  - Compares desired state vs actual state                       │
│  - Reconciles differences automatically                         │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼ Apply
┌─────────────────────────────────────────────────────────────────┐
│                    KUBERNETES CLUSTER                           │
│  (Actual State / Runtime Environment)                          │
└─────────────────────────────────────────────────────────────────┘

Push vs Pull Model

Key GitOps Benefits

1. Auditability: Git history = deployment history 2. Security: No external access to cluster required 3. Reliability: Automated drift correction 4. Speed: Deploy via PR merge 5. Rollback: Simple git revert 6. Disaster Recovery: Redeploy entire cluster from Git

Repository Strategies

Monorepo vs Polyrepo

Monorepo (Single repository for all environments):

gitops-repo/
├── apps/
│   ├── app-a/
│   │   ├── base/
│   │   └── overlays/
│   │       ├── dev/
│   │       ├── staging/
│   │       └── prod/
│   └── app-b/
└── infrastructure/
    ├── monitoring/
    └── networking/

Polyrepo (Separate repositories):

# Repository per concern
app-a-config/          # App A manifests
app-b-config/          # App B manifests
infrastructure/        # Shared infrastructure
cluster-bootstrap/     # Cluster setup

Multi-Repository Pattern (This Project)

Separates infrastructure from values for security boundaries:

infra-team/                    # Base configurations, ApplicationSets
├── applications/              # ArgoCD Application definitions
└── helm-base-values/          # Default Helm values

argo-cd-helm-values/           # Environment-specific overrides
├── dev/                       # Development values
├── stg/                       # Staging values
└── prd/                       # Production values

Benefits:

• Different access controls per repo
• Separation of concerns
• Environment-specific secrets isolated

Branching Strategies

Environment Branches

main ────────────────────────────────────► Production
  │
  └──► staging ──────────────────────────► Staging cluster
         │
         └──► develop ───────────────────► Development cluster

Trunk-Based with Overlays (Recommended)

main ────────────────────────────────────► All environments
  │
  ├── overlays/dev/       → Dev cluster
  ├── overlays/staging/   → Staging cluster
  └── overlays/prod/      → Prod cluster

Release Branches

main
  │
  ├── release/v1.0 ──────► Production (v1.0)
  ├── release/v1.1 ──────► Production (v1.1)
  └── release/v2.0 ──────► Production (v2.0)

Sync Policies and Strategies

Automated Sync

syncPolicy:
  automated:
    prune: true       # Delete resources not in Git
    selfHeal: true    # Revert manual changes

Manual Sync (Production Recommended)

syncPolicy:
  automated: null     # Require explicit sync

Sync Options

Declarative Configuration Patterns

Kustomize Pattern

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patchesStrategicMerge:
  - replica-patch.yaml
images:
  - name: myapp
    newTag: v1.2.3

Helm Pattern

# Application pointing to Helm chart
spec:
  source:
    repoURL: https://charts.example.com
    chart: my-app
    targetRevision: 1.2.3
    helm:
      releaseName: my-app
      valueFiles:
        - values.yaml
        - values-prod.yaml

Multi-Source Pattern

spec:
  sources:
    - repoURL: https://charts.bitnami.com/bitnami
      chart: nginx
      targetRevision: 15.0.0
      helm:
        valueFiles:
          - $values/nginx/values-prod.yaml
    - repoURL: https://github.com/org/values.git
      targetRevision: main
      ref: values

Progressive Delivery Integration

GitOps enables progressive delivery patterns:

Blue-Green Deployments

# Two applications, traffic shift via Ingress/Service
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-blue
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-green

Canary with Argo Rollouts

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: {duration: 5m}
        - setWeight: 50
        - pause: {duration: 10m}

Environment Promotion (Kargo)

Warehouse → Dev Stage → Staging Stage → Production Stage
    │           │              │               │
    └── Freight promotion through environments ───┘

Cloud Provider Integration

Azure Arc-enabled Kubernetes & AKS

Azure provides a managed ArgoCD experience through the Microsoft.ArgoCD cluster extension:

# Simple installation (single node)
az k8s-extension create \
  --resource-group <rg> --cluster-name <cluster> \
  --cluster-type managedClusters \
  --name argocd \
  --extension-type Microsoft.ArgoCD \
  --release-train preview \
  --config deployWithHighAvailability=false

# Production with workload identity (recommended)
# Use Bicep template - see references/azure-arc-integration.md

Key Benefits:

Prerequisites:

• Azure Arc-connected cluster OR MSI-based AKS cluster
• Microsoft.KubernetesConfiguration provider registered
• k8s-extension CLI extension installed

See references/azure-arc-integration.md for complete setup guide.

---

Security Considerations

Secrets Management

Never store secrets in Git! Use:

RBAC Best Practices

# Limit ArgoCD to specific namespaces
apiVersion: argoproj.io/v1alpha1
kind: AppProject
spec:
  destinations:
    - namespace: 'team-a-*'
      server: https://kubernetes.default.svc
  sourceRepos:
    - 'https://github.com/org/team-a-*'

Network Policies

• GitOps controller should be only component with Git access
• Restrict egress from application namespaces
• Use network policies to isolate environments

Observability and Debugging

Health Status Interpretation

Common Issues Checklist

1. Sync Failed: Check YAML syntax, RBAC permissions 2. OutOfSync: Compare diff, check ignoreDifferences 3. Degraded: Check Pod logs, resource limits 4. Missing: Verify namespace, check pruning settings

Drift Detection

# Check application diff
argocd app diff myapp

# Force refresh from Git
argocd app get myapp --refresh

Quick Decision Guide

When to Use GitOps

• Kubernetes-native workloads
• Multiple environments (dev/staging/prod)
• Need audit trail for deployments
• Team collaboration on infrastructure
• Disaster recovery requirements

When GitOps May Not Fit

• Rapidly changing development environments
• Legacy systems without declarative configs
• Real-time configuration changes required
• Single developer, single environment

References

For detailed information, see:

• references/core-principles.md - Deep dive into the 4 pillars
• references/patterns-and-practices.md - Branching and repo patterns
• references/tooling-ecosystem.md - ArgoCD vs Flux vs Kargo
• references/anti-patterns.md - Common mistakes to avoid
• references/troubleshooting.md - Debugging guide
• references/azure-arc-integration.md - Azure Arc & AKS GitOps setup

Templates

Ready-to-use templates in templates/:

• application.yaml - ArgoCD Application example
• applicationset.yaml - Multi-cluster deployment
• kustomization.yaml - Kustomize overlay structure

Scripts

Utility scripts in scripts/:

• gitops-health-check.sh - Validate GitOps setup

External Resources

• OpenGitOps Principles
• ArgoCD Documentation
• Flux Documentation
• Kargo Documentation
• GitOps Working Group
• Azure Arc GitOps with ArgoCD
• Azure Arc-enabled Kubernetes

---

Gotchas

• `selfHeal: true` fights kubectl edit: Engineers patching live resources during incident response will see their changes reverted within seconds. Either pause auto-sync first or commit the fix to Git.
• Monorepo + ApplicationSet git generator scales the controller hard: Every directory change re-renders every app. A 200-app repo with a noisy main branch can pin controller CPU. Split repos or use webhooks instead of polling.
• `PruneLast=true` and Helm hooks collide: Hooks run during sync; prune happens after. Resources from post-install hooks get deleted on next sync because they're not in the rendered manifest. Annotate hooks with Prune=false.
• Drift correction depends on the controller noticing: Resources outside ArgoCD's tracked set (e.g. dynamic Secrets from External Secrets Operator) are NOT reconciled — drift in them is invisible. Use explicit ignoreDifferences.
• Sealed Secrets are cluster-scoped: A SealedSecret encrypted for cluster A will not decrypt on cluster B. Multi-cluster GitOps needs per-cluster keys or a shared key (which defeats the security model).
• `git revert` rollback assumes immutable image tags: Reverting a manifest reverts the tag string, but a mutable tag (e.g. :latest) now points at a different image. Pin to digests for true rollback safety.

GitOps Anti-Patterns

Common mistakes and pitfalls to avoid when implementing GitOps, with guidance on proper practices.

Configuration Anti-Patterns

Anti-Pattern 1: Imperative Commands in Production

The Problem:

# DON'T DO THIS
kubectl scale deployment nginx --replicas=5
kubectl set image deployment/nginx nginx=nginx:1.22
kubectl edit configmap app-config

Why It's Bad:

• Changes are not tracked in Git
• Drift occurs between Git and cluster
• No audit trail
• Changes lost on next sync

The Fix:

# DO THIS: Update Git, let GitOps sync
# manifests/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 5  # Changed from 3
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.22  # Updated version

---

Anti-Pattern 2: Mutable Image Tags

The Problem:

# DON'T DO THIS
containers:
- name: app
  image: myapp:latest
  # or
  image: myapp:dev
  # or
  image: myapp:stable

Why It's Bad:

• Same tag, different content over time
• No way to track what's actually deployed
• Rollback doesn't work (:latest changed)
• Cache issues across nodes

The Fix:

# DO THIS: Use immutable tags or digests
containers:
- name: app
  image: myapp:v1.2.3
  # or even better
  image: myapp@sha256:abc123def456...

Automated Fix with Kargo/Flux:

# Warehouse subscription
subscriptions:
  - image:
      repoURL: myregistry/myapp
      imageSelectionStrategy: SemVer
      constraint: ^1.0.0

---

Anti-Pattern 3: Secrets in Git

The Problem:

# DON'T DO THIS - NEVER COMMIT SECRETS
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: bXlzZWNyZXRwYXNzd29yZA==  # base64 != encryption!

Why It's Bad:

• Git history is forever
• base64 is encoding, not encryption
• Secrets exposed to anyone with repo access
• Compliance violations

The Fix:

Option 1: External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: azure-keyvault
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: database-password

Option 2: Sealed Secrets

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
spec:
  encryptedData:
    password: AgBy8hCi...  # Actually encrypted

Option 3: SOPS

# Encrypted with SOPS - safe to commit
password: ENC[AES256_GCM,data:xxx,iv:yyy,tag:zzz,type:str]

---

Anti-Pattern 4: Hardcoded Environment Values

The Problem:

# DON'T DO THIS - Hardcoded for each environment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 3  # What about dev? staging?
  template:
    spec:
      containers:
      - name: app
        env:
        - name: DATABASE_URL
          value: "postgres://prod-db:5432/app"  # Hardcoded!
        resources:
          limits:
            memory: "2Gi"  # Same for all envs?

The Fix:

# base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1  # Overridden per environment
  template:
    spec:
      containers:
      - name: app
        envFrom:
        - configMapRef:
            name: app-config

# overlays/production/kustomization.yaml
replicas:
  - name: app
    count: 3

# overlays/production/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_URL: "postgres://prod-db:5432/app"

---

Workflow Anti-Patterns

Anti-Pattern 5: Bypassing Git for "Quick Fixes"

The Problem:

# "It's just a quick fix, I'll update Git later"
kubectl apply -f hotfix.yaml
# ... 3 months later, no Git update, forgotten

Why It's Bad:

• "Later" never comes
• Creates undocumented drift
• Next sync overwrites the fix
• Knowledge lost

The Fix:

1. Disable direct kubectl access to production 2. Enable self-heal in ArgoCD:

   syncPolicy:
     automated:
       selfHeal: true

3. Fast-track PR process for hotfixes 4. Emergency runbook that includes Git steps

---

Anti-Pattern 6: No Sync Windows

The Problem:

# Automated sync with no restrictions
syncPolicy:
  automated:
    prune: true
    selfHeal: true
# Deploys at 3 AM on Friday before a holiday...

Why It's Bad:

• Deployments during low-staffing periods
• No change control
• Compliance issues
• Incidents during off-hours

The Fix:

# ArgoCD Project with sync windows
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
spec:
  syncWindows:
    # Allow syncs Monday-Thursday, 9 AM - 5 PM
    - kind: allow
      schedule: "0 9 * * 1-4"
      duration: 8h
      applications: ["*"]
    # Deny all syncs on weekends
    - kind: deny
      schedule: "0 0 * * 0,6"
      duration: 24h
      applications: ["*"]

---

Anti-Pattern 7: Monolithic Applications

The Problem:

# Single Application for entire platform
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: everything
spec:
  source:
    path: manifests/  # 500+ resources!

Why It's Bad:

• Single failure affects everything
• Long sync times
• Difficult to track changes
• No granular rollback
• Complex RBAC

The Fix:

# App of Apps pattern
# Root application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: platform
spec:
  source:
    path: apps/
---
# Individual applications
# apps/frontend.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: frontend
spec:
  source:
    path: manifests/frontend/
---
# apps/backend.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: backend
spec:
  source:
    path: manifests/backend/

---

Anti-Pattern 8: Ignoring Drift Detection

The Problem:

# "OutOfSync is normal, we ignore it"
syncPolicy:
  automated: null  # No auto-sync
  # No alerts configured
  # No regular reconciliation

Why It's Bad:

• Security vulnerabilities unpatched
• Configuration creep
• Disaster recovery compromised
• Git becomes stale

The Fix:

1. Configure alerts for OutOfSync status 2. Schedule regular syncs even if manual 3. Use diff commands in CI:

   argocd app diff myapp --exit-code
   if [ $? -ne 0 ]; then
     echo "Drift detected!"
     # Send alert
   fi

4. Review OutOfSync apps weekly

---

Architecture Anti-Patterns

Anti-Pattern 9: Single Repository for All Environments

The Problem:

monorepo/
├── prod-secrets.yaml    # Production secrets
├── dev-secrets.yaml     # Dev secrets
├── manifests/           # Same access for all

Why It's Bad:

• Everyone with repo access sees production secrets
• No separation of duties
• Compliance violations
• Accidental production changes

The Fix:

# Separate repositories with different access
infra-config/          # Platform team only
├── applications/
└── base-values/

prod-values/           # Production team + approvals
├── secrets/
└── values/

dev-values/            # Developers
├── secrets/
└── values/

---

Anti-Pattern 10: No Health Checks

The Problem:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  # Syncs and reports "Healthy" immediately
  # Doesn't wait for pods to be ready

Why It's Bad:

• Deployment appears successful when it's not
• Rolling updates continue despite failures
• No automatic rollback trigger

The Fix:

# Proper health checks in Deployment
spec:
  template:
    spec:
      containers:
      - name: app
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20

# ArgoCD sync with health check
syncPolicy:
  automated:
    prune: true
    selfHeal: true
  syncOptions:
    - CreateNamespace=true
# ArgoCD will wait for resources to be healthy

---

Anti-Pattern 11: No Rollback Strategy

The Problem:

# On deployment failure:
# "Let me just push another commit to fix it"
# ... 30 minutes of debugging while production is down

Why It's Bad:

• Extended downtime
• Panic-driven changes
• More errors from rushed fixes

The Fix:

Immediate rollback via Git:

# Option 1: Revert commit
git revert HEAD
git push

# Option 2: Reset to known good
git reset --hard v1.2.2
git push --force  # If protected, use revert

# Option 3: ArgoCD CLI
argocd app rollback myapp 2  # Rollback to revision 2

Automated rollback with Argo Rollouts:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: {duration: 5m}
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: myapp
      # Automatic rollback on analysis failure

---

Anti-Pattern Checklist

Before deploying, verify:

• [ ] No kubectl apply to production
• [ ] No :latest or mutable tags
• [ ] No secrets in Git (plain or base64)
• [ ] Environment-specific values in overlays
• [ ] Sync windows configured for production
• [ ] Applications are granular (not monolithic)
• [ ] Drift alerts configured
• [ ] Health checks defined
• [ ] Rollback procedure documented
• [ ] Repository access properly scoped

Summary

Azure Arc GitOps Integration with ArgoCD

Comprehensive guide for deploying and managing ArgoCD via Azure Arc-enabled Kubernetes and Azure Kubernetes Service (AKS) using Azure's managed GitOps extension.

Overview

Azure provides a managed ArgoCD experience through the Microsoft.ArgoCD cluster extension. This enables GitOps workflows on:

• Azure Arc-enabled Kubernetes: On-premises, multi-cloud, or edge

Kubernetes clusters connected to Azure

• Azure Kubernetes Service (AKS): Azure's managed Kubernetes offering

Key Benefits

---

Prerequisites

Azure Arc-enabled Kubernetes

1. Kubernetes cluster connected to Azure Arc:

   # Connect cluster to Azure Arc
   az connectedk8s connect --name <cluster-name> \
     --resource-group <resource-group>

2. Required permissions:

• Microsoft.Kubernetes/connectedClusters (read/write)
• Microsoft.KubernetesConfiguration/extensions (read/write)

Azure Kubernetes Service (AKS)

1. MSI-based AKS cluster (not SPN):

   # Create MSI-based AKS cluster
   az aks create --resource-group <rg> --name <cluster> \
     --enable-managed-identity

   # Convert existing SPN cluster to MSI
   az aks update -g <rg> -n <cluster> --enable-managed-identity

2. Required permissions:

• Microsoft.ContainerService/managedClusters (read/write)
• Microsoft.KubernetesConfiguration/extensions (read/write)

Common Requirements

# Register Azure providers
az provider register --namespace Microsoft.Kubernetes
az provider register --namespace Microsoft.ContainerService
az provider register --namespace Microsoft.KubernetesConfiguration

# Install CLI extensions
az extension add -n k8s-configuration
az extension add -n k8s-extension

# Verify registration (wait for 'Registered' state)
az provider show -n Microsoft.KubernetesConfiguration -o table

---

Network Requirements

The GitOps agents require outbound access to:

---

Installation Methods

Method 1: Simple Installation (Single Node)

For development or single-node clusters:

az k8s-extension create \
  --resource-group <resource-group> \
  --cluster-name <cluster-name> \
  --cluster-type managedClusters \
  --name argocd \
  --extension-type Microsoft.ArgoCD \
  --release-train preview \
  --config deployWithHighAvailability=false \
  --config namespaceInstall=false \
  --config "config-maps.argocd-cmd-params-cm.data.application\.namespaces=namespace1,namespace2"

Parameters:

Method 2: High Availability Installation (Production)

For production with 3+ nodes:

az k8s-extension create \
  --resource-group <resource-group> \
  --cluster-name <cluster-name> \
  --cluster-type managedClusters \
  --name argocd \
  --extension-type Microsoft.ArgoCD \
  --release-train preview \
  --config namespaceInstall=false \
  --config "config-maps.argocd-cmd-params-cm.data.application\.namespaces=default,argocd"

Method 3: Namespace-Scoped Installation

For multi-tenant clusters with isolated ArgoCD instances:

az k8s-extension create \
  --resource-group <resource-group> \
  --cluster-name <cluster-name> \
  --cluster-type managedClusters \
  --name argocd-team-a \
  --extension-type Microsoft.ArgoCD \
  --release-train preview \
  --config namespaceInstall=true \
  --target-namespace team-a-argocd

---

Workload Identity Integration (Recommended for Production)

Workload identity enables Azure AD authentication without managing secrets.

Bicep Template

var clusterName = '<aks-or-arc-cluster-name>'
var workloadIdentityClientId = '<managed-identity-client-id>'
var ssoWorkloadIdentityClientId = '<sso-managed-identity-client-id>'
var url = 'https://<public-ip-for-argocd-ui>/'

var oidcConfig = '''
name: Azure
issuer: https://login.microsoftonline.com/<your-tenant-id>/v2.0
clientID: <sso-client-id>
azure:
  useWorkloadIdentity: true
requestedIDTokenClaims:
  groups:
    essential: true
requestedScopes:
  - openid
  - profile
  - email
'''

var defaultPolicy = 'role:readonly'
var policy = '''
p, role:org-admin, applications, *, */*, allow
p, role:org-admin, clusters, get, *, allow
p, role:org-admin, repositories, get, *, allow
p, role:org-admin, repositories, create, *, allow
p, role:org-admin, repositories, update, *, allow
p, role:org-admin, repositories, delete, *, allow
g, <entra-group-id>, role:org-admin
'''

resource cluster 'Microsoft.ContainerService/managedClusters@2024-10-01' existing = {
  name: clusterName
}

resource extension 'Microsoft.KubernetesConfiguration/extensions@2023-05-01' = {
  name: 'argocd'
  scope: cluster
  properties: {
    extensionType: 'Microsoft.ArgoCD'
    releaseTrain: 'preview'
    configurationSettings: {
      'workloadIdentity.enable': 'true'
      'workloadIdentity.clientId': workloadIdentityClientId
      'workloadIdentity.entraSSOClientId': ssoWorkloadIdentityClientId
      'config-maps.argocd-cm.data.oidc\\.config': oidcConfig
      'config-maps.argocd-cm.data.url': url
      'config-maps.argocd-rbac-cm.data.policy\\.default': defaultPolicy
      'config-maps.argocd-rbac-cm.data.policy\\.csv': policy
      'config-maps.argocd-cmd-params-cm.data.application\\.namespaces': 'default, argocd'
    }
  }
}

Deploy with Bicep

az deployment group create \
  --resource-group <resource-group> \
  --template-file argocd-extension.bicep

Setup Workload Identity Credentials

1. Retrieve OIDC issuer URL:

   # For AKS
   az aks show -n <cluster> -g <rg> --query "oidcIssuerProfile.issuerUrl" -o tsv

   # For Arc-enabled Kubernetes
   az connectedk8s show -n <cluster> -g <rg> --query "oidcIssuerProfile.issuerUrl" -o tsv

2. Create managed identity:

   az identity create --name argocd-identity --resource-group <rg>

3. Create federated credential:

   az identity federated-credential create \
     --name argocd-federated \
     --identity-name argocd-identity \
     --resource-group <rg> \
     --issuer <oidc-issuer-url> \
     --subject system:serviceaccount:argocd:source-controller \
     --audience api://AzureADTokenExchange

4. Grant ACR permissions (if using Azure Container Registry):

   # For ABAC-enabled registries
   az role assignment create \
     --role "Container Registry Repository Reader" \
     --assignee <identity-client-id> \
     --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ContainerRegistry/registries/<acr>

   # For non-ABAC registries
   az role assignment create \
     --role "AcrPull" \
     --assignee <identity-client-id> \
     --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ContainerRegistry/registries/<acr>

---

Accessing ArgoCD UI

Option 1: LoadBalancer Service

kubectl -n argocd expose service argocd-server \
  --type LoadBalancer \
  --name argocd-server-lb \
  --port 80 \
  --target-port 8080

Option 2: Ingress Controller

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  rules:
    - host: argocd.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 443

Option 3: Port Forward (Development)

kubectl port-forward svc/argocd-server -n argocd 8080:443

---

Deploying Applications

Example: AKS Store Demo

kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: aks-store-demo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/Azure-Samples/aks-store-demo.git
    targetRevision: HEAD
    path: kustomize/overlays/dev
  syncPolicy:
    automated: {}
  destination:
    namespace: pets
    server: https://kubernetes.default.svc
EOF

Multi-Source with Azure Container Registry

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  sources:
    # Helm chart from ACR
    - repoURL: <acr-name>.azurecr.io/helm
      chart: myapp
      targetRevision: 1.0.0
      helm:
        valueFiles:
          - $values/overlays/prod/values.yaml
    # Values from Git
    - repoURL: https://github.com/org/config.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp

---

Updating Configuration

Update ArgoCD configmaps through the extension (not directly via kubectl):

az k8s-extension update \
  --resource-group <resource-group> \
  --cluster-name <cluster-name> \
  --cluster-type managedClusters \
  --name argocd \
  --config "config-maps.argocd-cm.data.url=https://<new-public-ip>/auth/callback"

---

Connecting to Private ACR

For private Azure Container Registry access with workload identity:

1. Use workload identity (configured above) 2. Add repository in ArgoCD:

   argocd repo add <acr-name>.azurecr.io \
     --type helm \
     --name azure-acr \
     --enable-oci

---

Deleting the Extension

az k8s-extension delete \
  -g <resource-group> \
  -c <cluster-name> \
  -n argocd \
  -t managedClusters \
  --yes

---

Comparison: Azure Extension vs Manual Installation

---

Troubleshooting

Extension Installation Failed

# Check extension status
az k8s-extension show \
  -g <rg> -c <cluster> -t managedClusters \
  -n argocd

# Check ArgoCD pods
kubectl get pods -n argocd

# Check extension operator logs
kubectl logs -n azure-arc -l app.kubernetes.io/component=extension-manager

Workload Identity Issues

# Verify federated credential
az identity federated-credential list \
  --identity-name argocd-identity \
  --resource-group <rg>

# Check service account annotation
kubectl get sa -n argocd source-controller -o yaml

Sync Failures

# Check ArgoCD application status
argocd app get <app-name>

# Check repo server logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server

---

Best Practices for Azure GitOps

1. Use Workload Identity: Avoid storing secrets; use Azure AD authentication 2. Private Endpoints: Use Azure Private Link for ACR and Key Vault 3. Azure Policy: Enforce GitOps compliance with Azure Policy 4. Azure Monitor: Integrate ArgoCD metrics with Azure Monitor 5. Separate Environments: Use different resource groups for dev/staging/prod 6. RBAC: Map Azure AD groups to ArgoCD roles

---

References

• Microsoft Learn: GitOps with ArgoCD Tutorial
• Azure Arc-enabled Kubernetes Documentation
• AKS Workload Identity
• ArgoCD Azure AD OIDC Configuration

GitOps Core Principles

Deep dive into the four foundational principles of GitOps as defined by the OpenGitOps project (CNCF).

The Four Pillars of GitOps

Principle 1: Declarative

"A system managed by GitOps must have its desired state expressed declaratively."

What This Means

• Declarative: Describe WHAT you want, not HOW to achieve it
• Imperative (opposite): Step-by-step instructions to reach a state

Examples

Declarative (GitOps):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.21
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"

Imperative (NOT GitOps):

kubectl run nginx --image=nginx:1.21
kubectl scale deployment nginx --replicas=3
kubectl set resources deployment nginx -c=nginx --limits=memory=128Mi,cpu=500m

Why Declarative Matters

Declarative Tools

---

Principle 2: Versioned and Immutable

"Desired state is stored in a way that enforces immutability, versioning, and retains a complete version history."

Git as the Version Control System

Git provides:

• Immutability: Commits are content-addressed (SHA)
• Versioning: Full history of changes
• Branching: Parallel development streams
• Audit trail: Who changed what, when, why

Version Control Best Practices

# Good commit message structure
git commit -m "feat(nginx): increase replicas to 3 for high availability

- Scaling nginx deployment from 1 to 3 replicas
- Adding pod anti-affinity for distribution
- Tested in staging environment

Relates to: TICKET-123"

Immutability Patterns

Container Images:

# GOOD: Immutable tag
image: nginx:1.21.6

# BAD: Mutable tag
image: nginx:latest

Git References:

# GOOD: Specific commit or tag
targetRevision: v1.2.3
targetRevision: abc123def

# RISKY: Branch (mutable)
targetRevision: main

Version History Benefits

# View deployment history
git log --oneline manifests/

# Compare versions
git diff v1.0.0..v1.1.0 -- manifests/

# Find when change was introduced
git bisect start
git bisect bad HEAD
git bisect good v1.0.0

---

Principle 3: Pulled Automatically

"Software agents automatically pull the desired state declarations from the source."

Pull vs Push Model

┌──────────────────────────────────────────────────────────────────┐
│                     PUSH MODEL (Traditional)                      │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│   CI/CD Server                                                   │
│       │                                                          │
│       │ kubectl apply                                            │
│       │ (requires cluster credentials)                           │
│       ▼                                                          │
│   Kubernetes Cluster                                             │
│                                                                  │
│   Issues:                                                        │
│   - Credentials exposed in CI                                    │
│   - No continuous reconciliation                                 │
│   - Drift goes undetected                                        │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────┐
│                     PULL MODEL (GitOps)                           │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Git Repository ◄──────────────┐                                │
│       │                         │                                │
│       │ (pull/watch)            │ Developer pushes               │
│       ▼                         │                                │
│   GitOps Controller ────────────┘                                │
│   (inside cluster)                                               │
│       │                                                          │
│       │ kubectl apply                                            │
│       │ (internal credentials)                                   │
│       ▼                                                          │
│   Kubernetes Cluster                                             │
│                                                                  │
│   Benefits:                                                      │
│   - Credentials stay in cluster                                  │
│   - Continuous reconciliation                                    │
│   - Automatic drift detection                                    │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

Automatic Pull Mechanisms

Polling (Default):

# ArgoCD application controller settings
spec:
  syncPolicy:
    automated: {}
# Default poll interval: 3 minutes

Webhook (Recommended for Production):

# Configure webhook for instant updates
apiVersion: v1
kind: Secret
metadata:
  name: argocd-webhook
  namespace: argocd
data:
  github.secret: <base64-encoded-secret>

Git Generators (ApplicationSets):

spec:
  generators:
    - git:
        repoURL: https://github.com/org/repo.git
        revision: HEAD
        directories:
          - path: apps/*

Security Benefits of Pull

---

Principle 4: Continuously Reconciled

"Software agents continuously observe actual system state and attempt to apply the desired state."

The Reconciliation Loop

┌─────────────────────────────────────────────────────────────────┐
│                    CONTINUOUS RECONCILIATION                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   ┌──────────────┐                                              │
│   │ Git (Desired)│                                              │
│   │    State     │                                              │
│   └──────┬───────┘                                              │
│          │                                                      │
│          ▼                                                      │
│   ┌──────────────┐     ┌──────────────┐                        │
│   │   Compare    │◄────│   Observe    │                        │
│   │    (Diff)    │     │    Actual    │                        │
│   └──────┬───────┘     └──────┬───────┘                        │
│          │                    │                                 │
│          │                    │                                 │
│          ▼                    │                                 │
│   ┌──────────────┐           │                                 │
│   │    Apply     │───────────┘                                  │
│   │   Changes    │     Repeat continuously                      │
│   └──────────────┘                                              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Self-Healing Capabilities

spec:
  syncPolicy:
    automated:
      selfHeal: true    # Automatically revert manual changes
      prune: true       # Remove resources not in Git

Self-Heal Scenarios:

Drift Detection

Types of Drift:

1. Configuration Drift: Resource spec differs from Git 2. State Drift: Resource status unhealthy 3. Missing Resources: Resources deleted manually 4. Extra Resources: Resources created outside Git

Detecting Drift:

# ArgoCD diff command
argocd app diff myapp

# Flux reconciliation
flux reconcile kustomization myapp --with-source

Reconciliation Intervals

Example Configuration:

# ArgoCD ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  timeout.reconciliation: 180s  # 3 minutes

# Flux Kustomization
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
spec:
  interval: 5m  # Check every 5 minutes

---

Putting It All Together

The Complete GitOps Workflow

1. Developer creates PR with DECLARATIVE changes
                    │
                    ▼
2. Changes reviewed and VERSIONED in Git
                    │
                    ▼
3. GitOps controller PULLS changes automatically
                    │
                    ▼
4. Controller CONTINUOUSLY RECONCILES cluster state
                    │
                    ▼
5. Drift detected? → Auto-heal OR alert

Compliance with Principles Checklist

• [ ] All configurations are YAML/HCL (declarative)
• [ ] All changes go through Git PR (versioned)
• [ ] No kubectl apply from laptops (pulled)
• [ ] Self-heal enabled (continuously reconciled)
• [ ] Drift alerts configured (continuously reconciled)

References

• OpenGitOps Principles
• CNCF GitOps Working Group
• GitOps Principles v1.0.0

GitOps Patterns and Practices

Comprehensive guide to repository structures, branching strategies, and deployment patterns for GitOps implementations.

Repository Structure Patterns

Pattern 1: Monorepo

All applications and environments in a single repository.

gitops-monorepo/
├── apps/
│   ├── frontend/
│   │   ├── base/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── kustomization.yaml
│   │   └── overlays/
│   │       ├── dev/
│   │       │   ├── kustomization.yaml
│   │       │   └── replica-patch.yaml
│   │       ├── staging/
│   │       └── production/
│   ├── backend/
│   └── database/
├── infrastructure/
│   ├── cert-manager/
│   ├── ingress-nginx/
│   └── monitoring/
├── clusters/
│   ├── dev/
│   ├── staging/
│   └── production/
└── README.md

Pros:

• Single source of truth
• Easy cross-application changes
• Atomic multi-app deployments
• Simplified tooling

Cons:

• Large repository over time
• Broad access permissions needed
• CI/CD triggers for all changes
• Potential merge conflicts

Best For: Small-medium teams, tightly coupled applications

---

Pattern 2: Polyrepo (Multi-Repository)

Separate repositories per application or concern.

# Application repositories
app-frontend/
├── src/
├── Dockerfile
└── k8s/
    ├── base/
    └── overlays/

app-backend/
├── src/
├── Dockerfile
└── k8s/

# Infrastructure repository
platform-infrastructure/
├── cert-manager/
├── ingress/
└── monitoring/

# Cluster configuration
cluster-config/
├── dev/
├── staging/
└── production/

Pros:

• Fine-grained access control
• Independent release cycles
• Smaller, focused repositories
• Team autonomy

Cons:

• Harder to coordinate changes
• More repositories to manage
• Complex dependency tracking
• Potential version drift

Best For: Large organizations, microservices, multiple teams

---

Pattern 3: App of Apps (Umbrella Pattern)

A parent Application manages child Applications.

# Root Application (apps/root-app.yaml)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/gitops-config.git
    targetRevision: HEAD
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd

gitops-config/
├── apps/
│   ├── frontend.yaml      # Application CR
│   ├── backend.yaml       # Application CR
│   ├── monitoring.yaml    # Application CR
│   └── kustomization.yaml
└── manifests/
    ├── frontend/
    ├── backend/
    └── monitoring/

Benefits:

• Hierarchical organization
• Single sync point
• Environment-specific app sets
• Easy to add/remove apps

---

Pattern 4: Multi-Repository with Values Separation

Separates infrastructure definitions from environment-specific values.

# Repository 1: Infrastructure (infra-team/)
infra-team/
├── applications/
│   ├── nginx/
│   │   ├── applicationset.yaml
│   │   └── base-values.yaml
│   └── prometheus/
└── applicationsets/

# Repository 2: Values (argo-cd-helm-values/)
argo-cd-helm-values/
├── dev/
│   ├── nginx/
│   │   └── values.yaml
│   └── prometheus/
├── staging/
└── production/

ApplicationSet with Multi-Source:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: nginx
spec:
  generators:
    - list:
        elements:
          - cluster: dev
            url: https://dev-cluster
          - cluster: prod
            url: https://prod-cluster
  template:
    spec:
      sources:
        - repoURL: https://charts.bitnami.com/bitnami
          chart: nginx
          targetRevision: 15.0.0
          helm:
            valueFiles:
              - $values/{{cluster}}/nginx/values.yaml
        - repoURL: https://github.com/org/argo-cd-helm-values.git
          targetRevision: main
          ref: values

Benefits:

• Security boundary between repos
• Different RBAC per environment
• Separation of concerns
• Easier secret management

---

Branching Strategies

Strategy 1: Environment Branches

main ──────────────────────────────────────► Production
  │
  └── staging ─────────────────────────────► Staging
        │
        └── develop ───────────────────────► Development

Workflow:

1. Develop on develop branch 2. Merge to staging for testing 3. Merge to main for production

Pros: Clear environment mapping Cons: Merge conflicts, branch maintenance

---

Strategy 2: Trunk-Based with Directory Overlays

main (single branch)
├── base/                 # Shared configuration
├── overlays/
│   ├── dev/             # Dev-specific patches
│   ├── staging/         # Staging-specific patches
│   └── production/      # Prod-specific patches

Workflow:

1. All changes go to main 2. Kustomize overlays handle environment differences 3. GitOps controller watches specific paths

Pros: Simple, fewer branches, atomic changes Cons: Requires good overlay discipline

---

Strategy 3: Release Branches

main
  │
  ├── release/v1.0.0 ──► Production (v1.0)
  ├── release/v1.1.0 ──► Production (v1.1)
  └── release/v2.0.0 ──► Production (v2.0)

Workflow:

1. Develop on main 2. Create release branch for deployment 3. Hotfixes on release branches 4. Cherry-pick to main

Pros: Clear versioning, rollback by switching branches Cons: Branch proliferation, merge complexity

---

Strategy 4: GitFlow for GitOps

main ────────────────────────────────────────► Production
  ▲
  │
  │ merge
  │
develop ─────────────────────────────────────► Development
  ▲     ▲
  │     │
feature/  release/
branches  branches

Best For: Complex release processes, multiple parallel versions

---

Deployment Patterns

Progressive Delivery Patterns

Blue-Green Deployment

# Blue deployment (current production)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-blue
spec:
  source:
    path: manifests/blue
  destination:
    namespace: myapp-blue

---
# Green deployment (new version)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-green
spec:
  source:
    path: manifests/green
  destination:
    namespace: myapp-green

Traffic Switch:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
spec:
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /
            backend:
              service:
                name: myapp-green  # Switch from blue to green
                port:
                  number: 80

Canary Deployment

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        - setWeight: 10        # 10% traffic to canary
        - pause: {duration: 5m}
        - setWeight: 30
        - pause: {duration: 10m}
        - setWeight: 50
        - pause: {duration: 10m}
      trafficRouting:
        nginx:
          stableIngress: myapp-stable

Wave-Based Deployment

# Wave 1: Infrastructure
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

# Wave 2: Database migrations
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"

# Wave 3: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

# Wave 4: Post-deployment jobs
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"

---

Multi-Cluster Patterns

Hub and Spoke

                    ┌─────────────────┐
                    │   Hub Cluster   │
                    │   (ArgoCD)      │
                    └────────┬────────┘
                             │
            ┌────────────────┼────────────────┐
            │                │                │
            ▼                ▼                ▼
    ┌───────────┐    ┌───────────┐    ┌───────────┐
    │  Spoke 1  │    │  Spoke 2  │    │  Spoke 3  │
    │  (Dev)    │    │  (Staging)│    │  (Prod)   │
    └───────────┘    └───────────┘    └───────────┘

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: production
  template:
    spec:
      destination:
        server: '{{server}}'
        namespace: myapp

Pull-Based Multi-Cluster

Each cluster has its own GitOps controller:

Git Repository
      │
      ├──────────────────┬──────────────────┐
      │                  │                  │
      ▼                  ▼                  ▼
┌───────────┐    ┌───────────┐    ┌───────────┐
│ Cluster 1 │    │ Cluster 2 │    │ Cluster 3 │
│ ArgoCD    │    │ Flux      │    │ ArgoCD    │
└───────────┘    └───────────┘    └───────────┘

---

Environment Promotion Pattern

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│     Dev      │────►│   Staging    │────►│  Production  │
│              │     │              │     │              │
│ Auto-deploy  │     │ Auto-deploy  │     │ Manual gate  │
│ from main    │     │ after dev    │     │ approval     │
└──────────────┘     └──────────────┘     └──────────────┘

Kargo Implementation:

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: production
spec:
  requestedFreight:
    - origin:
        kind: Warehouse
        name: main-warehouse
      sources:
        stages:
          - staging
        requiredSoakTime: 24h  # Must be stable in staging for 24h

---

Best Practices Summary

Repository Best Practices

Branching Best Practices

Deployment Best Practices

Quick Reference

Choose Your Pattern

GitOps Tooling Ecosystem

Comprehensive comparison of GitOps tools, their architectures, and use cases.

Tool Comparison Overview

---

ArgoCD

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        ArgoCD Components                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────────┐    ┌──────────────────┐                  │
│  │   API Server     │◄──►│    Web UI        │                  │
│  │   (gRPC/REST)    │    │                  │                  │
│  └────────┬─────────┘    └──────────────────┘                  │
│           │                                                     │
│           ▼                                                     │
│  ┌──────────────────┐    ┌──────────────────┐                  │
│  │ Repo Server      │    │ Redis            │                  │
│  │ (Git operations) │    │ (Caching)        │                  │
│  └────────┬─────────┘    └──────────────────┘                  │
│           │                                                     │
│           ▼                                                     │
│  ┌──────────────────┐    ┌──────────────────┐                  │
│  │ Application      │    │ Dex              │                  │
│  │ Controller       │    │ (SSO/OIDC)       │                  │
│  │ (Reconciliation) │    │                  │                  │
│  └──────────────────┘    └──────────────────┘                  │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Key Features

Application CRD:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/repo.git
    targetRevision: HEAD
    path: manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

ApplicationSet (Multi-App Generation):

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-apps
spec:
  generators:
    - clusters: {}  # All registered clusters
  template:
    spec:
      destination:
        server: '{{server}}'

Strengths

• Rich web UI with visualization
• Multi-cluster management from single control plane
• SSO integration (OIDC, SAML, LDAP)
• ApplicationSets for templated deployments
• Extensive sync options and hooks
• Large community and ecosystem

Considerations

• Resource-intensive for large deployments
• Single point of failure (hub cluster)
• Requires dedicated namespace

Installation

Standard Installation:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Azure Arc / AKS Managed Extension:

# Register providers
az provider register --namespace Microsoft.KubernetesConfiguration
az extension add -n k8s-extension

# Install ArgoCD as managed extension
az k8s-extension create \
  --resource-group <rg> --cluster-name <cluster> \
  --cluster-type managedClusters \
  --name argocd \
  --extension-type Microsoft.ArgoCD \
  --release-train preview \
  --config deployWithHighAvailability=false

Benefits of Azure managed extension:

• Managed upgrades and maintenance
• Native Azure AD workload identity integration
• Consistent multi-cluster management via Azure Arc
• See azure-arc-integration.md for complete guide

---

Flux

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Flux Components                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────────┐    ┌──────────────────┐                  │
│  │ Source Controller│    │ Kustomize        │                  │
│  │ (Git, Helm, OCI) │    │ Controller       │                  │
│  └────────┬─────────┘    └────────┬─────────┘                  │
│           │                       │                             │
│           └───────────┬───────────┘                             │
│                       ▼                                         │
│  ┌──────────────────┐    ┌──────────────────┐                  │
│  │ Helm Controller  │    │ Notification     │                  │
│  │                  │    │ Controller       │                  │
│  └──────────────────┘    └──────────────────┘                  │
│                                                                 │
│  ┌──────────────────┐    ┌──────────────────┐                  │
│  │ Image Automation │    │ Image Reflector  │                  │
│  │ Controller       │    │ Controller       │                  │
│  └──────────────────┘    └──────────────────┘                  │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Key Features

GitRepository Source:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/org/repo.git
  ref:
    branch: main
  secretRef:
    name: git-credentials

Kustomization (Flux CRD, not Kustomize):

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: myapp
  path: ./manifests
  prune: true
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: myapp
      namespace: default

HelmRelease:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: nginx
  namespace: default
spec:
  interval: 5m
  chart:
    spec:
      chart: nginx
      version: '15.x'
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
  values:
    replicaCount: 2

Strengths

• Lightweight, modular architecture
• No single point of failure
• Native image automation (update manifests on new image)
• OCI registry support for storing configs
• Multi-tenancy via namespaces
• Terraform Controller integration

Considerations

• No built-in UI (requires Weave GitOps or similar)
• Steeper learning curve for CRD relationships
• Each cluster needs its own Flux instance

Installation

flux bootstrap github \
  --owner=my-org \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/my-cluster \
  --personal

---

Kargo

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Kargo Components                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────────┐    ┌──────────────────┐                  │
│  │   Warehouse      │    │   Stages         │                  │
│  │   (Artifact      │    │   (Promotion     │                  │
│  │    Discovery)    │    │    Targets)      │                  │
│  └────────┬─────────┘    └────────┬─────────┘                  │
│           │                       │                             │
│           └───────────┬───────────┘                             │
│                       │                                         │
│                       ▼                                         │
│  ┌──────────────────────────────────────────┐                  │
│  │              Freight                      │                  │
│  │  (Versioned artifact collection)          │                  │
│  └──────────────────────────────────────────┘                  │
│                       │                                         │
│                       ▼                                         │
│  ┌──────────────────────────────────────────┐                  │
│  │           Promotions                      │                  │
│  │  (Move Freight through Stages)            │                  │
│  └──────────────────────────────────────────┘                  │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Key Features

Warehouse (Artifact Discovery):

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: main-warehouse
  namespace: myproject
spec:
  subscriptions:
    - image:
        repoURL: ghcr.io/org/myapp
        imageSelectionStrategy: SemVer
    - git:
        repoURL: https://github.com/org/config.git
        branch: main

Stage (Promotion Target):

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: staging
  namespace: myproject
spec:
  requestedFreight:
    - origin:
        kind: Warehouse
        name: main-warehouse
      sources:
        direct: true
  promotionTemplate:
    spec:
      steps:
        - uses: git-clone
        - uses: kustomize-set-image
        - uses: git-commit
        - uses: git-push
        - uses: argocd-update

Strengths

• Purpose-built for environment promotion
• Coordinates image + config updates
• Verification (testing) between stages
• Soak time requirements
• Works alongside ArgoCD
• Progressive delivery native

Considerations

• Newer project (less mature)
• Requires ArgoCD or Flux for actual deployment
• Additional complexity layer

Installation

helm install kargo \
  oci://ghcr.io/akuity/kargo-charts/kargo \
  --namespace kargo \
  --create-namespace

---

Tool Selection Guide

Decision Matrix

Architecture Recommendations

Small Team (< 5 developers):

Single cluster
    │
    └── Flux or ArgoCD (standalone)

Medium Team (5-20 developers):

ArgoCD (hub cluster)
    │
    ├── Dev cluster
    ├── Staging cluster
    └── Prod cluster

Large Organization (20+ developers):

Kargo + ArgoCD
    │
    ├── ArgoCD manages deployments
    └── Kargo manages promotions
        │
        ├── Dev stages
        ├── Staging stages (with verification)
        └── Prod stages (with approval gates)

---

Complementary Tools

Secrets Management

Progressive Delivery

Policy Enforcement

Observability

---

Migration Paths

From Helm/kubectl to ArgoCD

1. Export existing Helm releases as values files 2. Create ArgoCD Applications pointing to charts 3. Disable Helm Tiller/manual deployments 4. Enable automated sync

From ArgoCD to Flux

1. Export Applications as Flux Kustomizations 2. Deploy Flux controllers 3. Migrate repo credentials 4. Decommission ArgoCD

Adding Kargo to Existing ArgoCD

1. Install Kargo alongside ArgoCD 2. Create Warehouses for artifact sources 3. Create Stages matching your environments 4. Add kargo.akuity.io/authorized-stage annotations to ArgoCD Applications 5. Define promotion templates

---

Quick CLI Reference

ArgoCD

argocd app list
argocd app sync myapp
argocd app get myapp
argocd app diff myapp
argocd app history myapp
argocd app rollback myapp 2

Flux

flux get kustomizations
flux reconcile kustomization myapp
flux get sources git
flux logs --kind=Kustomization --name=myapp
flux suspend kustomization myapp
flux resume kustomization myapp

Kargo

kargo get stages --project myproject
kargo get freight --project myproject
kargo promote --project myproject --freight <id> --stage prod
kargo approve --project myproject --freight <id> --stage prod

GitOps Troubleshooting Guide

Comprehensive debugging guide for common GitOps issues with ArgoCD, Flux, and Kubernetes.

Quick Diagnostic Commands

ArgoCD Quick Checks

# Application status overview
argocd app list

# Detailed app info
argocd app get myapp

# Show diff between Git and cluster
argocd app diff myapp

# Force refresh from Git
argocd app get myapp --refresh

# View sync history
argocd app history myapp

# Check ArgoCD components health
kubectl get pods -n argocd

Flux Quick Checks

# Overall Flux status
flux check

# Kustomization status
flux get kustomizations -A

# Source status
flux get sources git -A

# Reconcile immediately
flux reconcile kustomization myapp --with-source

# View logs
flux logs --kind=Kustomization --name=myapp

Kubernetes Quick Checks

# Pod status
kubectl get pods -n myapp

# Recent events
kubectl get events -n myapp --sort-by='.lastTimestamp'

# Describe problematic resource
kubectl describe deployment myapp -n myapp

# Pod logs
kubectl logs -l app=myapp -n myapp --tail=100

---

Common Issues and Solutions

Issue 1: Application Stuck in "OutOfSync"

Symptoms:

• Application shows OutOfSync status
• Sync button doesn't resolve the issue
• Diff shows unexpected differences

Diagnostic Steps:

# Step 1: View the diff
argocd app diff myapp

# Step 2: Check for ignored differences
argocd app get myapp -o yaml | grep -A 20 ignoreDifferences

# Step 3: Force a hard refresh
argocd app get myapp --hard-refresh

Common Causes and Fixes:

Cause A: Server-side modifications (mutating webhooks, controllers)

# Fix: Add ignoreDifferences to Application
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # Ignored by HPA
    - group: ""
      kind: Service
      jsonPointers:
        - /spec/clusterIP  # Auto-assigned

Cause B: Defaulting by Kubernetes API

# Fix: Use Server-Side Apply
spec:
  syncPolicy:
    syncOptions:
      - ServerSideApply=true

Cause C: Resource created outside Git

# Identify extra resources
argocd app resources myapp

# Either:
# 1. Add to Git
# 2. Enable pruning
# 3. Add to exclude patterns

---

Issue 2: Sync Failed

Symptoms:

• Application shows Sync Failed
• Error message in sync operation

Diagnostic Steps:

# Step 1: Get sync status details
argocd app get myapp

# Step 2: View sync operation result
argocd app sync myapp --dry-run

# Step 3: Check ArgoCD controller logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller

Common Causes and Fixes:

Cause A: Invalid YAML/Manifest

# Validate manifests locally
kustomize build ./overlays/prod | kubectl apply --dry-run=client -f -

# Or for Helm
helm template myrelease ./chart --validate

Cause B: RBAC Permissions

# Check ArgoCD service account permissions
kubectl auth can-i create deployments --as=system:serviceaccount:argocd:argocd-application-controller -n myapp

# Fix: Add namespace to AppProject destinations
spec:
  destinations:
    - namespace: myapp
      server: https://kubernetes.default.svc

Cause C: Resource Conflict

# Check if resource exists with different manager
kubectl get deployment myapp -n myapp -o yaml | grep -A 5 managedFields

# Fix: Force replace
spec:
  syncPolicy:
    syncOptions:
      - Replace=true

---

Issue 3: Application Degraded

Symptoms:

• Application shows Degraded health status
• Pods not running correctly

Diagnostic Steps:

# Step 1: Get health details
argocd app get myapp

# Step 2: Check pod status
kubectl get pods -n myapp
kubectl describe pod <pod-name> -n myapp

# Step 3: Check pod logs
kubectl logs <pod-name> -n myapp --previous  # For crashed containers

Common Causes and Fixes:

Cause A: Image Pull Failure

# Check events
kubectl get events -n myapp | grep -i pull

# Verify image exists
docker pull myregistry/myapp:v1.0.0

# Check imagePullSecrets
kubectl get deployment myapp -n myapp -o yaml | grep -A 5 imagePullSecrets

Cause B: Resource Limits

# Check for OOMKilled
kubectl get pods -n myapp -o jsonpath='{.items[*].status.containerStatuses[*].lastState.terminated.reason}'

# Check resource usage
kubectl top pods -n myapp

Cause C: Readiness Probe Failure

# Check probe configuration
kubectl get deployment myapp -n myapp -o yaml | grep -A 10 readinessProbe

# Test endpoint manually
kubectl exec -it <pod-name> -n myapp -- curl localhost:8080/health

---

Issue 4: Repository Connection Failed

Symptoms:

• ArgoCD can't connect to Git repository
• ComparisonError or Unable to fetch repository

Diagnostic Steps:

# Step 1: Check repository status
argocd repo list

# Step 2: Test connection
argocd repo get https://github.com/org/repo.git

# Step 3: Check repo-server logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server

Common Causes and Fixes:

Cause A: Authentication Failure

# Update credentials
argocd repo add https://github.com/org/repo.git \
  --username git \
  --password $GITHUB_TOKEN \
  --upsert

Cause B: SSH Key Issues

# Check known hosts
argocd cert list

# Add SSH key
argocd repo add git@github.com:org/repo.git \
  --ssh-private-key-path ~/.ssh/id_rsa

Cause C: Network/Firewall

# Test from repo-server pod
kubectl exec -it -n argocd <repo-server-pod> -- \
  git ls-remote https://github.com/org/repo.git

---

Issue 5: Webhook Not Triggering

Symptoms:

• Changes pushed to Git but no sync
• Waiting for poll interval

Diagnostic Steps:

# Step 1: Check webhook configuration in Git provider

# Step 2: Verify ArgoCD webhook endpoint
curl -X POST https://argocd.example.com/api/webhook

# Step 3: Check API server logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server | grep webhook

Common Causes and Fixes:

Cause A: Wrong Webhook URL

# Correct URL format
https://argocd.example.com/api/webhook

# NOT
https://argocd.example.com/webhook
https://argocd.example.com/api/v1/webhook

Cause B: Secret Mismatch

# Verify webhook secret in argocd-secret
kubectl get secret argocd-secret -n argocd -o yaml | grep webhook

Cause C: Ingress/Load Balancer Issue

# Check if webhook endpoint is reachable
curl -v https://argocd.example.com/api/webhook

---

Issue 6: Slow Sync Performance

Symptoms:

• Syncs take a long time
• Timeouts during sync
• High resource usage on ArgoCD

Diagnostic Steps:

# Step 1: Check resource usage
kubectl top pods -n argocd

# Step 2: Check number of resources
argocd app resources myapp | wc -l

# Step 3: Check controller metrics
kubectl port-forward -n argocd svc/argocd-metrics 8082:8082
curl localhost:8082/metrics | grep argocd_app

Optimization Steps:

1. Increase Controller Resources:

# argocd-application-controller deployment
resources:
  limits:
    cpu: "2"
    memory: "2Gi"
  requests:
    cpu: "500m"
    memory: "512Mi"

2. Split Large Applications:

# Instead of one app with 500 resources
# Create multiple smaller apps

3. Optimize Sync Options:

spec:
  syncPolicy:
    syncOptions:
      - ApplyOutOfSyncOnly=true  # Only sync changed resources

4. Adjust Reconciliation Timeout:

# In argocd-cm ConfigMap
data:
  timeout.reconciliation: 300s

---

Issue 7: Multi-Cluster Connection Issues

Symptoms:

• External cluster shows as disconnected
• Applications targeting external cluster fail

Diagnostic Steps:

# Step 1: List clusters
argocd cluster list

# Step 2: Check cluster status
argocd cluster get https://external-cluster:6443

# Step 3: Verify cluster secret
kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster

Common Causes and Fixes:

Cause A: Expired Credentials

# Rotate cluster credentials
argocd cluster rotate-auth https://external-cluster:6443

Cause B: Network Connectivity

# Test from ArgoCD pod
kubectl exec -it -n argocd <application-controller-pod> -- \
  curl -k https://external-cluster:6443/healthz

Cause C: Certificate Issues

# Re-add cluster with updated certs
argocd cluster add external-context --name external-cluster

---

Debugging Checklist

Pre-Sync Checklist

• [ ] Manifests are valid YAML
• [ ] Image tags exist and are pullable
• [ ] Secrets/ConfigMaps referenced exist
• [ ] Namespace exists or CreateNamespace=true
• [ ] RBAC allows ArgoCD to create resources
• [ ] Resource quotas won't block creation

Post-Failure Checklist

• [ ] Check ArgoCD UI for error messages
• [ ] Review argocd app diff output
• [ ] Check Kubernetes events in target namespace
• [ ] Review ArgoCD controller logs
• [ ] Verify Git repository is accessible
• [ ] Check for webhook delivery failures

Performance Checklist

• [ ] Applications are appropriately sized
• [ ] ApplyOutOfSyncOnly enabled where appropriate
• [ ] Controller resources adequate
• [ ] Redis cache functioning
• [ ] Repository server not overloaded

---

Log Locations

---

Emergency Procedures

Force Sync (Override Errors)

# ArgoCD
argocd app sync myapp --force --prune

# Flux
flux reconcile kustomization myapp --force

Disable Auto-Sync (Stop Reconciliation)

# ArgoCD - patch application
argocd app set myapp --sync-policy none

# Flux - suspend kustomization
flux suspend kustomization myapp

Emergency Rollback

# ArgoCD
argocd app rollback myapp <revision>

# Git-based (works for any tool)
git revert HEAD
git push

Nuclear Option (Delete and Recreate)

# WARNING: Causes downtime
argocd app delete myapp --cascade=false  # Keep resources
# Fix configuration
argocd app create myapp ...  # Recreate application