Implementing Gitops

GitOps Workflows

Implement GitOps continuous delivery for Kubernetes using declarative, pull-based deployment models where Git serves as the single source of truth for infrastructure and application configuration.

When to Use

Use GitOps workflows for:

• Kubernetes Deployments: Automating application and infrastructure deployments to Kubernetes clusters
• Multi-Cluster Management: Managing deployments across development, staging, production, and edge clusters
• Continuous Delivery: Implementing pull-based CD pipelines with automated reconciliation
• Drift Detection: Automatically detecting and correcting configuration drift from desired state
• Audit Requirements: Maintaining complete audit trails via Git commits for compliance
• Progressive Delivery: Implementing canary, blue-green, or rolling deployment strategies
• Disaster Recovery: Enabling rapid cluster recovery with GitOps bootstrap processes

Trigger keywords: "deploy to Kubernetes", "ArgoCD setup", "Flux bootstrap", "GitOps pipeline", "environment promotion", "multi-cluster deployment", "automated reconciliation"

Core GitOps Principles

1. Git as Single Source of Truth

All system configuration stored in Git repositories. No manual kubectl apply or cluster modifications. Declarative manifests (YAML) for all Kubernetes resources, environment-specific overlays, infrastructure configuration, and application deployments.

2. Pull-Based Deployment

Operators running inside clusters pull changes from Git and apply them automatically. Benefits include no cluster credentials in CI/CD pipelines, support for air-gapped environments, self-healing through continuous reconciliation, and simplified CI/CD.

3. Automated Reconciliation

GitOps operators continuously compare actual cluster state with desired state in Git and reconcile differences through a continuous loop: watch Git, compare live state, apply differences, report status, repeat.

4. Declarative Configuration

Use declarative Kubernetes manifests (not imperative scripts) to define desired state.

Tool Selection

ArgoCD vs Flux

Hybrid Approach: Some teams use Flux for infrastructure and ArgoCD for applications.

For ArgoCD implementation patterns, see references/argocd-patterns.md For Flux implementation patterns, see references/flux-patterns.md For Kustomize overlay patterns, see references/kustomize-overlays.md

Quick Start

ArgoCD Installation

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Basic Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/repo.git
    targetRevision: HEAD
    path: k8s/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Flux Bootstrap

flux bootstrap github \
  --owner=myorg \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/production

Basic Kustomization:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  path: "./k8s/prod"
  prune: true
  sourceRef:
    kind: GitRepository
    name: myapp

For complete examples, see examples/argocd/ and examples/flux/

Environment Promotion

Branch-Based Strategy: dev branch → staging branch → main branch (prod) Kustomize-Based Strategy: k8s/base/ → k8s/overlays/{dev,staging,prod}/

Promotion Process: 1. Merge code changes to main branch 2. CI builds container image with tag 3. Update image tag in environment overlay (Git commit) 4. GitOps operator detects change and deploys 5. Test in environment 6. Promote to next environment by updating Git

For multi-environment ApplicationSet patterns, see references/argocd-patterns.md

Multi-Cluster Management

ArgoCD: Register external clusters with argocd CLI, use ApplicationSets to generate Applications per cluster, manage from single ArgoCD instance.

Flux: Bootstrap Flux per cluster, use same Git repo with cluster-specific paths, configure remote clusters via kubeConfig secrets.

For detailed multi-cluster patterns, see references/multi-cluster.md

Progressive Delivery

Canary Deployments: Gradually shift traffic to new version, monitor metrics during rollout, automated rollback on failures.

Blue-Green Deployments: Deploy new version alongside old, switch traffic atomically, instant rollback if issues detected.

ArgoCD: Use Argo Rollouts for progressive delivery Flux: Integrate Flagger for automated canary analysis

For progressive delivery strategies and Argo Rollouts examples, see references/progressive-delivery.md

Secret Management

GitOps requires storing configuration in Git, but secrets must be protected.

For secret management integration patterns, see references/secret-management.md

Drift Detection and Remediation

GitOps operators continuously monitor for drift between Git (desired state) and cluster (actual state).

ArgoCD Automatic Self-Healing:

syncPolicy:
  automated:
    prune: true      # Remove resources not in Git
    selfHeal: true   # Revert manual changes

Flux Automatic Reconciliation:

spec:
  interval: 10m    # Check every 10 minutes
  prune: true      # Remove resources not in Git
  force: true      # Force apply on conflicts

Manual Operations:

# ArgoCD
argocd app get myapp           # View sync status
argocd app diff myapp          # Show differences
argocd app sync myapp          # Manually trigger sync

# Flux
flux get kustomizations              # View sync status
flux reconcile kustomization myapp   # Force immediate sync

For drift detection strategies and troubleshooting, see references/drift-remediation.md

Sync Hooks and Lifecycle

Execute operations before/after syncs using hooks.

PreSync Hook (Database Migration):

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded

PostSync Hook (Smoke Test):

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    argocd.argoproj.io/hook: PostSync

For complete sync hook examples, see examples/argocd/sync-hooks.yaml

Monitoring and Observability

Key Metrics

• Sync Status: OutOfSync, Synced, Unknown
• Sync Frequency: How often reconciliation occurs
• Drift Detection: Time to detect configuration drift
• Sync Duration: Time to apply changes
• Failure Rate: Failed syncs and causes

ArgoCD Metrics: Exposed at /metrics endpoint (argocd_app_sync_total, argocd_app_info) Flux Metrics: From controllers (gotk_reconcile_condition, gotk_reconcile_duration_seconds)

Troubleshooting

Common Issues

Sync Stuck/OutOfSync:

• Check Git repository accessibility
• Verify manifests are valid YAML
• Review sync logs for errors
• Check resource finalizers

Self-Heal Not Working:

• Verify selfHeal enabled in syncPolicy
• Check operator has write permissions
• Review resource ownership labels

Secrets Not Decrypting:

• Verify SOPS/ESO controllers installed
• Check KMS/Vault credentials
• Review encryption key configuration

CLI Quick Reference

ArgoCD Commands

argocd app create <name>          # Create application
argocd app get <name>              # View status
argocd app sync <name>             # Trigger sync
argocd app diff <name>             # Show drift
argocd app list                    # List all applications

Flux Commands

flux create source git <name>      # Create Git source
flux create kustomization <name>   # Create kustomization
flux get all                       # View all resources
flux reconcile <kind> <name>       # Force reconciliation
flux logs                          # View controller logs

Kustomize Commands

kustomize build k8s/overlays/prod  # Preview generated YAML
kubectl apply -k k8s/overlays/prod # Apply directly
kubectl diff -k k8s/overlays/prod  # Show differences

Installation Scripts

Use the provided installation scripts for quick setup:

# Install ArgoCD
./scripts/install-argocd.sh

# Bootstrap Flux
export GITHUB_TOKEN=<token>
export GITHUB_OWNER=<org>
export GITHUB_REPO=fleet-infra
./scripts/install-flux.sh

# Check for drift
./scripts/check-drift.sh

# Promote environment
./scripts/promote-env.sh dev staging

Example Files

Complete working examples provided in examples/ directory:

ArgoCD Examples:

• examples/argocd/application.yaml - Basic Application
• examples/argocd/applicationset.yaml - Multi-environment ApplicationSet
• examples/argocd/progressive-rollout.yaml - Progressive rollout strategy
• examples/argocd/sync-hooks.yaml - PreSync/PostSync hooks

Flux Examples:

• examples/flux/gitrepository.yaml - Git source configuration
• examples/flux/kustomization.yaml - Kustomization controller
• examples/flux/helmrelease.yaml - Helm release management
• examples/flux/ocirepository.yaml - OCI artifact source

Kustomize Examples:

• examples/kustomize/base/ - Base configuration
• examples/kustomize/overlays/{dev,staging,prod}/ - Environment overlays

Rollout Examples:

• examples/rollouts/canary.yaml - Canary deployment with Argo Rollouts
• examples/rollouts/blue-green.yaml - Blue-green deployment strategy

Related Skills

• kubernetes-operations: Kubernetes fundamentals and resource management
• infrastructure-as-code: Provisioning clusters that GitOps deploys to
• building-ci-pipelines: CI builds images, GitOps deploys them
• secret-management: Vault/ESO integration with GitOps
• deploying-applications: GitOps as the deployment mechanism

Summary

GitOps provides automated, declarative continuous delivery for Kubernetes with Git as the single source of truth. Choose ArgoCD for UI-driven workflows or Flux for CLI/API-first approaches. Implement automated reconciliation, drift detection, and progressive delivery for reliable deployments at scale. Integrate secret management, multi-cluster orchestration, and disaster recovery for production-grade GitOps workflows.

ArgoCD Implementation Patterns

Complete guide to implementing ArgoCD for GitOps workflows.

Table of Contents

1. Installation and Configuration 2. Application Patterns 3. ApplicationSet Patterns 4. Sync Policies 5. Sync Hooks 6. Multi-Tenancy 7. CLI Operations

Installation and Configuration

Standard Installation

# Create namespace
kubectl create namespace argocd

# Install ArgoCD
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Access UI via port-forward
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Get initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

High Availability Installation

# Install HA version with multiple replicas
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml

Ingress Configuration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  rules:
  - host: argocd.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              name: https
  tls:
  - hosts:
    - argocd.example.com
    secretName: argocd-secret

Application Patterns

Basic Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/myapp.git
    targetRevision: HEAD
    path: k8s/base
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Helm Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: nginx
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.bitnami.com/bitnami
    chart: nginx
    targetRevision: 13.2.0
    helm:
      releaseName: nginx
      values: |
        replicaCount: 3
        service:
          type: LoadBalancer
  destination:
    server: https://kubernetes.default.svc
    namespace: nginx
  syncPolicy:
    automated:
      prune: true

Kustomize Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/myapp.git
    targetRevision: main
    path: k8s/overlays/prod
    kustomize:
      namePrefix: prod-
      commonLabels:
        environment: production
      images:
      - name: myapp
        newTag: v1.2.3
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp-prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Multi-Source Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-multi
  namespace: argocd
spec:
  project: default
  sources:
  - repoURL: https://github.com/myorg/myapp.git
    targetRevision: main
    path: k8s/base
  - repoURL: https://charts.example.com
    chart: common-library
    targetRevision: 1.0.0
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp

ApplicationSet Patterns

List Generator (Static Environments)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp-envs
  namespace: argocd
spec:
  goTemplate: true
  generators:
  - list:
      elements:
      - env: dev
        cluster: https://kubernetes.default.svc
        replicas: 1
        imageTag: latest
      - env: staging
        cluster: https://kubernetes.default.svc
        replicas: 2
        imageTag: staging-v1.2.3
      - env: prod
        cluster: https://prod-cluster.example.com
        replicas: 5
        imageTag: v1.2.3
  template:
    metadata:
      name: 'myapp-{{.env}}'
      labels:
        environment: '{{.env}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/myapp.git
        targetRevision: HEAD
        path: 'k8s/overlays/{{.env}}'
        kustomize:
          images:
          - name: myapp
            newTag: '{{.imageTag}}'
      destination:
        server: '{{.cluster}}'
        namespace: 'myapp-{{.env}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true

Cluster Generator (Dynamic Multi-Cluster)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-apps
  namespace: argocd
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          environment: production
  template:
    metadata:
      name: 'myapp-{{name}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/myapp.git
        targetRevision: HEAD
        path: k8s/base
      destination:
        server: '{{server}}'
        namespace: myapp
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Git Directory Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: repo-apps
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: https://github.com/myorg/apps.git
      revision: HEAD
      directories:
      - path: apps/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/apps.git
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'
      syncPolicy:
        automated:
          prune: true

Progressive Rollout Strategy

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: progressive-rollout
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: dev
        url: https://dev-cluster.example.com
        env: development
      - cluster: staging
        url: https://staging-cluster.example.com
        env: staging
      - cluster: prod-us-east
        url: https://prod-us-east.example.com
        env: production
      - cluster: prod-us-west
        url: https://prod-us-west.example.com
        env: production
  strategy:
    type: RollingSync
    rollingSync:
      steps:
      - matchExpressions:
        - key: envLabel
          operator: In
          values:
          - development
        # Deploy to dev immediately (100%)
      - matchExpressions:
        - key: envLabel
          operator: In
          values:
          - staging
        maxUpdate: 1
        # Deploy to staging one at a time
      - matchExpressions:
        - key: envLabel
          operator: In
          values:
          - production
        maxUpdate: 50%
        # Deploy to 50% of prod clusters at once
  template:
    metadata:
      name: 'myapp-{{.cluster}}'
      labels:
        envLabel: '{{.env}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/myapp.git
        targetRevision: HEAD
        path: 'k8s/{{.cluster}}'
      destination:
        server: '{{.url}}'
        namespace: myapp
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Sync Policies

Automated Sync (Dev/Staging)

syncPolicy:
  automated:
    prune: true      # Delete resources removed from Git
    selfHeal: true   # Revert manual changes
  syncOptions:
  - CreateNamespace=true
  retry:
    limit: 5
    backoff:
      duration: 5s
      factor: 2
      maxDuration: 3m

Manual Sync (Production)

syncPolicy:
  syncOptions:
  - CreateNamespace=true
  - PrunePropagationPolicy=foreground
  - PruneLast=true
  retry:
    limit: 2
    backoff:
      duration: 5s
      maxDuration: 1m

Selective Sync

syncPolicy:
  automated:
    prune: true
    selfHeal: true
  syncOptions:
  - CreateNamespace=true
  - ApplyOutOfSyncOnly=true  # Only sync out-of-sync resources
  managedNamespaceMetadata:
    labels:
      managed-by: argocd

Sync Hooks

PreSync: Database Migration

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: myapp:latest
        command: ["/bin/sh"]
        args:
        - -c
        - |
          echo "Running database migrations..."
          /app/migrate up
      restartPolicy: Never
  backoffLimit: 3

PostSync: Smoke Test

apiVersion: batch/v1
kind: Job
metadata:
  name: smoke-test
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      containers:
      - name: test
        image: curlimages/curl:latest
        command: ["/bin/sh"]
        args:
        - -c
        - |
          echo "Running smoke tests..."
          curl -f http://myapp-service/health || exit 1
          echo "Smoke tests passed!"
      restartPolicy: Never

SyncFail: Rollback

apiVersion: batch/v1
kind: Job
metadata:
  name: rollback-notification
  annotations:
    argocd.argoproj.io/hook: SyncFail
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
      - name: notify
        image: curlimages/curl:latest
        command: ["/bin/sh"]
        args:
        - -c
        - |
          curl -X POST https://slack.com/api/chat.postMessage \
            -H "Authorization: Bearer $SLACK_TOKEN" \
            -d "channel=deployments" \
            -d "text=Deployment failed for myapp"
      restartPolicy: Never

Skip: Resource Lifecycle Management

apiVersion: batch/v1
kind: Job
metadata:
  name: cleanup-job
  annotations:
    argocd.argoproj.io/hook: Skip
    # Resource ignored during sync
spec:
  template:
    spec:
      containers:
      - name: cleanup
        image: busybox
        command: ["echo", "Cleanup task"]

Multi-Tenancy

Project-Based Isolation

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-frontend
  namespace: argocd
spec:
  description: Frontend team project
  sourceRepos:
  - https://github.com/myorg/frontend-*
  destinations:
  - namespace: 'frontend-*'
    server: https://kubernetes.default.svc
  clusterResourceWhitelist:
  - group: ''
    kind: Namespace
  namespaceResourceWhitelist:
  - group: 'apps'
    kind: Deployment
  - group: ''
    kind: Service
  - group: ''
    kind: ConfigMap
  roles:
  - name: frontend-admin
    policies:
    - p, proj:team-frontend:frontend-admin, applications, *, team-frontend/*, allow
    groups:
    - frontend-team

RBAC Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # Admin role
    p, role:admin, applications, *, */*, allow
    p, role:admin, clusters, *, *, allow
    p, role:admin, repositories, *, *, allow
    g, admins-group, role:admin

    # Developer role
    p, role:developer, applications, get, */*, allow
    p, role:developer, applications, sync, */*, allow
    g, developers-group, role:developer

    # Read-only role
    p, role:readonly, applications, get, */*, allow
    g, viewers-group, role:readonly

CLI Operations

Application Management

# Create application
argocd app create myapp \
  --repo https://github.com/myorg/myapp.git \
  --path k8s/base \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace myapp

# Get application status
argocd app get myapp

# List applications
argocd app list

# Sync application
argocd app sync myapp

# Show diff
argocd app diff myapp

# Delete application
argocd app delete myapp

Cluster Management

# Add cluster
argocd cluster add prod-cluster \
  --kubeconfig ~/.kube/prod-config \
  --name prod-cluster

# List clusters
argocd cluster list

# Remove cluster
argocd cluster rm https://prod-cluster.example.com

Repository Management

# Add Git repository
argocd repo add https://github.com/myorg/myapp.git \
  --username myuser \
  --password mytoken

# Add Helm repository
argocd repo add https://charts.example.com \
  --type helm \
  --name example-charts

# List repositories
argocd repo list

Advanced Operations

# Hard refresh (force Git fetch)
argocd app get myapp --hard-refresh

# Sync specific resources
argocd app sync myapp --resource apps:Deployment:myapp

# Rollback to previous version
argocd app rollback myapp 12

# View application history
argocd app history myapp

# Set application parameters
argocd app set myapp \
  --helm-set replicaCount=5

# Patch application
argocd app patch myapp \
  --patch '{"spec":{"syncPolicy":{"automated":{"prune":true}}}}'

Best Practices

Application Organization

• Use ApplicationSets for multi-environment deployments
• One Application per microservice
• Group related apps using labels
• Use meaningful application names

Sync Strategy

• Enable automated sync for dev/staging
• Use manual sync for production
• Enable selfHeal for stability
• Configure retry policies

Resource Management

• Use resource limits in manifests
• Enable prune to clean up deleted resources
• Use sync waves for ordered deployment
• Implement health checks

Security

• Use Projects for multi-tenancy
• Configure RBAC policies
• Store secrets using Sealed Secrets or ESO
• Enable audit logging

Monitoring

• Track sync status and duration
• Alert on sync failures
• Monitor drift detection
• Review application health metrics

Drift Detection and Remediation

Detecting and correcting configuration drift in GitOps workflows.

Table of Contents

• Drift Detection Concepts
• ArgoCD Drift Detection
• View Sync Status
• Automatic Self-Healing
• Manual Sync Operations
• Flux Drift Detection
• View Kustomization Status
• Force Reconciliation
• Automatic Drift Correction
• Disaster Recovery
• ArgoCD Recovery
• Flux Recovery
• Troubleshooting
• OutOfSync Issues
• Resource Finalizers

Drift Detection Concepts

Drift: Divergence between desired state (Git) and actual state (cluster).

Common Causes:

• Manual kubectl apply commands
• Cluster operators modifying resources
• External controllers changing state
• Resource finalizers preventing deletion
• CRD updates requiring migration

ArgoCD Drift Detection

View Sync Status

# Check application sync status
argocd app get myapp

# View detailed diff
argocd app diff myapp

# Hard refresh (fetch latest Git)
argocd app get myapp --hard-refresh

Automatic Self-Healing

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  syncPolicy:
    automated:
      prune: true      # Remove resources not in Git
      selfHeal: true   # Revert manual changes
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

Manual Sync Operations

# Sync specific resource
argocd app sync myapp --resource apps:Deployment:myapp

# Force sync (ignore hooks)
argocd app sync myapp --force

# Sync with prune
argocd app sync myapp --prune

# Dry-run sync
argocd app sync myapp --dry-run

Flux Drift Detection

View Kustomization Status

# Check all kustomizations
flux get kustomizations

# Check specific kustomization
flux get kustomization myapp

# View events
flux events --for Kustomization/myapp

Force Reconciliation

# Reconcile immediately
flux reconcile kustomization myapp

# Reconcile with source update
flux reconcile kustomization myapp --with-source

# Suspend reconciliation
flux suspend kustomization myapp

# Resume reconciliation
flux resume kustomization myapp

Automatic Drift Correction

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
spec:
  interval: 10m       # Check every 10 minutes
  prune: true         # Remove resources not in Git
  force: true         # Force apply on conflicts
  wait: true          # Wait for resources to be ready
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: myapp
  path: ./k8s/prod

Disaster Recovery

ArgoCD Recovery

# 1. Reinstall ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# 2. Re-register clusters
argocd cluster add prod-cluster

# 3. Restore applications from Git
kubectl apply -f git-repo/argocd/applications/

# 4. Sync all applications
argocd app sync --all

Flux Recovery

# Bootstrap Flux (idempotent operation)
flux bootstrap github \
  --owner=myorg \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/production

# Flux automatically reconciles all resources from Git

Troubleshooting

OutOfSync Issues

Check Git connectivity:

argocd repo get https://github.com/myorg/myapp
flux get sources git

Validate manifests:

kubectl apply --dry-run=server -f manifest.yaml
kustomize build overlays/prod | kubectl apply --dry-run=server -f -

Review sync logs:

argocd app logs myapp
flux logs --kind=Kustomization --name=myapp

Resource Finalizers

# View finalizers
kubectl get deployment myapp -o yaml | grep finalizers -A 5

# Remove finalizer (if stuck)
kubectl patch deployment myapp -p '{"metadata":{"finalizers":[]}}' --type=merge

Flux CD Implementation Patterns

Complete guide to implementing Flux CD for GitOps workflows.

Table of Contents

1. Installation and Bootstrap 2. Source Controllers 3. Kustomization Controller 4. Helm Controller 5. Notification Controller 6. Image Automation 7. Multi-Tenancy

Installation and Bootstrap

Flux CLI Installation

# macOS
brew install fluxcd/tap/flux

# Linux
curl -s https://fluxcd.io/install.sh | sudo bash

# Verify installation
flux --version

Bootstrap GitHub

# Export GitHub token
export GITHUB_TOKEN=<your-token>

# Bootstrap Flux
flux bootstrap github \
  --owner=myorg \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/production \
  --personal

Bootstrap GitLab

# Export GitLab token
export GITLAB_TOKEN=<your-token>

# Bootstrap Flux
flux bootstrap gitlab \
  --owner=myorg \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/production \
  --token-auth

Bootstrap Generic Git

# For any Git server
flux bootstrap git \
  --url=ssh://git@git.example.com/myorg/fleet-infra \
  --branch=main \
  --path=clusters/production \
  --private-key-file=/path/to/private-key

Source Controllers

GitRepository Source

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/myapp
  ref:
    branch: main
  secretRef:
    name: git-credentials
  ignore: |
    # Exclude files from sync
    .git/
    .github/
    *.md

GitRepository with SSH

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: myapp-ssh
  namespace: flux-system
spec:
  interval: 5m
  url: ssh://git@github.com/myorg/myapp.git
  ref:
    branch: main
  secretRef:
    name: ssh-credentials
---
apiVersion: v1
kind: Secret
metadata:
  name: ssh-credentials
  namespace: flux-system
type: Opaque
stringData:
  identity: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----
  known_hosts: |
    github.com ssh-rsa AAAA...

GitRepository with Tag

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: myapp-release
  namespace: flux-system
spec:
  interval: 10m
  url: https://github.com/myorg/myapp
  ref:
    tag: v1.2.3

GitRepository with SemVer

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: myapp-semver
  namespace: flux-system
spec:
  interval: 10m
  url: https://github.com/myorg/myapp
  ref:
    semver: ">=1.0.0 <2.0.0"

OCIRepository Source

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: myapp-oci
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/myorg/myapp-config
  ref:
    tag: latest
  provider: generic

OCIRepository with Authentication

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: private-oci
  namespace: flux-system
spec:
  interval: 5m
  url: oci://registry.example.com/myapp/config
  ref:
    tag: v1.0.0
  secretRef:
    name: oci-credentials
  provider: generic
---
apiVersion: v1
kind: Secret
metadata:
  name: oci-credentials
  namespace: flux-system
type: kubernetes.io/dockerconfigjson
stringData:
  .dockerconfigjson: |
    {
      "auths": {
        "registry.example.com": {
          "username": "myuser",
          "password": "mytoken"
        }
      }
    }

HelmRepository Source

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  interval: 1h
  url: https://charts.bitnami.com/bitnami

HelmRepository with Authentication

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: private-charts
  namespace: flux-system
spec:
  interval: 1h
  url: https://charts.example.com
  secretRef:
    name: helm-credentials
---
apiVersion: v1
kind: Secret
metadata:
  name: helm-credentials
  namespace: flux-system
type: Opaque
stringData:
  username: myuser
  password: mytoken

Kustomization Controller

Basic Kustomization

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  retryInterval: 2m
  timeout: 5m
  path: "./k8s/base"
  prune: true
  sourceRef:
    kind: GitRepository
    name: myapp
  targetNamespace: myapp
  wait: true

Kustomization with Dependencies

# Base infrastructure
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 10m
  path: "./infrastructure"
  prune: true
  sourceRef:
    kind: GitRepository
    name: fleet-infra
---
# Application depends on infrastructure
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 5m
  dependsOn:
  - name: infrastructure
  path: "./apps/myapp"
  prune: true
  sourceRef:
    kind: GitRepository
    name: fleet-infra

Kustomization with Health Checks

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  path: "./k8s/prod"
  prune: true
  sourceRef:
    kind: GitRepository
    name: myapp
  healthChecks:
  - apiVersion: apps/v1
    kind: Deployment
    name: myapp
    namespace: myapp
  - apiVersion: apps/v1
    kind: StatefulSet
    name: myapp-db
    namespace: myapp
  wait: true
  timeout: 5m

Kustomization with Patches

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp-prod
  namespace: flux-system
spec:
  interval: 10m
  path: "./k8s/base"
  prune: true
  sourceRef:
    kind: GitRepository
    name: myapp
  targetNamespace: myapp-prod
  patches:
  - patch: |-
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: myapp
      spec:
        replicas: 5
    target:
      kind: Deployment
      name: myapp
  - patch: |-
      apiVersion: v1
      kind: Service
      metadata:
        name: myapp
      spec:
        type: LoadBalancer
    target:
      kind: Service
      name: myapp

Kustomization with Post-Build Variables

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  path: "./k8s/base"
  prune: true
  sourceRef:
    kind: GitRepository
    name: myapp
  postBuild:
    substitute:
      APP_VERSION: "v1.2.3"
      ENVIRONMENT: "production"
      REPLICAS: "5"
    substituteFrom:
    - kind: ConfigMap
      name: cluster-vars

Kustomization with Force Apply

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  path: "./k8s/prod"
  prune: true
  force: true  # Force apply on conflicts
  sourceRef:
    kind: GitRepository
    name: myapp

Helm Controller

Basic HelmRelease

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: nginx
  namespace: flux-system
spec:
  interval: 10m
  chart:
    spec:
      chart: nginx
      version: ">=13.0.0 <14.0.0"
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
  values:
    replicaCount: 3
    service:
      type: LoadBalancer

HelmRelease with ValuesFrom

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  chart:
    spec:
      chart: myapp
      sourceRef:
        kind: HelmRepository
        name: myorg-charts
  valuesFrom:
  - kind: ConfigMap
    name: myapp-values
  - kind: Secret
    name: myapp-secrets
    valuesKey: values.yaml
  values:
    replicas: 5
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-values
  namespace: flux-system
data:
  values.yaml: |
    service:
      type: LoadBalancer
    ingress:
      enabled: true

HelmRelease with Rollback

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  chart:
    spec:
      chart: myapp
      sourceRef:
        kind: HelmRepository
        name: myorg-charts
  upgrade:
    remediation:
      retries: 3
  rollback:
    recreate: true
    force: true
    cleanupOnFail: true
  test:
    enable: true

HelmRelease with Dependencies

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  dependsOn:
  - name: postgresql
    namespace: flux-system
  - name: redis
    namespace: flux-system
  chart:
    spec:
      chart: myapp
      sourceRef:
        kind: HelmRepository
        name: myorg-charts

Notification Controller

Slack Notifications

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: deployments
  secretRef:
    name: slack-webhook-url
---
apiVersion: v1
kind: Secret
metadata:
  name: slack-webhook-url
  namespace: flux-system
type: Opaque
stringData:
  address: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: slack-deployments
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: info
  eventSources:
  - kind: GitRepository
    name: '*'
  - kind: Kustomization
    name: '*'

Microsoft Teams Notifications

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: teams
  namespace: flux-system
spec:
  type: msteams
  secretRef:
    name: teams-webhook-url
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: teams-alerts
  namespace: flux-system
spec:
  providerRef:
    name: teams
  eventSeverity: error
  eventSources:
  - kind: Kustomization
    name: '*'

Git Commit Status

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: github
  namespace: flux-system
spec:
  type: github
  address: https://github.com/myorg/myapp
  secretRef:
    name: github-token
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: github-status
  namespace: flux-system
spec:
  providerRef:
    name: github
  eventSeverity: info
  eventSources:
  - kind: Kustomization
    name: myapp

Image Automation

ImageRepository

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: myapp
  namespace: flux-system
spec:
  image: ghcr.io/myorg/myapp
  interval: 1m
  secretRef:
    name: ghcr-credentials

ImagePolicy

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: myapp
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: myapp
  policy:
    semver:
      range: ">=1.0.0 <2.0.0"

ImageUpdateAutomation

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: myapp
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: flux@example.com
        name: Flux Bot
      messageTemplate: |
        Update image to {{range .Updated.Images}}{{println .}}{{end}}
    push:
      branch: main
  update:
    path: ./k8s
    strategy: Setters

Image Marker in Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: ghcr.io/myorg/myapp:v1.0.0 # {"$imagepolicy": "flux-system:myapp"}
        ports:
        - containerPort: 8080

Multi-Tenancy

Namespace Isolation

# Tenant namespace
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    toolkit.fluxcd.io/tenant: tenant-a
---
# Tenant GitRepository
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: tenant-a-repo
  namespace: tenant-a
spec:
  interval: 1m
  url: https://github.com/tenant-a/apps
  ref:
    branch: main
---
# Tenant Kustomization
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenant-a-apps
  namespace: tenant-a
spec:
  interval: 10m
  path: "./apps"
  prune: true
  serviceAccountName: kustomize-controller
  sourceRef:
    kind: GitRepository
    name: tenant-a-repo
  targetNamespace: tenant-a

Service Account with RBAC

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tenant-a-reconciler
  namespace: tenant-a
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: tenant-a-reconciler
  namespace: tenant-a
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-a-reconciler
  namespace: tenant-a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: tenant-a-reconciler
subjects:
- kind: ServiceAccount
  name: tenant-a-reconciler
  namespace: tenant-a

CLI Operations

Source Commands

# List all sources
flux get sources all

# Get GitRepository status
flux get sources git myapp

# Reconcile source immediately
flux reconcile source git myapp

# Suspend/resume source
flux suspend source git myapp
flux resume source git myapp

Kustomization Commands

# List kustomizations
flux get kustomizations

# View kustomization details
flux get kustomization myapp --with-source

# Reconcile immediately
flux reconcile kustomization myapp --with-source

# Suspend/resume
flux suspend kustomization myapp
flux resume kustomization myapp

HelmRelease Commands

# List HelmReleases
flux get helmreleases

# View HelmRelease details
flux get helmrelease nginx

# Reconcile immediately
flux reconcile helmrelease nginx

# Suspend/resume
flux suspend helmrelease nginx
flux resume helmrelease nginx

Troubleshooting Commands

# View controller logs
flux logs --all-namespaces
flux logs --kind=Kustomization --name=myapp

# Check system status
flux check

# View events
flux events --for Kustomization/myapp

# Export configuration
flux export source git myapp
flux export kustomization myapp

Best Practices

Repository Structure

Organize repositories with clear separation:

fleet-infra/
├── clusters/
│   ├── production/
│   │   └── flux-system/
│   ├── staging/
│   └── dev/
├── infrastructure/
│   ├── sources/
│   ├── crds/
│   └── controllers/
└── apps/
    ├── base/
    └── production/

Source Management

• Use appropriate intervals (1m for dev, 5-10m for prod)
• Implement authentication for private repositories
• Use semver for production deployments
• Enable verification for signed commits

Kustomization Strategy

• Use dependencies for ordered deployment
• Enable health checks for critical apps
• Set appropriate timeouts
• Use prune carefully in production

Helm Management

• Pin chart versions in production
• Use valuesFrom for environment-specific config
• Enable rollback for safety
• Test charts before production deployment

Monitoring

• Configure notifications for failures
• Track reconciliation metrics
• Monitor Git source health
• Alert on sync failures

Kustomize Overlay Patterns

Template-free Kubernetes configuration management using Kustomize base and overlay pattern.

Table of Contents

1. Basic Concepts 2. Base Configuration 3. Environment Overlays 4. Advanced Patterns 5. Components

Basic Concepts

Kustomize Principles

• Declarative: Pure YAML, no templating
• Composable: Base + overlays for environments
• Patchable: Strategic merge and JSON patches
• Reusable: Components for cross-cutting concerns

Directory Structure

k8s/
├── base/                    # Common configuration
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── dev/                # Development overlay
│   │   └── kustomization.yaml
│   ├── staging/            # Staging overlay
│   │   └── kustomization.yaml
│   └── prod/               # Production overlay
│       ├── kustomization.yaml
│       └── patches/
└── components/             # Reusable components
    ├── monitoring/
    └── security/

Base Configuration

base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:latest
        ports:
        - containerPort: 8080
        env:
        - name: LOG_LEVEL
          value: info
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "200m"

base/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP

base/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
data:
  APP_NAME: myapp
  ENVIRONMENT: base

base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- deployment.yaml
- service.yaml
- configmap.yaml

commonLabels:
  app: myapp
  managed-by: kustomize

namespace: myapp

Environment Overlays

overlays/dev/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

namespace: myapp-dev

namePrefix: dev-

commonLabels:
  environment: development

replicas:
- name: myapp
  count: 1

images:
- name: myapp
  newTag: dev-latest

patches:
- patch: |-
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: myapp-config
    data:
      LOG_LEVEL: debug
      ENVIRONMENT: development
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: myapp
    spec:
      template:
        spec:
          containers:
          - name: myapp
            env:
            - name: DEBUG
              value: "true"

overlays/staging/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

namespace: myapp-staging

namePrefix: staging-

commonLabels:
  environment: staging

replicas:
- name: myapp
  count: 2

images:
- name: myapp
  newTag: staging-v1.2.3

patches:
- patch: |-
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: myapp-config
    data:
      LOG_LEVEL: info
      ENVIRONMENT: staging
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: myapp
    spec:
      template:
        spec:
          containers:
          - name: myapp
            resources:
              limits:
                memory: "256Mi"
                cpu: "500m"
              requests:
                memory: "128Mi"
                cpu: "250m"

overlays/prod/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

namespace: myapp-prod

namePrefix: prod-

commonLabels:
  environment: production

replicas:
- name: myapp
  count: 5

images:
- name: myapp
  newTag: v1.2.3

patches:
- path: patches/production-resources.yaml
- path: patches/production-service.yaml

configMapGenerator:
- name: myapp-config
  behavior: merge
  literals:
  - LOG_LEVEL=warn
  - ENVIRONMENT=production
  - ENABLE_MONITORING=true

secretGenerator:
- name: myapp-secrets
  files:
  - secrets/database-url
  - secrets/api-key

overlays/prod/patches/production-resources.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
      - name: myapp
        resources:
          limits:
            memory: "512Mi"
            cpu: "1000m"
          requests:
            memory: "256Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

overlays/prod/patches/production-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  type: LoadBalancer
  ports:
  - port: 443
    targetPort: 8080

Advanced Patterns

Strategic Merge Patches

# overlays/prod/kustomization.yaml
patches:
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: myapp
    spec:
      template:
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - myapp
                topologyKey: kubernetes.io/hostname

JSON 6902 Patches

# overlays/prod/kustomization.yaml
patches:
- target:
    kind: Deployment
    name: myapp
  patch: |-
    - op: add
      path: /spec/template/spec/containers/0/env/-
      value:
        name: FEATURE_FLAG
        value: "enabled"
    - op: replace
      path: /spec/replicas
      value: 10

Image Transformations

# overlays/prod/kustomization.yaml
images:
- name: myapp
  newName: registry.example.com/myapp
  newTag: v1.2.3
  digest: sha256:abc123...

Variable Substitution

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- deployment.yaml

vars:
- name: SERVICE_NAME
  objref:
    kind: Service
    name: myapp
    apiVersion: v1
  fieldref:
    fieldpath: metadata.name

Generators

# overlays/prod/kustomization.yaml
configMapGenerator:
- name: app-config
  literals:
  - DB_HOST=prod-db.example.com
  - CACHE_TTL=3600
  files:
  - configs/app.properties

secretGenerator:
- name: app-secrets
  envs:
  - secrets/.env.prod
  files:
  - tls.crt
  - tls.key

Components

Monitoring Component

# components/monitoring/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
- servicemonitor.yaml

labels:
- pairs:
    monitoring: prometheus

patches:
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: not-important
    spec:
      template:
        metadata:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "8080"
            prometheus.io/path: "/metrics"
  target:
    kind: Deployment

components/monitoring/servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-metrics
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: http
    path: /metrics
    interval: 30s

Security Component

# components/security/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
- networkpolicy.yaml
- podsecuritypolicy.yaml

patches:
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: not-important
    spec:
      template:
        spec:
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000
            fsGroup: 1000
          containers:
          - name: not-important
            securityContext:
              allowPrivilegeEscalation: false
              readOnlyRootFilesystem: true
              capabilities:
                drop:
                - ALL
  target:
    kind: Deployment

Using Components in Overlays

# overlays/prod/kustomization.yaml
resources:
- ../../base

components:
- ../../components/monitoring
- ../../components/security

replicas:
- name: myapp
  count: 5

Testing and Preview

Preview Generated Manifests

# Preview dev overlay
kustomize build overlays/dev

# Preview prod overlay
kustomize build overlays/prod

# Preview with kubectl
kubectl kustomize overlays/prod

Validate Manifests

# Validate YAML syntax
kustomize build overlays/prod | kubectl apply --dry-run=client -f -

# Validate with server
kustomize build overlays/prod | kubectl apply --dry-run=server -f -

# Diff against cluster
kubectl diff -k overlays/prod

Apply Overlays

# Apply dev overlay
kubectl apply -k overlays/dev

# Apply prod overlay
kubectl apply -k overlays/prod

# Delete resources
kubectl delete -k overlays/dev

Best Practices

Base Configuration

• Keep base minimal and generic
• Use sensible defaults
• Avoid environment-specific values
• Document customization points

Overlays

• One overlay per environment
• Use meaningful names (dev, staging, prod)
• Keep patches focused and small
• Document overlay purpose

Patches

• Prefer strategic merge over JSON 6902
• Keep patches maintainable
• Group related changes
• Use patch files for complex changes

Components

• Create components for cross-cutting concerns
• Make components optional
• Document component dependencies
• Test components in isolation

Repository Organization

• Base at repository root or k8s/base
• Overlays in k8s/overlays/
• Components in k8s/components/
• Use consistent naming conventions

GitOps Integration

Both ArgoCD and Flux support Kustomize natively:

ArgoCD:

spec:
  source:
    path: k8s/overlays/prod

Flux:

spec:
  path: "./k8s/overlays/prod"

Multi-Cluster Management

Manage deployments across multiple Kubernetes clusters using GitOps.

Table of Contents

• ArgoCD Multi-Cluster
• Cluster Registration
• ApplicationSet for Multi-Cluster
• Flux Multi-Cluster
• Bootstrap Multiple Clusters
• Remote Cluster Management

ArgoCD Multi-Cluster

Cluster Registration

# List available contexts
kubectl config get-contexts

# Add cluster to ArgoCD
argocd cluster add prod-cluster \
  --kubeconfig ~/.kube/prod-config \
  --name prod-cluster \
  --namespace argocd

# List registered clusters
argocd cluster list

# Get cluster details
argocd cluster get https://prod-cluster.example.com

ApplicationSet for Multi-Cluster

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: multi-cluster-app
  namespace: argocd
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          environment: production
  template:
    metadata:
      name: 'myapp-{{name}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/myapp.git
        targetRevision: HEAD
        path: k8s/base
      destination:
        server: '{{server}}'
        namespace: myapp
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Flux Multi-Cluster

Bootstrap Multiple Clusters

# Bootstrap production cluster
flux bootstrap github \
  --owner=myorg \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/production

# Bootstrap staging cluster
flux bootstrap github \
  --owner=myorg \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/staging

Remote Cluster Management

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp-remote
  namespace: flux-system
spec:
  interval: 10m
  kubeConfig:
    secretRef:
      name: remote-cluster-kubeconfig
  sourceRef:
    kind: GitRepository
    name: myapp
  path: ./k8s/prod
---
apiVersion: v1
kind: Secret
metadata:
  name: remote-cluster-kubeconfig
  namespace: flux-system
type: Opaque
stringData:
  value: |
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority-data: LS0t...
        server: https://remote-cluster.example.com
      name: remote-cluster
    contexts:
    - context:
        cluster: remote-cluster
        user: flux
      name: flux@remote-cluster
    current-context: flux@remote-cluster
    users:
    - name: flux
      user:
        token: eyJhbG...

Progressive Delivery Patterns

Advanced deployment strategies for gradual rollouts with automated validation and rollback.

Table of Contents

1. Deployment Strategies 2. Argo Rollouts 3. Flagger Integration 4. Metrics and Analysis

Deployment Strategies

Strategy Comparison

When to Use

• Canary: Gradual rollout with real traffic, metric-based validation
• Blue-Green: Zero-downtime with instant rollback capability
• Rolling: Standard Kubernetes default, simple progressive rollout
• A/B Testing: Feature testing with specific user segments

Argo Rollouts

Installation

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

# Install kubectl plugin
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-$(uname -s | tr '[:upper:]' '[:lower:]')-amd64
chmod +x ./kubectl-argo-rollouts-$(uname -s | tr '[:upper:]' '[:lower:]')-amd64
sudo mv ./kubectl-argo-rollouts-$(uname -s | tr '[:upper:]' '[:lower:]')-amd64 /usr/local/bin/kubectl-argo-rollouts

Basic Canary Rollout

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 1m}
      - setWeight: 40
      - pause: {duration: 1m}
      - setWeight: 60
      - pause: {duration: 1m}
      - setWeight: 80
      - pause: {duration: 1m}
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v2
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"

Canary with Analysis

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 30s}
      - analysis:
          templates:
          - templateName: success-rate
          args:
          - name: service-name
            value: myapp-canary
      - setWeight: 50
      - pause: {duration: 30s}
      - analysis:
          templates:
          - templateName: success-rate
          - templateName: latency
          args:
          - name: service-name
            value: myapp-canary
      - setWeight: 80
      - pause: {duration: 30s}
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v2
        ports:
        - containerPort: 8080

AnalysisTemplate: Success Rate

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    interval: 5m
    count: 3
    successCondition: result[0] >= 0.95
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(
            http_requests_total{service="{{args.service-name}}",status=~"2.."}[5m]
          )) /
          sum(rate(
            http_requests_total{service="{{args.service-name}}"}[5m]
          ))

AnalysisTemplate: Latency

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency
spec:
  args:
  - name: service-name
  metrics:
  - name: latency-p95
    interval: 5m
    count: 3
    successCondition: result[0] <= 500
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          histogram_quantile(0.95,
            sum(rate(
              http_request_duration_seconds_bucket{service="{{args.service-name}}"}[5m]
            )) by (le)
          ) * 1000

Blue-Green Deployment

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 5
  strategy:
    blueGreen:
      activeService: myapp-active
      previewService: myapp-preview
      autoPromotionEnabled: false
      scaleDownDelaySeconds: 300
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v2
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: myapp-active
spec:
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: myapp-preview
spec:
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 8080

Canary with Ingress Traffic Splitting

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 5
  strategy:
    canary:
      canaryService: myapp-canary
      stableService: myapp-stable
      trafficRouting:
        nginx:
          stableIngress: myapp-ingress
      steps:
      - setWeight: 10
      - pause: {duration: 1m}
      - setWeight: 20
      - pause: {duration: 1m}
      - setWeight: 50
      - pause: {duration: 2m}
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v2
---
apiVersion: v1
kind: Service
metadata:
  name: myapp-stable
spec:
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: myapp-canary
spec:
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 8080

Rollout CLI Commands

# List rollouts
kubectl argo rollouts list rollouts

# Get rollout status
kubectl argo rollouts get rollout myapp

# Watch rollout progress
kubectl argo rollouts get rollout myapp --watch

# Promote rollout (manual step)
kubectl argo rollouts promote myapp

# Abort rollout
kubectl argo rollouts abort myapp

# Retry failed rollout
kubectl argo rollouts retry rollout myapp

# View rollout history
kubectl argo rollouts history rollout myapp

# Rollback to previous version
kubectl argo rollouts undo myapp

# Restart rollout
kubectl argo rollouts restart myapp

Flagger Integration

Installation

# Add Flagger Helm repository
helm repo add flagger https://flagger.app

# Install Flagger with Prometheus
helm upgrade -i flagger flagger/flagger \
  --namespace flux-system \
  --set prometheus.install=true \
  --set meshProvider=kubernetes

# Install Flagger Grafana dashboards
helm upgrade -i flagger-grafana flagger/grafana \
  --namespace flux-system \
  --set url=http://prometheus.flux-system:9090

Canary with Flagger

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
  namespace: myapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  service:
    port: 80
    targetPort: 8080
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500
      interval: 1m
    webhooks:
    - name: load-test
      url: http://flagger-loadtester.test/
      timeout: 5s
      metadata:
        type: cmd
        cmd: "hey -z 1m -q 10 -c 2 http://myapp-canary.myapp/"

Flagger with Service Mesh (Istio)

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
  namespace: myapp
spec:
  provider: istio
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  service:
    port: 80
    targetPort: 8080
    gateways:
    - myapp-gateway
    hosts:
    - myapp.example.com
  analysis:
    interval: 30s
    threshold: 10
    maxWeight: 50
    stepWeight: 5
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500
      interval: 1m

Flagger Blue-Green

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
  namespace: myapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  service:
    port: 80
  analysis:
    interval: 1m
    threshold: 3
    iterations: 10
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m
    webhooks:
    - name: smoke-test
      url: http://flagger-loadtester/
      timeout: 30s
      metadata:
        type: cmd
        cmd: "curl -f http://myapp-canary.myapp/ || exit 1"
  strategy:
    type: bluegreen

A/B Testing with Flagger

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
  namespace: myapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  service:
    port: 80
  analysis:
    interval: 1m
    threshold: 10
    iterations: 10
    match:
    - headers:
        x-canary:
          exact: "insider"
    - headers:
        cookie:
          regex: "^(.*?;)?(canary=always)(;.*)?$"
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m

Metrics and Analysis

Prometheus Metrics for Analysis

# Success rate metric
sum(rate(http_requests_total{status=~"2.."}[5m])) /
sum(rate(http_requests_total[5m]))

# Latency P95
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
) * 1000

# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) /
sum(rate(http_requests_total[5m]))

# Request rate
sum(rate(http_requests_total[5m]))

Custom Metrics Provider

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: custom-metrics
spec:
  args:
  - name: service-name
  metrics:
  - name: custom-metric
    interval: 5m
    successCondition: result >= 0.95
    provider:
      web:
        url: "http://metrics-service/api/metrics?service={{args.service-name}}"
        jsonPath: "{$.success_rate}"

Analysis with Multiple Providers

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: multi-metrics
spec:
  metrics:
  - name: prometheus-metric
    provider:
      prometheus:
        address: http://prometheus:9090
        query: "sum(rate(http_requests_total[5m]))"
  - name: datadog-metric
    provider:
      datadog:
        apiKey:
          secretKeyRef:
            name: datadog-secret
            key: api-key
        query: "avg:system.cpu.user{service:myapp}"
  - name: newrelic-metric
    provider:
      newRelic:
        profile: myapp-profile
        query: "SELECT average(duration) FROM Transaction WHERE appName = 'myapp'"

Best Practices

Strategy Selection

• Canary: Use for production deployments with real traffic validation
• Blue-Green: Use when instant rollback is critical
• Rolling: Use for non-critical services with simple requirements
• A/B Testing: Use for feature validation with specific user segments

Metrics Selection

• Monitor success rate (errors per request)
• Track latency percentiles (P50, P95, P99)
• Measure resource utilization (CPU, memory)
• Watch custom business metrics

Rollout Configuration

• Start with small traffic percentages (5-10%)
• Use multiple analysis steps
• Set appropriate failure thresholds
• Configure reasonable timeouts
• Enable automatic rollback

Testing

• Test rollout strategies in staging first
• Validate analysis templates with historical data
• Simulate failures to verify rollback
• Monitor resource usage during rollouts

Secret Management Integration

Secure secret handling in GitOps workflows.

Table of Contents

• Sealed Secrets
• Installation
• Create Sealed Secret
• Sealed Secret Manifest
• External Secrets Operator
• Installation
• Vault SecretStore
• ExternalSecret
• SOPS Encryption
• Installation
• Encrypt File with SOPS
• Flux SOPS Integration

Sealed Secrets

Installation

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml

# Install kubeseal CLI
brew install kubeseal

Create Sealed Secret

# Create secret
kubectl create secret generic mysecret \
  --from-literal=password=mypassword \
  --dry-run=client -o yaml > secret.yaml

# Seal the secret
kubeseal -f secret.yaml -w sealedsecret.yaml

# Commit sealed secret to Git
git add sealedsecret.yaml
git commit -m "Add sealed secret"

Sealed Secret Manifest

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
  namespace: myapp
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySY...

External Secrets Operator

Installation

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets-system --create-namespace

Vault SecretStore

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: myapp
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "myapp"

ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
  namespace: myapp
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: myapp-secrets
    creationPolicy: Owner
  data:
  - secretKey: database-password
    remoteRef:
      key: secret/myapp/database
      property: password
  - secretKey: api-key
    remoteRef:
      key: secret/myapp/api
      property: key

SOPS Encryption

Installation

# Install SOPS
brew install sops

# Install age for encryption
brew install age

# Generate age key
age-keygen -o key.txt

Encrypt File with SOPS

# secrets/prod-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: myapp-secrets
stringData:
  database-url: postgresql://prod-db:5432/myapp
  api-key: sk_prod_abc123xyz

# Encrypt with age
sops --encrypt --age $(cat key.txt.pub) secrets/prod-secrets.yaml > secrets/prod-secrets.enc.yaml

# Commit encrypted file
git add secrets/prod-secrets.enc.yaml

Flux SOPS Integration

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp-secrets
  namespace: flux-system
spec:
  interval: 10m
  path: ./secrets
  prune: true
  sourceRef:
    kind: GitRepository
    name: myapp
  decryption:
    provider: sops
    secretRef:
      name: sops-age
---
apiVersion: v1
kind: Secret
metadata:
  name: sops-age
  namespace: flux-system
stringData:
  age.agekey: |
    # created: 2025-01-01T00:00:00Z
    # public key: age1...
    AGE-SECRET-KEY-1...