Progressive Delivery

Progressive Delivery

Two complementary tools for moving versions safely through Kubernetes environments:

• Argo Rollouts — replaces Deployment with a Rollout CRD that supports canary, blue-green, and metric-gated automated analysis. Handles traffic shaping at the cluster/service-mesh layer.
• Kargo — extends GitOps with promotion logic. Tracks Freight (versioned bundles of artifacts) and promotes it through Stages (dev → staging → prod) via Warehouses (sources) and verification steps.

ArgoCD continues to do what it does (sync desired state → cluster). Rollouts decides traffic split during a single deploy. Kargo decides when the next stage gets the new version.

Scope routing

Mental model

Kargo:    Freight v1.2.3 -> [dev stage] -> verify -> [stg stage] -> verify -> [prod stage]
                                                                              |
                                                                              v
ArgoCD:                                                                   syncs Rollout manifest
                                                                              |
                                                                              v
Rollouts:                                                          canary @ 10% -> analysis -> 50% -> 100%

When NOT to use

• Simple Deployment rollouts that don't need traffic shaping or analysis gates — vanilla Kubernetes Deployments are fine.
• Manual promotion via PRs editing target revision — that's the core argocd skill, not this one.
• Feature flags and runtime percentage rollouts inside the app — that's an application concern (LaunchDarkly, Unleash, etc.), not a deployment one.

Gotchas

• Rollouts replaces Deployment; it is not an addition. Migrating an existing app means changing the resource kind. Plan for one revision of downtime if not handled with kubectl-argo-rollouts conversion.
• AnalysisTemplate metrics queries are scoped to the Rollouts controller's permissions. If your Prometheus is in another namespace, the controller needs RBAC or a service-account token.
• Kargo Freight is immutable. Once produced, you don't edit it — you produce new Freight. Trying to "patch" a Stage's current Freight is an anti-pattern.
• Kargo + ArgoCD integration requires Kargo's controller to have permission to update ArgoCD `Application` CRs. Default install doesn't grant this — read the Helm values for argocd.permissions.
• Verification steps run between stages, not within them. A failing verification doesn't roll back the prior stage — it just blocks promotion forward. If you need rollback, that's a separate Rollouts-level analysis.

Argo Rollouts Skill

Comprehensive guide for Argo Rollouts - a Kubernetes controller providing advanced deployment capabilities including blue-green, canary, and experimentation for Kubernetes.

Quick Reference

Core Concepts

Rollout CRD

The Rollout resource replaces standard Kubernetes Deployment and provides:

• Blue-Green Strategy: Instant traffic switching between versions
• Canary Strategy: Gradual traffic shifting with analysis gates
• Traffic Management: Integration with service meshes and ingress controllers
• Automated Analysis: Metrics-based promotion/rollback decisions

Deployment Strategies

Blue-Green:

strategy:
  blueGreen:
    activeService: my-app-active
    previewService: my-app-preview
    autoPromotionEnabled: false

Canary:

strategy:
  canary:
    steps:
    - setWeight: 20
    - pause: {duration: 5m}
    - setWeight: 50
    - analysis:
        templates:
        - templateName: success-rate

Traffic Management Integrations

CLI Commands (kubectl-argo-rollouts)

# Installation
kubectl argo rollouts version

# Rollout Management
kubectl argo rollouts get rollout <name>
kubectl argo rollouts status <name>
kubectl argo rollouts promote <name>
kubectl argo rollouts abort <name>
kubectl argo rollouts retry <name>
kubectl argo rollouts undo <name>
kubectl argo rollouts pause <name>
kubectl argo rollouts restart <name>

# Dashboard
kubectl argo rollouts dashboard

# Validation
kubectl argo rollouts lint <file>

Analysis Providers

Reference Documentation

• Summary - Overview and architecture
• Deployment Strategies - Blue-green and canary details
• CLI Commands - kubectl plugin reference
• Analysis & Metrics - AnalysisTemplate configuration
• Examples - Complete YAML examples

Common Patterns

Canary with Automated Analysis

steps:
- setWeight: 10
- pause: {duration: 1m}
- analysis:
    templates:
    - templateName: success-rate
    args:
    - name: service-name
      value: my-service
- setWeight: 50
- pause: {duration: 2m}

Blue-Green with Pre-Promotion Analysis

strategy:
  blueGreen:
    activeService: active-svc
    previewService: preview-svc
    prePromotionAnalysis:
      templates:
      - templateName: smoke-tests
    autoPromotionEnabled: false

Troubleshooting

Best Practices

1. Always use analysis gates for production canaries 2. Set appropriate pause durations between weight increases 3. Configure rollback thresholds in AnalysisTemplates 4. Use preview services for blue-green validation 5. Monitor AnalysisRuns during deployments 6. Version your AnalysisTemplates alongside application code

Analysis & Metrics Integration

Overview

Argo Rollouts Analysis system allows automated promotion or rollback decisions based on metrics from various providers.

Analysis CRDs

AnalysisTemplate

Namespace-scoped template defining metrics and success criteria.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    interval: 1m
    count: 5
    successCondition: result[0] >= 0.95
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(
            http_requests_total{service="{{args.service-name}}",status=~"2.*"}[5m]
          )) / sum(rate(
            http_requests_total{service="{{args.service-name}}"}[5m]
          ))

ClusterAnalysisTemplate

Cluster-scoped version for organization-wide templates.

apiVersion: argoproj.io/v1alpha1
kind: ClusterAnalysisTemplate
metadata:
  name: global-success-rate
spec:
  args:
  - name: service-name
  - name: namespace
  metrics:
  - name: success-rate
    successCondition: result[0] >= 0.95
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(
            http_requests_total{service="{{args.service-name}}",namespace="{{args.namespace}}"}[5m]
          ))

AnalysisRun

Instantiated analysis (created automatically by Rollouts).

apiVersion: argoproj.io/v1alpha1
kind: AnalysisRun
metadata:
  name: my-rollout-abc123-1
spec:
  args:
  - name: service-name
    value: my-service
  metrics:
  - name: success-rate
    # ... copied from template

Metric Configuration

Common Metric Fields

Success/Failure Conditions

# Numeric comparison
successCondition: result[0] >= 0.95

# Boolean result
successCondition: result[0] == true

# Multiple conditions
successCondition: result[0] >= 0.95 && result[1] < 500

# Array access (for multiple values)
successCondition: result[0] >= 0.90 && result[1] >= 0.95

# Failure condition
failureCondition: result[0] < 0.80

Metric Providers

Prometheus

provider:
  prometheus:
    address: http://prometheus.monitoring:9090
    timeout: 30
    insecure: false
    headers:
    - key: Authorization
      value: Bearer token
    query: |
      sum(rate(http_requests_total{status=~"5.*"}[5m]))
      / sum(rate(http_requests_total[5m])) * 100

Datadog

Credentials: Datadog requires DD_API_KEY and DD_APP_KEY configured via a Kubernetes Secret

referenced in the argo-rollouts controller deployment, or set as environment variables.

provider:
  datadog:
    interval: 5m
    query: avg:kubernetes.cpu.usage{service:{{args.service-name}}}
    apiVersion: v2

New Relic

Profiles: New Relic profiles are configured in a ConfigMap (argo-rollouts-config) with

personal-api-key and account-id. The profile field references a named configuration.

provider:
  newRelic:
    profile: default
    query: |
      SELECT average(duration)
      FROM Transaction
      WHERE appName = '{{args.service-name}}'

CloudWatch

IAM Permissions: The argo-rollouts controller needs IAM permissions for cloudwatch:GetMetricData.

Use IRSA (IAM Roles for Service Accounts) or node instance profiles.

provider:
  cloudWatch:
    interval: 5m
    metricDataQueries:
    - id: errors
      expression: "errors / requests * 100"
      label: "Error Rate"
    - id: errors
      metricStat:
        metric:
          namespace: AWS/ApplicationELB
          metricName: HTTPCode_Target_5XX_Count
          dimensions:
          - name: LoadBalancer
            value: "{{args.alb-name}}"
        period: 300
        stat: Sum

Wavefront

provider:
  wavefront:
    address: https://company.wavefront.com
    query: |
      mavg(5m, sum(rate(ts("requests.errors", service="{{args.service-name}}"))))

Kayenta (Automated Canary Analysis)

provider:
  kayenta:
    address: https://kayenta.example.com
    application: my-app
    canaryConfigName: my-canary-config
    metricsAccountName: prometheus
    configurationAccountName: s3
    storageAccountName: s3
    threshold:
      pass: 95
      marginal: 75
    scopes:
    - name: default
      controlScope:
        scope: baseline
        region: us-west-2
      experimentScope:
        scope: canary
        region: us-west-2

Web (HTTP)

⚠️ Security Warning: The Web provider makes HTTP requests to the specified URL.

Never use user-controlled input for URLs. Ensure proper network policies to prevent SSRF attacks.

Always validate and sanitize any templated arguments used in URLs.

provider:
  web:
    url: "http://my-service.default.svc/health"
    method: GET
    headers:
    - key: Authorization
      value: "Bearer {{args.token}}"
    timeoutSeconds: 30
    jsonPath: "{$.status}"

Job (Kubernetes Job)

provider:
  job:
    metadata:
      generateName: load-test-
    spec:
      backoffLimit: 1
      template:
        spec:
          containers:
          - name: load-test
            image: grafana/k6
            args: ["run", "/scripts/test.js"]
          restartPolicy: Never

Using Analysis in Rollouts

Inline Analysis (Canary Step)

spec:
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 5m}
      - analysis:
          templates:
          - templateName: success-rate
          args:
          - name: service-name
            value: my-service
      - setWeight: 50

Background Analysis (Continuous)

spec:
  strategy:
    canary:
      analysis:
        templates:
        - templateName: continuous-success-rate
        startingStep: 2
        args:
        - name: service-name
          value: my-service
      steps:
      - setWeight: 20
      - pause: {duration: 2m}
      - setWeight: 50
      - pause: {duration: 2m}

Pre-Promotion Analysis (Blue-Green)

spec:
  strategy:
    blueGreen:
      activeService: my-app-active
      previewService: my-app-preview
      prePromotionAnalysis:
        templates:
        - templateName: smoke-test
        - templateName: load-test
        args:
        - name: preview-url
          value: http://my-app-preview

Post-Promotion Analysis

spec:
  strategy:
    blueGreen:
      postPromotionAnalysis:
        templates:
        - templateName: post-deploy-health

Common AnalysisTemplates

HTTP Success Rate (Prometheus)

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: http-success-rate
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    interval: 1m
    count: 5
    successCondition: result[0] >= 0.95
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(
            http_requests_total{service="{{args.service-name}}",code=~"2.*"}[5m]
          )) / sum(rate(
            http_requests_total{service="{{args.service-name}}"}[5m]
          ))

Latency Check (Prometheus)

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency-check
spec:
  args:
  - name: service-name
  - name: latency-threshold
    value: "500"
  metrics:
  - name: p99-latency
    interval: 1m
    count: 5
    successCondition: result[0] < {{args.latency-threshold}}
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          histogram_quantile(0.99, sum(rate(
            http_request_duration_seconds_bucket{service="{{args.service-name}}"}[5m]
          )) by (le)) * 1000

Error Rate Check

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  args:
  - name: service-name
  - name: max-error-rate
    value: "0.05"
  metrics:
  - name: error-rate
    interval: 1m
    count: 5
    successCondition: result[0] <= {{args.max-error-rate}}
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(
            http_requests_total{service="{{args.service-name}}",code=~"5.*"}[5m]
          )) / sum(rate(
            http_requests_total{service="{{args.service-name}}"}[5m]
          ))

Health Check (Web)

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: health-check
spec:
  args:
  - name: host
  metrics:
  - name: health
    interval: 30s
    count: 10
    successCondition: result == "healthy"
    failureLimit: 3
    provider:
      web:
        url: "http://{{args.host}}/health"
        jsonPath: "{$.status}"

Dry-Run Analysis

Test analysis without affecting rollout:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisRun
metadata:
  name: test-analysis
spec:
  metrics:
  - name: test-success-rate
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(http_requests_total{status="200"}[5m]))
    successCondition: result[0] > 100
    count: 1
  dryRun:
  - metricName: test-success-rate

Troubleshooting Analysis

Check AnalysisRun Status

# List analysis runs
kubectl argo rollouts list analysisruns

# Get specific run
kubectl argo rollouts get analysisrun <name>

# Check measurements
kubectl get analysisrun <name> -o jsonpath='{.status.metricResults}'

Common Issues

CLI Commands Reference

Installation

kubectl-argo-rollouts Plugin

# macOS (Homebrew)
brew install argoproj/tap/kubectl-argo-rollouts

# Linux/macOS (curl)
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-$(uname -s | tr '[:upper:]' '[:lower:]')-amd64
chmod +x kubectl-argo-rollouts-*
sudo mv kubectl-argo-rollouts-* /usr/local/bin/kubectl-argo-rollouts

# Windows (PowerShell)
Invoke-WebRequest -Uri https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-windows-amd64 -OutFile kubectl-argo-rollouts.exe

Verify Installation

kubectl argo rollouts version

Rollout Management Commands

Get Rollout Status

# Basic status
kubectl argo rollouts get rollout <rollout-name>

# Watch mode (live updates)
kubectl argo rollouts get rollout <rollout-name> -w

# All namespaces
kubectl argo rollouts get rollout <rollout-name> -A

# Specific namespace
kubectl argo rollouts get rollout <rollout-name> -n <namespace>

# Output as YAML
kubectl argo rollouts get rollout <rollout-name> -o yaml

Status Check

# Check rollout status
kubectl argo rollouts status <rollout-name>

# Wait for rollout to complete
kubectl argo rollouts status <rollout-name> --watch

# Timeout after specified duration
kubectl argo rollouts status <rollout-name> --timeout 5m

List Rollouts

# List all rollouts
kubectl argo rollouts list rollouts

# All namespaces
kubectl argo rollouts list rollouts -A

# Specific namespace
kubectl argo rollouts list rollouts -n <namespace>

# Watch mode
kubectl argo rollouts list rollouts -w

Rollout Control Commands

Promote

# Promote to next step or full promotion
kubectl argo rollouts promote <rollout-name>

# Skip current step only
kubectl argo rollouts promote <rollout-name> --skip-current-step

# Skip all remaining steps (full promotion)
kubectl argo rollouts promote <rollout-name> --full

Pause

# Pause rollout
kubectl argo rollouts pause <rollout-name>

Resume

# Resume paused rollout (continues to next step)
kubectl argo rollouts promote <rollout-name>

Abort

# Abort rollout (scales down canary, routes to stable)
kubectl argo rollouts abort <rollout-name>

Retry

# Retry a failed rollout
kubectl argo rollouts retry <rollout-name>

Undo (Rollback)

# Undo to previous revision
kubectl argo rollouts undo <rollout-name>

# Undo to specific revision
kubectl argo rollouts undo <rollout-name> --to-revision=2

Restart

# Restart rollout (triggers new rollout with same spec)
kubectl argo rollouts restart <rollout-name>

Set Image

# Update container image
kubectl argo rollouts set image <rollout-name> <container>=<image>:<tag>

# Example
kubectl argo rollouts set image my-rollout app=nginx:1.21

Analysis Commands

List AnalysisRuns

# List all analysis runs
kubectl argo rollouts list analysisruns

# Watch mode
kubectl argo rollouts list analysisruns -w

Get AnalysisRun

# Get specific analysis run
kubectl argo rollouts get analysisrun <analysisrun-name>

# Watch mode
kubectl argo rollouts get analysisrun <analysisrun-name> -w

Experiment Commands

List Experiments

# List all experiments
kubectl argo rollouts list experiments

# Watch mode
kubectl argo rollouts list experiments -w

Get Experiment

# Get specific experiment
kubectl argo rollouts get experiment <experiment-name>

# Watch mode
kubectl argo rollouts get experiment <experiment-name> -w

Dashboard

Start Dashboard

# Start web dashboard on localhost:3100
kubectl argo rollouts dashboard

# Custom port
kubectl argo rollouts dashboard --port 8080

# Custom address
kubectl argo rollouts dashboard --address 0.0.0.0

Validation Commands

Lint

# Validate rollout YAML
kubectl argo rollouts lint -f rollout.yaml

# Validate with specific namespace
kubectl argo rollouts lint -f rollout.yaml -n <namespace>

Notifications

Create Notification

# Create notification ConfigMap (for notification controller)
kubectl argo rollouts notifications template get <template-name>

# List notification templates
kubectl argo rollouts notifications template list

# List triggers
kubectl argo rollouts notifications trigger list

Common Usage Patterns

Deploy New Version

# Update image and monitor
kubectl argo rollouts set image my-rollout app=myapp:v2
kubectl argo rollouts get rollout my-rollout -w

Manual Canary Progression

# Start rollout (image update triggers rollout)
kubectl argo rollouts set image my-rollout app=myapp:v2

# Wait for first step
kubectl argo rollouts status my-rollout

# Promote to next step
kubectl argo rollouts promote my-rollout

# Continue promoting...
kubectl argo rollouts promote my-rollout

Emergency Rollback

# Abort current deployment
kubectl argo rollouts abort my-rollout

# Or undo to previous
kubectl argo rollouts undo my-rollout

# Verify stable
kubectl argo rollouts get rollout my-rollout

Debug Failed Rollout

# Check rollout status
kubectl argo rollouts get rollout my-rollout

# Check analysis runs
kubectl argo rollouts list analysisruns

# Get failed analysis details
kubectl argo rollouts get analysisrun <failed-run-name>

# Retry after fixing issue
kubectl argo rollouts retry my-rollout

Output Formats

Namespace Flags

Common Options

# Help for any command
kubectl argo rollouts <command> --help

# Verbose output
kubectl argo rollouts <command> -v=6

# Dry run (where applicable)
kubectl argo rollouts <command> --dry-run

Deployment Strategies

Overview

Argo Rollouts supports two primary deployment strategies:

1. Canary: Gradual traffic shifting with analysis gates 2. Blue-Green: Instant traffic switching between versions

Canary Strategy

Basic Canary

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-rollout
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 1m}
      - setWeight: 40
      - pause: {duration: 1m}
      - setWeight: 60
      - pause: {duration: 1m}
      - setWeight: 80
      - pause: {duration: 1m}
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: my-app:v2

Canary Step Types

Canary with Analysis

spec:
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 2m}
      - analysis:
          templates:
          - templateName: success-rate
          args:
          - name: service-name
            value: my-service
      - setWeight: 30
      - pause: {duration: 2m}
      - analysis:
          templates:
          - templateName: latency-check
      - setWeight: 50
      - pause: {duration: 5m}

Canary with Traffic Management (Istio)

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-istio
spec:
  replicas: 5
  strategy:
    canary:
      canaryService: my-app-canary
      stableService: my-app-stable
      trafficRouting:
        istio:
          virtualService:
            name: my-app-vsvc
            routes:
            - primary
      steps:
      - setWeight: 10
      - pause: {duration: 1m}
      - setWeight: 30
      - pause: {duration: 1m}
      - setWeight: 50
      - pause: {}  # Manual promotion required
  selector:
    matchLabels:
      app: my-app
  template:
    # ...

Canary with NGINX Ingress

spec:
  strategy:
    canary:
      canaryService: my-app-canary
      stableService: my-app-stable
      trafficRouting:
        nginx:
          stableIngress: my-app-ingress
          annotationPrefix: nginx.ingress.kubernetes.io
          additionalIngressAnnotations:
            canary-by-header: X-Canary
      steps:
      - setWeight: 20
      - pause: {duration: 5m}
      - setWeight: 50
      - pause: {}

Blue-Green Strategy

Basic Blue-Green

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: blue-green-rollout
spec:
  replicas: 3
  strategy:
    blueGreen:
      activeService: my-app-active
      previewService: my-app-preview
      autoPromotionEnabled: false
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: my-app:v2

Blue-Green Configuration Options

Blue-Green with Pre-Promotion Analysis

spec:
  strategy:
    blueGreen:
      activeService: my-app-active
      previewService: my-app-preview
      autoPromotionEnabled: false
      prePromotionAnalysis:
        templates:
        - templateName: smoke-tests
        args:
        - name: preview-url
          value: http://my-app-preview.default.svc.cluster.local

Blue-Green with Post-Promotion Analysis

spec:
  strategy:
    blueGreen:
      activeService: my-app-active
      previewService: my-app-preview
      autoPromotionEnabled: true
      autoPromotionSeconds: 30
      postPromotionAnalysis:
        templates:
        - templateName: post-deploy-checks
        args:
        - name: service-name
          value: my-app-active

Blue-Green with Anti-Affinity

spec:
  strategy:
    blueGreen:
      activeService: my-app-active
      previewService: my-app-preview
      antiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          weight: 100
      scaleDownDelaySeconds: 30

Comparison: Canary vs Blue-Green

Advanced Patterns

Canary with Header-Based Routing

spec:
  strategy:
    canary:
      canaryService: my-app-canary
      stableService: my-app-stable
      trafficRouting:
        istio:
          virtualService:
            name: my-app-vsvc
            routes:
            - primary
      steps:
      - setHeaderRoute:
          name: canary-header
          match:
          - headerName: X-Canary
            headerValue:
              exact: "true"
      - pause: {}  # Test with header
      - setWeight: 20
      - pause: {duration: 5m}

Canary with Experiment

spec:
  strategy:
    canary:
      steps:
      - experiment:
          duration: 5m
          templates:
          - name: baseline
            specRef: stable
            replicas: 1
          - name: canary
            specRef: canary
            replicas: 1
          analyses:
          - name: compare
            templateName: ab-test-analysis
      - setWeight: 50
      - pause: {duration: 5m}

Gradual Scale with Canary

spec:
  strategy:
    canary:
      steps:
      - setCanaryScale:
          replicas: 1
      - pause: {duration: 2m}
      - setCanaryScale:
          weight: 25  # 25% of stable replicas
      - setWeight: 25
      - pause: {duration: 5m}
      - setCanaryScale:
          matchTrafficWeight: true  # Match traffic weight
      - setWeight: 50

Rollback Behavior

Automatic Rollback (Analysis Failure)

When an AnalysisRun fails, the Rollout automatically:

1. Aborts the current update 2. Scales down canary/preview ReplicaSet 3. Routes all traffic to stable version 4. Sets status to "Degraded"

Manual Rollback

# Abort current rollout and rollback
kubectl argo rollouts abort my-rollout

# Undo to previous version
kubectl argo rollouts undo my-rollout

# Undo to specific revision
kubectl argo rollouts undo my-rollout --to-revision=2

Complete YAML Examples

Basic Canary Rollout

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: basic-canary
  namespace: default
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: nginx:1.21
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi
        readinessProbe:
          httpGet:
            path: /healthz
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 30s}
      - setWeight: 40
      - pause: {duration: 30s}
      - setWeight: 60
      - pause: {duration: 30s}
      - setWeight: 80
      - pause: {duration: 30s}

Basic Blue-Green Rollout

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: basic-blue-green
  namespace: default
spec:
  replicas: 3
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: nginx:1.21
        ports:
        - containerPort: 80
  strategy:
    blueGreen:
      activeService: my-app-active
      previewService: my-app-preview
      autoPromotionEnabled: false
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-active
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-preview
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 80

Note on Service Selectors: Both activeService and previewService use the same base selector (app: my-app).

Argo Rollouts automatically manages rollouts-pod-template-hash labels on pods to distinguish between stable and preview ReplicaSets.

You do NOT need to add hash-based selectors to your services—the Rollouts controller handles traffic routing automatically.

Canary with Istio Traffic Management

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-istio
  namespace: default
spec:
  replicas: 5
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
        istio-injection: enabled
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: app
        image: myregistry/myapp:v1
        ports:
        - containerPort: 8080
        env:
        - name: VERSION
          value: "v1"
  strategy:
    canary:
      stableService: my-app-stable
      canaryService: my-app-canary
      trafficRouting:
        istio:
          virtualService:
            name: my-app-vsvc
            routes:
            - primary
      steps:
      - setWeight: 5
      - pause: {duration: 1m}
      - setWeight: 20
      - pause: {duration: 2m}
      - setWeight: 50
      - pause: {duration: 2m}
      - setWeight: 80
      - pause: {duration: 2m}
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-stable
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-canary
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app-vsvc
spec:
  hosts:
  - my-app.example.com
  gateways:
  - my-gateway
  http:
  - name: primary
    route:
    - destination:
        host: my-app-stable
      weight: 100
    - destination:
        host: my-app-canary
      weight: 0

Canary with Analysis

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-with-analysis
  namespace: default
spec:
  replicas: 5
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: myregistry/myapp:v1
        ports:
        - containerPort: 8080
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 2m}
      - analysis:
          templates:
          - templateName: success-rate-analysis
          args:
          - name: service-name
            value: my-app
      - setWeight: 30
      - pause: {duration: 2m}
      - analysis:
          templates:
          - templateName: latency-analysis
      - setWeight: 50
      - pause: {duration: 5m}
      - analysis:
          templates:
          - templateName: success-rate-analysis
          - templateName: latency-analysis
          args:
          - name: service-name
            value: my-app
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-analysis
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    interval: 1m
    count: 5
    successCondition: result[0] >= 0.95
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(
            http_requests_total{service="{{args.service-name}}",status=~"2.*"}[5m]
          )) / sum(rate(
            http_requests_total{service="{{args.service-name}}"}[5m]
          ))
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency-analysis
spec:
  metrics:
  - name: p99-latency
    interval: 1m
    count: 5
    successCondition: result[0] < 500
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          histogram_quantile(0.99, sum(rate(
            http_request_duration_seconds_bucket[5m]
          )) by (le)) * 1000

Blue-Green with Pre/Post Promotion Analysis

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: blue-green-with-analysis
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: myregistry/myapp:v1
        ports:
        - containerPort: 8080
  strategy:
    blueGreen:
      activeService: my-app-active
      previewService: my-app-preview
      autoPromotionEnabled: false
      scaleDownDelaySeconds: 30
      prePromotionAnalysis:
        templates:
        - templateName: smoke-test
        args:
        - name: preview-host
          value: my-app-preview.default.svc.cluster.local
      postPromotionAnalysis:
        templates:
        - templateName: post-deploy-check
        args:
        - name: active-host
          value: my-app-active.default.svc.cluster.local
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: smoke-test
spec:
  args:
  - name: preview-host
  metrics:
  - name: smoke-test
    count: 1
    successCondition: result == "ok"
    provider:
      web:
        url: "http://{{args.preview-host}}/health"
        jsonPath: "{$.status}"
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: post-deploy-check
spec:
  args:
  - name: active-host
  metrics:
  - name: post-deploy
    interval: 30s
    count: 5
    successCondition: result == "ok"
    failureLimit: 1
    provider:
      web:
        url: "http://{{args.active-host}}/health"
        jsonPath: "{$.status}"

Canary with Background Analysis

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-background-analysis
spec:
  replicas: 5
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: myregistry/myapp:v1
        ports:
        - containerPort: 8080
  strategy:
    canary:
      # Background analysis runs continuously during rollout
      analysis:
        templates:
        - templateName: continuous-analysis
        startingStep: 1  # Start after first step
        args:
        - name: service-name
          value: my-app
      steps:
      - setWeight: 10
      - pause: {duration: 5m}
      - setWeight: 30
      - pause: {duration: 5m}
      - setWeight: 50
      - pause: {duration: 5m}
      - setWeight: 80
      - pause: {duration: 5m}
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: continuous-analysis
spec:
  args:
  - name: service-name
  metrics:
  - name: error-rate
    interval: 1m
    failureLimit: 5
    successCondition: result[0] < 0.05
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(http_requests_total{service="{{args.service-name}}",status=~"5.*"}[1m]))
          / sum(rate(http_requests_total{service="{{args.service-name}}"}[1m]))

Canary with Experiment (A/B Testing)

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-experiment
spec:
  replicas: 5
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: myregistry/myapp:v1
        ports:
        - containerPort: 8080
  strategy:
    canary:
      steps:
      - experiment:
          duration: 10m
          templates:
          - name: baseline
            specRef: stable
            replicas: 1
          - name: canary
            specRef: canary
            replicas: 1
          analyses:
          - name: compare
            templateName: ab-test
            args:
            - name: baseline-hash
              valueFrom:
                podTemplateHashValue: Stable
            - name: canary-hash
              valueFrom:
                podTemplateHashValue: Latest
      - setWeight: 50
      - pause: {duration: 5m}
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: ab-test
spec:
  args:
  - name: baseline-hash
  - name: canary-hash
  metrics:
  - name: conversion-rate-comparison
    interval: 1m
    count: 10
    successCondition: result[0] >= result[1]
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(conversions_total{pod_hash="{{args.canary-hash}}"}[5m]))
          / sum(rate(page_views_total{pod_hash="{{args.canary-hash}}"}[5m]))

Canary with NGINX Ingress

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-nginx
spec:
  replicas: 5
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: myregistry/myapp:v1
        ports:
        - containerPort: 8080
  strategy:
    canary:
      stableService: my-app-stable
      canaryService: my-app-canary
      trafficRouting:
        nginx:
          stableIngress: my-app-ingress
          annotationPrefix: nginx.ingress.kubernetes.io
          additionalIngressAnnotations:
            canary-by-header: X-Canary
            canary-by-header-value: "true"
      steps:
      - setWeight: 10
      - pause: {duration: 2m}
      - setWeight: 30
      - pause: {duration: 2m}
      - setWeight: 50
      - pause: {}
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-stable
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-canary
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: my-app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app-stable
            port:
              number: 80

ClusterAnalysisTemplate Example

apiVersion: argoproj.io/v1alpha1
kind: ClusterAnalysisTemplate
metadata:
  name: org-wide-success-rate
spec:
  args:
  - name: service-name
  - name: namespace
  - name: threshold
    value: "0.95"
  metrics:
  - name: success-rate
    interval: 1m
    count: 5
    successCondition: result[0] >= {{args.threshold}}
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(
            http_requests_total{
              service="{{args.service-name}}",
              namespace="{{args.namespace}}",
              status=~"2.*"
            }[5m]
          )) / sum(rate(
            http_requests_total{
              service="{{args.service-name}}",
              namespace="{{args.namespace}}"
            }[5m]
          ))

Using ClusterAnalysisTemplate in Rollout

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-rollout
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: myregistry/myapp:v1
  strategy:
    canary:
      steps:
      - setWeight: 20
      - analysis:
          templates:
          - templateName: org-wide-success-rate
            clusterScope: true  # Reference ClusterAnalysisTemplate
          args:
          - name: service-name
            value: my-app
          - name: namespace
            value: production
          - name: threshold
            value: "0.98"
      - setWeight: 50
      - pause: {duration: 5m}

HPA Integration

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-with-hpa
spec:
  replicas: 5
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: myregistry/myapp:v1
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 5m}
      - setWeight: 50
      - pause: {duration: 5m}
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: rollout-with-hpa
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Argo Rollouts Summary

Overview

Argo Rollouts is a Kubernetes controller and set of CRDs that provides advanced deployment capabilities including:

• Blue-Green Deployments: Instant traffic switching between versions
• Canary Deployments: Gradual traffic shifting with configurable steps
• Progressive Delivery: Automated analysis and promotion/rollback
• Traffic Management: Native integration with service meshes and ingress controllers
• Experimentation: A/B testing with multiple ReplicaSets

Architecture

┌─────────────────────────────────────────────────────────┐
│                  Argo Rollouts Controller                │
├─────────────────────────────────────────────────────────┤
│  Rollout CRD    │  AnalysisTemplate  │  Experiment CRD  │
│  (replaces      │  (defines metrics  │  (A/B testing    │
│   Deployment)   │   for analysis)    │   workloads)     │
├─────────────────────────────────────────────────────────┤
│              Traffic Management Layer                    │
│  ┌─────────┬─────────┬─────────┬─────────┬──────────┐  │
│  │  Istio  │  NGINX  │   ALB   │ Traefik │ Linkerd  │  │
│  └─────────┴─────────┴─────────┴─────────┴──────────┘  │
├─────────────────────────────────────────────────────────┤
│              Analysis & Metrics Providers               │
│  ┌──────────┬─────────┬──────────┬──────────────────┐  │
│  │Prometheus│ Datadog │ Wavefront│ CloudWatch/NewRelic│  │
│  └──────────┴─────────┴──────────┴──────────────────┘  │
└─────────────────────────────────────────────────────────┘

Core CRDs

Key Features

1. Progressive Delivery

• Gradual rollout with configurable weight steps
• Automatic promotion based on metrics
• Automated rollback on failure detection

2. Traffic Management Integration

• Native integration with Istio VirtualService
• NGINX Ingress Controller support
• AWS ALB Ingress Controller
• Ambassador, Traefik, Linkerd, SMI support

3. Metrics-Based Analysis

• Query metrics from multiple providers
• Define success criteria and thresholds
• Automated pass/fail decisions

4. Rollout Strategies

• Canary: Gradual traffic shifting (1% → 5% → 20% → 100%)
• Blue-Green: Instant switch with preview environment
• Hybrid: Combine strategies with analysis gates

Installation

# Install Argo Rollouts controller
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

# Install kubectl plugin
brew install argoproj/tap/kubectl-argo-rollouts
# or
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-darwin-amd64
chmod +x kubectl-argo-rollouts-darwin-amd64
sudo mv kubectl-argo-rollouts-darwin-amd64 /usr/local/bin/kubectl-argo-rollouts

How It Works

1. Rollout Created: Controller creates initial ReplicaSet 2. Update Triggered: New ReplicaSet created with updated spec 3. Traffic Shifted: Based on strategy (canary weights or blue-green switch) 4. Analysis Runs: Metrics queried against defined thresholds 5. Promotion/Rollback: Automatic decision based on analysis results

Comparison with Kubernetes Deployment

When to Use Argo Rollouts

Good fit when:

• Need gradual rollouts with traffic control
• Want automated canary analysis
• Require instant rollback capability
• Using service mesh for traffic management
• Need A/B testing capabilities

May not need if:

• Simple rolling updates are sufficient
• No service mesh or advanced ingress
• Don't require metrics-based promotion

Kargo Skill

Complete guide for Kargo - an unopinionated continuous promotion platform that extends GitOps principles with progressive delivery capabilities.

Overview

Kargo manages the promotion of desired state through environments while tools like ArgoCD handle syncing actual state to desired state in Git. Kargo complements ArgoCD by handling promotion logic.

Installation

Prerequisites

• Helm v3.13.1 or later
• Kubernetes cluster with cert-manager pre-installed
• Optional: ArgoCD v2.13.0+, Argo Rollouts v1.7.2+

Basic Installation (Helm)

# Generate required values
export ADMIN_PASSWORD_HASH=$(htpasswd -bnBC 10 admin <password> | cut -d: -f2)
export TOKEN_SIGNING_KEY=$(openssl rand -base64 48)

# Install Kargo
helm install kargo \
  oci://ghcr.io/akuity/kargo-charts/kargo \
  --namespace kargo \
  --create-namespace \
  --set api.adminAccount.passwordHash="$ADMIN_PASSWORD_HASH" \
  --set api.adminAccount.tokenSigningKey="$TOKEN_SIGNING_KEY" \
  --wait

Quick Start (All-in-One)

curl -L https://raw.githubusercontent.com/akuity/kargo/main/hack/quickstart/install.sh | sh

Troubleshooting

• 401 errors: Update Helm to v3.13.1+
• 403 errors: Run docker logout ghcr.io

Core Concepts

Projects

Units of tenancy for organizing promotion pipelines. Each project maps to a Kubernetes namespace.

apiVersion: kargo.akuity.io/v1alpha1
kind: Project
metadata:
  name: my-project

Warehouses

Monitor repositories for new artifact revisions and package them into Freight.

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: my-warehouse
  namespace: my-project
spec:
  subscriptions:
  - image:
      repoURL: public.ecr.aws/nginx/nginx
      imageSelectionStrategy: SemVer
      constraint: ^1.26.0
  - git:
      repoURL: https://github.com/example/repo.git
      branch: main
      commitSelectionStrategy: NewestFromBranch
  - chart:
      repoURL: https://charts.example.com
      name: my-chart
      semverConstraint: ^1.0.0

Image Selection Strategies

Git Commit Selection Strategies

Git Expression Filters

# Exclude bot commits
expressionFilter: !(author contains '<bot@example.com>')

# Filter by commit message
expressionFilter: subject contains 'feat:' || subject contains 'fix:'

# Filter by date
expressionFilter: creatorDate.Year() >= 2024

Path Filtering

includePaths:
  - apps/guestbook
  - glob:apps/*/config
excludePaths:
  - apps/guestbook/README.md
  - regex:.*\.test\.yaml$

Freight

Meta-artifacts containing references to specific artifact revisions. Ensures related artifacts move together through the pipeline.

# Update freight alias
kargo update freight \
  --project my-project \
  --name <freight-hash> \
  --new-alias frozen-tauntaun

# Manual approval
kargo approve \
  --project my-project \
  --freight <freight-id> \
  --stage prod

Stages

Promotion targets that link together to form pipelines.

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: test
  namespace: my-project
spec:
  vars:
    - name: gitopsRepo
      value: https://github.com/example/repo.git
    - name: targetBranch
      value: stage/test

  requestedFreight:
    - origin:
        kind: Warehouse
        name: my-warehouse
      sources:
        direct: true

  promotionTemplate:
    spec:
      steps:
        - uses: git-clone
        - uses: kustomize-set-image
        - uses: git-commit
        - uses: git-push
        - uses: argocd-update

  verification:
    analysisTemplates:
      - name: integration-test

Freight Availability Strategies

• OneOf (default): Verification needed in at least one upstream stage
• All: Verification required across all upstream stages

Auto-Promotion Policies

• NewestFreight: Continuously promotes newest verified/approved freight
• MatchUpstream: Promotes freight matching upstream stage's current version

Promotions

Move code and configuration changes through application lifecycle stages using GitOps.

# Promote via CLI
kargo promote \
  --project my-project \
  --freight <freight-id> \
  --stage prod

# Using alias
kargo promote \
  --project my-project \
  --freight-alias frozen-tauntaun \
  --stage prod

Promotion Steps Reference

Git Operations

git-clone

- uses: git-clone
  config:
    repoURL: https://github.com/example/repo.git
    author:
      name: "Kargo Bot"
      email: "kargo@example.com"
    checkout:
      - branch: main
        path: ./out
      - commit: abc123def456
        path: ./config
        create: true  # Create orphaned branch if missing

git-commit

- uses: git-commit
  as: commit
  config:
    path: ./out
    message: |
      Update image to ${{ imageFrom(vars.imageRepo).Tag }}
    author:
      name: "Kargo Automation"
      email: "kargo@example.com"

git-push

- uses: git-push
  as: push
  config:
    path: ./out
    targetBranch: ${{ vars.targetBranch }}
    maxAttempts: 50
    # OR for PR workflow:
    generateTargetBranch: true
    provider: github

Output: branch, commit, commitURL

git-open-pr

- uses: git-open-pr
  as: open-pr
  config:
    repoURL: https://github.com/example/repo.git
    provider: github
    sourceBranch: ${{ outputs['push'].branch }}
    targetBranch: main
    title: "Promote to ${{ ctx.stage }}"
    labels:
      - kargo
      - automated

Output: pr.id, pr.url

git-wait-for-pr

- uses: git-wait-for-pr
  config:
    repoURL: https://github.com/example/repo.git
    prNumber: ${{ outputs['open-pr'].pr.id }}
    provider: github

git-merge-pr

- uses: git-merge-pr
  config:
    repoURL: https://github.com/example/repo.git
    prNumber: ${{ outputs['open-pr'].pr.id }}
    wait: true

git-clear

- uses: git-clear
  config:
    path: ./out

Configuration Management

kustomize-set-image

- uses: kustomize-set-image
  config:
    path: ./out
    images:
    - image: ghcr.io/example/app
      tag: ${{ imageFrom(vars.imageRepo).Tag }}
    - image: ghcr.io/example/other
      newName: registry.example.com/other
      digest: ${{ imageFrom('ghcr.io/example/other').Digest }}

kustomize-build

- uses: kustomize-build
  config:
    path: ./src/overlays/test
    outPath: ./out/manifests.yaml
    plugin.helm.kubeVersion: "1.28.0"

helm-template

- uses: helm-template
  config:
    path: ./charts/my-chart
    outPath: ./out/manifests.yaml
    releaseName: my-release
    namespace: default
    outLayout: helm  # or 'flat'
    valuesFiles:
      - ./values-prod.yaml
    buildDependencies: true
    includeCRDs: true
    setValues:
      - key: image.tag
        value: ${{ imageFrom(vars.imageRepo).Tag }}

helm-update-chart

- uses: helm-update-chart
  config:
    path: ./src/my-chart
    charts:
      - repository: https://charts.example.com
        name: dependency-chart
        version: 2.0.0

yaml-update

- uses: yaml-update
  config:
    path: ./src/values.yaml
    updates:
      - key: image.tag
        value: ${{ imageFrom(vars.imageRepo).Tag }}
      - key: replicas
        value: "3"

json-update

- uses: json-update
  config:
    path: ./src/config.json
    updates:
      - key: version
        value: ${{ imageFrom(vars.imageRepo).Tag }}

ArgoCD Integration

argocd-update

- uses: argocd-update
  config:
    apps:
    - name: my-app
      namespace: argocd
      sources:
      - repoURL: https://github.com/example/repo.git
        desiredRevision: ${{ outputs.push.commit }}
        updateTargetRevision: true

Required Application Annotation:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  annotations:
    kargo.akuity.io/authorized-stage: "my-project:my-stage"

File Operations

copy

- uses: copy
  config:
    inPath: ./overlay/kustomization.yaml
    outPath: ./src/kustomization.yaml

delete

- uses: delete
  config:
    path: ./temp

External Integrations

http

- uses: http
  config:
    method: POST
    url: https://api.example.com/deploy
    headers:
      - name: Authorization
        value: "Bearer ${{ secrets.apiToken }}"
    body: '{"version":"${{ ctx.freight.displayID }}"}'
    timeout: 5m
    successExpression: response.status >= 200 && response.status < 300
    failureExpression: response.status >= 500

compose-output

- uses: compose-output
  config:
    pr_url: "${{ vars.repoURL }}/pull/${{ outputs['open-pr'].pr.id }}"
    commit_sha: "${{ outputs['commit'].commit }}"

Expression Language

Syntax

${{ expression }}

Built-in Variables

Functions

Artifact Functions

# Git commits
${{ commitFrom("https://github.com/example/repo.git").ID }}
${{ commitFrom("https://github.com/example/repo.git").Branch }}
${{ commitFrom("https://github.com/example/repo.git").Message }}
${{ commitFrom("https://github.com/example/repo.git").Author }}

# Container images
${{ imageFrom("public.ecr.aws/nginx/nginx").Tag }}
${{ imageFrom("public.ecr.aws/nginx/nginx").Digest }}
${{ imageFrom("public.ecr.aws/nginx/nginx").RepoURL }}

# Helm charts
${{ chartFrom("https://example.com/charts", "my-chart").Version }}
${{ chartFrom("https://example.com/charts", "my-chart").RepoURL }}

Status Functions

# Conditional execution
if: ${{ success() }}      # All preceding steps succeeded
if: ${{ failure() }}      # Any preceding step failed
if: ${{ always() }}       # Always execute
if: ${{ status("my-step") == "Succeeded" }}

Utility Functions

${{ quote(42) }}                    # Convert to quoted string
${{ configMap("my-config").key }}   # Read ConfigMap data
${{ secret("my-secret").password }} # Read Secret data
${{ warehouse("my-warehouse") }}    # Get Warehouse freight origin
${{ semverDiff("1.2.3", "1.3.0") }} # Returns: "Minor"

Verification

AnalysisTemplate Structure

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: integration-test
  namespace: my-project
spec:
  args:
    - name: commit
  metrics:
  - name: test-job
    provider:
      job:
        spec:
          template:
            spec:
              containers:
              - name: test
                image: alpine:latest
                command: ["sh", "-c", "sleep 10 && exit 0"]
              restartPolicy: Never
          backoffLimit: 1

Stage Verification Configuration

spec:
  verification:
    analysisTemplates:
      - name: integration-test
    analysisRunMetadata:
      labels:
        env: test
    args:
      - name: commit
        value: ${{ commitFrom("https://github.com/example/repo.git").ID }}

Soak Time

spec:
  requestedFreight:
    - origin:
        kind: Warehouse
        name: my-warehouse
      sources:
        stages:
          - uat
        requiredSoakTime: 24h

Patterns

Image Updater Pattern

Single Warehouse monitors image repository, produces Freight for each new version.

Config Updater Pattern

Warehouse tracks Git commits, stages combine base config with overlays.

Common Case Pattern

Single Warehouse subscribes to both image AND Git repositories, promoting both together.

Multiple Warehouses Pattern

Separate Warehouses for images and configs, enabling independent promotion cadences.

Grouped Services Pattern

Single Warehouse subscribes to multiple repositories, ensuring coordinated promotion.

Fanning Out/In Pattern

Non-linear pipeline with branching (e.g., A/B testing) and convergence.

PR Workflow Pattern

steps:
  - uses: git-push
    config:
      generateTargetBranch: true
      provider: github

  - uses: git-open-pr
    as: open-pr
    config:
      targetBranch: main
      title: "Promote to ${{ ctx.stage }}"

  - uses: git-wait-for-pr
    config:
      prNumber: ${{ outputs['open-pr'].pr.id }}
      timeout: 48h

Security

OIDC Configuration

api:
  oidc:
    enabled: true
    issuerURL: https://idp.example.com
    clientID: kargo-ui
    cliClientID: kargo-cli
    admins:
      claims:
        groups: [devops]
    projectCreators:
      claims:
        groups: [leads]

User-to-ServiceAccount Mapping

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
  namespace: my-project
  annotations:
    rbac.kargo.akuity.io/claims: |
      {
        "sub": ["alice", "bob"],
        "groups": ["devops", "kargo-admin"]
      }

Pre-defined Project Roles

• kargo-admin: Full project management permissions
• kargo-viewer: Read-only access
• default: Kubernetes-managed baseline

Credential Management

Git Credentials

apiVersion: v1
kind: Secret
metadata:
  name: git-credentials
  namespace: my-project
  labels:
    kargo.akuity.io/cred-type: git
stringData:
  repoURL: https://github.com/example/repo.git
  username: my-username
  password: my-personal-access-token

SSH Key Authentication

apiVersion: v1
kind: Secret
metadata:
  name: git-ssh-credentials
  labels:
    kargo.akuity.io/cred-type: git
stringData:
  repoURL: git@github.com:example/repo.git
  sshPrivateKey: <base64-encoded-ssh-key>

GitHub App Authentication

stringData:
  repoURL: https://github.com/example/repo.git
  githubAppClientID: "1234567"
  githubAppPrivateKey: <base64-encoded-app-key>
  githubAppInstallationID: "98765432"

Container Registry Credentials

AWS ECR:

metadata:
  labels:
    kargo.akuity.io/cred-type: image
stringData:
  repoURL: 123456789.dkr.ecr.us-west-2.amazonaws.com
  awsRegion: us-west-2
  awsAccessKeyID: AKIA...
  awsSecretAccessKey: ...

Google Artifact Registry:

stringData:
  repoURL: us-central1-docker.pkg.dev/my-project/my-repo
  gcpServiceAccountKey: |
    { "type": "service_account", ... }

CLI Credential Management

# Create credentials
kargo create credentials \
  --project my-project my-creds \
  --git \
  --repo-url https://github.com/example/repo.git \
  --username my-username \
  --password my-pat

# List credentials
kargo get credentials --project my-project

# Delete credentials
kargo delete credentials --project my-project my-creds

Project Configuration

ProjectConfig Resource

apiVersion: kargo.akuity.io/v1alpha1
kind: ProjectConfig
metadata:
  name: my-project
  namespace: my-project
spec:
  promotionPolicies:
    - stages: ["test", "uat"]
      autoPromotionEnabled: true
    - stages: ["prod"]
      autoPromotionEnabled: false

  webhookReceivers:
    - name: github-webhook
      github:
        secretRef:
          name: github-secret

Namespace Management

# Adopt pre-existing namespace
metadata:
  labels:
    kargo.akuity.io/project: "true"

# Preserve namespace after project deletion
metadata:
  annotations:
    kargo.akuity.io/keep-namespace: "true"

CLI Commands

# Authentication
kargo login https://kargo.example.com --admin
kargo login https://kargo.example.com --sso

# Projects
kargo create project my-project
kargo get project my-project
kargo delete project my-project

# Stages
kargo create -f stage.yaml
kargo get stage my-stage --project my-project
kargo refresh stage my-stage --project my-project
kargo delete stage my-stage --project my-project

# Freight
kargo get freight --project my-project
kargo approve --project my-project --freight <id> --stage prod

# Promotions
kargo promote --project my-project --freight <id> --stage prod
kargo promote --project my-project --freight-alias my-alias --stage prod

# Verification
kargo verify stage my-stage --project my-project
kargo verify stage my-stage --project my-project --abort

# Roles
kargo get roles --project my-project
kargo create role developer --project my-project
kargo grant --role developer --claim groups=dev --project my-project
kargo delete role developer --project my-project

# Credentials
kargo get credentials --project my-project
kargo create credentials --project my-project my-creds --git --repo-url <url> --username <user> --password <token>
kargo update credentials --project my-project my-creds --password <new-token>
kargo delete credentials --project my-project my-creds

Complete Stage Example

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: production
  namespace: my-project
spec:
  vars:
    - name: imageRepo
      value: ghcr.io/example/app
    - name: gitRepo
      value: https://github.com/example/deployment.git
    - name: gitBranch
      value: main
    - name: appName
      value: my-app

  requestedFreight:
    - origin:
        kind: Warehouse
        name: prod-warehouse
      sources:
        stages:
          - staging
        requiredSoakTime: 24h

  verification:
    analysisTemplates:
      - name: smoke-tests
    args:
      - name: imageTag
        value: ${{ imageFrom(vars.imageRepo).Tag }}

  promotionTemplate:
    spec:
      steps:
        - uses: git-clone
          as: clone
          config:
            repoURL: ${{ vars.gitRepo }}
            checkout:
              - branch: ${{ vars.gitBranch }}
                path: ./out

        - uses: kustomize-set-image
          config:
            path: ./out
            images:
              - image: ${{ vars.imageRepo }}
                tag: ${{ imageFrom(vars.imageRepo).Tag }}

        - uses: git-commit
          as: commit
          config:
            path: ./out
            message: |
              Promote to production
              Image: ${{ imageFrom(vars.imageRepo).Tag }}

        - uses: git-push
          as: push
          config:
            path: ./out
            targetBranch: ${{ vars.gitBranch }}

        - uses: argocd-update
          config:
            apps:
              - name: ${{ vars.appName }}
                sources:
                  - repoURL: ${{ vars.gitRepo }}
                    desiredRevision: ${{ outputs.push.commit }}
                    updateTargetRevision: true

References

For detailed documentation, see:

• references/promotion-steps.md - Complete promotion steps reference
• references/expressions.md - Expression language reference
• references/patterns.md - Deployment patterns
• references/security.md - Security configuration

Official Documentation

• Main Docs: <https://docs.kargo.io/>
• GitHub: <https://github.com/akuity/kargo>
• Examples: <https://github.com/akuity/kargo-examples>

Kargo Expressions Reference

Complete reference for Kargo's expression language based on expr-lang.

Syntax

All expressions use the ${{ }} delimiter:

config:
  message: ${{ "Hello, world!" }}
  tag: ${{ imageFrom(vars.imageRepo).Tag }}

Pre-defined Variables

Promotion Context (ctx)

Step Outputs (outputs)

Access output from previous steps by alias:

${{ outputs['step-alias'].fieldName }}
${{ outputs.push.commit }}
${{ outputs['open-pr'].pr.id }}

User Variables (vars)

Access variables defined at Stage or PromotionTemplate level:

${{ vars.gitRepo }}
${{ vars.targetBranch }}
${{ vars.imageRepo }}

Task Context (task)

Access outputs from previous steps within the same PromotionTask:

${{ task.previousStep.output }}

Built-in Functions

Artifact Functions

commitFrom

Get Git commit information from freight.

# Basic usage
${{ commitFrom("https://github.com/example/repo.git").ID }}
${{ commitFrom("https://github.com/example/repo.git").Branch }}
${{ commitFrom("https://github.com/example/repo.git").Message }}
${{ commitFrom("https://github.com/example/repo.git").Author }}
${{ commitFrom("https://github.com/example/repo.git").Committer }}
${{ commitFrom("https://github.com/example/repo.git").Tag }}

# With warehouse origin
${{ commitFrom("https://github.com/example/repo.git", warehouse("my-warehouse")).ID }}

Available Fields:

imageFrom

Get container image information from freight.

${{ imageFrom("public.ecr.aws/nginx/nginx").Tag }}
${{ imageFrom("public.ecr.aws/nginx/nginx").Digest }}
${{ imageFrom("public.ecr.aws/nginx/nginx").RepoURL }}
${{ imageFrom("public.ecr.aws/nginx/nginx").Annotations }}

# With warehouse origin
${{ imageFrom("public.ecr.aws/nginx/nginx", warehouse("my-warehouse")).Tag }}

Available Fields:

chartFrom

Get Helm chart information from freight.

${{ chartFrom("https://charts.example.com", "my-chart").Version }}
${{ chartFrom("https://charts.example.com", "my-chart").RepoURL }}
${{ chartFrom("https://charts.example.com", "my-chart").Name }}

# OCI charts
${{ chartFrom("oci://registry.example.com/charts", "my-chart").Version }}

Available Fields:

Origin Functions

warehouse

Get warehouse freight origin for artifact lookups.

${{ warehouse("my-warehouse") }}

# Usage with artifact functions
${{ imageFrom("ghcr.io/example/app", warehouse("my-warehouse")).Tag }}

Metadata Functions

freightMetadata

Retrieve freight metadata.

${{ freightMetadata("freight-id").label }}
${{ freightMetadata(ctx.targetFreight.name).annotation }}

stageMetadata

Retrieve stage metadata.

${{ stageMetadata("dev").labels.environment }}
${{ stageMetadata(ctx.stage).annotations.owner }}

Kubernetes Resources

configMap

Read ConfigMap data.

${{ configMap("my-config").someKey }}
${{ configMap("my-config", "custom-namespace").data }}

secret

Read Secret data.

${{ secret("my-secret").password }}
${{ secret("my-secret", "custom-namespace").apiKey }}

Status Functions

success

Returns true if all preceding steps succeeded.

if: ${{ success() }}

failure

Returns true if any preceding step failed.

if: ${{ failure() }}

always

Always returns true (for unconditional execution).

if: ${{ always() }}

status

Get status of a specific step by alias.

if: ${{ status("my-step") == "Succeeded" }}
if: ${{ status("my-step") == "Errored" }}
if: ${{ status("my-step") == "Skipped" }}

Status Values:

• Succeeded
• Errored
• Skipped
• Running
• Pending

Utility Functions

quote

Convert value to quoted string.

${{ quote(42) }}  # "42"
${{ quote(true) }}  # "true"

unsafeQuote

Convert to string with escaped quotes (use with caution).

${{ unsafeQuote("hello \"world\"") }}

semverDiff

Compare two semantic versions and return difference type.

${{ semverDiff("1.2.3", "1.3.0") }}  # "Minor"
${{ semverDiff("1.2.3", "2.0.0") }}  # "Major"
${{ semverDiff("1.2.3", "1.2.4") }}  # "Patch"
${{ semverDiff("1.2.3", "1.2.3") }}  # "None"

Return Values:

• Major - Major version changed
• Minor - Minor version changed
• Patch - Patch version changed
• Metadata - Only metadata/prerelease changed
• None - Versions are identical
• Incomparable - Versions cannot be compared

Expression Operators

Comparison Operators

${{ vars.value == "expected" }}
${{ vars.count != 0 }}
${{ vars.count > 5 }}
${{ vars.count >= 10 }}
${{ vars.count < 100 }}
${{ vars.count <= 50 }}

Logical Operators

${{ vars.enabled && vars.ready }}
${{ vars.dev || vars.test }}
${{ !vars.disabled }}

String Operations

${{ vars.name + "-suffix" }}
${{ vars.message contains "error" }}
${{ vars.name startsWith "prod" }}
${{ vars.name endsWith "-v1" }}
${{ vars.name matches "^prod-.*" }}

Ternary Operator

${{ vars.prod ? "production" : "development" }}

Nil Coalescing

${{ vars.optional ?? "default" }}

Complex Expressions

Conditional Logic

# Major version check
if: ${{ semverDiff(imageFrom(vars.imageRepo).Tag, outputs['read-version'].current) == 'Major' }}

# Combined conditions
if: ${{ success() && outputs['test'].passed == true }}

# Null-safe access
message: ${{ outputs['step']?.value ?? "default" }}

String Interpolation

message: "Updated ${{ ctx.stage }} to image ${{ imageFrom(vars.imageRepo).Tag }}"

body: |
  {
    "project": "${{ ctx.project }}",
    "stage": "${{ ctx.stage }}",
    "version": "${{ imageFrom(vars.imageRepo).Tag }}"
  }

JSON Construction

body: ${{ quote({
  "channel": vars.slackChannel,
  "text": "Deployed " + ctx.freight.displayID + " to " + ctx.stage
}) }}

Warehouse Expression Filters

Git Commit Filters

Available fields for expressionFilter:

• id - Commit SHA
• commitDate - Commit timestamp
• author - Author identity
• committer - Committer identity
• subject - First line of commit message

# Exclude bot commits
expressionFilter: !(author contains '<bot@example.com>')

# Filter by message pattern
expressionFilter: subject contains 'feat:' || subject contains 'fix:'

# Multiple conditions
expressionFilter: !(subject contains '[skip-ci]') && author != 'dependabot'

Git Tag Filters

Additional fields for tag-based selection:

• tag - Tag name
• creatorDate - Tag creation date
• tagger - Tagger identity
• annotation - Tag annotation message

# Filter by creation date
expressionFilter: creatorDate.Year() >= 2024

# Filter by tag pattern
expressionFilter: tag matches '^v[0-9]+\\.[0-9]+\\.[0-9]+$'

HTTP Response Expressions

For http step success/failure conditions:

successExpression: response.status >= 200 && response.status < 300
failureExpression: response.status >= 500

# Body checks (JSON)
successExpression: response.body.status == "success"
failureExpression: response.body.error != nil

# Header checks
successExpression: response.header("X-Request-Id") != ""

Note: Success/failure expressions should NOT be wrapped in ${{ }}.

Variable Scoping

Priority Order (highest to lowest)

1. Step-level variables 2. PromotionTask variables 3. PromotionTemplate variables 4. Stage variables

Example

# Stage
spec:
  vars:
    - name: repo
      value: https://github.com/example/repo.git
    - name: branch
      value: main

# PromotionTemplate (overrides stage vars)
spec:
  vars:
    - name: branch
      value: develop  # Overrides stage value

# Step (can reference both)
steps:
  - uses: git-clone
    config:
      repoURL: ${{ vars.repo }}     # From stage
      branch: ${{ vars.branch }}    # From template (overridden)

Type Handling

# Numeric
numField: ${{ 40 + 2 }}  # 42

# String
strField: ${{ quote(40 + 2) }}  # "42"

# Boolean
enabled: ${{ vars.prod == true }}

# Array access
first: ${{ ctx.freight.images[0].tag }}

# Map access
value: ${{ ctx.freight.commits["repo-url"].ID }}

Best Practices

1. Use `quote()` for JSON strings - Ensures proper escaping 2. Validate expressions in expr-lang playground - Test complex expressions before deployment 3. Use descriptive variable names - Improves readability 4. Handle nil values - Use ?? operator for optional values 5. Keep expressions simple - Break complex logic into multiple steps

Kargo Deployment Patterns Reference

Comprehensive guide to Kargo deployment patterns and architectural approaches.

1. Image Updater Pattern

Single Warehouse monitors image repository, produces Freight for each new version.

Use Case: Rolling out container image updates across environments.

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: image-warehouse
  namespace: my-project
spec:
  subscriptions:
  - image:
      repoURL: ghcr.io/example/app
      imageSelectionStrategy: SemVer
      constraint: ^1.0.0
---
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: test
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: image-warehouse
    sources:
      direct: true
  promotionTemplate:
    spec:
      steps:
      - uses: git-clone
        config:
          repoURL: https://github.com/example/repo.git
          checkout:
          - branch: main
            path: ./out
      - uses: kustomize-set-image
        config:
          path: ./out/overlays/test
          images:
          - image: ghcr.io/example/app
            tag: ${{ imageFrom('ghcr.io/example/app').Tag }}
      - uses: git-commit
        config:
          path: ./out
          message: "Update image to ${{ imageFrom('ghcr.io/example/app').Tag }}"
      - uses: git-push
        config:
          path: ./out
          targetBranch: stage/test
      - uses: argocd-update
        config:
          apps:
          - name: app-test

2. Config Updater Pattern

Warehouse tracks Git commits, stages combine base config with overlays.

Use Case: Rolling out configuration changes across environments.

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: config-warehouse
spec:
  subscriptions:
  - git:
      repoURL: https://github.com/example/config.git
      branch: main
      commitSelectionStrategy: NewestFromBranch
      includePaths:
      - base/
      excludePaths:
      - "*.md"

Critical: Avoid feedback loops by writing to different branches or excluding output paths.

3. Common Case Pattern (Image + Config)

Single Warehouse subscribes to both image AND Git repositories.

Use Case: Promoting both application code and configuration together.

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: combined-warehouse
spec:
  subscriptions:
  - image:
      repoURL: ghcr.io/example/app
      imageSelectionStrategy: SemVer
      constraint: ^1.0.0
  - git:
      repoURL: https://github.com/example/config.git
      branch: main
      includePaths:
      - config/

Both artifacts referenced by single Freight, promoted together as a unit.

4. Multiple Warehouses Pattern

Separate Warehouses for images and configs with independent promotion cadences.

Use Case: When image updates occur frequently but configuration changes are rare.

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: image-warehouse
spec:
  subscriptions:
  - image:
      repoURL: ghcr.io/example/app
---
apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: config-warehouse
spec:
  subscriptions:
  - git:
      repoURL: https://github.com/example/config.git
---
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: test
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: image-warehouse
    sources:
      direct: true
  - origin:
      kind: Warehouse
      name: config-warehouse
    sources:
      direct: true

5. Grouped Services Pattern

Single Warehouse subscribes to multiple repositories for coordinated promotion.

Use Case: Microservices that must be deployed together.

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: microservices-warehouse
spec:
  subscriptions:
  - image:
      repoURL: ghcr.io/example/frontend
      imageSelectionStrategy: SemVer
  - image:
      repoURL: ghcr.io/example/backend
      imageSelectionStrategy: SemVer
  - image:
      repoURL: ghcr.io/example/api-gateway
      imageSelectionStrategy: SemVer
  freightCreationPolicy: Automatic
  freightCreationCriteria:
    expression: |
      imageFrom('ghcr.io/example/frontend').Tag ==
      imageFrom('ghcr.io/example/backend').Tag

Warning: Avoid over-coupling unrelated repositories.

6. Ordered Services Pattern

Deploy services in a specific order.

Option A: ArgoCD Sync Waves

# In manifests
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"  # Database first
---
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"  # Backend second
---
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "3"  # Frontend last

Option B: Sequential Steps

promotionTemplate:
  spec:
    steps:
    - uses: argocd-update
      as: deploy-db
      config:
        apps:
        - name: database
    - uses: argocd-update
      as: deploy-backend
      config:
        apps:
        - name: backend
    - uses: argocd-update
      config:
        apps:
        - name: frontend

7. Control Flow Stages Pattern

Stages without promotion processes for organization and interaction points.

Use Case: De-cluttering pipelines, providing approval gates.

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: approval-gate
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: my-warehouse
    sources:
      stages:
      - uat
  # No promotionTemplate - manual approval only

8. Fanning Out/In Pattern

Non-linear pipelines with branching and convergence.

Use Case: A/B testing, parallel validation environments.

# Test stage feeds two parallel stages
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: test
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: my-warehouse
    sources:
      direct: true
---
# Parallel stage A
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: variant-a
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: my-warehouse
    sources:
      stages:
      - test
---
# Parallel stage B
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: variant-b
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: my-warehouse
    sources:
      stages:
      - test
---
# Convergence stage
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: production
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: my-warehouse
    sources:
      stages:
      - variant-a
      - variant-b
    availabilityStrategy: All  # Require both variants verified

9. PR Workflow Pattern

Use pull requests for production promotions.

apiVersion: kargo.akuity.io/v1alpha1
kind: PromotionTask
metadata:
  name: pr-promotion
spec:
  vars:
  - name: repo
    value: https://github.com/example/repo.git
  steps:
  - uses: git-clone
    config:
      repoURL: ${{ vars.repo }}
      checkout:
      - branch: main
        path: ./out

  - uses: kustomize-set-image
    config:
      path: ./out
      images:
      - image: ghcr.io/example/app
        tag: ${{ imageFrom('ghcr.io/example/app').Tag }}

  - uses: git-commit
    config:
      path: ./out
      message: "Promote to production: ${{ ctx.freight.displayID }}"

  - uses: git-push
    as: push
    config:
      path: ./out
      generateTargetBranch: true
      provider: github

  - uses: git-open-pr
    as: open-pr
    config:
      repoURL: ${{ vars.repo }}
      sourceBranch: ${{ outputs.push.branch }}
      targetBranch: main
      title: "Promote to production"
      labels:
      - kargo
      - production

  - uses: git-wait-for-pr
    config:
      repoURL: ${{ vars.repo }}
      prNumber: ${{ outputs['open-pr'].pr.id }}
      timeout: 48h

10. Gatekeeper Stage Pattern

A stage where failures block invalid combinations.

Use Case: Automatic filtering of incompatible artifact combinations.

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: gatekeeper
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: my-warehouse
    sources:
      direct: true
  promotionTemplate:
    spec:
      steps:
      - uses: http
        config:
          url: https://api.example.com/validate
          method: POST
          body: '{"version": "${{ imageFrom(vars.repo).Tag }}"}'
          successExpression: response.body.valid == true
  verification:
    analysisTemplates:
    - name: compatibility-check

Auto-promotion enabled; failures block incompatible Freight from progressing.

11. Rendered Configs Pattern

Use helm template or kustomize build to generate plain YAML.

Benefits:

• Improved GitOps agent performance
• Complete visibility of changes in PR reviews
• Simplified debugging

promotionTemplate:
  spec:
    steps:
    - uses: git-clone
      config:
        repoURL: ${{ vars.repo }}
        checkout:
        - branch: main
          path: ./src

    - uses: helm-template
      config:
        path: ./src/charts/app
        outPath: ./src/rendered/manifests.yaml
        releaseName: my-app
        namespace: production
        valuesFiles:
        - ./src/values-prod.yaml
        setValues:
        - key: image.tag
          value: ${{ imageFrom(vars.imageRepo).Tag }}

    - uses: git-commit
      config:
        path: ./src
        message: "Render manifests for ${{ ctx.stage }}"

    - uses: git-push
      config:
        path: ./src
        targetBranch: rendered/${{ ctx.stage }}

Repository Layout Patterns

Helm Chart Layout

.
├── Chart.yaml
├── values.yaml
├── templates/
│   ├── deployment.yaml
│   └── service.yaml
└── stages/
    ├── test/values.yaml
    ├── uat/values.yaml
    └── prod/values.yaml

Kustomize Layout

.
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   └── service.yaml
└── stages/
    ├── test/kustomization.yaml
    ├── uat/kustomization.yaml
    └── prod/kustomization.yaml

Monorepo Layout

.
├── guestbook/
│   ├── base/
│   └── stages/
└── portal/
    ├── base/
    └── stages/

Important: Configure Warehouse path filters for monorepos.

Storage Options

Option 1: Stage-Specific Branches (Recommended)

Treat branches as storage, not merge targets.

- uses: git-push
  config:
    targetBranch: stage/${{ ctx.stage }}

Option 2: Writing to Main Branch

Separate input and output paths to avoid feedback loops.

# Warehouse excludes output
spec:
  subscriptions:
  - git:
      includePaths:
      - src/
      excludePaths:
      - builds/

Option 3: Separate Repository

Write output to different repository entirely.

- uses: git-clone
  config:
    repoURL: https://github.com/example/output-repo.git
    checkout:
    - branch: main
      path: ./output

Auto-Promotion Configuration

Project-Level Policy

apiVersion: kargo.akuity.io/v1alpha1
kind: ProjectConfig
metadata:
  name: my-project
spec:
  promotionPolicies:
  - stages:
    - test
    - uat
    autoPromotionEnabled: true
  - stages:
    - prod
    autoPromotionEnabled: false

Pattern-Based Matching

promotionPolicies:
- stages:
  - regex:^dev-.*
  autoPromotionEnabled: true
- stages:
  - glob:prod-*
  autoPromotionEnabled: false

Label Selectors

promotionPolicies:
- stageSelector:
    matchLabels:
      tier: development
  autoPromotionEnabled: true

Best Practices

1. Start Simple: Begin with Image Updater pattern, add complexity as needed 2. Avoid Feedback Loops: Use separate branches/paths for input and output 3. Use Path Filters: Essential for monorepos to prevent unnecessary Freight 4. Enable Auto-Promotion Carefully: Auto-promote to dev/test, manual for prod 5. Implement Verification: Add AnalysisTemplates for critical stages 6. Use Soak Times: Require minimum duration in pre-prod stages 7. Document Patterns: Maintain clear documentation of your pipeline topology

Kargo Promotion Steps Reference

Complete reference for all 34 Kargo promotion steps.

Git Operations

git-clone

Clones a remote Git repository and checks out specified revisions.

- uses: git-clone
  as: clone
  config:
    repoURL: https://github.com/example/repo.git
    insecureSkipTLSVerify: false
    author:
      - name: "Kargo Bot"
        email: "kargo@example.com"
        signingKey: "..."  # Optional GPG key
    checkout:
      - as: freight
        commit: ${{ commitFrom(vars.repo).ID }}
        path: /workspace/source
      - as: stage-config
        branch: ${{ vars.targetBranch }}
        path: /workspace/target
        create: true  # Create orphaned branch if missing

Parameters:

Output: commits map containing checkout keys mapped to HEAD commit hashes.

git-commit

Commits working tree changes to the checked out branch.

- uses: git-commit
  as: commit
  config:
    path: ./out
    message: |
      Update image to ${{ imageFrom(vars.imageRepo).Tag }}

      Freight: ${{ ctx.freight.displayID }}
    author:
      name: "Kargo Automation"
      email: "kargo@example.com"
      signingKey: "..."

Parameters:

Output: commit - SHA of created commit (or existing HEAD if no changes).

git-push

Pushes committed changes to remote repository.

- uses: git-push
  as: push
  config:
    path: ./out
    targetBranch: ${{ vars.targetBranch }}
    maxAttempts: 50
    # OR for PR workflow:
    generateTargetBranch: true
    provider: github  # azure, bitbucket, gitea, github, gitlab

Parameters:

Output:

• branch - Remote branch name
• commit - Commit SHA pushed
• commitURL - URL to pushed commit

git-open-pr

Opens a pull request.

- uses: git-open-pr
  as: open-pr
  config:
    repoURL: https://github.com/example/repo.git
    provider: github
    sourceBranch: ${{ outputs['push'].branch }}
    targetBranch: main
    createTargetBranch: false
    title: "Promote to ${{ ctx.stage }}"
    description: "Auto-generated promotion PR"
    labels:
      - kargo
      - automated

Output:

• pr.id - PR number
• pr.url - PR URL

git-wait-for-pr

Waits for a PR to be merged or closed.

- uses: git-wait-for-pr
  config:
    repoURL: https://github.com/example/repo.git
    prNumber: ${{ outputs['open-pr'].pr.id }}
    provider: github
    timeout: 48h

Output: commit - Merge commit SHA

git-merge-pr

Merges an open pull request.

- uses: git-merge-pr
  config:
    repoURL: https://github.com/example/repo.git
    prNumber: ${{ outputs['open-pr'].pr.id }}
    provider: github
    wait: true  # Retry if not mergeable

git-clear

Deletes entire contents of Git working tree.

- uses: git-clear
  config:
    path: ./out

Configuration Management

kustomize-set-image

Updates container image references in kustomization.yaml.

- uses: kustomize-set-image
  config:
    path: ./out/overlays/test
    images:
    - image: ghcr.io/example/app
      tag: ${{ imageFrom(vars.imageRepo).Tag }}
    - image: ghcr.io/example/other
      newName: registry.example.com/other
      digest: ${{ imageFrom('ghcr.io/example/other').Digest }}

Parameters:

Output: commitMessage - Description of changes

kustomize-build

Renders manifests from kustomization.yaml.

- uses: kustomize-build
  config:
    path: ./src/overlays/test
    outPath: ./out/manifests.yaml
    plugin.helm.kubeVersion: "1.28.0"
    plugin.helm.apiVersions:
      - apps/v1

helm-template

Renders Helm charts.

- uses: helm-template
  config:
    path: ./charts/my-chart
    outPath: ./out/manifests.yaml
    releaseName: my-release
    namespace: default
    outLayout: helm  # or 'flat'
    valuesFiles:
      - ./values-prod.yaml
    buildDependencies: true
    includeCRDs: true
    disableHooks: false
    skipTests: false
    kubeVersion: "1.28.0"
    apiVersions:
      - apps/v1
    setValues:
      - key: image.tag
        value: ${{ imageFrom(vars.imageRepo).Tag }}

helm-update-chart

Updates Helm chart dependencies in Chart.yaml.

- uses: helm-update-chart
  config:
    path: ./src/my-chart
    charts:
      - repository: https://charts.example.com
        name: dependency-chart
        version: 2.0.0
      - repository: oci://registry.example.com
        name: oci-dependency
        version: 1.5.0

Output: commitMessage

yaml-update

Updates YAML file values.

- uses: yaml-update
  config:
    path: ./src/values.yaml
    updates:
      - key: image.tag
        value: ${{ imageFrom(vars.imageRepo).Tag }}
      - key: replicas
        value: "3"
      - key: config.nested.value
        value: updated-value

Output: commitMessage

yaml-parse

Extracts values from YAML files.

- uses: yaml-parse
  as: get-version
  config:
    filePath: ./src/source.yaml
    expression: version

Output: value - Extracted value

json-update

Updates JSON file values.

- uses: json-update
  config:
    path: ./src/config.json
    updates:
      - key: version
        value: ${{ imageFrom(vars.imageRepo).Tag }}
      - key: enabled
        value: true

json-parse

Extracts values from JSON files.

- uses: json-parse
  as: get-config
  config:
    filePath: ./src/config.json
    expression: settings.version

ArgoCD Integration

argocd-update

Updates ArgoCD Application resources.

- uses: argocd-update
  config:
    apps:
    - name: my-app
      namespace: argocd
      sources:
      - repoURL: https://github.com/example/repo.git
        desiredRevision: ${{ outputs.push.commit }}
        updateTargetRevision: true
      - repoURL: ghcr.io/example/app
        tag: v1.2.3
        newName: registry.example.com/app
        updateTargetRevision: true
      # Helm parameters
      - repoURL: https://github.com/example/repo.git
        helm:
          parameters:
          - key: image.tag
            value: v1.2.3
        updateTargetRevision: true

Kustomize Images:

sources:
- repoURL: ghcr.io/example/app
  tag: v1.2.3
  newName: registry.example.com/app

Helm Parameters:

sources:
- repoURL: https://github.com/example/repo.git
  helm:
    parameters:
    - key: image.tag
      value: v1.2.3

Required Application Annotation:

annotations:
  kargo.akuity.io/authorized-stage: "project:stage"

File Operations

copy

Copies files or directories.

- uses: copy
  config:
    inPath: ./overlay/kustomization.yaml
    outPath: ./src/kustomization.yaml

delete

Removes files or directories.

- uses: delete
  config:
    path: ./temp

untar

Extracts tar/gzipped archives.

- uses: untar
  config:
    inPath: ./archive.tar.gz
    outPath: ./extracted

External Integrations

http

Makes HTTP/S requests.

- uses: http
  config:
    method: POST
    url: https://api.example.com/deploy
    headers:
      - name: Authorization
        value: "Bearer ${{ secrets.apiToken }}"
      - name: Content-Type
        value: application/json
    queryParams:
      - name: stage
        value: ${{ ctx.stage }}
    body: '{"version":"${{ ctx.freight.displayID }}"}'
    timeout: 5m
    successExpression: response.status >= 200 && response.status < 300
    failureExpression: response.status >= 500

Response Object (in expressions):

• response.status - HTTP status code
• response.headers - Header map
• response.header("name") - Header accessor
• response.body - Unmarshaled JSON

http-download

Downloads files from HTTP/S URLs.

- uses: http-download
  config:
    url: https://example.com/file.tar.gz
    outPath: ./downloads/file.tar.gz
    headers:
      - name: Authorization
        value: "Bearer ${{ secrets.token }}"

oci-download

Downloads OCI artifacts.

- uses: oci-download
  config:
    repoURL: ghcr.io/example/artifact
    tag: latest
    outPath: ./artifacts

gha-dispatch-workflow

Dispatches GitHub Actions workflows.

- uses: gha-dispatch-workflow
  as: dispatch
  config:
    repoURL: https://github.com/example/repo
    workflowFileName: deploy.yaml
    ref: main
    inputs:
      environment: production
      version: ${{ imageFrom(vars.imageRepo).Tag }}

gha-wait-for-workflow

Waits for GitHub Actions workflow completion.

- uses: gha-wait-for-workflow
  config:
    repoURL: https://github.com/example/repo
    runID: ${{ outputs['dispatch'].runID }}
    timeout: 30m

jira

Manages Jira issues.

- uses: jira
  config:
    url: https://example.atlassian.net
    projectKey: DEPLOY
    issueType: Task
    summary: "Deployment to ${{ ctx.stage }}"
    description: "Deploying ${{ ctx.freight.displayID }}"

send-message

Sends notifications to Slack, email, etc.

- uses: send-message
  config:
    channel: slack-notifications
    message: |
      Deployed ${{ ctx.freight.displayID }} to ${{ ctx.stage }}

Workflow Utilities

compose-output

Combines outputs from multiple steps.

- uses: compose-output
  config:
    pr_url: "${{ vars.repoURL }}/pull/${{ outputs['open-pr'].pr.id }}"
    commit_sha: "${{ outputs['commit'].commit }}"
    summary: "Deployed ${{ ctx.freight.displayID }}"

set-metadata

Updates Stage or Freight resource metadata.

- uses: set-metadata
  config:
    stage:
      labels:
        last-promoted: ${{ ctx.freight.displayID }}
    freight:
      annotations:
        deployed-at: ${{ now() }}

Conditional Execution & Error Handling

Conditional Steps

steps:
  - uses: some-step
    as: my-step

  - uses: another-step
    if: ${{ success() }}  # Only if all previous succeeded

  - uses: error-handler
    if: ${{ failure() }}  # Only if any previous failed

  - uses: cleanup
    if: ${{ always() }}   # Always execute

  - uses: conditional
    if: ${{ status('my-step') == 'Errored' }}

Error Handling

steps:
  - uses: git-wait-for-pr
    continueOnError: true
    retry:
      errorThreshold: 3
      timeout: 48h

Step Aliasing & Output References

steps:
  - uses: git-clone
    as: clone
    config:
      repoURL: ${{ vars.repo }}

  - uses: git-commit
    as: commit
    config:
      path: ./out
      message: "Update"

  - uses: git-push
    as: push
    config:
      path: ./out

  - uses: argocd-update
    config:
      apps:
      - name: my-app
        sources:
        - repoURL: ${{ vars.repo }}
          desiredRevision: ${{ outputs.push.commit }}

Kargo Security Reference

Comprehensive guide to Kargo security configuration, access controls, and credential management.

Authentication Overview

Kargo implements a two-tier security model:

1. IDP Authentication: External identity providers via OpenID Connect (OIDC) 2. Kubernetes RBAC Authorization: Users mapped to ServiceAccounts for access control

OpenID Connect (OIDC) Configuration

Basic OIDC Setup

# Helm values
api:
  oidc:
    enabled: true
    issuerURL: https://idp.example.com
    clientID: kargo-ui
    cliClientID: kargo-cli  # Optional, if different client
    additionalScopes:
      - groups

IDP Callback URLs

For OIDC + PKCE compatible IDPs:

https://<api-server>/login           # UI
http://localhost/auth/callback       # CLI

For IDPs requiring Dex:

https://<api-server>/dex/callback    # Both UI and CLI

Dex Integration (for incompatible IDPs)

api:
  oidc:
    enabled: true
    dex:
      enabled: true
      connectors:
        - type: github
          id: github
          name: GitHub
          config:
            clientID: <github-client-id>
            clientSecret: $CLIENT_SECRET
            redirectURI: https://<api-server>/dex/callback
            orgs:
              - name: my-org
                teams:
                  - devops
      env:
        - name: CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: github-dex
              key: clientSecret

System Role Configuration

api:
  oidc:
    admins:
      claims:
        email:
          - alice@example.com
          - bob@example.com
        groups:
          - kargo-admins
    projectCreators:
      claims:
        groups:
          - leads
    viewers:
      claims:
        groups:
          - devops
    users:
      claims:
        groups:
          - developers

System Roles:

User-to-ServiceAccount Mapping

Annotation-Based Mapping

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
  namespace: my-project
  annotations:
    rbac.kargo.akuity.io/claims: |
      {
        "sub": ["alice", "bob"],
        "email": "carl@example.com",
        "groups": ["devops", "kargo-admin"]
      }

Alternative Format (Backward Compatible)

annotations:
  rbac.kargo.akuity.io/claim.sub: alice,bob
  rbac.kargo.akuity.io/claim.email: carl@example.com
  rbac.kargo.akuity.io/claim.groups: devops,kargo-admin

Multi-ServiceAccount Binding

Users mapped to multiple ServiceAccounts get the union of all permissions:

# Developer ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: developer
  annotations:
    rbac.kargo.akuity.io/claims: '{"groups":["developers"]}'
---
# Promoter ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: promoter
  annotations:
    rbac.kargo.akuity.io/claims: '{"groups":["devops"]}'

Project-Level RBAC

Pre-defined Roles

Custom Role with Permissions

apiVersion: v1
kind: ServiceAccount
metadata:
  name: promoter
  namespace: my-project
  annotations:
    rbac.kargo.akuity.io/claims: '{"groups":["devops"]}'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: promoter
  namespace: my-project
rules:
- apiGroups: [kargo.akuity.io]
  resources: [promotions]
  verbs: [create, patch, update]
- apiGroups: [kargo.akuity.io]
  resources: [stages]
  verbs: [promote]  # Custom Kargo verb
  resourceNames: [dev, staging]  # Specific stages only
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: promoter
  namespace: my-project
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: promoter
subjects:
- kind: ServiceAccount
  name: promoter
  namespace: my-project

Custom Kargo RBAC Verbs

Kargo extends standard Kubernetes verbs:

Cross-Namespace RoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: global-developer
  namespace: my-project
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kargo-admin
subjects:
- kind: ServiceAccount
  name: team-x-developers
  namespace: kargo-global-service-accounts

Global ServiceAccount Namespaces

api:
  oidc:
    globalServiceAccounts:
      namespaces:
        - kargo-global-service-accounts

CLI Role Management

# List roles
kargo get roles --project my-project

# Create role
kargo create role developer --project my-project

# Grant OIDC claims
kargo grant --role developer \
  --claim groups=developer \
  --project my-project

# Grant resource permissions
kargo grant --role developer \
  --verb '*' --resource-type stages \
  --project my-project

# View as Kubernetes resources
kargo get role developer --as-kubernetes-resources --project my-project

# Delete role
kargo delete role developer --project my-project

Credential Management

Credential Secret Format

apiVersion: v1
kind: Secret
metadata:
  name: my-credentials
  namespace: my-project
  labels:
    kargo.akuity.io/cred-type: git  # git, helm, image, generic
type: Opaque
stringData:
  repoURL: https://github.com/example/repo.git
  username: my-username
  password: my-token

Git Credentials

HTTPS with Token:

stringData:
  repoURL: https://github.com/example/repo.git
  username: my-username
  password: my-personal-access-token

SSH Key:

stringData:
  repoURL: git@github.com:example/repo.git
  sshPrivateKey: <base64-encoded-ssh-key>

GitHub App:

stringData:
  repoURL: https://github.com/example/repo.git
  githubAppClientID: "1234567"
  githubAppPrivateKey: <base64-encoded-app-key>
  githubAppInstallationID: "98765432"

Container Registry Credentials

Basic Auth:

metadata:
  labels:
    kargo.akuity.io/cred-type: image
stringData:
  repoURL: registry.example.com
  username: my-user
  password: my-password

AWS ECR (Long-lived):

stringData:
  repoURL: 123456789.dkr.ecr.us-west-2.amazonaws.com
  awsRegion: us-west-2
  awsAccessKeyID: AKIA...
  awsSecretAccessKey: ...

AWS ECR via IRSA (Operator Setup):

# Helm values
controller:
  serviceAccount:
    iamRole: arn:aws:iam::ACCOUNT_ID:role/kargo-controller-role

Token cached for 10 hours.

Google Artifact Registry:

stringData:
  repoURL: us-central1-docker.pkg.dev/my-project/my-repo
  gcpServiceAccountKey: |
    {
      "type": "service_account",
      "project_id": "my-project",
      ...
    }

Token cached for 40 minutes.

Azure ACR:

stringData:
  repoURL: myregistry.azurecr.io
  username: <repository-scoped-token-username>
  password: <repository-scoped-token-password>

Helm Repository Credentials

metadata:
  labels:
    kargo.akuity.io/cred-type: helm
stringData:
  repoURL: https://charts.example.com
  username: my-user
  password: my-token

Regex Pattern Matching

Match multiple repositories with one credential:

stringData:
  repoURL: '^https://github\.com/myorg/.*\.git$'
  repoURLIsRegex: 'true'
  username: my-username
  password: my-pat

CLI Credential Management

# Create
kargo create credentials \
  --project my-project my-creds \
  --git \
  --repo-url https://github.com/example/repo.git \
  --username my-username \
  --password my-pat

# List
kargo get credentials --project my-project

# View
kargo get credentials --project my-project my-creds

# Update
kargo update credentials \
  --project my-project my-creds \
  --password new-token

# Update with regex
kargo update credentials \
  --project my-project my-creds \
  --repo-url '^https://github.com/' \
  --regex

# Delete
kargo delete credentials --project my-project my-creds

Credential Precedence

1. Exact repoURL matches in project namespace (lexical order) 2. Regex pattern matches in project namespace (lexical order) 3. Global credentials from operator-designated namespaces

Global Credentials (Operator Setup)

# Helm values
controller:
  globalCredentials:
    namespaces:
      - kargo-global-creds

Required RBAC:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kargo-controller-read-secrets
  namespace: kargo-global-creds
subjects:
- kind: ServiceAccount
  name: kargo-controller
  namespace: kargo
roleRef:
  kind: ClusterRole
  name: kargo-controller-read-secrets
  apiGroup: rbac.authorization.k8s.io

Secure Configuration (Production)

Disable Admin Account

api:
  adminAccount:
    enabled: false  # Requires SSO

TLS Configuration

# Production - use proper certificates
api:
  tls:
    selfSignedCert: false
# Create Secret: kargo-api-cert (TLS type)

Secret Access Controls

# Disable API server secret management
api:
  secretManagementEnabled: false

# Restrict controller secret reading
controller:
  serviceAccount:
    clusterWideSecretReadingEnabled: false  # Default

ArgoCD Authorization

Applications must explicitly authorize Kargo:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    kargo.akuity.io/authorized-stage: "my-project:my-stage"

Multiple stages:

annotations:
  kargo.akuity.io/authorized-stage: "my-project:test,my-project:uat,my-project:prod"

Security Best Practices

1. Use OIDC: Always integrate with external identity providers 2. Principle of Least Privilege: Grant minimal required permissions 3. Group-Based Access: Use IDP groups for scalable access control 4. Separate Global and Project Roles: Use global namespaces for infrastructure ServiceAccounts 5. Credential Rotation: Regularly rotate repository credentials and API keys 6. Regex Patterns: Use patterns for broad credential coverage 7. Ambient Credentials: Prefer IRSA/Workload Identity over long-lived keys 8. Monitor RBAC: Regularly audit ServiceAccount mappings 9. Disable Admin in Production: Force SSO authentication 10. Use TLS: Never skip TLS in production environments 11. GitOps for Credentials: Use Sealed Secrets or External Secrets Operator