Deployment Strategy

Deployment Strategy

Overview

Covers deployment strategy selection, rollback safety, and progressive delivery patterns. Focuses on zero-downtime release techniques, blast radius management, and environment promotion workflows from development through production.

When to use: Choosing between blue-green, canary, or rolling deployments, implementing rollback procedures, configuring health checks and readiness gates, planning environment promotion workflows, integrating feature flags for progressive delivery.

When NOT to use: CI/CD pipeline mechanics and GitHub Actions workflows (use ci-cd-architecture skill), infrastructure provisioning and cloud platform selection (use ci-cd-architecture skill), application architecture decisions (use framework-specific skills).

Quick Reference

Common Mistakes

Delegation

• Audit deployment safety of existing infrastructure: Use Explore agent to review deployment configs, health check definitions, and rollback procedures
• Implement a specific deployment strategy: Use Task agent to configure blue-green, canary, or rolling deployment for a target platform
• Plan migration from one deployment strategy to another: Use Plan agent to evaluate current strategy, define migration steps, and identify risks

If the ci-cd-architecture skill is available, delegate CI/CD pipeline setup and GitHub Actions workflow configuration to it.

Otherwise, recommend: npx skills add oakoss/agent-skills --skill ci-cd-architecture

References

• Blue-green, canary, rolling, and recreate deployment strategies with configuration examples
• Rollback procedures, health checks, feature flags, and blast radius management
• Environment promotion workflows, configuration management, and secrets handling

Deployment Patterns

Strategy Comparison

Blue-Green Deployment

Two identical environments run in parallel. Traffic switches atomically from the current (blue) to the new (green) environment. The old environment stays available for instant rollback.

When to Use

• Applications requiring instant rollback capability
• Stateless services where environment duplication is straightforward
• Releases that need full validation before any user traffic

Kubernetes Blue-Green with Service Selector

apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  selector:
    app: my-app
    version: green
  ports:
    - port: 80
      targetPort: 8080

Deploy the green version alongside blue, validate it, then update the Service selector from version: blue to version: green. Roll back by reverting the selector.

Kubernetes Blue-Green Deployment Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-green
  labels:
    app: my-app
    version: green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      version: green
  template:
    metadata:
      labels:
        app: my-app
        version: green
    spec:
      containers:
        - name: my-app
          image: my-app:2.0.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 20

Blue-Green Traffic Switch Script

#!/usr/bin/env bash
set -euo pipefail

NEW_VERSION="${1:?Usage: switch-traffic.sh <green|blue>}"
SERVICE_NAME="my-app"
NAMESPACE="production"

kubectl patch service "$SERVICE_NAME" \
  -n "$NAMESPACE" \
  -p "{\"spec\":{\"selector\":{\"version\":\"$NEW_VERSION\"}}}"

echo "Traffic switched to $NEW_VERSION"

kubectl rollout status deployment/"$SERVICE_NAME-$NEW_VERSION" \
  -n "$NAMESPACE" --timeout=120s

AWS ALB Blue-Green with Target Groups

Resources:
  BlueTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: my-app-blue
      Port: 8080
      Protocol: HTTP
      VpcId: !Ref VpcId
      HealthCheckPath: /healthz
      HealthCheckIntervalSeconds: 10
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 3

  GreenTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: my-app-green
      Port: 8080
      Protocol: HTTP
      VpcId: !Ref VpcId
      HealthCheckPath: /healthz
      HealthCheckIntervalSeconds: 10
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 3

  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref ALB
      Port: 443
      Protocol: HTTPS
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref BlueTargetGroup

Switch traffic by updating the listener's default action to point to GreenTargetGroup.

Canary Deployment

Routes a small percentage of traffic to the new version while the majority continues hitting the stable version. Monitors error rates, latency, and business metrics before gradually increasing the canary percentage.

When to Use

• High-traffic services where full rollout risk is unacceptable
• Releases requiring real-world validation under production load
• Services with thorough monitoring and alerting

Canary Progression Example

Phase 1:  5% canary, 95% stable  (10 min observation)
Phase 2: 25% canary, 75% stable  (15 min observation)
Phase 3: 50% canary, 50% stable  (15 min observation)
Phase 4: 100% canary              (full rollout)

Halt and roll back at any phase if error rate exceeds threshold.

Argo Rollouts Canary Configuration

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  replicas: 5
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - setWeight: 25
        - pause: { duration: 15m }
        - setWeight: 50
        - pause: { duration: 15m }
        - setWeight: 100
      canaryService: my-app-canary
      stableService: my-app-stable
      trafficRouting:
        istio:
          virtualService:
            name: my-app-vsvc
            routes:
              - primary
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-app:2.0.0
          ports:
            - containerPort: 8080

Istio Virtual Service for Traffic Splitting

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app-vsvc
spec:
  hosts:
    - my-app.example.com
  http:
    - route:
        - destination:
            host: my-app-stable
          weight: 95
        - destination:
            host: my-app-canary
          weight: 5

Nginx Ingress Canary Annotations

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: 'true'
    nginx.ingress.kubernetes.io/canary-weight: '5'
spec:
  rules:
    - host: my-app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app-canary
                port:
                  number: 80

Rolling Deployment

Incrementally replaces old instances with new ones. The default strategy in Kubernetes. Controls the pace of updates using maxSurge and maxUnavailable parameters.

When to Use

• Standard application updates without special rollback requirements
• Services that can tolerate running mixed versions briefly
• Default choice when blue-green or canary complexity is not justified

Kubernetes Rolling Update Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-app:2.0.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10

Rolling Update Parameter Guide

• maxSurge: 1, maxUnavailable: 0 ensures full capacity at all times (safest, slowest)
• maxSurge: 25%, maxUnavailable: 25% balances speed and safety (Kubernetes default)

Rollback a Rolling Update

kubectl rollout undo deployment/my-app

kubectl rollout undo deployment/my-app --to-revision=3

kubectl rollout history deployment/my-app

Recreate Deployment

Terminates all existing pods before creating new ones. Accepts brief downtime in exchange for simplicity and avoiding mixed-version states.

When to Use

• Development and staging environments where downtime is acceptable
• Applications that cannot run two versions simultaneously (incompatible schemas, singleton resources)
• Batch processing workloads with no user-facing traffic

Kubernetes Recreate Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-app:2.0.0

Strategy Decision Framework

Need instant rollback?
  YES -> Blue-green
  NO  -> Continue

Can you monitor canary metrics?
  YES -> High-traffic service?
    YES -> Canary
    NO  -> Rolling
  NO  -> Continue

Can you tolerate brief downtime?
  YES -> Recreate
  NO  -> Rolling (with maxUnavailable: 0)

Environment Management

Environment Promotion Model

A promotion model defines how changes flow from development through to production. Each stage acts as a quality gate that catches issues before they reach users.

Three-Environment Model

Development -> Staging -> Production
     |             |            |
  Feature      Pre-prod      Live
  testing      validation    users

Promotion Gates Between Environments

Each promotion requires passing automated and manual gates before proceeding.

Dev -> Staging gates:
  - All unit tests pass
  - Linting and type checks pass
  - Container image builds successfully

Staging -> Production gates:
  - Integration tests pass against staging
  - Performance benchmarks within thresholds
  - Security scan passes (no critical vulnerabilities)
  - Manual approval from on-call engineer (production only)
  - Database migration tested against staging data

GitHub Actions Promotion Workflow

name: Promote to Production

on:
  workflow_dispatch:
    inputs:
      staging-sha:
        description: 'Staging deployment SHA to promote'
        required: true

jobs:
  verify-staging:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
        with:
          ref: ${{ inputs.staging-sha }}

      - name: Run integration tests against staging
        run: |
          npm ci
          npm run test:integration -- --target=staging

      - name: Verify staging health
        run: |
          curl -sf https://staging.example.com/healthz || exit 1

  promote:
    needs: verify-staging
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Deploy to production
        run: |
          kubectl set image deployment/my-app \
            my-app=my-registry/my-app:${{ inputs.staging-sha }} \
            -n production

      - name: Wait for rollout
        run: |
          kubectl rollout status deployment/my-app \
            -n production --timeout=300s

      - name: Verify production health
        run: |
          sleep 30
          curl -sf https://app.example.com/healthz || exit 1

GitOps Promotion with Argo CD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-production
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/deployment-manifests
    targetRevision: main
    path: environments/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

In GitOps, promotion happens by updating the image tag in the environment-specific directory of the deployment manifests repository. A pull request to change environments/production/kustomization.yaml triggers review and automated sync.

Kustomize Environment Overlays

deployment-manifests/
  base/
    deployment.yaml
    service.yaml
    kustomization.yaml
  environments/
    development/
      kustomization.yaml
    staging/
      kustomization.yaml
    production/
      kustomization.yaml

# environments/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: my-app
    newName: my-registry/my-app
    newTag: 'abc123'
replicas:
  - name: my-app
    count: 5
patches:
  - target:
      kind: Deployment
      name: my-app
    patch: |
      - op: replace
        path: /spec/template/spec/containers/0/resources/requests/memory
        value: "512Mi"
      - op: replace
        path: /spec/template/spec/containers/0/resources/limits/memory
        value: "1Gi"

Environment Parity

Minimize differences between environments to catch issues early. Drift between staging and production is the most common cause of "works in staging, breaks in production" failures.

Parity Checklist

What Should Differ

Configuration Management

Environment Variable Patterns

Separate configuration into layers: base defaults, environment-specific overrides, and secrets.

# base-config.yaml (committed to repo)
app:
  name: my-app
  port: 8080
  log_format: json
  cors_origins: []
  rate_limit_rpm: 100

# environment overrides (committed to repo, no secrets)
# staging.yaml
app:
  cors_origins:
    - https://staging.example.com
  rate_limit_rpm: 1000
  log_level: info

# production.yaml
app:
  cors_origins:
    - https://app.example.com
  rate_limit_rpm: 10000
  log_level: warn

Twelve-Factor Config Approach

Configuration that changes between environments belongs in environment variables, not in code. Code should read config from environment and fail fast if required values are missing.

interface AppConfig {
  databaseUrl: string;
  redisUrl: string;
  apiKey: string;
  environment: 'development' | 'staging' | 'production';
  logLevel: string;
}

function loadConfig(): AppConfig {
  const required = ['DATABASE_URL', 'REDIS_URL', 'API_KEY'] as const;
  const missing = required.filter((key) => !process.env[key]);

  if (missing.length > 0) {
    throw new Error(
      `Missing required environment variables: ${missing.join(', ')}`,
    );
  }

  return {
    databaseUrl: process.env.DATABASE_URL!,
    redisUrl: process.env.REDIS_URL!,
    apiKey: process.env.API_KEY!,
    environment:
      (process.env.NODE_ENV as AppConfig['environment']) ?? 'development',
    logLevel: process.env.LOG_LEVEL ?? 'info',
  };
}

Secrets Management

Secrets must never be committed to source control. Use platform-provided secret stores with least-privilege access.

Secrets Management Options

Kubernetes External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-secrets
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: my-app-secrets
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: production/my-app/database-url
    - secretKey: API_KEY
      remoteRef:
        key: production/my-app/api-key

Secret Rotation Pattern

1. Generate new secret value
2. Update secret store (both old and new values valid)
3. Deploy application that uses new secret
4. Verify application works with new secret
5. Revoke old secret value
6. Audit: confirm no processes still use old secret

Never rotate secrets and deploy application changes simultaneously. Rotate secrets as a separate, isolated operation.

Deployment Concurrency

Prevent overlapping deployments that can cause unpredictable states.

GitHub Actions Concurrency Control

name: Deploy
on:
  push:
    branches: [main]

concurrency:
  group: deploy-production
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v6
      - name: Deploy
        run: ./deploy.sh

Setting cancel-in-progress: false ensures the current deployment completes before the next one starts, preventing partial deployments.

Kubernetes Resource Quotas for Deployment Safety

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

A PodDisruptionBudget ensures a minimum number of pods remain available during voluntary disruptions like deployments, node drains, or cluster upgrades.

Rollback and Safety

Rollback Procedures

Every production deployment must have a tested rollback plan before the deploy begins. The rollback method depends on the deployment strategy and platform.

Rollback by Platform

Kubernetes Rollback Commands

kubectl rollout undo deployment/my-app

kubectl rollout undo deployment/my-app --to-revision=3

kubectl rollout history deployment/my-app

kubectl rollout status deployment/my-app

Automated Rollback with Argo Rollouts

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: error-rate-check
            args:
              - name: service-name
                value: my-app
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-check
spec:
  metrics:
    - name: error-rate
      interval: 60s
      successCondition: result[0] < 0.05
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))

If the error rate exceeds 5% in three consecutive checks, the rollout aborts and traffic reverts to the stable version automatically.

AWS CodeDeploy Automatic Rollback

Resources:
  DeploymentGroup:
    Type: AWS::CodeDeploy::DeploymentGroup
    Properties:
      ApplicationName: !Ref Application
      DeploymentGroupName: production
      DeploymentStyle:
        DeploymentOption: WITH_TRAFFIC_CONTROL
        DeploymentType: BLUE_GREEN
      AutoRollbackConfiguration:
        Enabled: true
        Events:
          - DEPLOYMENT_FAILURE
          - DEPLOYMENT_STOP_ON_ALARM
      AlarmConfiguration:
        Alarms:
          - Name: HighErrorRate
          - Name: HighLatency
        Enabled: true

Health Checks

Health checks verify that application instances are functioning correctly. Configure three types of probes for thorough health monitoring.

Probe Types

Kubernetes Health Probe Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: my-app
          image: my-app:2.0.0
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 0
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 0
            periodSeconds: 10
            failureThreshold: 3

Health Check Endpoint Implementation

import { type Request, type Response } from 'express';

interface HealthStatus {
  status: 'healthy' | 'degraded' | 'unhealthy';
  checks: Record<string, { status: string; latencyMs?: number }>;
}

async function checkDatabase(pool: unknown): Promise<boolean> {
  try {
    await (pool as { query: (q: string) => Promise<unknown> }).query(
      'SELECT 1',
    );
    return true;
  } catch {
    return false;
  }
}

async function checkRedis(client: unknown): Promise<boolean> {
  try {
    const result = await (client as { ping: () => Promise<string> }).ping();
    return result === 'PONG';
  } catch {
    return false;
  }
}

export function healthzHandler(_req: Request, res: Response): void {
  res.status(200).json({ status: 'ok' });
}

export async function readinessHandler(
  _req: Request,
  res: Response,
  deps: { pool: unknown; redis: unknown },
): Promise<void> {
  const start = Date.now();
  const dbOk = await checkDatabase(deps.pool);
  const dbLatency = Date.now() - start;

  const redisStart = Date.now();
  const redisOk = await checkRedis(deps.redis);
  const redisLatency = Date.now() - redisStart;

  const health: HealthStatus = {
    status: dbOk && redisOk ? 'healthy' : 'unhealthy',
    checks: {
      database: { status: dbOk ? 'up' : 'down', latencyMs: dbLatency },
      redis: { status: redisOk ? 'up' : 'down', latencyMs: redisLatency },
    },
  };

  res.status(health.status === 'healthy' ? 200 : 503).json(health);
}

Feature Flags for Progressive Delivery

Feature flags decouple deployment from release. Code ships to production with new functionality disabled, then activates incrementally based on flag configuration.

Feature Flag Lifecycle

1. Create flag with owner and planned removal date
2. Implement behind flag (default: OFF)
3. Deploy to production (flag still OFF)
4. Enable for internal users / beta testers
5. Progressive rollout: 5% -> 25% -> 50% -> 100%
6. Monitor metrics at each stage
7. Remove flag and dead code after full rollout

Feature Flag Implementation

interface FeatureFlag {
  name: string;
  enabled: boolean;
  rolloutPercentage: number;
  allowlist: string[];
}

function isFeatureEnabled(flag: FeatureFlag, userId: string): boolean {
  if (!flag.enabled) return false;
  if (flag.allowlist.includes(userId)) return true;

  const hash = simpleHash(userId + flag.name);
  return hash % 100 < flag.rolloutPercentage;
}

function simpleHash(input: string): number {
  let hash = 0;
  for (let i = 0; i < input.length; i++) {
    const char = input.charCodeAt(i);
    hash = (hash << 5) - hash + char;
    hash |= 0;
  }
  return Math.abs(hash);
}

Feature Flag with Safe Defaults

async function getFlags(
  flagService: { getFlag: (name: string) => Promise<FeatureFlag> },
  flagName: string,
): Promise<FeatureFlag> {
  try {
    return await flagService.getFlag(flagName);
  } catch {
    return {
      name: flagName,
      enabled: false,
      rolloutPercentage: 0,
      allowlist: [],
    };
  }
}

When the flag service is unreachable, default to the safe state (feature disabled). This circuit breaker pattern prevents flag service outages from cascading into application failures.

Feature Flag Hygiene

Blast Radius Management

Blast radius is the scope of impact when a deployment fails. Reduce it by limiting who and what is affected by any single change.

Techniques to Reduce Blast Radius

Region-Based Rollout Order

Phase 1: Internal staging environment (all engineers)
Phase 2: Smallest production region (lowest traffic)
Phase 3: Secondary production regions
Phase 4: Primary production region (highest traffic)

Gate between each phase: error rate < 0.1%, p99 latency < target

Database Migration Safety

Database changes are the most common source of failed rollbacks. Use the expand-contract pattern to keep both old and new application versions compatible with the database schema.

Expand-Contract Pattern

Step 1 (Expand): Add new column/table, deploy app that writes to BOTH old and new
Step 2 (Migrate): Backfill existing data from old to new location
Step 3 (Switch): Deploy app that reads from new, writes to BOTH
Step 4 (Contract): Remove old column/table after all instances use new schema

Safe Migration Example

-- Step 1: Add new column (backward compatible)
ALTER TABLE users ADD COLUMN email_normalized VARCHAR(255);

-- Step 2: Backfill data
UPDATE users SET email_normalized = LOWER(TRIM(email)) WHERE email_normalized IS NULL;

-- Step 3: App now reads from email_normalized (deploy and verify)

-- Step 4: Drop old usage (only after all app instances use new column)
-- ALTER TABLE users DROP COLUMN email;  -- do this in a SEPARATE migration

Never combine schema changes with data migrations in the same deployment. Never drop columns or tables in the same release that stops writing to them.