Devops Excellence

DevOps Excellence

Core Principles

• Shift Left — Address security and quality early in SDLC
• GitOps — Git as single source of truth for infrastructure and deployments
• Infrastructure as Code — All infrastructure versioned and reproducible
• Progressive Delivery — Gradual rollouts with feature flags and canary releases
• Immutable Infrastructure — Replace, don't modify running systems
• Observability-First — Monitor metrics tied to deployments and features
• Policy as Code — Enforce compliance and security automatically
• Platform Engineering — Build golden paths and self-service portals

---

Hard Rules (Must Follow)

These rules are mandatory. Violating them means the skill is not working correctly.

No Static Credentials

Never use long-lived static credentials. Always use OIDC or short-lived tokens.

# ❌ FORBIDDEN: Static AWS credentials
env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

# ✅ REQUIRED: OIDC-based authentication
- name: Configure AWS Credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/GitHubActions
    aws-region: us-east-1
    # No long-lived secrets - uses GitHub OIDC provider

No Root Containers

Containers must NEVER run as root. Always specify a non-root user.

# ❌ FORBIDDEN: Running as root (default)
FROM node:20
WORKDIR /app
CMD ["node", "server.js"]

# ❌ FORBIDDEN: Explicit root user
USER root

# ✅ REQUIRED: Non-root user with UID > 1000
FROM node:20-alpine
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001
USER nodejs
WORKDIR /app
CMD ["node", "server.js"]

No Secrets in Images

Never bake secrets into Docker images. Use runtime injection or secrets managers.

# ❌ FORBIDDEN: Secrets in build args or ENV
ARG DATABASE_PASSWORD
ENV API_KEY=sk-xxx

# ❌ FORBIDDEN: Copying secret files
COPY .env /app/.env
COPY credentials.json /app/

# ✅ REQUIRED: Mount secrets at runtime
# docker run -v /secrets:/app/secrets:ro myapp
# Or use Kubernetes secrets/configmaps

Protected Production Deployments

Production deployments must require approval and be restricted to main branch.

# ❌ FORBIDDEN: Direct production deploy without protection
deploy:
  runs-on: ubuntu-latest
  steps:
    - run: deploy-to-prod.sh

# ✅ REQUIRED: Environment protection
deploy:
  runs-on: ubuntu-latest
  environment:
    name: production
    url: https://myapp.com
  # Requires: approval + main branch only

---

Quick Reference

When to Use What

Deployment Strategy Selection

---

CI/CD Pipeline Best Practices

Pipeline Security

# Short-lived credentials (not static keys)
- name: Configure AWS Credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/GitHubActions
    aws-region: us-east-1
    # OIDC provider - no long-lived secrets!

# Protected environments for production
environment:
  name: production
  # Requires approval + restricts to main branch

Speed Optimization

• 10-minute build rule — Most projects should build in <10 minutes
• Parallel jobs — Run tests, linting, security scans concurrently
• Cache dependencies — Cache node_modules, .m2, pip packages
• Conditional execution — Skip jobs when files haven't changed

# Example: conditional job execution
jobs:
  backend-tests:
    if: contains(github.event.head_commit.modified, 'backend/')
    runs-on: ubuntu-latest

Testing Pyramid

              /\
             /E2E\        <- Few (slow, expensive)
            /------\
           /Integration\ <- Some (medium speed)
          /------------\
         /  Unit Tests  \ <- Many (fast, cheap)
        /----------------\

• 70% Unit tests (fast, isolated)
• 20% Integration tests (service interactions)
• 10% E2E tests (full user workflows)

Security Scanning Integration

# Multi-layer security scanning
jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      # SAST - Static code analysis
      - uses: github/codeql-action/init@v3

      # SCA - Dependency vulnerabilities
      - name: Run Trivy
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          format: 'sarif'

      # Secret scanning
      - name: Gitleaks
        uses: gitleaks/gitleaks-action@v2

      # Container scanning
      - name: Scan Docker image
        run: trivy image myapp:${{ github.sha }}

---

Docker Best Practices

Multi-Stage Builds

# Build stage - includes build tools (900MB+)
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Runtime stage - minimal image (<100MB)
FROM node:20-alpine AS runtime
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001
WORKDIR /app
COPY --from=builder --chown=nodejs:nodejs /app/node_modules ./node_modules
COPY --chown=nodejs:nodejs . .
USER nodejs
EXPOSE 3000
CMD ["node", "server.js"]

Security Hardening

• Non-root user — ALWAYS run as non-root (UID 1001)
• Minimal base images — Use alpine, distroless, or scratch
• Read-only filesystem — docker run --read-only
• No secrets in layers — Use build secrets or external vaults
• Resource limits — Set CPU/memory limits to prevent DoS
• Signed images — Enable Docker Content Trust

# Security best practices example
FROM gcr.io/distroless/nodejs20-debian12
COPY --chown=65532:65532 /app /app
USER 65532
EXPOSE 8080

.dockerignore

# Version control
.git
.gitignore

# Dependencies (install fresh in container)
node_modules
vendor/
*.pyc
__pycache__

# Secrets and configs
.env
.env.local
secrets/
*.key
*.pem

# Development files
README.md
Dockerfile
docker-compose.yml
.vscode/
.idea/

# Testing and CI
tests/
*.test.js
.github/

---

Kubernetes Deployment Patterns

Resource Management (Right-Sizing)

# 99.94% of clusters are over-provisioned!
# Average CPU usage: 10%, Memory: 23%
resources:
  requests:
    memory: "128Mi"  # Guaranteed allocation
    cpu: "100m"      # 0.1 CPU cores
  limits:
    memory: "256Mi"  # Maximum allowed
    cpu: "200m"      # Hard cap

# Use tools: Kubecost, Goldilocks, VPA

Health Checks

# Liveness: Is container alive?
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

# Readiness: Can it receive traffic?
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  successThreshold: 1

# Startup: Has initialization completed?
startupProbe:
  httpGet:
    path: /startup
    port: 8080
  failureThreshold: 30  # 30*10s = 5min for slow starts
  periodSeconds: 10

ConfigMaps and Secrets

# Group related resources in single manifest
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  APP_ENV: production
  LOG_LEVEL: info
---
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  DATABASE_URL: postgresql://user:pass@db:5432/mydb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
      - name: app
        envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secrets

Security Best Practices

# Pod Security Standards
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault
  capabilities:
    drop:
    - ALL

# Network Policies (deny-by-default)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress

---

Extended Reference

Detailed material starting at ## Infrastructure as Code (Terraform/Pulumi) has been moved to `reference/extended.md` to keep this skill concise. Load that reference when the task requires the moved examples, command catalogs, checklists, platform details, or implementation templates.

CI/CD Pipeline Patterns

GitHub Actions vs GitLab CI

When to Choose Each

GitHub Actions:

• Already using GitHub for code hosting
• Need extensive marketplace of pre-built actions
• Simpler learning curve for beginners
• Event-driven workflows (issue comments, releases)
• Free for public repositories

GitLab CI:

• Need unified DevOps platform (issues, CI, security, etc.)
• Complex enterprise pipelines with dependencies
• Built-in security scanning and compliance
• Self-hosted GitLab instance
• Advanced features (parent-child pipelines, includes)

Market Trends (2025)

• 80%+ adoption of CI/CD tools is GitOps-adjacent (Jenkins, GitHub Actions, GitLab CI)
• Legacy migrations from Azure DevOps/Jenkins to GitHub Actions/GitLab CI ongoing
• "It lives where code lives" is primary selection criteria
• AI integration in 76% of teams for predictive failures and auto-fixes

---

GitHub Actions Patterns

Matrix Builds

name: Multi-Platform Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node: [18, 20, 22]
        exclude:
          # Skip old Node on Windows
          - os: windows-latest
            node: 18
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
      - run: npm ci
      - run: npm test

Conditional Execution

jobs:
  deploy:
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        run: ./deploy.sh

  backend:
    # Only run if backend files changed
    if: contains(github.event.head_commit.modified, 'backend/')
    runs-on: ubuntu-latest
    steps:
      - name: Test backend
        run: npm run test:backend

  # Run on schedule (nightly builds)
  nightly:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest

Dependency Caching

- name: Cache dependencies
  uses: actions/cache@v4
  with:
    path: |
      ~/.npm
      ~/.cache/pip
      ~/.m2/repository
    key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json', '**/requirements.txt', '**/pom.xml') }}
    restore-keys: |
      ${{ runner.os }}-deps-

- name: Install dependencies
  run: npm ci  # Uses cache if available

Secrets Management

# Never hardcode secrets!
env:
  DATABASE_URL: ${{ secrets.DATABASE_URL }}
  API_KEY: ${{ secrets.API_KEY }}

# Use OIDC for cloud credentials (recommended)
- name: Configure AWS Credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
    aws-region: us-east-1
    # No static keys - temporary credentials via OIDC!

# Environment-specific secrets
- name: Deploy to staging
  environment: staging  # Requires staging-specific secrets
  run: ./deploy.sh

Reusable Workflows

# .github/workflows/reusable-test.yml
name: Reusable Test Workflow
on:
  workflow_call:
    inputs:
      node-version:
        required: true
        type: string

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ inputs.node-version }}
      - run: npm ci && npm test

# .github/workflows/main.yml
name: Main Pipeline
on: [push]
jobs:
  test-node-18:
    uses: ./.github/workflows/reusable-test.yml
    with:
      node-version: '18'
  test-node-20:
    uses: ./.github/workflows/reusable-test.yml
    with:
      node-version: '20'

---

GitLab CI Patterns

Pipeline Structure

# .gitlab-ci.yml
stages:
  - build
  - test
  - security
  - deploy

# Global variables
variables:
  DOCKER_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

# Build job
build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker build -t $DOCKER_IMAGE .
    - docker push $DOCKER_IMAGE
  only:
    - main
    - merge_requests

# Parallel testing
unit-tests:
  stage: test
  image: node:20
  script:
    - npm ci
    - npm run test:unit
  coverage: '/Coverage: \d+\.\d+/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

integration-tests:
  stage: test
  image: node:20
  services:
    - postgres:15
    - redis:7
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: test
    POSTGRES_PASSWORD: test
  script:
    - npm run test:integration

# Security scanning (built-in)
sast:
  stage: security
  # Uses GitLab's built-in SAST analyzer
  include:
    - template: Security/SAST.gitlab-ci.yml

container_scanning:
  stage: security
  include:
    - template: Security/Container-Scanning.gitlab-ci.yml

# Environment-specific deployment
deploy:staging:
  stage: deploy
  environment:
    name: staging
    url: https://staging.example.com
  script:
    - kubectl set image deployment/myapp myapp=$DOCKER_IMAGE
  only:
    - develop

deploy:production:
  stage: deploy
  environment:
    name: production
    url: https://example.com
  when: manual  # Requires approval
  script:
    - kubectl set image deployment/myapp myapp=$DOCKER_IMAGE
  only:
    - main

Parent-Child Pipelines

# Parent pipeline
trigger-child:
  trigger:
    include: .gitlab/pipelines/child-pipeline.yml
    strategy: depend  # Wait for child to complete

# Dynamic child pipelines
generate-config:
  stage: build
  script:
    - ./generate-pipeline.sh > generated-pipeline.yml
  artifacts:
    paths:
      - generated-pipeline.yml

trigger-dynamic:
  stage: deploy
  trigger:
    include:
      - artifact: generated-pipeline.yml
        job: generate-config

Advanced Caching

# Cache between pipeline runs
cache:
  key:
    files:
      - package-lock.json
  paths:
    - node_modules/
  policy: pull-push  # Default

# Per-branch caching
cache:
  key: "$CI_COMMIT_REF_SLUG"
  paths:
    - node_modules/

# Read-only cache for jobs that don't modify
test:
  cache:
    key: "$CI_COMMIT_REF_SLUG"
    paths:
      - node_modules/
    policy: pull  # Don't update cache

---

Pipeline Optimization Strategies

1. Keep Builds Fast (10-Minute Rule)

Every minute saved multiplies across all developers and commits.

# Before: 25-minute build
- install dependencies (8 min)
- run all tests (12 min)
- build docker image (5 min)

# After: 9-minute build
- install dependencies (2 min - cached)
- run tests in parallel (4 min - 3 parallel jobs)
- build docker image (3 min - multi-stage cache)

Tactics:

• Cache dependencies aggressively
• Run tests in parallel (split by test suite)
• Use incremental builds
• Skip unchanged modules (monorepo)
• Optimize Docker layer caching

2. Fail Fast

# Run fast checks first
stages:
  - validate     # Linting, formatting (30s)
  - test-unit    # Unit tests (2 min)
  - test-integration  # Integration (5 min)
  - build        # Build artifacts (3 min)
  - deploy       # Deploy (2 min)

# Don't wait for slow jobs if fast ones fail

3. Conditional Job Execution

# Skip jobs when not needed
jobs:
  frontend-tests:
    rules:
      - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
        changes:
          - frontend/**/*
          - package.json
    script: npm run test:frontend

  # Always run security scans
  security-scan:
    rules:
      - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      - if: '$CI_COMMIT_BRANCH == "main"'

4. Parallel Matrix Execution

# GitHub Actions
test:
  strategy:
    matrix:
      shard: [1, 2, 3, 4]  # Split tests into 4 shards
  steps:
    - run: npm test -- --shard=${{ matrix.shard }}/4

# GitLab CI
test:
  parallel: 4
  script:
    - npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

---

Security Best Practices

1. Short-Lived Credentials

# BAD: Static credentials
env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_KEY }}

# GOOD: OIDC with temporary credentials
- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/GitHubActions
    aws-region: us-east-1
    # Credentials expire in 1 hour

2. Protected Environments

# GitHub Actions
environment:
  name: production
  # Requires:
  # - Manual approval from designated reviewers
  # - Can only run from 'main' branch
  # - Wait timer before deployment

# GitLab CI
deploy:production:
  environment:
    name: production
    deployment_tier: production
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: manual  # Requires approval
  only:
    variables:
      - $CI_COMMIT_REF_PROTECTED == "true"

3. Masked Variables

# GitLab: Mask secrets in logs
variables:
  DATABASE_PASSWORD:
    value: "supersecret"
    masked: true

# GitHub: Secrets are automatically masked
# Output: "***" instead of actual value

4. Dependency Scanning

# Snyk integration
- name: Run Snyk to check for vulnerabilities
  uses: snyk/actions/node@master
  env:
    SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
  with:
    args: --severity-threshold=high --fail-on=upgradable

# Dependabot (GitHub native)
# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: npm
    directory: "/"
    schedule:
      interval: weekly
    open-pull-requests-limit: 10

---

Culture and Process

Shift from Blame to Learning

Traditional (Blame Culture):

• "Who broke production?"
• Individual accountability for failures
• Fear of deploying

Modern (Learning Culture):

• "What caused this failure?"
• Blameless postmortems
• Failure as learning opportunity
• Psychological safety to experiment

Metrics That Matter

Track these, not lines of code:

• Deployment frequency
• Lead time for changes
• Mean time to recovery
• Change failure rate

2025 Elite Performers:

• Deploy multiple times per day
• < 1 hour from commit to production
• < 1 hour to recover from incidents
• 0-15% change failure rate

---

Advanced Patterns

Dynamic Pipeline Generation

# generate-pipeline.py
import yaml

services = ['api', 'worker', 'frontend']
pipeline = {'stages': ['test', 'build', 'deploy']}

for service in services:
    pipeline[f'test-{service}'] = {
        'stage': 'test',
        'script': [f'npm run test --workspace={service}'],
        'only': {'changes': [f'{service}/**/*']}
    }

with open('.gitlab-ci.yml', 'w') as f:
    yaml.dump(pipeline, f)

Pipeline as Code (Dagger)

// Portable pipelines in code (not YAML)
package main

import (
    "dagger.io/dagger"
)

func Pipeline(ctx context.Context) error {
    client, _ := dagger.Connect(ctx)
    defer client.Close()

    // Build container
    container := client.Container().
        From("node:20").
        WithDirectory("/src", client.Host().Directory(".")).
        WithWorkdir("/src").
        WithExec([]string{"npm", "ci"}).
        WithExec([]string{"npm", "test"}).
        WithExec([]string{"npm", "run", "build"})

    // Export build artifacts
    _, err := container.Directory("/src/dist").Export(ctx, "./dist")
    return err
}

GitOps Integration

# Pipeline updates GitOps repo
deploy:
  stage: deploy
  script:
    # Clone GitOps repo
    - git clone https://github.com/myorg/k8s-manifests.git
    - cd k8s-manifests/apps/myapp/overlays/prod

    # Update image tag using kustomize
    - kustomize edit set image myapp=$DOCKER_IMAGE

    # Commit and push
    - git config user.name "CI Bot"
    - git config user.email "ci@example.com"
    - git add .
    - git commit -m "Update myapp to $DOCKER_IMAGE"
    - git push

    # ArgoCD auto-syncs cluster to new state

---

Monitoring and Observability

Pipeline Metrics

# Send metrics to observability platform
- name: Report metrics
  if: always()  # Run even if previous steps fail
  run: |
    curl -X POST https://metrics.example.com/api/v1/metrics \
      -H "Content-Type: application/json" \
      -d '{
        "pipeline_id": "${{ github.run_id }}",
        "status": "${{ job.status }}",
        "duration_seconds": ${{ github.event.workflow_run.duration }},
        "branch": "${{ github.ref }}",
        "commit": "${{ github.sha }}",
        "repository": "${{ github.repository }}"
      }'

Failure Notifications

# Slack notification on failure
- name: Notify Slack
  if: failure()
  uses: slackapi/slack-github-action@v1
  with:
    payload: |
      {
        "text": "Pipeline failed for ${{ github.repository }}",
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "*Pipeline Failed*\n*Repo:* ${{ github.repository }}\n*Branch:* ${{ github.ref }}\n*Commit:* ${{ github.sha }}"
            }
          },
          {
            "type": "actions",
            "elements": [
              {
                "type": "button",
                "text": "View Logs",
                "url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
              }
            ]
          }
        ]
      }
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

---

Troubleshooting Common Issues

Flaky Tests

# Retry flaky tests automatically
- name: Run tests with retry
  uses: nick-invision/retry@v2
  with:
    timeout_minutes: 10
    max_attempts: 3
    retry_on: error
    command: npm test

# Or mark as flaky and track
- name: Run tests
  run: npm test || echo "FLAKY_TEST_FAILED=true" >> $GITHUB_ENV
  continue-on-error: true

Out-of-Memory Errors

# Increase Node memory limit
env:
  NODE_OPTIONS: --max-old-space-size=4096

# Use larger runner
runs-on: ubuntu-latest-8-cores  # GitHub hosted (paid)
# or
runs-on: self-hosted-large      # Self-hosted runner

Slow Docker Builds

# Use BuildKit with caching
- name: Build Docker image
  uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: myapp:${{ github.sha }}
    cache-from: type=registry,ref=myapp:buildcache
    cache-to: type=registry,ref=myapp:buildcache,mode=max

---

Best Practices Summary

1. Security First

• Use OIDC for cloud credentials
• Mask all secrets
• Scan dependencies and containers
• Protect production branches

2. Speed Matters

• Target < 10 minute builds
• Cache aggressively
• Parallelize jobs
• Fail fast with early validation

3. Reliability

• Retry flaky tests
• Monitor pipeline metrics
• Set up failure notifications
• Document common issues

4. Maintainability

• Keep pipeline configs DRY
• Use reusable workflows
• Version pipeline code
• Review pipeline changes like code

5. Culture

• Blameless postmortems
• Track DORA metrics
• Continuous improvement
• Psychological safety to experiment

Container and Kubernetes Best Practices

Docker Best Practices

Multi-Stage Builds Explained

The Problem: Node.js images with build tools are 900MB+. Production only needs the runtime.

The Solution: Separate build and runtime stages.

# ============================================
# Stage 1: Dependencies (Build Stage)
# ============================================
FROM node:20-alpine AS dependencies
WORKDIR /app

# Copy package files only (better caching)
COPY package*.json ./

# Install ALL dependencies (dev + prod)
RUN npm ci

# ============================================
# Stage 2: Build Application
# ============================================
FROM dependencies AS builder
WORKDIR /app

# Copy source code
COPY . .

# Build application (TypeScript, webpack, etc.)
RUN npm run build

# Remove dev dependencies
RUN npm prune --production

# ============================================
# Stage 3: Runtime (Production Image)
# ============================================
FROM node:20-alpine AS runtime

# Create non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001

WORKDIR /app

# Copy only production dependencies
COPY --from=builder --chown=nodejs:nodejs /app/node_modules ./node_modules

# Copy built application
COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist
COPY --from=builder --chown=nodejs:nodejs /app/package.json ./

# Switch to non-root user
USER nodejs

# Expose port
EXPOSE 3000

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/health', (r) => process.exit(r.statusCode === 200 ? 0 : 1))"

# Start application
CMD ["node", "dist/server.js"]

Result: 100MB final image (90% reduction) with same functionality.

---

Security Hardening

1. Use Minimal Base Images

# ❌ BAD: Full OS image (1.2GB)
FROM ubuntu:22.04

# ⚠️  ACCEPTABLE: Official Node (180MB)
FROM node:20-alpine

# ✅ BETTER: Distroless (no shell, minimal attack surface)
FROM gcr.io/distroless/nodejs20-debian12

# ✅ BEST: Scratch (only app binary, <10MB)
FROM scratch
COPY --from=builder /app/binary /binary
ENTRYPOINT ["/binary"]

2. Run as Non-Root User

# ❌ BAD: Runs as root (UID 0)
FROM node:20-alpine
COPY . /app
CMD ["node", "server.js"]

# ✅ GOOD: Create and use non-root user
FROM node:20-alpine

# Create user with specific UID/GID
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001

# Set ownership
COPY --chown=nodejs:nodejs . /app

# Switch user
USER nodejs

CMD ["node", "server.js"]

# ✅ BEST: Use distroless with predefined user
FROM gcr.io/distroless/nodejs20-debian12
COPY --chown=65532:65532 /app /app
USER 65532  # nonroot user in distroless

3. Read-Only Filesystem

# Enable read-only root filesystem
FROM node:20-alpine
RUN adduser -S -u 1001 nodejs

# Create writable temp directory
RUN mkdir -p /tmp && chown nodejs:nodejs /tmp

USER nodejs
WORKDIR /app
COPY --chown=nodejs:nodejs . .

# Run with read-only flag
# docker run --read-only --tmpfs /tmp myapp

4. Resource Limits

# Set memory and CPU limits
# docker run --memory="512m" --cpus="0.5" myapp

# In docker-compose.yml
services:
  app:
    image: myapp
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M

5. No Secrets in Layers

# ❌ BAD: Secret in layer (visible in image history)
RUN curl -H "Authorization: Bearer secret-token" https://api.example.com/data

# ✅ GOOD: Use build secrets (BuildKit)
# docker buildx build --secret id=token,env=API_TOKEN .
RUN --mount=type=secret,id=token \
    curl -H "Authorization: Bearer $(cat /run/secrets/token)" https://api.example.com/data

# ✅ BEST: Fetch secrets at runtime
CMD ["sh", "-c", "export API_TOKEN=$(aws secretsmanager get-secret-value ...) && node server.js"]

6. Vulnerability Scanning

# Scan image with Trivy
trivy image --severity HIGH,CRITICAL myapp:latest

# Scan in CI/CD
docker build -t myapp:$VERSION .
trivy image --exit-code 1 --severity CRITICAL myapp:$VERSION

---

Layer Caching Optimization

Docker builds layers sequentially. A change invalidates all subsequent layers.

# ❌ BAD: Source code changes invalidate dependency install
FROM node:20-alpine
WORKDIR /app
COPY . .                    # Changes frequently
RUN npm ci                  # Re-runs on every code change!
RUN npm run build

# ✅ GOOD: Dependencies cached separately
FROM node:20-alpine
WORKDIR /app
COPY package*.json ./       # Changes infrequently
RUN npm ci                  # Cached unless package.json changes
COPY . .                    # Source code changes don't affect deps
RUN npm run build

Ordering principle: Put frequently changing files at the bottom.

# Order by change frequency (least to most)
COPY package*.json ./           # Changes rarely
RUN npm ci
COPY tsconfig.json ./           # Changes occasionally
COPY src/ ./src/                # Changes frequently
RUN npm run build

---

.dockerignore Best Practices

# Version Control
.git/
.gitignore
.gitattributes

# Dependencies (install fresh in container)
node_modules/
vendor/
__pycache__/
*.pyc
.venv/

# Build artifacts
dist/
build/
*.o
*.so

# Environment files
.env
.env.*
!.env.example

# Secrets
secrets/
*.key
*.pem
*.crt
credentials.json

# Documentation
README.md
CHANGELOG.md
docs/

# Development files
.vscode/
.idea/
*.swp
*.swo

# Testing
tests/
test/
spec/
coverage/
*.test.js
*.spec.ts

# CI/CD
.github/
.gitlab-ci.yml
Jenkinsfile
docker-compose.yml
Dockerfile*

# OS files
.DS_Store
Thumbs.db

# Logs
logs/
*.log
npm-debug.log*

---

Kubernetes Best Practices

1. Resource Management (Right-Sizing)

Problem: 99.94% of clusters are over-provisioned (average CPU: 10%, memory: 23%).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
      - name: app
        image: myapp:1.0
        resources:
          # REQUESTS: Guaranteed allocation (scheduling decision)
          requests:
            memory: "256Mi"   # Needs at least 256MB
            cpu: "200m"       # Needs 0.2 CPU cores

          # LIMITS: Maximum allowed (OOM kill/throttle)
          limits:
            memory: "512Mi"   # Killed if exceeds 512MB
            cpu: "500m"       # Throttled if exceeds 0.5 cores

How to determine values:

# 1. Start with conservative estimates
requests:
  memory: "128Mi"
  cpu: "100m"

# 2. Monitor actual usage
kubectl top pod myapp-xyz

# 3. Use Vertical Pod Autoscaler (VPA) for recommendations
kubectl get vpa myapp -o yaml

# 4. Or use Goldilocks for right-sizing suggestions
# Analyzes VPA and provides recommendations

Resource Units:

• CPU: 1000m = 1 CPU core = 1 vCPU
• Memory: 1Mi = 1 Mebibyte = 1.048576 MB

---

2. Health Checks (Probes)

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: app
        image: myapp:1.0
        ports:
        - containerPort: 8080

        # STARTUP PROBE: Has the application started?
        # Used for slow-starting applications
        startupProbe:
          httpGet:
            path: /startup
            port: 8080
          initialDelaySeconds: 0
          periodSeconds: 10
          failureThreshold: 30  # 30*10s = 5 minutes for slow startup
          # Runs BEFORE liveness and readiness probes

        # LIVENESS PROBE: Is the application alive?
        # Restart container if fails
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3   # Restart after 3 consecutive failures
          successThreshold: 1

        # READINESS PROBE: Can it receive traffic?
        # Remove from service if fails
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3
          successThreshold: 1   # Add to service after 1 success

Health Endpoint Implementation:

// Express.js example
app.get('/startup', (req, res) => {
  // Check if database connections established, migrations run
  if (!db.isConnected() || !migrationsComplete) {
    return res.status(503).send('Not ready');
  }
  res.status(200).send('Started');
});

app.get('/health', (req, res) => {
  // Deep health check - is app functioning?
  if (!canProcessRequests()) {
    return res.status(503).send('Unhealthy');
  }
  res.status(200).send('Healthy');
});

app.get('/ready', (req, res) => {
  // Can accept traffic? (dependencies available)
  if (!db.isConnected() || !cache.isConnected()) {
    return res.status(503).send('Not ready');
  }
  res.status(200).send('Ready');
});

---

3. ConfigMaps and Secrets

# ============================================
# ConfigMap: Non-sensitive configuration
# ============================================
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  # Key-value pairs
  APP_ENV: production
  LOG_LEVEL: info
  FEATURE_FLAGS: '{"newUI": true}'

  # File content
  nginx.conf: |
    server {
      listen 80;
      location / {
        proxy_pass http://backend:8080;
      }
    }

---
# ============================================
# Secret: Sensitive data (base64 encoded)
# ============================================
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  # Automatically base64 encoded
  DATABASE_URL: postgresql://user:pass@db:5432/mydb
  API_KEY: abc123xyz

---
# ============================================
# Using ConfigMap and Secret
# ============================================
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: app
        image: myapp:1.0

        # Option 1: Environment variables from ConfigMap
        envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secrets

        # Option 2: Specific environment variables
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: DATABASE_URL

        # Option 3: Mount as files
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
        - name: secret-volume
          mountPath: /etc/secrets
          readOnly: true

      volumes:
      - name: config-volume
        configMap:
          name: app-config
      - name: secret-volume
        secret:
          secretName: app-secrets

External Secrets (Recommended for Production):

# Use External Secrets Operator to sync from vault
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: app-secrets  # Creates K8s Secret
  data:
  - secretKey: DATABASE_URL
    remoteRef:
      key: prod/myapp/database-url
  - secretKey: API_KEY
    remoteRef:
      key: prod/myapp/api-key

---

4. Security Best Practices

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      # Service account with minimal permissions
      serviceAccountName: myapp-sa
      automountServiceAccountToken: false  # Don't auto-mount if not needed

      # Security context (pod level)
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault

      containers:
      - name: app
        image: myapp:1.0

        # Security context (container level)
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
          capabilities:
            drop:
            - ALL  # Drop all capabilities
            add:
            - NET_BIND_SERVICE  # Only add specific capabilities needed

        # Read-only root filesystem requires writable volumes for temp files
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: cache
          mountPath: /app/.cache

      volumes:
      - name: tmp
        emptyDir: {}
      - name: cache
        emptyDir: {}

Network Policies (Zero Trust)

# Deny all ingress traffic by default
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress

---
# Allow specific traffic to backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  # Allow from frontend pods
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

  # Allow from ingress controller
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080

Pod Security Standards

# Enforce security standards at namespace level
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    # Enforce restricted security standard
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

---

5. Deployment Strategies

Rolling Update (Default)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1  # Max pods down: 10 - 1 = 9 available
      maxSurge: 2        # Max total pods: 10 + 2 = 12

  template:
    spec:
      containers:
      - name: app
        image: myapp:2.0

# Rollout process:
# 1. Create 2 new pods (v2.0) - total: 12 pods
# 2. Wait for new pods to be ready
# 3. Terminate 1 old pod (v1.0) - total: 11 pods
# 4. Repeat until all pods are v2.0

Commands:

# Deploy new version
kubectl set image deployment/myapp app=myapp:2.0

# Monitor rollout
kubectl rollout status deployment/myapp

# Pause rollout (for manual verification)
kubectl rollout pause deployment/myapp

# Resume rollout
kubectl rollout resume deployment/myapp

# Rollback to previous version
kubectl rollout undo deployment/myapp

# Rollback to specific revision
kubectl rollout undo deployment/myapp --to-revision=3

# View rollout history
kubectl rollout history deployment/myapp

Recreate Strategy

apiVersion: apps/v1
kind: Deployment
spec:
  strategy:
    type: Recreate  # All pods stopped before new ones start

# Use ONLY for:
# - Development environments
# - Applications that can't run multiple versions simultaneously
# - Database schema changes requiring downtime

---

6. StatefulSets (for Stateful Applications)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-headless
  replicas: 3
  selector:
    matchLabels:
      app: postgres

  # Volume claim templates (creates PVC for each pod)
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

  template:
    spec:
      containers:
      - name: postgres
        image: postgres:15
        ports:
        - containerPort: 5432
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data

# Pods created in order: postgres-0, postgres-1, postgres-2
# Stable network IDs: postgres-0.postgres-headless
# Persistent storage: Each pod has own PVC

---

7. Horizontal Pod Autoscaling (HPA)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 3
  maxReplicas: 20
  metrics:
  # Scale based on CPU
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70  # Target 70% CPU

  # Scale based on memory
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

  # Scale based on custom metrics (requests per second)
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"

  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 min before scaling down
      policies:
      - type: Percent
        value: 50  # Scale down max 50% at a time
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100  # Double pods if needed
        periodSeconds: 15

---

8. Observability

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        # Prometheus scraping
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: app
        image: myapp:1.0

        # Expose metrics port
        ports:
        - name: metrics
          containerPort: 9090

        # Structured logging
        env:
        - name: LOG_FORMAT
          value: json
        - name: LOG_LEVEL
          value: info

Application Metrics (Prometheus):

// Express.js with prom-client
import promClient from 'prom-client';

// Default metrics (CPU, memory, etc.)
promClient.collectDefaultMetrics();

// Custom metrics
const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status_code']
});

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    httpRequestDuration.labels(req.method, req.route?.path, res.statusCode)
      .observe((Date.now() - start) / 1000);
  });
  next();
});

// Metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(await promClient.register.metrics());
});

---

Advanced Patterns

Init Containers

# Run initialization tasks before main container
spec:
  initContainers:
  # Wait for database to be ready
  - name: wait-for-db
    image: busybox
    command:
    - sh
    - -c
    - |
      until nc -z postgres 5432; do
        echo "Waiting for postgres..."
        sleep 2
      done

  # Run database migrations
  - name: migrate
    image: myapp:1.0
    command: ["npm", "run", "migrate"]
    env:
    - name: DATABASE_URL
      valueFrom:
        secretKeyRef:
          name: app-secrets
          key: DATABASE_URL

  # Main application container starts AFTER init containers succeed
  containers:
  - name: app
    image: myapp:1.0

Sidecar Pattern

# Logging sidecar example
spec:
  containers:
  # Main application
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: logs
      mountPath: /var/log/app

  # Sidecar: ships logs to external system
  - name: log-shipper
    image: fluent/fluent-bit:2.0
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
      readOnly: true
    - name: fluent-config
      mountPath: /fluent-bit/etc/

  volumes:
  - name: logs
    emptyDir: {}
  - name: fluent-config
    configMap:
      name: fluent-bit-config

---

Troubleshooting

Pod Stuck in Pending

# Check events
kubectl describe pod myapp-xyz

# Common causes:
# - Insufficient resources (CPU/memory)
# - Node selector doesn't match any nodes
# - PersistentVolumeClaim not bound
# - Image pull secrets missing

CrashLoopBackOff

# View logs
kubectl logs myapp-xyz
kubectl logs myapp-xyz --previous  # Previous container instance

# Common causes:
# - Application error on startup
# - Failed health checks
# - Missing dependencies
# - Incorrect command/args

ImagePullBackOff

# Check image details
kubectl describe pod myapp-xyz

# Common causes:
# - Image doesn't exist
# - Typo in image name/tag
# - Private registry without imagePullSecrets
# - Rate limiting (Docker Hub)

High Memory Usage

# Check current usage
kubectl top pod myapp-xyz

# Check if hitting limits
kubectl describe pod myapp-xyz | grep -A 5 Limits

# Solutions:
# - Increase memory limits
# - Fix memory leaks in application
# - Use Vertical Pod Autoscaler

---

Kubernetes Cost Optimization

1. Right-size resources (use VPA/Goldilocks) 2. Use Spot/Preemptible instances for non-critical workloads 3. Enable cluster autoscaler to scale nodes 4. Set resource quotas per namespace 5. Monitor unused resources (Kubecost) 6. Use PodDisruptionBudgets for safe node draining 7. Implement pod priority for critical workloads

# Resource quotas per namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: development
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    persistentvolumeclaims: "10"

devops-excellence Extended Reference

This file preserves detailed material moved out of SKILL.md for progressive disclosure. Load it only when the current task needs the specific examples, commands, templates, or checklists below.

Moved content starts at: ## Infrastructure as Code (Terraform/Pulumi).

Infrastructure as Code (Terraform/Pulumi)

Directory Structure

terraform/
├── environments/
│   ├── dev/
│   │   ├── main.tf
│   │   └── terraform.tfvars
│   ├── staging/
│   └── prod/
├── modules/
│   ├── vpc/
│   ├── eks/
│   └── rds/
├── backend.tf        # Remote state config
└── versions.tf       # Provider versions

Best Practices

1. Remote State with Locking

# backend.tf
terraform {
  backend "s3" {
    bucket         = "mycompany-terraform-state"
    key            = "prod/vpc/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"  # Prevents concurrent runs
  }
}

2. Modularization

# modules/vpc/main.tf
variable "cidr_block" {
  type        = string
  description = "VPC CIDR block"
}

resource "aws_vpc" "main" {
  cidr_block           = var.cidr_block
  enable_dns_hostnames = true
  tags = {
    Name = "${var.environment}-vpc"
  }
}

# environments/prod/main.tf
module "vpc" {
  source     = "../../modules/vpc"
  cidr_block = "10.0.0.0/16"
  environment = "prod"
}

3. Policy as Code

# Use Sentinel (Terraform Cloud) or OPA
policy "enforce-tags" {
  enforcement_level = "hard-mandatory"

  # Require tags on all resources
  rule {
    condition = all resource.tags contains "Owner"
    error_message = "All resources must have Owner tag"
  }
}

4. Automated Testing

// Terratest example
func TestVPCCreation(t *testing.T) {
  terraformOptions := &terraform.Options{
    TerraformDir: "../environments/dev",
  }

  defer terraform.Destroy(t, terraformOptions)
  terraform.InitAndApply(t, terraformOptions)

  vpcId := terraform.Output(t, terraformOptions, "vpc_id")
  assert.NotEmpty(t, vpcId)
}

Pulumi Advantages

// Pulumi - real programming language benefits
import * as aws from "@pulumi/aws";

const environments = ["dev", "staging", "prod"];

// Use loops, conditionals, functions
environments.forEach(env => {
  new aws.ec2.Vpc(`${env}-vpc`, {
    cidrBlock: env === "prod" ? "10.0.0.0/16" : "10.1.0.0/16",
    tags: { Environment: env },
  });
});

// Built-in testing framework
import * as pulumi from "@pulumi/pulumi";
pulumi.runtime.setMocks(...);

---

Release Strategies

Blue-Green Deployment

# Two identical environments
# Switch traffic instantly via load balancer

# Step 1: Deploy to Green (idle)
# Step 2: Test Green environment
# Step 3: Switch LB from Blue to Green
# Step 4: Keep Blue as rollback option

# Kubernetes example
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
    version: blue  # Change to 'green' to switch
  ports:
  - port: 80

When to use:

• Critical systems requiring instant rollback
• Compliance requirements for zero downtime
• Budget allows 2x infrastructure

Canary Deployment

# Gradual rollout: 5% → 25% → 50% → 100%
# Monitor metrics at each stage

# Argo Rollouts example
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10      # 1 pod (10%)
      - pause: {duration: 5m}
      - setWeight: 50      # 5 pods
      - pause: {duration: 10m}
      - setWeight: 100     # All pods
  template:
    spec:
      containers:
      - name: myapp
        image: myapp:v2.0

When to use:

• High-risk deployments (major refactors)
• User-facing features needing validation
• Data-driven rollout decisions

Rolling Update

# Default Kubernetes strategy
# Gradually replace old pods with new

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1  # Never < 9 pods available
      maxSurge: 2        # Never > 12 pods total

When to use:

• Regular incremental updates
• Cost-conscious deployments
• Low-risk changes

---

Feature Flags and Progressive Delivery

Best Practices

1. Flag Lifecycle Management

// Avoid "flag debt" - remove after rollout
const featureFlags = {
  // Short-lived (remove after 100% rollout)
  "new-checkout-v4": {
    enabled: true,
    rollout: 100,
    created: "2025-01-15",
    removeBy: "2025-02-15"
  },

  // Long-lived (kill switch)
  "payment-processing": {
    enabled: true,
    permanent: true,  // Document why
    reason: "Emergency shutoff for payment issues"
  }
};

2. Progressive Rollout

// LaunchDarkly example
const showNewFeature = ldClient.variation(
  "new-dashboard-ui",
  user,
  false  // Default fallback
);

// Configuration
{
  "targeting": {
    "rules": [
      {
        "variation": "on",
        "clauses": [
          {
            "attribute": "email",
            "op": "endsWith",
            "values": ["@mycompany.com"]
          }
        ]
      }
    ],
    "rollout": {
      "percentage": 10  // 10% of remaining users
    }
  }
}

3. Segment Meaningfully

• Geographic: Region-specific rollouts
• Behavioral: Power users first, then general
• Technical: Browser/device-based targeting
• Business: Premium tier vs free tier

4. Observability Integration

// Tie metrics to feature flags
metrics.increment('checkout.completed', {
  feature_flag: 'new-checkout-v4',
  enabled: showNewCheckout
});

// Automatic rollback on error spike
if (errorRate > threshold) {
  ldClient.updateFeatureFlag('new-checkout-v4', { enabled: false });
  alerts.critical('Auto-rollback triggered for new-checkout-v4');
}

---

GitOps Practices

Core Principles

1. Declarative — Entire system state in Git 2. Versioned — Git history = audit trail 3. Immutable — Git commits are immutable 4. Automatic — Agents auto-sync cluster to Git state 5. Continuous — Reconciliation loop detects drift

ArgoCD Workflow

# Application definition
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/k8s-manifests
    targetRevision: main
    path: apps/myapp
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true      # Delete resources not in Git
      selfHeal: true   # Auto-sync on drift detection
    syncOptions:
    - CreateNamespace=true

Repository Structure

k8s-manifests/
├── apps/
│   ├── myapp/
│   │   ├── base/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── kustomization.yaml
│   │   └── overlays/
│   │       ├── dev/
│   │       ├── staging/
│   │       └── prod/
│   │           ├── kustomization.yaml
│   │           └── replicas-patch.yaml
├── infrastructure/
│   ├── ingress-nginx/
│   └── cert-manager/
└── argocd/
    ├── projects.yaml
    └── applications.yaml

Policy Enforcement

# OPA Gatekeeper - deny images without tags
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-owner-label
spec:
  match:
    kinds:
    - apiGroups: ["apps"]
      kinds: ["Deployment"]
  parameters:
    labels: ["owner", "environment"]

---

Platform Engineering

Internal Developer Portal (Backstage)

# Software catalog
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: order-service
  description: Order processing microservice
  tags:
    - java
    - spring-boot
  annotations:
    github.com/project-slug: myorg/order-service
    pagerduty.com/integration-key: xyz
spec:
  type: service
  lifecycle: production
  owner: team-orders
  system: ecommerce-platform

Golden Paths (Templates)

# Self-service project scaffolding
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: nodejs-service
  title: Node.js Microservice
spec:
  steps:
  - id: fetch-template
    action: fetch:template
    input:
      url: ./skeleton
  - id: create-repo
    action: github:repo:create
  - id: setup-pipeline
    action: github:actions:create
  - id: provision-k8s
    action: argocd:create-app

Benefits

• Setup time — Days to minutes (40% reduction in tickets)
• Consistency — Standardized patterns across teams
• Security — Policies enforced at platform level
• Autonomy — Self-service without DevOps bottleneck

---

Security Scanning (SAST/DAST/SCA)

Testing Types

Complete Pipeline Integration

# GitHub Actions security workflow
name: Security Scan
on: [push, pull_request]

jobs:
  sast:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: github/codeql-action/init@v3
      with:
        languages: javascript, python
    - uses: github/codeql-action/analyze@v3

  sca:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Run Trivy SCA
      uses: aquasecurity/trivy-action@master
      with:
        scan-type: 'fs'
        severity: 'CRITICAL,HIGH'

  secrets:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0  # Full history
    - uses: gitleaks/gitleaks-action@v2

  container:
    runs-on: ubuntu-latest
    steps:
    - name: Build image
      run: docker build -t myapp:${{ github.sha }} .
    - name: Scan image
      uses: aquasecurity/trivy-action@master
      with:
        image-ref: myapp:${{ github.sha }}
        severity: 'CRITICAL,HIGH'
        exit-code: 1  # Fail on vulnerabilities

Runtime Security (Falco)

# Detect suspicious container activity
- rule: Shell in Container
  desc: Unexpected shell execution in container
  condition: >
    spawned_process and
    container and
    proc.name in (bash, sh, zsh)
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name
    command=%proc.cmdline)
  priority: WARNING

---

Metrics and Observability

DORA Metrics (2025 Benchmarks)

Key Metrics to Track

# Deployment metrics
deployment.frequency: counter
deployment.duration: histogram
deployment.rollback: counter

# Pipeline metrics
pipeline.success_rate: gauge
pipeline.duration: histogram
pipeline.queue_time: histogram

# Feature flag metrics
feature_flag.evaluation: counter
feature_flag.enabled_users: gauge
feature_flag.error_rate: gauge (by flag)

# Resource metrics
pod.cpu_usage: gauge
pod.memory_usage: gauge
pod.restart_count: counter

---

Checklist

## CI/CD Pipeline
- [ ] Short-lived credentials (OIDC, not static keys)
- [ ] Protected branches for production
- [ ] Parallel jobs for speed
- [ ] Dependency caching configured
- [ ] Build completes in < 10 minutes
- [ ] Security scanning (SAST, SCA, secrets)

## Containers
- [ ] Multi-stage Dockerfile
- [ ] Non-root user (UID > 1000)
- [ ] Minimal base image (alpine/distroless)
- [ ] .dockerignore configured
- [ ] Image scanning in CI
- [ ] Resource limits defined

## Kubernetes
- [ ] Resource requests/limits set
- [ ] Liveness and readiness probes
- [ ] Security context (runAsNonRoot)
- [ ] Network policies defined
- [ ] ConfigMaps/Secrets for config
- [ ] Deployment strategy chosen
- [ ] Image pull policy configured

## Infrastructure as Code
- [ ] Remote state with locking
- [ ] Modular architecture
- [ ] Policy as Code enforcement
- [ ] Automated tests (Terratest/Pulumi tests)
- [ ] Version pinning for providers
- [ ] Environment parity

## Deployments
- [ ] Deployment strategy selected
- [ ] Rollback plan documented
- [ ] Feature flags for large changes
- [ ] Gradual rollout configured
- [ ] Metrics tied to deployments
- [ ] Automated rollback on errors

## Security
- [ ] SAST in pipeline
- [ ] SCA for dependencies
- [ ] Secret scanning enabled
- [ ] Container vulnerability scanning
- [ ] Runtime security monitoring
- [ ] Supply chain security (signed images)

## Observability
- [ ] Deployment frequency tracked
- [ ] Lead time measured
- [ ] MTTR tracked
- [ ] Change failure rate monitored
- [ ] Feature flag metrics
- [ ] Resource utilization dashboards

---

See Also

• reference/cicd.md — CI/CD pipeline patterns and examples
• reference/containers.md — Docker and Kubernetes deep dive
• reference/release-strategies.md — Deployment patterns comparison
• templates/github-actions.yaml — Production-ready workflow
• templates/Dockerfile — Secure multi-stage Dockerfile

Release Strategies and Deployment Patterns

Overview

Key Statistics (2025):

• 80% of Kubernetes outages stem from deployment errors
• Elite performers deploy multiple times per day with 0-15% failure rate
• 65% of organizations with real-time monitoring report faster incident resolution

Strategy Comparison Matrix

---

Rolling Deployment

How It Works

Gradually replace old version pods with new version, maintaining minimum availability.

Initial:  [v1] [v1] [v1] [v1] [v1] [v1] [v1] [v1] [v1] [v1]  (10 pods)

Step 1:   [v1] [v1] [v1] [v1] [v1] [v1] [v1] [v1] [v2] [v2]  (create 2 v2)
Step 2:   [v1] [v1] [v1] [v1] [v1] [v1] [v1] [v2] [v2]       (remove 1 v1)
Step 3:   [v1] [v1] [v1] [v1] [v1] [v1] [v2] [v2] [v2]       (repeat...)
...
Final:    [v2] [v2] [v2] [v2] [v2] [v2] [v2] [v2] [v2] [v2]  (all v2)

Kubernetes Implementation

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  replicas: 10

  # Rolling update strategy
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1  # Max 1 pod down at a time (90% availability)
      maxSurge: 2        # Max 2 extra pods during rollout

  selector:
    matchLabels:
      app: myapp

  template:
    metadata:
      labels:
        app: myapp
        version: v2.0
    spec:
      containers:
      - name: app
        image: myapp:v2.0
        ports:
        - containerPort: 8080

        # Health checks ensure new pods are healthy before old ones removed
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10

Deployment Process

# 1. Deploy new version
kubectl apply -f deployment.yaml
# or
kubectl set image deployment/myapp app=myapp:v2.0

# 2. Monitor rollout
kubectl rollout status deployment/myapp
# Output: Waiting for deployment "myapp" rollout to finish: 3 of 10 updated replicas are available...

# 3. Watch pods being replaced
kubectl get pods -w -l app=myapp

# 4. Pause rollout for manual verification (optional)
kubectl rollout pause deployment/myapp

# Verify new version is working correctly
curl http://myapp-v2-pod-ip:8080/health

# 5. Resume rollout
kubectl rollout resume deployment/myapp

# 6. Rollback if issues detected
kubectl rollout undo deployment/myapp

# Rollback to specific revision
kubectl rollout history deployment/myapp
kubectl rollout undo deployment/myapp --to-revision=3

Pros and Cons

Advantages:

• No extra infrastructure required
• Built into Kubernetes (no additional tools)
• Gradual rollout reduces blast radius
• Automatic rollback on health check failures
• Works with autoscaling

Disadvantages:

• Both versions running simultaneously (compatibility required)
• Slower rollback compared to blue-green
• Can't test full new version in isolation
• Database schema changes tricky (must be backward compatible)

When to Use:

• Regular incremental updates
• Cost-conscious environments
• Applications supporting multiple concurrent versions
• Low-to-medium risk changes

---

Blue-Green Deployment

How It Works

Maintain two identical production environments. Route all traffic to one (Blue), deploy to the other (Green), then swap.

Initial State:
  [Load Balancer] → Blue Environment (v1.0) [LIVE]
                    Green Environment (idle)

Deploy to Green:
  [Load Balancer] → Blue Environment (v1.0) [LIVE]
                    Green Environment (v2.0) [TESTING]

Switch Traffic:
  [Load Balancer] → Green Environment (v2.0) [LIVE]
                    Blue Environment (v1.0) [STANDBY - instant rollback]

After Validation:
  [Load Balancer] → Green Environment (v2.0) [LIVE]
                    Blue Environment (v1.0) → becomes new Green (idle)

Kubernetes Implementation

Method 1: Service Selector Switching

# ============================================
# Blue Deployment (Current Production)
# ============================================
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-blue
spec:
  replicas: 10
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: app
        image: myapp:v1.0
        ports:
        - containerPort: 8080

---
# ============================================
# Green Deployment (New Version)
# ============================================
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-green
spec:
  replicas: 10
  selector:
    matchLabels:
      app: myapp
      version: green
  template:
    metadata:
      labels:
        app: myapp
        version: green
    spec:
      containers:
      - name: app
        image: myapp:v2.0
        ports:
        - containerPort: 8080

---
# ============================================
# Service (Points to Blue Initially)
# ============================================
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
    version: blue  # CHANGE TO 'green' TO SWITCH
  ports:
  - port: 80
    targetPort: 8080
  type: LoadBalancer

Switching Process:

# 1. Deploy green environment
kubectl apply -f myapp-green-deployment.yaml

# 2. Wait for green pods to be ready
kubectl wait --for=condition=ready pod -l version=green --timeout=300s

# 3. Test green environment directly (before switching traffic)
kubectl port-forward deployment/myapp-green 8080:8080
curl http://localhost:8080/health

# 4. Switch service to green
kubectl patch service myapp -p '{"spec":{"selector":{"version":"green"}}}'

# 5. Monitor for issues (keep blue running)
# If problems detected:
kubectl patch service myapp -p '{"spec":{"selector":{"version":"blue"}}}'  # Instant rollback!

# 6. After validation, scale down blue
kubectl scale deployment myapp-blue --replicas=0

Method 2: Ingress/Route Switching

# ============================================
# Blue Service
# ============================================
apiVersion: v1
kind: Service
metadata:
  name: myapp-blue
spec:
  selector:
    app: myapp
    version: blue
  ports:
  - port: 80
    targetPort: 8080

---
# ============================================
# Green Service
# ============================================
apiVersion: v1
kind: Service
metadata:
  name: myapp-green
spec:
  selector:
    app: myapp
    version: green
  ports:
  - port: 80
    targetPort: 8080

---
# ============================================
# Ingress (Switch backend)
# ============================================
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
spec:
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-blue  # CHANGE TO myapp-green TO SWITCH
            port:
              number: 80

AWS/Cloud Implementation

AWS with Application Load Balancer:

# 1. Create two target groups (blue and green)
aws elbv2 create-target-group --name myapp-blue-tg ...
aws elbv2 create-target-group --name myapp-green-tg ...

# 2. Deploy to green target group
# (Update Auto Scaling Group or ECS service)

# 3. Switch ALB listener to green target group
aws elbv2 modify-listener \
  --listener-arn arn:aws:elasticloadbalancing:... \
  --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:.../myapp-green-tg

# 4. Instant rollback if needed
aws elbv2 modify-listener ... TargetGroupArn=.../myapp-blue-tg

Database Considerations

-- Problem: Blue uses schema v1, Green uses schema v2
-- Solution: Make migrations backward compatible

-- GOOD: Additive changes (v2 adds column, v1 ignores it)
ALTER TABLE users ADD COLUMN phone VARCHAR(20);

-- BAD: Breaking changes
ALTER TABLE users DROP COLUMN email;  -- v1 will fail!

-- Pattern: Multi-phase migration
-- Phase 1: Add new column (both versions work)
ALTER TABLE users ADD COLUMN email_v2 VARCHAR(255);

-- Phase 2: Dual-write in code (write to both email and email_v2)

-- Phase 3: Backfill data
UPDATE users SET email_v2 = email WHERE email_v2 IS NULL;

-- Phase 4: Switch reads to email_v2

-- Phase 5: Drop old column (after both environments use email_v2)
ALTER TABLE users DROP COLUMN email;

Pros and Cons

Advantages:

• Zero downtime
• Instant rollback (just switch back)
• Full testing of new environment before cutover
• Clean separation between versions
• Database can be tested in isolation

Disadvantages:

• Requires 2x infrastructure (expensive)
• Database migrations complex
• Syncing state between environments tricky
• Session/state management challenges

When to Use:

• Critical systems requiring instant rollback
• High-compliance environments
• Major version upgrades
• Budget allows for duplicate infrastructure

---

Canary Deployment

How It Works

Gradually shift traffic from old version to new version: 5% → 25% → 50% → 100%.

Step 1: Deploy canary (5% traffic)
  95% traffic → v1.0 [9 pods]
   5% traffic → v2.0 [1 pod]  ← Monitor metrics

Step 2: Increase to 25% (if metrics good)
  75% traffic → v1.0 [7 pods]
  25% traffic → v2.0 [3 pods]

Step 3: Increase to 50%
  50% traffic → v1.0 [5 pods]
  50% traffic → v2.0 [5 pods]

Step 4: Promote to 100%
 100% traffic → v2.0 [10 pods]
   0% traffic → v1.0 [removed]

Implementation with Argo Rollouts

# Install Argo Rollouts
# kubectl create namespace argo-rollouts
# kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 10

  strategy:
    canary:
      # Canary steps
      steps:
      - setWeight: 10        # Send 10% traffic to canary
      - pause: {duration: 5m}  # Wait 5 minutes

      - setWeight: 25        # Increase to 25%
      - pause: {duration: 10m}

      - setWeight: 50        # Increase to 50%
      - pause: {duration: 10m}

      - setWeight: 75        # Increase to 75%
      - pause: {duration: 5m}

      # If we reach here, promote to 100%

      # Optional: Analysis template (automated promotion/rollback)
      analysis:
        templates:
        - templateName: success-rate
        startingStep: 1  # Run analysis after first step
        args:
        - name: service-name
          value: myapp

      # Traffic routing (choose one)
      trafficRouting:
        # Option 1: Istio
        istio:
          virtualService:
            name: myapp-vsvc
          destinationRule:
            name: myapp-dest
            canarySubsetName: canary
            stableSubsetName: stable

        # Option 2: NGINX Ingress
        nginx:
          stableIngress: myapp
          annotationPrefix: nginx.ingress.kubernetes.io

  selector:
    matchLabels:
      app: myapp

  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: app
        image: myapp:v2.0
        ports:
        - containerPort: 8080

---
# AnalysisTemplate: Automated success rate check
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
  - name: service-name

  metrics:
  - name: success-rate
    interval: 1m
    successCondition: result >= 0.95  # 95% success rate required
    failureLimit: 3  # Rollback after 3 failed checks
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(http_requests_total{
            service="{{args.service-name}}",
            status=~"2.."
          }[1m]))
          /
          sum(rate(http_requests_total{
            service="{{args.service-name}}"
          }[1m]))

Canary Commands

# Deploy canary
kubectl apply -f rollout.yaml

# Watch rollout progress
kubectl argo rollouts get rollout myapp --watch

# Promote to next step (if paused)
kubectl argo rollouts promote myapp

# Abort and rollback
kubectl argo rollouts abort myapp
kubectl argo rollouts undo myapp

# Set image
kubectl argo rollouts set image myapp app=myapp:v2.1

Manual Canary with Kubernetes

# ============================================
# Stable Deployment (90% of pods)
# ============================================
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-stable
spec:
  replicas: 9
  selector:
    matchLabels:
      app: myapp
      track: stable
  template:
    metadata:
      labels:
        app: myapp
        track: stable
        version: v1.0
    spec:
      containers:
      - name: app
        image: myapp:v1.0

---
# ============================================
# Canary Deployment (10% of pods)
# ============================================
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
      track: canary
  template:
    metadata:
      labels:
        app: myapp
        track: canary
        version: v2.0
    spec:
      containers:
      - name: app
        image: myapp:v2.0

---
# ============================================
# Service (routes to both stable and canary)
# ============================================
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp  # Matches both stable and canary
  ports:
  - port: 80
    targetPort: 8080

Gradually increase canary:

# 10% canary (1/10 pods)
kubectl scale deployment myapp-canary --replicas=1

# 25% canary (3/12 pods)
kubectl scale deployment myapp-stable --replicas=9
kubectl scale deployment myapp-canary --replicas=3

# 50% canary
kubectl scale deployment myapp-stable --replicas=5
kubectl scale deployment myapp-canary --replicas=5

# Promote (100% new version)
kubectl scale deployment myapp-stable --replicas=10
kubectl patch deployment myapp-stable -p '{"spec":{"template":{"spec":{"containers":[{"name":"app","image":"myapp:v2.0"}]}}}}'
kubectl scale deployment myapp-canary --replicas=0

Metrics-Based Automated Rollback

# Flagger (progressive delivery operator)
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp

  service:
    port: 80
    targetPort: 8080

  analysis:
    interval: 1m
    threshold: 5  # Number of checks
    maxWeight: 50
    stepWeight: 10

    # Metrics for automated decision
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99  # Rollback if < 99% success
      interval: 1m

    - name: request-duration
      thresholdRange:
        max: 500  # Rollback if p99 latency > 500ms
      interval: 1m

    # Alert on canary failure
    webhooks:
    - name: slack-alert
      url: https://hooks.slack.com/services/YOUR/WEBHOOK
      type: pre-rollout

Pros and Cons

Advantages:

• Lowest risk deployment strategy
• Gradual rollout limits blast radius
• Real production data for validation
• Automated rollback based on metrics
• Cost-effective (no duplicate infrastructure)

Disadvantages:

• Complex setup (requires service mesh or ingress controller)
• Longer deployment time
• Need robust monitoring and metrics
• Difficult to test all code paths with limited traffic

When to Use:

• High-risk changes (major refactors, new features)
• Customer-facing applications
• When you have strong observability
• Cost-conscious but need safety

---

A/B Testing

How It Works

Route traffic based on user attributes (not just percentage). Test different features with different user segments.

User Segment A (50%) → Version A (existing feature)
User Segment B (50%) → Version B (new feature)

Measure: conversion rate, engagement, revenue

Implementation

# Using Istio VirtualService for A/B testing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - myapp.example.com
  http:
  - match:
    # Route users with 'premium' cookie to version B
    - headers:
        cookie:
          regex: ".*tier=premium.*"
    route:
    - destination:
        host: myapp
        subset: version-b

  - match:
    # Route mobile users to version A
    - headers:
        user-agent:
          regex: ".*Mobile.*"
    route:
    - destination:
        host: myapp
        subset: version-a

  # Default: 50/50 split
  - route:
    - destination:
        host: myapp
        subset: version-a
      weight: 50
    - destination:
        host: myapp
        subset: version-b
      weight: 50

Feature Flags for A/B Testing

// LaunchDarkly example
import { LDClient } from 'launchdarkly-node-server-sdk';

const ldClient = LDClient.init(process.env.LD_SDK_KEY);

app.get('/checkout', async (req, res) => {
  const user = {
    key: req.user.id,
    email: req.user.email,
    custom: {
      tier: req.user.tier,  // 'free' or 'premium'
      country: req.user.country
    }
  };

  // A/B test: new checkout flow
  const showNewCheckout = await ldClient.variation(
    'new-checkout-ui',
    user,
    false  // Default
  );

  if (showNewCheckout) {
    res.render('checkout-v2', { user });
  } else {
    res.render('checkout-v1', { user });
  }

  // Track metrics
  analytics.track('checkout_viewed', {
    userId: user.key,
    variant: showNewCheckout ? 'new' : 'old'
  });
});

LaunchDarkly Configuration:

{
  "name": "new-checkout-ui",
  "kind": "boolean",
  "targeting": {
    "rules": [
      {
        "variation": 1,
        "clauses": [
          {
            "attribute": "tier",
            "op": "in",
            "values": ["premium"]
          }
        ]
      },
      {
        "variation": 1,
        "clauses": [
          {
            "attribute": "country",
            "op": "in",
            "values": ["US", "CA"]
          }
        ]
      }
    ],
    "fallthrough": {
      "rollout": {
        "variations": [
          {"variation": 0, "weight": 50000},
          {"variation": 1, "weight": 50000}
        ]
      }
    }
  }
}

---

Shadow Deployment (Dark Launch)

How It Works

Send live traffic to new version WITHOUT exposing results to users. Compare responses/metrics.

Production Traffic → Primary (v1.0) → Response to User
                   ↘ Shadow (v2.0)  → Logged (not returned)

Compare: latency, errors, response differences

Implementation with Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - myapp
  http:
  - route:
    - destination:
        host: myapp
        subset: v1  # Primary version
      weight: 100

    mirror:
      host: myapp
      subset: v2  # Shadow version (receives copy of traffic)
    mirrorPercentage:
      value: 100  # Mirror 100% of traffic

Use Cases:

• Performance testing with real traffic
• Validating new algorithms/models
• Testing infrastructure changes
• Comparing response times/resource usage

---

Choosing the Right Strategy

Decision Tree

Is this a critical production system?
├─ YES → Budget for 2x infrastructure?
│        ├─ YES → Blue-Green (instant rollback)
│        └─ NO  → Canary (gradual, metrics-based)
└─ NO  → Risk level?
         ├─ HIGH → Canary with feature flags
         ├─ MEDIUM → Rolling update
         └─ LOW → Rolling update or Recreate (dev/test)

Need to test business hypotheses?
└─ A/B Testing with feature flags

Need to validate performance before launch?
└─ Shadow deployment

By Use Case

---

Best Practices

1. Always Have a Rollback Plan

# Document rollback procedure
# Example:
# Rollback for Blue-Green:
kubectl patch service myapp -p '{"spec":{"selector":{"version":"blue"}}}'

# Rollback for Rolling:
kubectl rollout undo deployment/myapp

# Rollback for Canary (Argo Rollouts):
kubectl argo rollouts abort myapp
kubectl argo rollouts undo myapp

2. Monitor Key Metrics

# Essential metrics to track during deployment
- Error rate (4xx, 5xx responses)
- Latency (p50, p95, p99)
- Request rate
- Resource usage (CPU, memory)
- Business metrics (conversion, signups)

3. Automated Rollback Triggers

# Set thresholds for auto-rollback
error_rate_threshold: 5%  # Rollback if > 5% errors
latency_p99_threshold: 1000ms  # Rollback if p99 > 1s
availability_threshold: 99.9%  # Rollback if < 99.9% uptime

4. Feature Flags for Large Changes

// Decouple deployment from release
if (featureFlags.isEnabled('new-payment-flow', user)) {
  return newPaymentFlow(req);
} else {
  return oldPaymentFlow(req);
}

// Deploy with flag OFF → test in production → enable for 5% → ramp to 100%

5. Database Migration Strategy

-- Expand-Contract Pattern
-- Phase 1: EXPAND (add new schema, both versions work)
ALTER TABLE users ADD COLUMN phone_number VARCHAR(20);

-- Phase 2: Dual-write (application writes to both old and new)

-- Phase 3: Backfill data

-- Phase 4: CONTRACT (remove old schema after all apps migrated)
ALTER TABLE users DROP COLUMN phone;

---

Common Pitfalls

1. Stateful Apps Without Session Management

• Problem: User session on v1, next request hits v2
• Solution: Sticky sessions or external session store (Redis)

2. Database Compatibility Issues

• Problem: v2 schema breaks v1
• Solution: Backward-compatible migrations, expand-contract pattern

3. No Rollback Plan

• Problem: Deployment fails, no way to recover quickly
• Solution: Document and test rollback before deploying

4. Insufficient Monitoring

• Problem: Issues not detected during canary phase
• Solution: Monitor error rates, latency, business metrics

5. Too Aggressive Canary Ramp

• Problem: Jump from 10% to 100% too quickly
• Solution: Gradual steps (10% → 25% → 50% → 75% → 100%)

6. Ignoring Non-Functional Changes

• Problem: "Just a config change" causes outage
• Solution: Treat all changes with same rigor