Devops Engineer

DevOps Engineer

Senior DevOps engineer specializing in CI/CD pipelines, infrastructure as code, and deployment automation.

Role Definition

You are a senior DevOps engineer with 10+ years of experience. You operate with three perspectives:

• Build Hat: Automating build, test, and packaging
• Deploy Hat: Orchestrating deployments across environments
• Ops Hat: Ensuring reliability, monitoring, and incident response

When to Use This Skill

• Setting up CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins)
• Containerizing applications (Docker, Docker Compose)
• Kubernetes deployments and configurations
• Infrastructure as code (Terraform, Pulumi)
• Cloud platform configuration (AWS, GCP, Azure)
• Deployment strategies (blue-green, canary, rolling)
• Building internal developer platforms and self-service tools
• Incident response, on-call, and production troubleshooting
• Release automation and artifact management

Core Workflow

1. Assess - Understand application, environments, requirements 2. Design - Pipeline structure, deployment strategy 3. Implement - IaC, Dockerfiles, CI/CD configs 4. Validate - Run terraform plan, lint configs, execute unit/integration tests; confirm no destructive changes before proceeding 5. Deploy - Roll out with verification; run smoke tests post-deployment 6. Monitor - Set up observability, alerts; confirm rollback procedure is ready before going live

Reference Guide

Load detailed guidance based on context:

Constraints

MUST DO

• Use infrastructure as code (never manual changes)
• Implement health checks and readiness probes
• Store secrets in secret managers (not env files)
• Enable container scanning in CI/CD
• Document rollback procedures
• Use GitOps for Kubernetes (ArgoCD, Flux)

MUST NOT DO

• Deploy to production without explicit approval
• Store secrets in code or CI/CD variables
• Skip staging environment testing
• Ignore resource limits in containers
• Use latest tag in production
• Deploy on Fridays without monitoring

Output Templates

Provide: CI/CD pipeline config, Dockerfile, K8s/Terraform files, deployment verification, rollback procedure

Minimal GitHub Actions Example

name: CI
on:
  push:
    branches: [main]
jobs:
  build-test-push:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .
      - name: Run tests
        run: docker run --rm myapp:${{ github.sha }} pytest
      - name: Scan image
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:${{ github.sha }}
      - name: Push to registry
        run: |
          docker tag myapp:${{ github.sha }} ghcr.io/org/myapp:${{ github.sha }}
          docker push ghcr.io/org/myapp:${{ github.sha }}

Minimal Dockerfile Example

FROM python:3.12-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.12-slim
WORKDIR /app
COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
COPY . .
USER nonroot
HEALTHCHECK --interval=30s --timeout=5s CMD curl -f http://localhost:8080/health || exit 1
CMD ["python", "main.py"]

Rollback Procedure Example

# Kubernetes: roll back to previous deployment revision
kubectl rollout undo deployment/myapp -n production
kubectl rollout status deployment/myapp -n production

# Verify rollback succeeded
kubectl get pods -n production -l app=myapp
curl -f https://myapp.example.com/health

Always document the rollback command and verification step in the PR or change ticket before deploying.

Knowledge Reference

GitHub Actions, GitLab CI, Jenkins, CircleCI, Docker, Kubernetes, Helm, ArgoCD, Flux, Terraform, Pulumi, Crossplane, AWS/GCP/Azure, Prometheus, Grafana, PagerDuty, Backstage, LaunchDarkly, Flagger

Deployment Strategies

Strategy Comparison

Rolling Deployment (Kubernetes)

apiVersion: apps/v1
kind: Deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25% # Max pods above desired
      maxUnavailable: 25% # Max pods unavailable

Blue-Green with Ingress

# Blue deployment (current)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-blue
  labels:
    version: blue
---
# Green deployment (new)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-green
  labels:
    version: green
---
# Service pointing to active version
apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  selector:
    version: blue # Switch to 'green' for cutover

Canary with Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app
spec:
  hosts:
    - app
  http:
    - match:
        - headers:
            canary:
              exact: "true"
      route:
        - destination:
            host: app-canary
    - route:
        - destination:
            host: app-stable
          weight: 90
        - destination:
            host: app-canary
          weight: 10

Rollback Procedures

Kubernetes Rollback

# View rollout history
kubectl rollout history deployment/app

# Rollback to previous
kubectl rollout undo deployment/app

# Rollback to specific revision
kubectl rollout undo deployment/app --to-revision=2

# Check status
kubectl rollout status deployment/app

ArgoCD Rollback

argocd app rollback app-prod --revision=123

Terraform Rollback

# Identify previous state
terraform state list

# Import previous configuration
git checkout HEAD~1 -- main.tf
terraform apply

Pre-deployment Checklist

• [ ] Database migrations are backward compatible
• [ ] Feature flags for new functionality
• [ ] Monitoring dashboards updated
• [ ] Alert thresholds reviewed
• [ ] Rollback procedure documented
• [ ] Staging tested and approved
• [ ] Team notified of deployment window

Post-deployment Verification

# Check pod status
kubectl get pods -l app=app

# Check logs for errors
kubectl logs -l app=app --tail=100 | grep -i error

# Verify endpoints
curl -f https://app.example.com/health

# Check metrics
# - Error rate < 1%
# - Latency p99 < 500ms
# - No memory/CPU spikes

Deployment Metrics (DORA)

Track four key metrics:

• Deployment Frequency: Target 10+/day
• Lead Time for Changes: Target <1 hour
• Change Failure Rate: Target <5%
• MTTR: Target <30 minutes

# Prometheus metrics for DORA tracking
- record: deployment:frequency:1d
  expr: count_over_time(deployment_completed[1d])

- record: deployment:lead_time:p95
  expr: histogram_quantile(0.95,
    rate(commit_to_deploy_seconds_bucket[1h]))

- record: deployment:failure_rate
  expr: |
    sum(rate(deployment_failed[1h]))
    / sum(rate(deployment_total[1h]))

Advanced Canary with Automated Analysis

# Flagger: Automated canary with rollback
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: api
spec:
  provider: istio
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  progressDeadlineSeconds: 60
  service:
    port: 8080
    trafficPolicy:
      tls:
        mode: ISTIO_MUTUAL
  analysis:
    interval: 30s
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: error-rate
        templateRef:
          name: error-rate
        thresholdRange:
          max: 1
      - name: latency
        templateRef:
          name: latency
        thresholdRange:
          max: 500
    webhooks:
      - name: acceptance-test
        type: pre-rollout
        url: http://test-runner/
      - name: load-test
        url: http://loadtester/
        timeout: 5s
        metadata:
          type: bash
          cmd: "hey -z 1m -q 10 http://api-canary:8080/"

Shadow Deployment

# Mirror traffic to shadow deployment
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api
spec:
  hosts:
    - api
  http:
    - match:
        - headers:
            x-test-version:
              exact: "v2"
      route:
        - destination:
            host: api
            subset: v2
      mirror:
        host: api
        subset: v2-shadow
      mirrorPercentage:
        value: 100
    - route:
        - destination:
            host: api
            subset: v1

Docker Patterns

Multi-stage Dockerfile (Node.js)

# Build stage
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production && npm cache clean --force
COPY . .
RUN npm run build

# Production stage
FROM node:20-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
RUN addgroup -g 1001 -S nodejs && adduser -S nodejs -u 1001
COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist
COPY --from=builder --chown=nodejs:nodejs /app/node_modules ./node_modules
USER nodejs
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s \
  CMD wget -qO- http://localhost:3000/health || exit 1
CMD ["node", "dist/main.js"]

Multi-stage Dockerfile (Python)

FROM python:3.12-slim AS builder
WORKDIR /app
RUN pip install --no-cache-dir poetry
COPY pyproject.toml poetry.lock ./
RUN poetry export -f requirements.txt --output requirements.txt
RUN pip wheel --no-cache-dir --wheel-dir /wheels -r requirements.txt

FROM python:3.12-slim AS runner
WORKDIR /app
RUN useradd -m -u 1001 appuser
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir /wheels/*
COPY --chown=appuser:appuser . .
USER appuser
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=3s \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Docker Compose (Development)

version: "3.8"
services:
  app:
    build:
      context: .
      target: builder # Use dev stage
    volumes:
      - .:/app
      - /app/node_modules
    ports:
      - "3000:3000"
    environment:
      - DATABASE_URL=postgres://user:pass@db:5432/app
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
      POSTGRES_DB: app
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user -d app"]
      interval: 5s
      timeout: 5s
      retries: 5

volumes:
  postgres_data:

Security Best Practices

.dockerignore Template

node_modules
.git
.env*
*.md
Dockerfile*
docker-compose*
.dockerignore
coverage
.nyc_output

GitHub Actions Pipelines

Complete CI/CD Pipeline

name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
      - run: npm ci
      - run: npm test
      - run: npm run lint

  build:
    needs: test
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    outputs:
      image-tag: ${{ steps.meta.outputs.tags }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=sha,prefix=
            type=ref,event=branch
      - uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  deploy-staging:
    needs: build
    if: github.ref == 'refs/heads/develop'
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - run: |
          kubectl set image deployment/app app=${{ needs.build.outputs.image-tag }}

  deploy-production:
    needs: build
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - run: |
          kubectl set image deployment/app app=${{ needs.build.outputs.image-tag }}

Common Workflow Patterns

Matrix Builds (Multi-version testing)

jobs:
  test:
    strategy:
      matrix:
        node-version: [18, 20, 22]
        os: [ubuntu-latest, macos-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}

Reusable Workflows

# .github/workflows/deploy.yml
on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
    secrets:
      DEPLOY_KEY:
        required: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    steps:
      - run: echo "Deploying to ${{ inputs.environment }}"

Caching Dependencies

- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

Quick Reference

Incident Response

Response Metrics

• MTTD (Mean Time to Detect): Target < 5 minutes
• MTTA (Mean Time to Acknowledge): Target < 5 minutes
• MTTR (Mean Time to Resolve): Target < 30 minutes
• MTBF (Mean Time Between Failures): Maximize

Severity Levels

Runbook Template

# Runbook: High API Error Rate

## Symptoms

- Alert: `api_error_rate > 0.05`
- Dashboard: https://grafana.example.com/d/api-errors

## Impact

Users cannot complete purchases (~$X per minute)

## Triage

1. Check dashboard for affected endpoints
2. Check recent deployments: `kubectl rollout history deployment/api`
3. Check dependencies: database, redis, external APIs

## Resolution

### Option 1: Rollback

kubectl rollout undo deployment/api -n production

### Option 2: Scale Up

kubectl scale deployment/api --replicas=10 -n production

### Option 3: Fix Config

kubectl set env deployment/api DB_POOL_SIZE=50 -n production

## Verification

- [ ] Error rate <1%
- [ ] P95 latency <500ms
- [ ] Health checks passing

## Communication

- Update status page
- Notify #incidents
- Post if user-facing

Auto-Remediation Script

#!/usr/bin/env python3
import kubernetes, prometheus_api_client

class IncidentRemediator:
    def check_high_error_rate(self):
        query = 'rate(http_requests_total{status=~"5.."}[5m]) > 0.05'
        result = self.prometheus.custom_query(query)
        return len(result) > 0

    def rollback_deployment(self, namespace, deployment):
        body = {'spec': {'rollbackTo': {'revision': 0}}}
        self.k8s.patch_namespaced_deployment(deployment, namespace, body)

    def remediate(self):
        if self.check_high_error_rate():
            if self.rollback_deployment('production', 'api'):
                time.sleep(120)
                if not self.check_high_error_rate():
                    return  # Success
            # Escalate if remediation fails
            self.create_incident("Auto-remediation failed")

Postmortem Template

# Postmortem: API Outage - 2024-01-15

**Date**: January 15, 2024
**Duration**: 45 minutes (14:23 - 15:08 UTC)
**Severity**: SEV1
**Impact**: Complete API outage, ~$25K revenue loss

## Summary

API became unresponsive due to database connection pool exhaustion
from slow query in v2.3.1.

## Timeline (UTC)

- 14:23 - Alert fired
- 14:27 - Incident declared SEV1
- 14:30 - Rollback initiated
- 14:45 - Identified slow query
- 14:50 - Killed queries
- 15:08 - Resolved

## Root Cause

New query missing index on `user_id`, causing full table scans that
exhausted connection pool under load.

## Impact

- 100% API failure for 45 minutes
- 15,000 users affected
- $25K revenue loss
- 200+ support tickets

## Action Items

| Action                   | Owner    | Deadline   |
| ------------------------ | -------- | ---------- |
| Add index on user_id     | DB team  | 2024-01-16 |
| Add query perf testing   | Platform | 2024-01-22 |
| Increase staging DB size | Infra    | 2024-01-30 |

## Lessons Learned

- Performance testing must use production-scale data
- Connection pool exhaustion needs active intervention
- Consider circuit breakers for DB operations

PagerDuty Configuration

schedules:
  - name: Primary On-Call
    time_zone: America/New_York
    layers:
      - rotation_turn_length_seconds: 604800 # 1 week
        users: [PXXXXXX, PXXXXXX, PXXXXXX]

escalation_policies:
  - name: Production
    rules:
      - escalation_delay_in_minutes: 0
        targets: [{ type: schedule, id: primary }]
      - escalation_delay_in_minutes: 15
        targets: [{ type: schedule, id: secondary }]
      - escalation_delay_in_minutes: 30
        targets: [{ type: user, id: manager }]

Chaos Engineering

# chaos-mesh: Pod failure test
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-failure-test
spec:
  action: pod-failure
  mode: one
  duration: "30s"
  selector:
    namespaces: [production]
    labelSelectors:
      app: api
  scheduler:
    cron: "@every 2h"

#!/bin/bash
# Game Day: Database failover drill

echo "🎮 Game Day: Database failover"
slack-cli -d incidents "Starting failover drill"

# Simulate failure
kubectl delete pod postgres-0 -n production

# Monitor recovery
start=$(date +%s)
while ! kubectl get pod postgres-1 | grep Running; do
  sleep 5
done
duration=$(($(date +%s) - start))

echo "Failover: ${duration}s" >> results.md
curl -f https://api.example.com/health || echo "FAIL"

Evidence Collection & Forensics

#!/bin/bash
# collect-evidence.sh - Preserve incident evidence

INCIDENT_ID=$1
EVIDENCE_DIR="incidents/${INCIDENT_ID}/evidence"
mkdir -p $EVIDENCE_DIR

# Preserve logs
kubectl logs -l app=api --all-containers --timestamps \
  --since=2h > $EVIDENCE_DIR/pod-logs.txt

# Capture current state
kubectl get all -n production -o yaml > $EVIDENCE_DIR/k8s-state.yaml
kubectl describe pods -n production > $EVIDENCE_DIR/pod-details.txt

# Network traces
kubectl exec -n production deploy/api -- \
  tcpdump -i any -w /tmp/capture.pcap -G 60 -W 5 &

# Memory/CPU snapshot
kubectl top pods -n production > $EVIDENCE_DIR/resource-usage.txt

# Git commit at time of incident
git log --since="2 hours ago" --oneline > $EVIDENCE_DIR/recent-commits.txt

# Database queries
psql -c "SELECT * FROM pg_stat_activity" > $EVIDENCE_DIR/db-activity.txt

# Create timeline
echo "$(date): Evidence collection completed" >> $EVIDENCE_DIR/timeline.txt

Communication Templates

## SEV1 Initial Notification

**INCIDENT ALERT - SEV1**

**Status**: Investigating
**Impact**: Payment API unavailable (100% error rate)
**Started**: 2024-01-15 14:23 UTC
**Affected**: All users (~15K active sessions)
**Lead**: @oncall-engineer
**War Room**: https://zoom.us/incident-123

Updates every 15 minutes or on major change.

---

## SEV1 Resolution Notification

**INCIDENT RESOLVED**

**Summary**: Payment API restored after database connection pool exhaustion
**Duration**: 45 minutes (14:23 - 15:08 UTC)
**Resolution**: Rollback to v2.3.0 + query optimization
**Impact**: 15K users, ~$25K revenue loss
**Next Steps**: Postmortem scheduled for Jan 16 10am

Thanks to @oncall-team for rapid response.

Incident Classification

Security Incident Specifics

# Compromise investigation checklist
□ Isolate affected systems
□ Preserve logs and memory dumps
□ Identify attack vector
□ Check for lateral movement
□ Scan for malware/backdoors
□ Review access logs for 30 days
□ Identify data accessed
□ Assess exfiltration risk
□ Check for persistence mechanisms
□ Coordinate with security team
□ Notify legal if PII involved
□ Document chain of custody

Compliance Requirements

# Incident notification requirements
gdpr:
  notification_deadline: 72h
  authority: Data Protection Officer
  required_info:
    - Nature of breach
    - Data categories affected
    - Number of individuals
    - Consequences
    - Remediation measures

sox:
  notification_deadline: immediate
  authority: Audit Committee
  documentation:
    - Financial impact
    - Control failures
    - Remediation plan

pci_dss:
  notification_deadline: 24h
  authority: Card brands + acquirer
  required_info:
    - Cardholder data affected
    - Incident timeline
    - Forensic investigation

Best Practices

• Maintain runbooks for critical services
• Practice with game days monthly
• Automate common remediation
• Keep postmortems blameless
• Track incident metrics
• Test recovery procedures
• Document all incidents
• Improve detection continuously
• Preserve evidence chain properly
• Coordinate communication clearly
• Escalate security incidents immediately
• Understand compliance obligations
• Train team on response procedures
• Review and update playbooks quarterly

Kubernetes Manifests

Complete Deployment Stack

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
        - name: app
          image: ghcr.io/org/app:latest
          ports:
            - containerPort: 3000
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "500m"
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 5
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: database-url
---
apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  selector:
    app: app
  ports:
    - port: 80
      targetPort: 3000
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts: [app.example.com]
      secretName: app-tls
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app
                port:
                  number: 80

ConfigMap and Secrets

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  API_TIMEOUT: "30s"
---
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  database-url: "postgres://user:pass@host:5432/db"

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Quick Reference

Common kubectl Commands

kubectl apply -f deployment.yaml
kubectl get pods -l app=app
kubectl describe pod <pod-name>
kubectl logs -f <pod-name>
kubectl exec -it <pod-name> -- /bin/sh
kubectl rollout status deployment/app
kubectl rollout undo deployment/app

Platform Engineering

Platform Principles

• Self-service first: Reduce manual work to <10%
• Golden paths: Pre-approved, opinionated templates
• Developer experience: Measure and optimize productivity
• Platform as product: Treat with product mindset

Self-Service with Crossplane

# Composition for self-service database
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: postgres-database
spec:
  compositeTypeRef:
    apiVersion: platform.example.com/v1alpha1
    kind: Database
  resources:
    - name: rds-instance
      base:
        apiVersion: rds.aws.crossplane.io/v1alpha1
        kind: DBInstance
        spec:
          forProvider:
            dbInstanceClass: db.t3.micro
            engine: postgres
            engineVersion: "15"
            masterUsername: admin
            allocatedStorage: 20

Terraform Self-Service Module

# modules/service/main.tf
variable "service_name" {}
variable "environment" {}

module "k8s_service" {
  source   = "./k8s-deployment"
  name     = var.service_name
  env      = var.environment
}

module "database" {
  source = "./postgres"
  name   = "${var.service_name}-db"
}

module "monitoring" {
  source  = "./monitoring-stack"
  service = var.service_name
}

output "service_url" {
  value = module.k8s_service.url
}

Backstage Service Template

# templates/microservice/template.yaml
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: microservice-template
  title: Microservice Golden Path
spec:
  owner: platform-team
  type: service
  parameters:
    - title: Service Info
      properties:
        name:
          type: string
        owner:
          type: string
          ui:field: OwnerPicker
        language:
          type: string
          enum: [go, python, nodejs, java]
  steps:
    - id: fetch
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
    - id: publish
      action: publish:github
      input:
        repoUrl: github.com?owner=org&repo=${{ parameters.name }}
    - id: register
      action: catalog:register

Service Catalog Info

# catalog-info.yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-service
  annotations:
    github.com/project-slug: org/payment-service
    pagerduty.com/integration-key: abc123
    grafana/dashboard-selector: service=payment
spec:
  type: service
  lifecycle: production
  owner: payments-team
  system: checkout
  dependsOn:
    - resource:default/payment-db
    - component:default/auth-service
  providesApis:
    - payment-api

Golden Path Scaffolding

#!/bin/bash
# create-service.sh - Golden path for new services

SERVICE=$1
LANG=$2

# Create from template
gh repo create "org/$SERVICE" --template "org/template-$LANG"
git clone "git@github.com:org/$SERVICE.git"
cd "$SERVICE"

# Setup CI/CD
cat > .github/workflows/ci.yml <<EOF
name: CI/CD
on: [push]
jobs:
  pipeline:
    uses: org/workflows/.github/workflows/standard.yml@v1
    with:
      service_name: $SERVICE
EOF

# Create infrastructure
cat > terraform/main.tf <<EOF
module "service" {
  source = "git::https://github.com/org/terraform//service"
  name   = "$SERVICE"
}
EOF

git add . && git commit -m "Golden path init" && git push

echo "✓ Service created! Merge to main to deploy."

GitOps Repository Structure

gitops/
├── apps/
│   ├── production/
│   │   ├── payment-service/
│   │   └── auth-service/
│   └── staging/
│       └── payment-service/
├── infrastructure/
│   ├── clusters/
│   │   ├── prod-us-east/
│   │   └── prod-eu-west/
│   └── base/
│       ├── ingress/
│       └── monitoring/
└── platform/
    ├── backstage/
    ├── argocd/
    └── vault/

ArgoCD Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-service
spec:
  project: default
  source:
    repoURL: https://github.com/org/gitops
    path: apps/production/payment-service
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      limit: 5
      backoff:
        duration: 5s
        maxDuration: 3m

Platform Metrics

# prometheus/platform-metrics.yaml
groups:
  - name: platform
    rules:
      # Self-service adoption rate
      - record: platform:self_service:rate
        expr: |
          sum(rate(platform_provision_automated[1h]))
          /
          sum(rate(platform_provision_total[1h]))

      # Provisioning time P95
      - record: platform:provision:p95
        expr: |
          histogram_quantile(0.95,
            rate(platform_provision_duration_bucket[5m]))

      # Golden path adoption
      - record: platform:golden_path:adoption
        expr: |
          count(service{template="golden-path"})
          / count(service)

Custom Backstage Plugin

// plugins/platform-stats/PlatformMetrics.tsx
import React from 'react';
import { InfoCard, Progress } from '@backstage/core-components';

export const PlatformMetrics = () => {
  const metrics = {
    selfServiceRate: 92,
    avgProvisionTime: '3.5min',
    uptime: '99.95%',
    satisfaction: 4.6
  };

  return (
    <InfoCard title="Platform Health">
      <Progress value={metrics.selfServiceRate} label="Self-Service" />
      <p>Provision Time: {metrics.avgProvisionTime}</p>
      <p>Uptime: {metrics.uptime}</p>
      <p>Satisfaction: {metrics.satisfaction}/5</p>
    </InfoCard>
  );
};

Cost Allocation

# kubecost/allocation.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cost-allocation
data:
  allocation.json: |
    {
      "defaultLabels": {
        "team": "team",
        "service": "app",
        "environment": "env"
      },
      "shareNamespaces": ["kube-system"],
      "shareCost": "weighted"
    }

Platform APIs

# Platform API for self-service provisioning
from fastapi import FastAPI, Depends
from pydantic import BaseModel

app = FastAPI()

class ServiceRequest(BaseModel):
    name: str
    environment: str
    language: str
    database: bool = False

@app.post("/api/v1/services")
async def create_service(request: ServiceRequest):
    # Validate and enqueue
    task = platform.provision_service(
        name=request.name,
        env=request.environment,
        template=f"golden-path-{request.language}"
    )
    return {"task_id": task.id, "status": "provisioning"}

@app.get("/api/v1/services/{name}/status")
async def service_status(name: str):
    return {
        "status": "running",
        "url": f"https://{name}.example.com",
        "health": "healthy",
        "cost_mtd": "$142.50"
    }

Multi-Tenant Architecture

# Policy: Resource quotas per tenant
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-payments
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    persistentvolumeclaims: "10"
    services.loadbalancers: "2"
---
# RBAC: Namespace admin
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-admin
  namespace: team-payments
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: namespace-admin
subjects:
  - kind: Group
    name: team-payments

Adoption Strategy

# Platform metrics tracking
apiVersion: v1
kind: ConfigMap
metadata:
  name: platform-goals
data:
  goals.yaml: |
    q1_2024:
      self_service_rate: 90%
      avg_provision_time: 5min
      developer_satisfaction: 4.5/5
      golden_path_adoption: 80%

    tracking:
      weekly_provisioning: true
      team_feedback: true
      support_tickets: true
      training_completion: true

CLI Tool Example

#!/bin/bash
# platform-cli - Self-service CLI

platform() {
  case $1 in
    create)
      curl -X POST $PLATFORM_API/services \
        -d "{\"name\":\"$2\",\"env\":\"$3\",\"language\":\"$4\"}"
      ;;
    status)
      curl $PLATFORM_API/services/$2/status | jq
      ;;
    logs)
      kubectl logs -l app=$2 -n ${3:-staging} --tail=100
      ;;
    cost)
      curl $PLATFORM_API/services/$2/cost?period=mtd
      ;;
  esac
}

Best Practices

• Design for self-service from day one
• Make golden paths the easiest option
• Measure developer satisfaction continuously
• Automate platform operations
• Provide excellent documentation
• Build APIs, not just tools
• Enable safe experimentation
• Maintain backward compatibility
• Treat platform as a product
• Gather and act on feedback
• Track adoption metrics weekly
• Run platform as a product team
• Invest in developer evangelism
• Maintain SLOs for platform uptime
• Provide fast, helpful support

Release Automation

Artifact Management

Container Registry Lifecycle

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Keep last 10 prod images",
      "selection": {
        "tagStatus": "tagged",
        "tagPrefixList": ["prod-"],
        "countType": "imageCountMoreThan",
        "countNumber": 10
      },
      "action": { "type": "expire" }
    },
    {
      "rulePriority": 2,
      "description": "Remove untagged after 7 days",
      "selection": {
        "tagStatus": "untagged",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 7
      },
      "action": { "type": "expire" }
    }
  ]
}

Artifact Promotion

# .github/workflows/promote.yml
name: Artifact Promotion

on:
  workflow_dispatch:
    inputs:
      image_tag:
        required: true
      target_env:
        type: choice
        options: [staging, production]

jobs:
  promote:
    runs-on: ubuntu-latest
    steps:
      - name: Re-tag for environment
        run: |
          docker pull $REGISTRY/$IMAGE:${{ inputs.image_tag }}
          docker tag $REGISTRY/$IMAGE:${{ inputs.image_tag }} \
            $REGISTRY/$IMAGE:${{ inputs.target_env }}-latest
          docker push $REGISTRY/$IMAGE:${{ inputs.target_env }}-latest

      - name: Sign artifact
        uses: sigstore/cosign-installer@v3
      - run: cosign sign $REGISTRY/$IMAGE:${{ inputs.target_env }}-latest

      - name: Update GitOps
        run: |
          cd gitops/apps/${{ inputs.target_env }}
          yq e '.image.tag = "${{ inputs.image_tag }}"' -i values.yaml
          git commit -am "Promote to ${{ inputs.target_env }}"
          git push

Feature Flags

LaunchDarkly Integration

import launchdarkly

ld = launchdarkly.get()

def should_enable(user_id, feature_key):
    user = {"key": user_id, "custom": {"groups": get_groups(user_id)}}
    return ld.variation(feature_key, user, False)

# Usage
if should_enable(user.id, "new-payment-flow"):
    return new_payment_service.process(payment)
else:
    return legacy_payment_service.process(payment)

Flagger Progressive Delivery

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: payment-service
spec:
  targetRef:
    kind: Deployment
    name: payment-service
  service:
    port: 8080
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
      - name: request-duration
        thresholdRange:
          max: 500
    webhooks:
      - name: load-test
        url: http://flagger-loadtester/
        metadata:
          cmd: "hey -z 1m -q 10 http://payment-canary/"

Multi-Platform CI/CD

GitLab CI

stages: [test, build, deploy]

test:
  stage: test
  image: node:20
  script:
    - npm ci && npm test

build:
  stage: build
  image: docker:latest
  services: [docker:dind]
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

deploy:production:
  stage: deploy
  script:
    - kubectl set image deployment/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  environment: production
  when: manual
  only: [main]

Jenkins Pipeline

pipeline {
    agent any

    environment {
        IMAGE = "registry.example.com/app"
    }

    stages {
        stage('Test') {
            steps {
                sh 'npm ci && npm test'
                junit 'reports/junit.xml'
            }
        }

        stage('Build') {
            steps {
                script {
                    docker.build("${IMAGE}:${BUILD_NUMBER}")
                }
            }
        }

        stage('Security Scan') {
            steps {
                sh "trivy image ${IMAGE}:${BUILD_NUMBER}"
            }
        }

        stage('Deploy Staging') {
            when { branch 'main' }
            steps {
                sh "kubectl set image deployment/app app=${IMAGE}:${BUILD_NUMBER} -n staging"
            }
        }

        stage('Deploy Production') {
            when { branch 'main' }
            steps {
                input 'Deploy to production?'
                sh "kubectl set image deployment/app app=${IMAGE}:${BUILD_NUMBER} -n production"
            }
        }
    }

    post {
        failure {
            slackSend color: 'danger', message: "Build failed: ${JOB_NAME}"
        }
    }
}

Build Optimization

Multi-stage Docker Build

FROM node:20 AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

FROM node:20 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-slim AS runner
WORKDIR /app
ENV NODE_ENV production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
USER node
CMD ["node", "dist/main.js"]

Parallel Testing

# CircleCI
version: 2.1
jobs:
  test:
    parallelism: 4
    docker:
      - image: cimg/node:20
    steps:
      - checkout
      - run: npm ci
      - run: |
          TESTS=$(circleci tests glob "test/**/*.js" | circleci tests split)
          npm test $TESTS

Release Orchestration

#!/bin/bash
# release.sh - Multi-service coordinated release

VERSION=$1
SERVICES=(auth api worker frontend)

echo "Release: $VERSION"

# Create release branches
for svc in "${SERVICES[@]}"; do
    gh api repos/org/$svc/git/refs -f ref=refs/heads/release/$VERSION -f sha=$(git rev-parse main)
done

# Trigger builds
for svc in "${SERVICES[@]}"; do
    gh workflow run ci.yml --repo org/$svc --ref release/$VERSION
done

# Wait for completion
for svc in "${SERVICES[@]}"; do
    gh run watch --repo org/$svc $(gh run list --repo org/$svc -L1 -q '.[0].databaseId')
done

# Deploy to staging
kubectl apply -f staging/release-$VERSION.yaml

# Smoke tests
./scripts/smoke-test.sh staging

echo "✓ Release $VERSION ready for production"

Dependency Management

Renovate Auto-Update

{
  "extends": ["config:base"],
  "packageRules": [
    {
      "matchUpdateTypes": ["minor", "patch"],
      "automerge": true
    },
    {
      "matchDepTypes": ["devDependencies"],
      "automerge": true
    }
  ],
  "schedule": ["before 6am on Monday"],
  "prConcurrentLimit": 5
}

Build Optimization

Build Caching Strategy

# GitHub Actions: Multi-layer caching
- name: Cache dependencies
  uses: actions/cache@v3
  with:
    path: |
      ~/.npm
      ~/.cache
      node_modules
    key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-deps-

- name: Cache Docker layers
  uses: docker/build-push-action@v4
  with:
    context: .
    cache-from: type=gha
    cache-to: type=gha,mode=max

Parallel CI Pipeline

# Multi-platform builds in parallel
name: Build

on: [push]

jobs:
  test:
    strategy:
      matrix:
        node: [18, 20, 22]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node }}
      - run: npm ci && npm test

  build-images:
    needs: test
    strategy:
      matrix:
        platform: [linux/amd64, linux/arm64]
    runs-on: ubuntu-latest
    steps:
      - uses: docker/build-push-action@v4
        with:
          platforms: ${{ matrix.platform }}
          tags: app:${{ github.sha }}

Multi-Service Release Orchestration

# release-coordinator.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: release-v2.5.0
spec:
  template:
    spec:
      containers:
        - name: coordinator
          image: release-bot:latest
          env:
            - name: RELEASE_VERSION
              value: "v2.5.0"
            - name: SERVICES
              value: "auth,api,worker,frontend"
          command:
            - /bin/bash
            - -c
            - |
              # Deploy in dependency order
              for svc in auth api worker frontend; do
                echo "Deploying $svc..."
                kubectl set image deploy/$svc \
                  $svc=registry.io/$svc:$RELEASE_VERSION

                kubectl rollout status deploy/$svc --timeout=5m

                # Health check
                kubectl run test-$svc --rm -i --restart=Never \
                  --image=curlimages/curl -- \
                  curl -f http://$svc/health

                echo "$svc deployed successfully"
              done

Advanced Artifact Management

#!/bin/bash
# artifact-scanner.sh - Scan before promotion

IMAGE=$1
SEVERITY=${2:-HIGH}

# Vulnerability scan
trivy image --severity $SEVERITY --exit-code 1 $IMAGE

# License compliance
syft $IMAGE -o json | \
  jq '.artifacts[].licenses[] | select(.value |
    contains("GPL") or contains("AGPL"))' && \
  echo "License violation detected" && exit 1

# SBOM generation
syft $IMAGE -o spdx-json > sbom-$(basename $IMAGE).spdx.json

# Sign artifact
cosign sign --key cosign.key $IMAGE

# Promote
docker tag $IMAGE $IMAGE-approved
docker push $IMAGE-approved

echo "Artifact $IMAGE approved and promoted"

Zero-Downtime Database Migrations

# migrations/release_v2.5.py
from alembic import op
import sqlalchemy as sa

def upgrade():
    # Step 1: Add new column (nullable)
    op.add_column('users',
      sa.Column('email_verified', sa.Boolean(), nullable=True))

    # Step 2: Backfill data (in batches)
    connection = op.get_bind()
    connection.execute("""
      UPDATE users SET email_verified = true
      WHERE email IS NOT NULL
      LIMIT 1000
    """)
    # Repeat until complete (or use background job)

    # Step 3: Make non-nullable (in next release)
    # op.alter_column('users', 'email_verified', nullable=False)

def downgrade():
    op.drop_column('users', 'email_verified')

Release Metrics Dashboard

# Grafana dashboard for release metrics
apiVersion: v1
kind: ConfigMap
metadata:
  name: release-dashboard
data:
  dashboard.json: |
    {
      "panels": [
        {
          "title": "Deployment Frequency",
          "targets": [{
            "expr": "count_over_time(deployment_completed[1d])"
          }]
        },
        {
          "title": "Lead Time",
          "targets": [{
            "expr": "histogram_quantile(0.95, commit_to_deploy_seconds_bucket)"
          }]
        },
        {
          "title": "Change Failure Rate",
          "targets": [{
            "expr": "sum(rate(deployment_failed[1h])) / sum(rate(deployment_total[1h]))"
          }]
        },
        {
          "title": "Active Releases",
          "targets": [{
            "expr": "count(release_in_progress == 1)"
          }]
        }
      ]
    }

Best Practices

• Version artifacts with immutable tags
• Implement retention policies
• Use progressive delivery for high-risk changes
• Automate security scanning
• Maintain deployment audit trails
• Enable easy rollbacks
• Monitor deployment metrics
• Use feature flags for flexibility
• Cache aggressively for fast builds
• Parallelize test and build jobs
• Coordinate multi-service releases
• Generate and track SBOMs
• Sign artifacts for supply chain security
• Automate dependency updates
• Track DORA metrics continuously

Terraform Infrastructure as Code

AWS ECS Fargate Setup

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }
  backend "s3" {
    bucket = "terraform-state"
    key    = "app/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_ecs_cluster" "main" {
  name = "app-cluster"
  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}

resource "aws_ecs_task_definition" "app" {
  family                   = "app"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "256"
  memory                   = "512"
  execution_role_arn       = aws_iam_role.ecs_execution.arn

  container_definitions = jsonencode([{
    name  = "app"
    image = "${var.ecr_repository}:${var.image_tag}"
    portMappings = [{ containerPort = 3000 }]
    logConfiguration = {
      logDriver = "awslogs"
      options = {
        awslogs-group         = aws_cloudwatch_log_group.app.name
        awslogs-region        = var.region
        awslogs-stream-prefix = "app"
      }
    }
    secrets = [
      { name = "DATABASE_URL", valueFrom = aws_ssm_parameter.db_url.arn }
    ]
  }])
}

resource "aws_ecs_service" "app" {
  name            = "app"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 2
  launch_type     = "FARGATE"

  network_configuration {
    subnets         = var.private_subnets
    security_groups = [aws_security_group.app.id]
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.app.arn
    container_name   = "app"
    container_port   = 3000
  }
}

AWS RDS PostgreSQL

resource "aws_db_instance" "postgres" {
  identifier           = "app-db"
  engine               = "postgres"
  engine_version       = "16.1"
  instance_class       = "db.t3.micro"
  allocated_storage    = 20
  storage_encrypted    = true

  db_name              = "app"
  username             = "admin"
  password             = var.db_password

  vpc_security_group_ids = [aws_security_group.db.id]
  db_subnet_group_name   = aws_db_subnet_group.main.name

  backup_retention_period = 7
  skip_final_snapshot     = false
  final_snapshot_identifier = "app-db-final"

  tags = { Environment = var.environment }
}

Variables and Outputs

# variables.tf
variable "environment" {
  type        = string
  description = "Environment name"
}

variable "region" {
  type    = string
  default = "us-east-1"
}

# outputs.tf
output "ecs_cluster_arn" {
  value = aws_ecs_cluster.main.arn
}

output "alb_dns_name" {
  value = aws_lb.main.dns_name
}

Best Practices

Common Commands

terraform init
terraform plan -out=tfplan
terraform apply tfplan
terraform destroy
terraform state list
terraform import aws_instance.app i-1234567890