Senior Devops

Senior DevOps Engineer

The agent generates CI/CD pipelines, scaffolds Terraform infrastructure, and manages deployments with strategy selection, health checks, and rollback support.

---

Quick Start

# Generate CI/CD pipeline from project analysis
python scripts/pipeline_generator.py <project-path> --platform github-actions --verbose

# Scaffold Terraform infrastructure
python scripts/terraform_scaffolder.py <target-path> --provider aws --env production --verbose

# Manage deployment with canary strategy
python scripts/deployment_manager.py <target-path> --strategy canary --verbose

Tools Overview

All tools support --json for machine-readable output and --output / -o for file writing.

---

Workflow 1: Containerize and Deploy

Step 1 -- Build a production Dockerfile.

The agent generates multi-stage Dockerfiles following this pattern:

# Stage 1: Build
FROM node:20-alpine AS builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --only=production && npm cache clean --force
COPY . .
RUN npm run build

# Stage 2: Production
FROM node:20-alpine AS production
WORKDIR /app
RUN addgroup -g 1001 appgroup && \
    adduser -u 1001 -G appgroup -s /bin/sh -D appuser
COPY --from=builder --chown=appuser:appgroup /app/dist ./dist
COPY --from=builder --chown=appuser:appgroup /app/node_modules ./node_modules
COPY --from=builder --chown=appuser:appgroup /app/package.json ./
USER appuser
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:3000/healthz || exit 1
CMD ["node", "dist/server.js"]

Validation checkpoint: Image builds with docker build -t app:test . and docker run --rm app:test returns healthy.

Step 2 -- Deploy to Kubernetes.

The agent creates a Deployment with probes, resource limits, and security context:

spec:
  containers:
    - name: app
      image: myapp:1.2.3
      resources:
        requests: { cpu: 250m, memory: 256Mi }
        limits: { cpu: "1", memory: 512Mi }
      livenessProbe:
        httpGet: { path: /healthz, port: 3000 }
        initialDelaySeconds: 15
        periodSeconds: 20
      readinessProbe:
        httpGet: { path: /ready, port: 3000 }
        initialDelaySeconds: 5
        periodSeconds: 10
      startupProbe:
        httpGet: { path: /healthz, port: 3000 }
        failureThreshold: 30
        periodSeconds: 10

Probe decision:

• startupProbe: Slow-starting apps (JVM, model loading). Prevents liveness from killing during startup.
• livenessProbe: Detects deadlocks. Keep simple -- do not check downstream dependencies.
• readinessProbe: Controls traffic routing. Include dependency checks here.

Validation checkpoint: kubectl get pods -l app=myapp shows all pods Running and Ready.

---

Workflow 2: Infrastructure as Code with Terraform

Step 1 -- Scaffold the module structure.

python scripts/terraform_scaffolder.py ./infrastructure --provider aws --env production --verbose

The agent produces:

infrastructure/
  modules/
    vpc/         # main.tf, variables.tf, outputs.tf
    eks/
    rds/
  environments/
    staging/     # main.tf, terraform.tfvars, backend.tf
    production/

Step 2 -- Configure remote state.

terraform {
  backend "s3" {
    bucket         = "mycompany-terraform-state"
    key            = "production/infrastructure.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

Step 3 -- Run drift detection in CI.

terraform plan -detailed-exitcode -out=plan.tfplan
# Exit 0 = clean, Exit 1 = error, Exit 2 = drift detected

Validation checkpoint: terraform plan shows no unexpected changes. Drift alerts fire within 24 hours.

Key rules:

• One state file per environment per component (blast radius control)
• Never store state locally or in git
• Run terraform plan in CI, terraform apply only after approval
• Use directories for environment separation, modules for shared logic

---

Workflow 3: CI/CD Pipeline Design

python scripts/pipeline_generator.py /path/to/project --platform github-actions --json

The agent generates pipelines following these principles:

1. Fail fast -- lint and unit tests before expensive integration tests 2. Cache aggressively -- node_modules, Docker layers, pip packages 3. Immutable artifacts -- build once, deploy the same artifact everywhere 4. Gate promotions -- manual approval or smoke tests before production 5. Parallel execution -- independent test suites and security scans run concurrently

Example: GitHub Actions with matrix testing and deployment gates

jobs:
  test:
    strategy:
      matrix:
        node-version: [18, 20]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: "${{ matrix.node-version }}", cache: npm }
      - run: npm ci && npm run lint && npm test -- --coverage

  build:
    needs: [test, security]
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: docker/build-push-action@v5
        with:
          push: true
          tags: "${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}"
          cache-from: type=gha
          cache-to: type=gha,mode=max

  deploy-staging:
    needs: build
    environment: staging
    steps:
      - run: helm upgrade --install app charts/myapp --set image.tag=${{ github.sha }} --wait

  deploy-production:
    needs: deploy-staging
    environment: production  # requires manual approval

Validation checkpoint: Pipeline runs in under 15 minutes. All stages produce exit code 0.

---

Deployment Strategy Selection

Canary promotion ladder: 1. Deploy at 5% traffic. Monitor error rate and latency for 10 min. 2. Promote to 25%. Monitor 10 min. 3. Promote to 50%. Monitor 15 min. 4. Promote to 100%. 5. Automated rollback if error rate exceeds baseline by 2x at any step.

---

Monitoring Essentials

Every service dashboard includes the Four Golden Signals:

1. Latency -- P50, P90, P99 response times 2. Traffic -- Requests per second by endpoint and status code 3. Errors -- 5xx rate, 4xx rate, application error codes 4. Saturation -- CPU, memory, connection pool, queue depth

SLO targets (example):

When the error budget is exhausted, the agent recommends freezing feature deployments until the budget recovers.

---

Anti-Patterns

1. Monolithic state -- one Terraform state for everything. Split by component and environment. 2. `latest` tag in production -- always use specific image tags. 3. Secrets in image layers -- inject at runtime via environment or mounted secrets. Verify with docker history --no-trunc. 4. No resource limits -- every container needs CPU/memory limits to prevent noisy-neighbor attacks. 5. Manual deployments -- automate with approval gates instead.

---

Troubleshooting

---

References

See also: references/kubernetes_patterns.md for Helm charts, HPA/VPA/KEDA decisions, network policies, and RBAC patterns. references/cloud_platform_guide.md for AWS/GCP/Azure service comparison, multi-cloud strategy, and cost optimization.

---

Integration Points

---

Last Updated: April 2026 Version: 2.1.0

Cicd Pipeline Guide

Overview

This reference guide provides comprehensive information for senior devops.

Patterns and Practices

Pattern 1: Best Practice Implementation

Description: Detailed explanation of the pattern.

When to Use:

• Scenario 1
• Scenario 2
• Scenario 3

Implementation:

// Example code implementation
export class Example {
  // Implementation details
}

Benefits:

• Benefit 1
• Benefit 2
• Benefit 3

Trade-offs:

• Consider 1
• Consider 2
• Consider 3

Pattern 2: Advanced Technique

Description: Another important pattern for senior devops.

Implementation:

// Advanced example
async function advancedExample() {
  // Code here
}

Guidelines

Code Organization

• Clear structure
• Logical separation
• Consistent naming
• Proper documentation

Performance Considerations

• Optimization strategies
• Bottleneck identification
• Monitoring approaches
• Scaling techniques

Security Best Practices

• Input validation
• Authentication
• Authorization
• Data protection

Common Patterns

Pattern A

Implementation details and examples.

Pattern B

Implementation details and examples.

Pattern C

Implementation details and examples.

Anti-Patterns to Avoid

Anti-Pattern 1

What not to do and why.

Anti-Pattern 2

What not to do and why.

Tools and Resources

Recommended Tools

• Tool 1: Purpose
• Tool 2: Purpose
• Tool 3: Purpose

Further Reading

• Resource 1
• Resource 2
• Resource 3

Conclusion

Key takeaways for using this reference guide effectively.

Cloud Platform Guide

Service Comparison Matrix

Multi-Cloud Decision Framework

When multi-cloud makes sense:

• Regulatory requirements mandate vendor diversity
• Acquisition brings workloads on a different cloud
• Best-of-breed services (GCP for ML, AWS for breadth)

When it does not:

• Avoiding lock-in as the sole motivation (operational tax exceeds savings)
• Small teams that cannot afford complexity overhead

If you go multi-cloud:

• Use Terraform for the abstraction layer
• Standardize on Kubernetes as compute plane
• Centralize observability (Datadog, Grafana Cloud)
• Invest in a platform engineering team

Cost Optimization

Right-Sizing Methodology

1. Collect 2-4 weeks of CPU/memory utilization 2. Identify instances below 40% average CPU 3. Recommend one size down 4. Validate in staging under load test 5. Apply in production during maintenance window 6. Monitor for 1 week post-change

Spot/Preemptible Strategy

FinOps Practices

• Tagging: Enforce team, environment, service, cost-center on all resources
• Budget alerts: 50%, 80%, 100% of monthly budget
• Reserved capacity: 1-year for baseline workloads (30-40% savings)
• Scheduled scaling: Scale down non-prod outside business hours
• Storage lifecycle: S3 lifecycle policies for Glacier/Archive tiers
• Unused resources: Weekly scan for unattached EBS, idle LBs, stale snapshots

Prometheus Alerting Rules

groups:
  - name: application
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "Error rate exceeds 5% for 5 minutes"

      - alert: HighLatencyP99
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 2.0
        for: 10m
        labels: { severity: warning }

      - alert: PodCrashLooping
        expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
        for: 5m
        labels: { severity: critical }

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.15
        for: 10m
        labels: { severity: warning }

Incident Response Quick Reference

Severity Classification

Runbook Template

## Symptoms
- What alerts fire, what users report

## Diagnosis
1. kubectl get pods -n production -l app=myapp
2. helm history myapp -n production
3. kubectl logs -l app=myapp --tail=100

## Quick Fix (< 5 min)
- kubectl rollout restart deployment/myapp
- kubectl scale deployment/myapp --replicas=10

## Rollback (< 10 min)
- helm rollback myapp [previous-revision]

Postmortem (required for SEV-1/SEV-2)

1. Timeline reconstruction 2. Root cause (5 Whys) 3. Impact assessment 4. What went well / poorly 5. Action items with owners and due dates

Deployment Strategies

Overview

This reference guide provides comprehensive information for senior devops.

Patterns and Practices

Pattern 1: Best Practice Implementation

Description: Detailed explanation of the pattern.

When to Use:

• Scenario 1
• Scenario 2
• Scenario 3

Implementation:

// Example code implementation
export class Example {
  // Implementation details
}

Benefits:

• Benefit 1
• Benefit 2
• Benefit 3

Trade-offs:

• Consider 1
• Consider 2
• Consider 3

Pattern 2: Advanced Technique

Description: Another important pattern for senior devops.

Implementation:

// Advanced example
async function advancedExample() {
  // Code here
}

Guidelines

Code Organization

• Clear structure
• Logical separation
• Consistent naming
• Proper documentation

Performance Considerations

• Optimization strategies
• Bottleneck identification
• Monitoring approaches
• Scaling techniques

Security Best Practices

• Input validation
• Authentication
• Authorization
• Data protection

Common Patterns

Pattern A

Implementation details and examples.

Pattern B

Implementation details and examples.

Pattern C

Implementation details and examples.

Anti-Patterns to Avoid

Anti-Pattern 1

What not to do and why.

Anti-Pattern 2

What not to do and why.

Tools and Resources

Recommended Tools

• Tool 1: Purpose
• Tool 2: Purpose
• Tool 3: Purpose

Further Reading

• Resource 1
• Resource 2
• Resource 3

Conclusion

Key takeaways for using this reference guide effectively.

Infrastructure As Code

Overview

This reference guide provides comprehensive information for senior devops.

Patterns and Practices

Pattern 1: Best Practice Implementation

Description: Detailed explanation of the pattern.

When to Use:

• Scenario 1
• Scenario 2
• Scenario 3

Implementation:

// Example code implementation
export class Example {
  // Implementation details
}

Benefits:

• Benefit 1
• Benefit 2
• Benefit 3

Trade-offs:

• Consider 1
• Consider 2
• Consider 3

Pattern 2: Advanced Technique

Description: Another important pattern for senior devops.

Implementation:

// Advanced example
async function advancedExample() {
  // Code here
}

Guidelines

Code Organization

• Clear structure
• Logical separation
• Consistent naming
• Proper documentation

Performance Considerations

• Optimization strategies
• Bottleneck identification
• Monitoring approaches
• Scaling techniques

Security Best Practices

• Input validation
• Authentication
• Authorization
• Data protection

Common Patterns

Pattern A

Implementation details and examples.

Pattern B

Implementation details and examples.

Pattern C

Implementation details and examples.

Anti-Patterns to Avoid

Anti-Pattern 1

What not to do and why.

Anti-Pattern 2

What not to do and why.

Tools and Resources

Recommended Tools

• Tool 1: Purpose
• Tool 2: Purpose
• Tool 3: Purpose

Further Reading

• Resource 1
• Resource 2
• Resource 3

Conclusion

Key takeaways for using this reference guide effectively.

Kubernetes Patterns Reference

Pod Design Patterns

Sidecar Pattern

Add capabilities without modifying the main container:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
    spec:
      serviceAccountName: app-sa
      securityContext:
        runAsNonRoot: true
        fsGroup: 1001
      containers:
        - name: app
          image: myapp:1.2.3
          ports:
            - containerPort: 3000
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits: { cpu: "1", memory: 512Mi }
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: db-password
        - name: log-shipper
          image: fluent/fluent-bit:2.2
          volumeMounts:
            - name: app-logs
              mountPath: /var/log/app
      volumes:
        - name: app-logs
          emptyDir: {}

Helm Chart Structure

charts/myapp/
  Chart.yaml
  values.yaml
  values-staging.yaml
  values-production.yaml
  templates/
    deployment.yaml
    service.yaml
    ingress.yaml
    hpa.yaml
    networkpolicy.yaml
    serviceaccount.yaml
    _helpers.tpl

Key values.yaml patterns:

replicaCount: 3
image:
  repository: myapp
  tag: "1.2.3"
  pullPolicy: IfNotPresent

resources:
  requests: { cpu: 250m, memory: 256Mi }
  limits: { cpu: "1", memory: 512Mi }

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 20
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: app.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: app-tls
      hosts:
        - app.example.com

HPA Configuration

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 120

HPA vs VPA vs KEDA

Network Policies

Default-deny with explicit allow:

# Default deny all
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

---
# Allow app traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: app-network-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress-nginx
      ports:
        - protocol: TCP
          port: 3000
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53

RBAC Best Practices

• Least privilege: grant minimum permissions needed
• Use ClusterRoles for cluster-wide, Roles for namespace-scoped
• Bind service accounts to roles, not users
• Audit: kubectl auth can-i --list --as=system:serviceaccount:production:app-sa
• Never grant cluster-admin to application service accounts

Container Security Checklist

• [ ] Base images from trusted registries (Docker Official, Chainguard, Distroless)
• [ ] Scanned with Trivy/Grype: trivy image --severity HIGH,CRITICAL myapp:latest
• [ ] No root processes -- USER directive required
• [ ] Read-only root filesystem: --read-only --tmpfs /tmp
• [ ] Resource limits enforced (CPU, memory)
• [ ] No secrets in image layers -- verify with docker history --no-trunc
• [ ] Minimal base images (Alpine, Distroless)

Secret Management Decision Matrix

Supply Chain Security

# Sign container images
cosign sign --key cosign.key ghcr.io/myorg/myapp:1.2.3
cosign verify --key cosign.pub ghcr.io/myorg/myapp:1.2.3

# Generate and scan SBOM
syft ghcr.io/myorg/myapp:1.2.3 -o spdx-json > sbom.json
grype sbom:sbom.json --fail-on high