Ops Devops Platform

DevOps Engineering — Quick Reference

This skill equips teams with actionable templates, checklists, and patterns for building self-service platforms, automating infrastructure with GitOps, deploying securely with DevSecOps, scaling with Kubernetes, ensuring reliability through SRE practices, and operating production systems with strong observability.

Modern baseline (2026): IaC (Terraform/OpenTofu/Pulumi), GitOps (Argo CD/Flux), Kubernetes (follow upstream supported releases), OpenTelemetry + Prometheus/Grafana, supply-chain security (SBOM + signing + provenance), policy-as-code (OPA/Gatekeeper or Kyverno), and eBPF-powered networking/security/observability (e.g., Cilium + Tetragon).

---

Quick Reference

---

Decision Tree: Choosing DevOps Approach

What do you need to accomplish?
    ├─ Infrastructure provisioning?
    │   ├─ Cloud-agnostic → Terraform or OpenTofu (OSS fork)
    │   ├─ Programming-first → Pulumi (TypeScript/Python/Go)
    │   ├─ AWS-specific → CloudFormation or Terraform/OpenTofu
    │   ├─ GCP-specific → Deployment Manager or Terraform/OpenTofu
    │   └─ Azure-specific → ARM/Bicep or Terraform/OpenTofu
    │
    ├─ Application deployment?
    │   ├─ Kubernetes cluster?
    │   │   ├─ Simple deploy → kubectl apply -f manifests/
    │   │   ├─ Complex app → Helm charts
    │   │   └─ GitOps workflow → ArgoCD or FluxCD
    │   └─ Serverless?
    │       ├─ AWS → Lambda + SAM/Serverless Framework
    │       ├─ GCP → Cloud Functions
    │       └─ Azure → Azure Functions
    │
    ├─ CI/CD pipeline setup?
    │   ├─ GitHub-based → GitHub Actions (template-github-actions.md)
    │   ├─ GitLab-based → GitLab CI
    │   ├─ Enterprise → Jenkins or Tekton
    │   └─ Security-first → Add SAST/DAST/SCA scans (template-ci-cd.md)
    │
    ├─ Observability & monitoring?
    │   ├─ Metrics → Prometheus + Grafana
    │   ├─ Distributed tracing → Jaeger or OpenTelemetry
    │   ├─ Logs → Loki or ELK stack
    │   ├─ eBPF-based → Cilium + Hubble (sidecarless)
    │   └─ Unified platform → Datadog or New Relic
    │
    ├─ Incident management?
    │   ├─ On-call rotation → PagerDuty or Opsgenie
    │   ├─ Postmortem → template-postmortem.md
    │   └─ Communication → template-incident-comm.md
    │
    ├─ Platform engineering?
    │   ├─ Self-service → Backstage or Port (internal developer portal)
    │   ├─ Policy enforcement → OPA/Gatekeeper
    │   └─ Golden paths → Template repositories + automation
    │
    └─ Security hardening?
        ├─ Container scanning → Trivy or Grype
        ├─ Runtime security → Falco or Sysdig
        ├─ Secrets management → HashiCorp Vault or cloud-native KMS
        └─ Compliance → CIS Benchmarks, template-security-hardening.md

---

When to Use This Skill

Claude should invoke this skill when users request:

• Platform engineering patterns (self-service developer platforms, internal tools)
• GitOps workflows (ArgoCD, FluxCD, declarative infrastructure management)
• Infrastructure as Code patterns (Terraform, K8s manifests, policy as code)
• CI/CD pipelines with DevSecOps (GitHub Actions, security scanning, SAST/DAST/SCA)
• SRE incident management, escalation, and postmortem templates
• eBPF-based observability (Cilium, Hubble, kernel-level insights, OpenTelemetry)
• Kubernetes operational patterns (day-2 operations, resource management, workload placement)
• Cloud-native monitoring (Prometheus, Grafana, unified observability platforms)
• Team workflow, communication, handover guides, and runbooks

---

Resources (Best Practices Guides)

Operational best practices by domain:

• DevOps/SRE Operations: references/devops-best-practices.md - Core patterns for safe infrastructure changes, deployments, and incident response
• Platform Engineering: references/platform-engineering-patterns.md - Self-service platforms, golden paths, internal developer portals, policy as code
• GitOps Workflows: references/gitops-workflows.md - Continuous reconciliation, multi-environment promotion, ArgoCD/FluxCD patterns, progressive delivery
• SRE Incident Management: references/sre-incident-management.md - Severity classification, escalation procedures, blameless postmortems, alert correlation, and runbooks
• Operational Standards: references/operational-patterns.md - Platform engineering blueprints, CI/CD safety, SLOs, and reliability drills
• AIOps: references/aiops-patterns.md - Self-healing systems, automated operations, AI-assisted analysis

---

Templates (Copy-Paste Ready)

Production templates organized by tech stack:

AWS Cloud

• assets/aws/template-aws-ops.md - AWS service operations and best practices
• assets/aws/template-aws-terraform.md - Terraform modules for AWS infrastructure
• assets/aws/template-cost-optimization.md - AWS cost optimization strategies

GCP Cloud

• assets/gcp/template-gcp-ops.md - GCP service operations
• assets/gcp/template-gcp-terraform.md - Terraform modules for GCP

Azure Cloud

• assets/azure/template-azure-ops.md - Azure service operations

Kubernetes

• assets/kubernetes/template-kubernetes-ops.md - Day-to-day K8s operations
• assets/kubernetes/template-ha-dr.md - High availability and disaster recovery
• assets/kubernetes/template-platform-api.md - Platform API patterns
• assets/kubernetes/template-k8s-deploy.yaml - Deployment manifests

Docker

• assets/docker/template-docker-ops.md - Container build, security, and operations

Kafka

• assets/kafka/template-kafka-ops.md - Kafka cluster operations and streaming

Terraform & IaC

• assets/terraform-iac/template-iac-terraform.md - Infrastructure as Code patterns
• assets/terraform-iac/template-module.md - Reusable Terraform modules
• assets/terraform-iac/template-env-promotion.md - Environment promotion strategies

CI/CD Pipelines

• assets/cicd-pipelines/template-ci-cd.md - General CI/CD patterns
• assets/cicd-pipelines/template-github-actions.md - GitHub Actions workflows
• assets/cicd-pipelines/template-gitops.md - GitOps deployment patterns
• assets/cicd-pipelines/template-release-safety.md - Safe release practices

Monitoring & Observability

• assets/monitoring-observability/template-slo.md - Service level objectives
• assets/monitoring-observability/template-alert-rules.md - Alert configuration
• assets/monitoring-observability/template-observability-slo.md - Observability patterns
• assets/monitoring-observability/template-loadtest-perf.md - Load testing and performance

Incident Response

• assets/incident-response/template-postmortem.md - Incident postmortems
• assets/incident-response/template-runbook-starter.md - Runbook starter template
• assets/incident-response/template-incident-comm.md - Incident communication
• assets/incident-response/template-incident-response.md - Incident response procedures

Security

• assets/security/template-security-hardening.md - Security hardening checklists

---

Shared Utilities

Centralized patterns from software-clean-code-standard — extract, don't duplicate:

• config-validation.md — Zod 3.24+, secrets management (Vault, 1Password, Doppler)
• resilience-utilities.md — p-retry v6, circuit breaker, OTel spans
• logging-utilities.md — pino v9 + OpenTelemetry integration
• observability-utilities.md — OpenTelemetry SDK, tracing, metrics

---

Related Skills

Operations & Infrastructure:

• ../qa-resilience/SKILL.md — Resilience, chaos engineering, and failure handling patterns
• ../data-sql-optimization/SKILL.md — Database tuning, high availability, and migrations
• ../qa-observability/SKILL.md — Monitoring, tracing, profiling, and performance optimization
• ../qa-debugging/SKILL.md — Production debugging, log analysis, and root cause investigation

Security & Compliance:

• ../software-security-appsec/SKILL.md — Application-layer security patterns and OWASP best practices

Software Development:

• ../software-backend/SKILL.md — Service-level design and integration patterns
• ../software-architecture-design/SKILL.md — System design, scalability, and architectural patterns
• ../dev-api-design/SKILL.md — RESTful API design and versioning
• ../dev-git-workflow/SKILL.md — Git branching strategies and CI/CD integration

Optional: AI/Automation (Related Skills):

• ../ai-mlops/SKILL.md — ML model deployment, monitoring, and lifecycle management

---

Cost Governance & Capacity Planning

[assets/cost-governance/template-cost-governance.md](assets/cost-governance/template-cost-governance.md) — Production cost control for cloud infrastructure.

Key Sections

• Cost Governance Framework — Tagging strategy, budget alerts, anomaly detection
• Cloud Cost Optimization — Right-sizing, reserved capacity, storage tiering
• Kubernetes Cost Control — Resource requests/limits, quotas, autoscaler config
• Capacity Planning — Utilization baseline, growth projections, scaling triggers
• FinOps Practices — Monthly review agenda, optimization workflow

---

Do / Avoid

Do

• Tag all resources at creation time
• Set budget alerts before hitting limits
• Review right-sizing recommendations monthly
• Use spot/preemptible for fault-tolerant workloads
• Set Kubernetes resource requests on all pods
• Enable cluster autoscaler with scale-down
• Document capacity planning assumptions
• Run blameless postmortems after every SEV1/2

Avoid

---

Optional: AI/Automation (AIOps)

AI can assist with analysis and triage, but infrastructure/cost/incident decisions require human approval and an audit trail.

See references/aiops-patterns.md for self-healing systems, automated operations, AI-assisted analysis, and bounded claims.

---

Operational Deep Dives

See references/operational-patterns.md for:

• Platform engineering blueprints and GitOps reconciliation checklists
• DevSecOps CI/CD gates, SLO/SLI playbooks, and rollout verification steps
• Observability patterns (eBPF), incident noise reduction, and reliability drills

---

External Resources

See data/sources.json for curated sources organized by tech stack:

• Cloud Platforms: AWS, GCP, Azure documentation and best practices
• Container Orchestration: Kubernetes, Helm, Kustomize, Docker
• Infrastructure as Code: Terraform, OpenTofu, Pulumi, CloudFormation, ARM templates
• CI/CD & GitOps: GitHub Actions, GitLab CI, Jenkins, ArgoCD, FluxCD
• Streaming: Apache Kafka, Confluent, Strimzi
• Monitoring: Prometheus, Grafana, Datadog, OpenTelemetry, Jaeger, Cilium/Hubble, Tetragon
• SRE: Google SRE books, incident response patterns
• Security: OWASP DevSecOps, CIS Benchmarks, Trivy, Falco
• Tools: kubectl, k9s, stern, Cosign, Syft, Terragrunt

---

Use this skill as a hub for safe, modern, and production-grade DevOps patterns. All templates and patterns are operational—no theory or book summaries.

---

Trend Awareness Protocol

When users ask recommendation questions about DevOps, platform engineering, or cloud infrastructure, validate time-sensitive details (versions, deprecations, licensing, major releases) against primary sources.

Trigger Conditions

• "What's the best tool for [Kubernetes/IaC/CI-CD/monitoring]?"
• "What should I use for [container orchestration/GitOps/observability]?"
• "What's the latest in DevOps/platform engineering?"
• "Current best practices for [Terraform/ArgoCD/Prometheus]?"
• "Is [tool/approach] still relevant in 2026?"
• "[Kubernetes] vs [alternative]?" or "[ArgoCD] vs [FluxCD]?"
• "Best cloud provider for [use case]?"
• "What orchestration/monitoring tool should I use?"

Minimum Verification (Preferred Order)

1. Check the official docs + release notes linked in data/sources.json for the specific tools you recommend. 2. If internet access is available, confirm recent releases, breaking changes, and deprecations from those release pages. 3. If internet access is not available, state that versions may have changed and focus on stable selection criteria (operational fit, ecosystem, maturity, team skills, compliance).

What to Report

After searching, provide:

• Current landscape: What tools/approaches are popular NOW (not 6 months ago)
• Emerging trends: New tools, patterns, or practices gaining traction
• Deprecated/declining: Tools/approaches losing relevance or support
• Recommendation: Based on fresh data, not just static knowledge

Example Topics (verify with fresh search)

• Kubernetes versions and ecosystem tools (1.33+, Cilium, Gateway API)
• Infrastructure as Code (Terraform, OpenTofu, Pulumi, CDK)
• GitOps platforms (ArgoCD, FluxCD, Codefresh)
• Observability stacks (OpenTelemetry, Grafana stack, Datadog)
• Platform engineering tools (Backstage, Port, Kratix)
• CI/CD platforms (GitHub Actions, GitLab CI, Dagger)
• Cloud-native security (Falco, Trivy, policy engines)

Fact-Checking

• Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

# AWS Operations Template (DevOps)

*Purpose: A comprehensive operational template for running, deploying, securing, diagnosing, and managing workloads on AWS.*

---

# 1. Overview

**Environment:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  
- [ ] multi-account  

**Task Type:**  
- [ ] Deploy service  
- [ ] IAM configuration  
- [ ] Networking (VPC, SGs, Subnets)  
- [ ] ECS/EKS operations  
- [ ] S3 operations  
- [ ] RDS/ElastiCache changes  
- [ ] Scaling  
- [ ] Monitoring/CloudWatch  
- [ ] Incident Response  

---

# 2. Core AWS Architecture Patterns

## 2.1 VPC Baseline

Checklist:
- [ ] Multi-AZ subnets (public, private)  
- [ ] Private subnets for services  
- [ ] Public subnets only for LBs  
- [ ] NAT Gateways for outbound traffic  
- [ ] SGs least privilege  
- [ ] NACLs optional unless strict segmentation required  

---

## 2.2 EKS Deployment Pattern

kubectl apply -f deployment.yaml kubectl rollout status deployment/app

Checklist:
- [ ] IAM roles for service accounts (IRSA) used  
- [ ] Cluster autoscaler installed  
- [ ] Nodegroups updated safely  
- [ ] ALB ingress or NLB configured  
- [ ] CloudWatch Container Insights enabled  

---

## 2.3 ECS Fargate Deployment

aws ecs update-service \ --cluster <cluster> \ --service <service> \ --force-new-deployment

Checklist:
- [ ] Task definitions versioned  
- [ ] CPU/memory set  
- [ ] Task role least-privilege  
- [ ] LB health checks configured  

---

## 2.4 Lambda Deployment

aws lambda update-function-code \ --function-name <name> \ --zip-file fileb://function.zip

Checklist:
- [ ] Timeout < 30s  
- [ ] Memory tuned  
- [ ] Retries & DLQ configured  
- [ ] CloudWatch alarms created  

---

# 3. IAM Operations

## 3.1 IAM Least Privilege Checklist

- [ ] No wildcard: “\*” permissions  
- [ ] Separate roles: Admin / Deploy / ReadOnly  
- [ ] Rotate IAM keys every 90 days or disable  
- [ ] Use IAM roles for workloads (EKS IRSA / ECS Task Roles)  
- [ ] MFA required for human users  
- [ ] No inline policies  

---

# 4. S3 Operations

## 4.1 Secure Bucket

Checklist:
- [ ] Block public access enabled  
- [ ] Versioning enabled  
- [ ] SSE-KMS encryption  
- [ ] Lifecycle rules active  
- [ ] Access restricted to IAM roles  

---

# 5. CloudWatch Monitoring

## 5.1 Key Metrics

- CPUUtilization  
- Memory (CW Agent/Container Insights)  
- ALB 5xx errors  
- API Gateway errors  
- Lambda duration & errors  
- RDS CPU, connections, free storage  

---

## 5.2 Logs

aws logs tail /aws/lambda/<function> --follow

Checklist:
- [ ] Log retention configured  
- [ ] Structured JSON logs  
- [ ] Metric filters created  

---

# 6. RDS & Database Ops

## 6.1 Failover

aws rds reboot-db-instance --db-instance-identifier <id> --force-failover

Checklist:
- [ ] Backups verified  
- [ ] Multi-AZ required for prod  
- [ ] Enhanced monitoring enabled  

---

## 6.2 Parameter Group Updates
- Apply during maintenance window unless safe  
- Reboot required depending on parameter type  

---

# 7. Scaling & Auto Scaling

## 7.1 EC2 ASG

Checklist:
- [ ] Health checks green  
- [ ] Warm pools optional  
- [ ] Scheduled scaling for predictable peaks  

## 7.2 DynamoDB Auto Scaling

Checklist:
- [ ] Read/write capacity policies set  
- [ ] Throttles monitored  

---

# 8. Incident Response (AWS)

## 8.1 High CPU on EC2

- Check `top`  
- Check CloudWatch metrics  
- Scale ASG or fix noisy neighbor  

## 8.2 ALB 5xx Spikes

- Check target health  
- Check EKS/ECS logs  
- Restart failing tasks  

## 8.3 S3 Access Denied

- Check IAM role  
- Check bucket policies  
- Check block public access  

---

# 9. Final AWS Ops Checklist

- [ ] IAM least privilege  
- [ ] Encryption everywhere  
- [ ] Monitoring/alerts configured  
- [ ] Multi-AZ for all critical resources  
- [ ] Backups validated  
- [ ] Versioning & lifecycle rules  
- [ ] Logs retained properly  
- [ ] Autoscaling configured  

---

# END

# AWS Terraform Module Template

*Purpose: Safely automate AWS resource provisioning (EC2, RDS, S3, IAM, etc.) with reusable modules.*

## When to Use
- New AWS service deployments
- IaC for scalable/ephemeral infrastructure
- Secure, auditable production changes

---

# TEMPLATE STARTS HERE

## variables.tf

variable "region" { type = string; default = "us-east-1" } variable "tags" { type = map(string); default = {} } variable "bucket_name" { type = string } main.tf (example: S3 Bucket)

provider "aws" { region = var.region }

resource "aws_s3_bucket" "main" { bucket = var.bucket_name tags = var.tags }

output "bucket_arn" { value = aws_s3_bucket.main.arn } README.md (snippet)

module "bucket" { source = "./modules/s3" bucket_name = "acme-app-prod-assets" tags = { Environment = "prod", Owner = "team-y" } } Quality Checklist

AWS credentials provided via environment or secrets manager Remote state (S3 + DynamoDB) configured for production Plan and apply peer-reviewed IAM roles use least privilege for module resources ---

assets/cloud/template-azure-terraform.md

# Azure Terraform Module Template

*Purpose: Automate and version Azure resource deployment (storage, compute, networking, RBAC) with secure, modular IaC.*

## When to Use
- Provisioning Azure resources via CI/CD
- Environment promotion and compliance
- RBAC, networking, storage automation

---

# TEMPLATE STARTS HERE

## variables.tf

variable "resource_group_name" { type = string } variable "location" { type = string; default = "eastus" } variable "tags" { type = map(string); default = {} } main.tf (example: Storage Account)

provider "azurerm" { features = {} }

resource "azurerm_resource_group" "main" { name = var.resource_group_name location = var.location tags = var.tags }

resource "azurerm_storage_account" "main" { name = "${var.resource_group_name}sa" resource_group_name = azurerm_resource_group.main.name location = azurerm_resource_group.main.location account_tier = "Standard" account_replication_type = "LRS" tags = var.tags } outputs.tf

output "storage_account_name" { value = azurerm_storage_account.main.name } README.md (snippet)

module "storage" { source = "./modules/azure-storage" resource_group_name = "my-app-prod" tags = { environment = "prod", owner = "team-z" } } Quality Checklist

Provider and resource group created with correct permissions State stored securely (Azure Blob, key vault, etc.) Secrets not hardcoded Peer review and CI plan before production apply

# Infrastructure Cost Optimization Template (DevOps)

*Purpose: A template for reviewing, analyzing, and optimizing cloud and platform costs across compute, storage, networking, and managed services.*

---

# 1. Overview

**Scope:**  
- [ ] Single service  
- [ ] Entire cluster  
- [ ] Project/account  
- [ ] Multi-cloud  

**Cloud Provider(s):**  
- [ ] AWS  
- [ ] GCP  
- [ ] Azure  
- [ ] Other: _______  

**Period Reviewed:**  
[Last 30 / 90 days]

**Owner:**  
[Name]

---

# 2. Cost Sources

- [ ] Compute (VMs, node pools, serverless)  
- [ ] Storage (block, object, DB)  
- [ ] Managed DB (RDS/CloudSQL/SQL DB)  
- [ ] Network egress  
- [ ] Load balancers & gateways  
- [ ] Observability tooling  
- [ ] CI/CD infrastructure  

---

# 3. High-Level Breakdown

| Category | Monthly Cost | % of Total |
|---------|--------------|------------|
| Compute | | |
| Storage | | |
| DB | | |
| Network | | |
| Other | | |

Identify top 5 cost drivers.

---

# 4. Compute Optimization

Checklist:
- [ ] Right-sizing instances or node pools  
- [ ] Remove idle/underutilized resources  
- [ ] Use autoscaling aggressively  
- [ ] Apply spot/preemptible workloads where safe  
- [ ] Consolidate workloads (bin-packing)  
- [ ] Reserved instances / savings plans / committed use discounts  

---

# 5. Kubernetes-Specific Optimization

Checklist:
- [ ] Resource requests tuned to real usage  
- [ ] No massive overprovisioning  
- [ ] Remove unused deployments/CRDs  
- [ ] Autoscaling for pods & nodes active  
- [ ] Remove zombie pods/namespaces  
- [ ] Right-size cluster node types  

---

# 6. Storage Optimization

Checklist:
- [ ] Unused volumes removed  
- [ ] S3/GCS/Blob lifecycle policies applied  
- [ ] Logs compressed & tiered to cheaper storage  
- [ ] Cold data moved to Glacier/Archive tiers  
- [ ] DB storage right-sized  
- [ ] Duplicate data reduced  

---

# 7. Database Cost Optimization

Checklist:
- [ ] Evaluate read replicas vs cache  
- [ ] Right-size DB instance classes  
- [ ] Storage auto-scaling limits verified  
- [ ] Index bloat under control  
- [ ] Delete stale test databases  
- [ ] Use managed backup retention wisely  

---

# 8. Network & Egress

Checklist:
- [ ] Minimize cross-region traffic  
- [ ] Cache external API calls  
- [ ] Use private connectivity where cheaper  
- [ ] Optimize CDN usage  
- [ ] Reduce unnecessary large payloads  

---

# 9. Observability & Tooling

Checklist:
- [ ] Log volume reduced with sampling / filters  
- [ ] Metrics cardinality under control  
- [ ] Retention periods reasonable  
- [ ] Multiple tools consolidated where possible  

---

# 10. Optimization Actions

List proposed changes:

| Action | Est. Savings | Impact Risk | Owner | ETA |
|-------|--------------|-------------|--------|-----|
| | | | | |

---

# 11. Validation Plan

- [ ] Apply changes in non-prod first  
- [ ] Monitor performance/SLOs after cost changes  
- [ ] Capture before/after cost graphs  
- [ ] Ensure no performance regressions  

---

# 12. Completed Example

**Scope:** K8s cluster + DB + S3 (prod)  
**Findings:**  
- Overprovisioned nodes (50% avg CPU)  
- S3 logs stored in Standard instead of IA  
- DB instance class larger than necessary  

**Actions:**  
- Right-size node pools  
- Apply lifecycle to S3 buckets  
- Reduce DB instance one size  

**Result:**  
- ~25% monthly cost reduction  
- No SLO impact  

---

# END

# Azure Cloud Operations Template (DevOps)

*Purpose: A full operational template for Azure Kubernetes Service (AKS), Azure Functions, networking, identity, monitoring, and infrastructure operations.*

---

# 1. Overview

**Environment:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  

**Task Type:**  
- [ ] AKS deployment  
- [ ] ACR image push  
- [ ] VMSS scaling  
- [ ] Networking (VNet/Subnets/NSG)  
- [ ] Key Vault operations  
- [ ] App Service / Functions deploy  
- [ ] Azure Monitor setup  
- [ ] Incident response  

---

# 2. Core Azure Architectures

## 2.1 VNet Architecture

Checklist:
- [ ] Hub-and-spoke or flat VNet defined  
- [ ] Private endpoints used  
- [ ] NSGs with least privilege rules  
- [ ] No public IP for internal services  

---

# 3. AKS Operations

## 3.1 Deployment

kubectl apply -f deployment.yaml kubectl rollout status deployment/app

Checklist:
- [ ] Managed identity for pods  
- [ ] Azure CNI vs Kubenet decision documented  
- [ ] Autoscaler enabled  
- [ ] Node pools separated (system/user)  
- [ ] Azure Monitor for containers enabled  

---

## 3.2 AKS Node Pool Upgrade

az aks nodepool upgrade \ --resource-group <rg> \ --cluster-name <cluster> \ --name <pool> \ --kubernetes-version <version>

Checklist:
- [ ] Zero-downtime validated  
- [ ] PDBs in place  
- [ ] Surge upgrades configured  

---

# 4. Azure Container Registry (ACR)

## Build & Push

az acr build --registry <acr-name> --image app:<tag> .

Checklist:
- [ ] ACR firewall rules configured  
- [ ] Only managed identities access  
- [ ] Image scanning enabled  

---

# 5. Azure App Service & Functions

## App Service Deploy

az webapp deploy \ --resource-group <rg> \ --name <app> \ --src-path app.zip

Checklist:
- [ ] Health checks configured  
- [ ] Autoscale rules defined  
- [ ] App Insights enabled  

---

## Azure Functions Deploy

func azure functionapp publish <app-name>

Checklist:
- [ ] Consumption vs Premium tier selected properly  
- [ ] Cold start impact measured  
- [ ] Logging enabled  

---

# 6. Identity & Key Vault

## Key Vault Checklist

- [ ] Secrets stored in Vault only  
- [ ] RBAC mode enabled  
- [ ] Private endpoints enabled  
- [ ] Soft delete + purge protection  
- [ ] Managed identities for apps  

---

# 7. Azure Monitor & Logging

## Monitor Metrics

- CPU / Memory  
- AKS node/pod health  
- App Insights performance  
- Storage queue length  
- API latency  

## Log Analytics Queries (KQL)

ContainerLog | where LogEntry contains "error"

Checklist:
- [ ] Alerts configured  
- [ ] Dashboards created  
- [ ] SLOs tracked  

---

# 8. Scaling & Autoscaling

## VMSS

az vmss scale \ --name <vmss> \ --new-capacity 5 \ --resource-group <rg>

Checklist:
- [ ] Autoscale rules defined  
- [ ] CPU/memory thresholds correct  
- [ ] Health probe validated  

---

# 9. Incident Response (Azure)

## App Down

- Check App Service health  
- Check logs in App Insights  
- Check regional outage notifications  
- Restart App Service  

## AKS Issues

- `kubectl describe pod`  
- Check ACR access  
- Node pool exhaustion  
- API server throttling  

## Storage Account Issues

- Check firewall/endpoint config  
- Check queue backlog  
- Check availability events  

---

# 10. Final Azure Ops Checklist

- [ ] Use managed identities  
- [ ] Network security (NSG + private endpoints)  
- [ ] ACR + App Services access locked down  
- [ ] Logging + Alerts configured  
- [ ] Application Insights dashboards  
- [ ] Scaling validated  
- [ ] Backups tested  

---

# END

# CI/CD Pipeline Template (DevOps)

*Purpose: A complete, production-ready CI/CD template for building, testing, securing, deploying, promoting, and rolling back software changes safely and repeatably.*

---

# 1. Overview

**Service / Application:**  
[name]

**Pipeline Type:**  
- [ ] Build-only  
- [ ] Build + deploy  
- [ ] Full promotion (dev → stage → prod)  
- [ ] GitOps (ArgoCD/Flux)  

**Deployment Target:**  
- [ ] Kubernetes  
- [ ] VM / Server  
- [ ] Serverless  
- [ ] Container registry only  
- [ ] Multi-region  

**Build Artifacts:**  
- [ ] Docker image  
- [ ] Binary  
- [ ] Package (npm/pip/etc.)  
- [ ] Terraform plan  
- [ ] Helm chart  

---

# 2. Pipeline Structure

A standard end-to-end CI/CD pipeline follows:

1. Source Control Trigger 2. Static Analysis 3. Build 4. Unit Tests 5. Integration Tests 6. Security Scans 7. Artifact Packaging 8. Deploy to Staging 9. Smoke Tests 10. Approval Gate 11. Deploy to Production 12. Verification & Monitoring 13. Rollback if required

---

# 3. Pipeline Triggers

### 3.1 CI (Build/Test)

Triggered on:
- Pull requests  
- Commits to main  
- Scheduled nightly tests  
- Dependency/security updates  

### 3.2 CD (Deploy)

Triggered on:
- Merge to main  
- Tag creation (e.g., `v1.2.3`)  
- Manual approval  
- GitOps reconciliation  

---

# 4. Build Stage

### Commands

npm ci npm run build go build mvn package docker build -t app:$SHA .

### Build Checklist

- [ ] Deterministic builds  
- [ ] Version embedded into artifact  
- [ ] Build cache enabled  
- [ ] Build reproducibility validated  

---

# 5. Test Stage

## 5.1 Unit Tests

npm test -- --coverage pytest go test ./...

Checklist:
- [ ] Test coverage target met  
- [ ] No flaky tests  
- [ ] Test time < target  

---

## 5.2 Integration Tests

- Run using docker-compose or ephemeral environment  
- Services mocked only when needed  

Checklist:
- [ ] API endpoints tested  
- [ ] Database setup isolated  
- [ ] Cleanup script runs on failure  

---

## 5.3 Smoke Tests (Staging)

curl -f <https://staging.example.com/healthz>

Checklist:
- [ ] Deployment healthy  
- [ ] Basic functionality validated  
- [ ] Errors logged to CI console  

---

# 6. Security Scans

## 6.1 Dependency Scans
- Snyk  
- Dependabot  
- Trivy  

## 6.2 Code Scans
- SAST (e.g., CodeQL)  
- Linting (flake8, eslint)  

## 6.3 Container Scans

trivy image app:$SHA

## 6.4 IaC Scans
- Checkov  
- Tfsec  

### Security Checklist

- [ ] No critical vulnerabilities  
- [ ] SBOM generated  
- [ ] Secrets scan clean  
- [ ] Image signed  

---

# 7. Artifact Packaging

## 7.1 Docker

docker build -t registry/app:$SHA . docker push registry/app:$SHA

## 7.2 Package Repos

npm publish pip upload mvn deploy

Checklist:
- [ ] Artifact immutable  
- [ ] Tagged with commit SHA  
- [ ] Stored in registry  
- [ ] Retention policies configured  

---

# 8. Staging Deployment

## Deployment Strategies

- [ ] Rolling  
- [ ] Blue/Green  
- [ ] Canary  
- [ ] GitOps (ArgoCD/Flux)  

Example command:

helm upgrade --install app ./charts/app --namespace staging \ --set image.tag=$SHA

Checklist:
- [ ] Staging auto-deployed  
- [ ] Smoke tests pass  
- [ ] Monitoring dashboard green  

---

# 9. Promotion & Approval

## 9.1 Approval Gate

Required for production:

- [ ] SRE/DevOps approval  
- [ ] Product approval  
- [ ] Security approval (if sensitive)  

Checklist:
- [ ] Error budgets respected  
- [ ] Release notes published  
- [ ] Rollback plan validated  

---

# 10. Production Deployment

### Example K8s Production Deploy

helm upgrade --install app ./charts/app \ --namespace prod \ --set image.tag=$SHA

### Example GitOps Deployment

git commit -am "prod deploy: $SHA" git push

ArgoCD/Flux syncs automatically.

### Production Checklist

- [ ] Health checks passing  
- [ ] p99 latency stable  
- [ ] No spike in error rate  
- [ ] Logs clean  
- [ ] Capacity within thresholds  

---

# 11. Deployment Strategies

## 11.1 Rolling Update

- Minimal risk  
- Good default  

Checklist:
- [ ] Readiness probe correct  
- [ ] MaxSurge/MaxUnavailable tuned  

---

## 11.2 Blue/Green

blue = live green = new version

Checklist:
- [ ] Full validation of green  
- [ ] Traffic switch atomic  
- [ ] Blue preserved for rollback  

---

## 11.3 Canary

1% → 5% → 20% → 50% → 100%

Checklist:
- [ ] Automated rollback  
- [ ] Metrics compared between cohorts  
- [ ] SLO-based gating  

---

# 12. Rollback

### Rollback Methods

- [ ] Deploy previous image  
- [ ] Revert Git commit (GitOps)  
- [ ] Helm rollback  
- [ ] Restore previous config  
- [ ] Disable feature flags  

### Rollback Plan Template

Rollback Trigger: Rollback Method: Rollback Steps: Validation After Rollback:

### Rollback Checklist

- [ ] Rollback < 2 minutes  
- [ ] No schema-incompatible changes  
- [ ] Post-rollback monitoring verified  

---

# 13. CI/CD Security

- [ ] No long-lived tokens  
- [ ] Use OIDC to cloud for access  
- [ ] Secrets in CI vaults only  
- [ ] Principle of least privilege  
- [ ] Logs don’t include secrets  
- [ ] Scoped permissions for pipelines  

---

# 14. Example Full Pipeline (Generic YAML)

stages:

• build
• test
• security
• deploy

build: script:

• npm ci
• npm run build

test: script:

• npm test

security: script:

• snyk test
• trivy fs .

deploy: script:

• helm upgrade --install app .

when: manual

---

# 15. Final Review Checklist

### Pipeline Quality
- [ ] Build deterministic  
- [ ] Tests reliable  
- [ ] Security checks automated  
- [ ] Promotion flow clear  

### Deployment Safety
- [ ] Rollback tested  
- [ ] Canary or rolling strategy  
- [ ] No manual steps in prod deploy  

### Reliability
- [ ] Observability integrated  
- [ ] Dashboards updated  
- [ ] Alerts tuned post-deploy  

---

# END

GitHub Actions CI/CD Template

Purpose: Build/test, deploy with approvals, and support rollback using a secure, auditable workflow.

When to Use

• Application/container build/test/deploy
• IaC workflows (Terraform/OpenTofu, CloudFormation, etc.)
• GitOps changes (manifests/Helm/Kustomize) gated by PR checks

---

TEMPLATE STARTS HERE

.github/workflows/ci-cd.yml (example)

name: CI/CD

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write # For cloud OIDC (optional)

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Replace with your stack (Node/Go/Python/etc.)
      - name: Run tests
        run: ./scripts/test.sh

  deploy:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    needs: ci
    environment: production # Configure environment protection rules in GitHub UI
    steps:
      - uses: actions/checkout@v4

      # Optional: authenticate via OIDC (AWS/GCP/Azure) instead of long-lived secrets.
      - name: Deploy
        run: ./scripts/deploy.sh

  rollback:
    if: failure()
    runs-on: ubuntu-latest
    needs: deploy
    steps:
      - uses: actions/checkout@v4
      - name: Roll back
        run: ./scripts/rollback.sh

Quality Checklist

• Secrets come from GitHub Actions secrets/OIDC (no plaintext in repo/logs)
• Deploys are gated by environment protection rules (approvals, required checks)
• Artifacts are immutable and versioned (image digest/tag, build provenance)
• Rollback is tested (automated where possible) and documented in a runbook
• Notifications exist for deploy/rollback outcomes (Slack/webhook/email)

# GitOps Template (ArgoCD / Flux)

*Purpose: A practical template for designing, operating, and troubleshooting GitOps workflows with ArgoCD or Flux.*

---

# 1. Overview

**Service / System:**  
[name]

**GitOps Tool:**  
- [ ] ArgoCD  
- [ ] Flux  

**Environment:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  
- [ ] multi-cluster  

**Scope:**  
- [ ] New GitOps app  
- [ ] Promotion flow  
- [ ] Rollback flow  
- [ ] Multi-tenant setup  
- [ ] Multi-cluster sync  

---

# 2. Repo & Structure

## 2.1 Git Layout

Common patterns:

- **App repo** (code)  
- **Ops/Env repo** (manifests, Helm releases, Kustomize)

Example:

app-repo/ src/ Dockerfile ...

ops-repo/ apps/ app1/ base/ overlays/ dev/ staging/ prod/ clusters/ prod-eu1/ prod-us1/

Checklist:
- [ ] Separation of code vs infra state  
- [ ] Environments defined as overlays  
- [ ] Kustomize or Helm used consistently  

---

# 3. ArgoCD Configuration

## 3.1 Application CR Example

apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: app1-prod spec: project: default source: repoURL: https://github.com/org/ops-repo.git path: apps/app1/overlays/prod targetRevision: main destination: server: https://kubernetes.default.svc namespace: app1-prod syncPolicy: automated: prune: true selfHeal: true syncOptions:

• CreateNamespace=true

Checklist:

- [ ] Source points to env overlay  
- [ ] Automated sync only if desired  
- [ ] SelfHeal enabled for true GitOps  
- [ ] Prune enabled once safe  

---

# 4. Flux Configuration

## 4.1 GitRepository + Kustomization Example

apiVersion: source.toolkit.fluxcd.io/v1 kind: GitRepository metadata: name: ops-repo spec: url: https://github.com/org/ops-repo.git branch: main --- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: app1-prod spec: interval: 1m path: ./apps/app1/overlays/prod prune: true sourceRef: kind: GitRepository name: ops-repo

Checklist:

- [ ] Interval set appropriately  
- [ ] Prune true when ready  
- [ ] SourceRef points to repo object  

---

# 5. Promotion Workflow

## 5.1 Git-based Promotion

Flow:

1. Build image from app repo  
2. Tag image with SHA  
3. Update `values.yaml` / Kustomize patch in ops repo  
4. Open PR to ops repo (dev → staging → prod)  
5. Merge PR → GitOps tool syncs  

Checklist:

- [ ] Promotion via Git, not kubectl  
- [ ] Image tags immutable  
- [ ] PR reviewed and approved  
- [ ] CI validates manifests before merge  

---

# 6. Rollback Workflow

GitOps rollback is **git revert**:

git revert <commit> git push

ArgoCD/Flux will sync to the reverted state.

Checklist:

- [ ] Revert tested in non-prod  
- [ ] Rollback < 2–3 minutes  
- [ ] SLOs monitored during rollback  

---

# 7. Sync & Health Checks

## ArgoCD

- Sync status: Synced / OutOfSync  
- Health: Healthy / Degraded  

Commands:

argocd app list argocd app get app1-prod argocd app sync app1-prod

## Flux

Commands:

flux get kustomizations flux reconcile kustomization app1-prod

Checklist:

- [ ] No manual kubectl in prod for managed resources  
- [ ] Sync failures alerted  
- [ ] Drift from Git resolved via Git, not cluster edits  

---

# 8. Security & Access

Checklist:

- [ ] GitOps tool uses least-privilege SA  
- [ ] Read-only access to manifests for developers (write via PR)  
- [ ] Secrets handled with SOPS/SealedSecrets  
- [ ] No plain-text secrets in ops repo  

---

# 9. Troubleshooting

## Common Issues

- **OutOfSync**: changes made manually → fix via Git  
- **Degraded**: manifests applied but app failing → check K8s events  
- **Permissions**: GitOps SA cannot apply resource → update RBAC  
- **Sync loops**: tool fighting manual changes → block kubectl changes in prod  

---

# 10. Final GitOps Checklist

- [ ] Desired state in Git only  
- [ ] No manual prod edits  
- [ ] Auditable PRs for all changes  
- [ ] Rollbacks via commits  
- [ ] Alerts for sync/health issues  
- [ ] Secrets encrypted at rest  

---

# END

# Release Safety Template (DevOps)

*Purpose: A complete operational template for planning, executing, validating, and safely rolling out production releases using modern DevOps and progressive delivery patterns.*

---

# 1. Release Overview

**Service / Component:**  
[name]

**Release Version / Build ID:**  
[example: v1.7.0 / SHA256: abc123]

**Environment:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  
- [ ] multi-region  

**Release Owner:**  
[name]

**Release Window:**  
[start / end time]

**Approvals Required:**  
- [ ] Engineering  
- [ ] SRE  
- [ ] Product  
- [ ] Security  

---

# 2. Release Strategy

Choose exactly one:

- [ ] Rolling update  
- [ ] Blue/Green  
- [ ] Canary  
- [ ] Shadow traffic  
- [ ] Feature-flag gated  
- [ ] GitOps promotion  

**Rationale:**  
[Why this strategy is appropriate]

---

# 3. Pre-Release Validation

## 3.1 Deployment Readiness Checklist

- [ ] CI pipeline green  
- [ ] Unit tests passed  
- [ ] Integration tests passed  
- [ ] Security scans passed (SAST/DAST/Image scan)  
- [ ] Dependencies validated  
- [ ] Docker image signed  
- [ ] Infrastructure drift-free  
- [ ] No “freeze” period in effect  
- [ ] Code freeze followed (if applicable)  

---

## 3.2 Observability Readiness

- [ ] Dashboards updated with new version tags  
- [ ] Alerts tuned & non-noisy  
- [ ] Synthetic checks configured  
- [ ] SLO/Error budgets in healthy range  

---

## 3.3 Database Migration Check

- [ ] Schema changes backward compatible  
- [ ] Migrations tested in staging  
- [ ] Safe expand → migrate → contract workflow  
- [ ] Rollback path exists  
- [ ] No destructive operations in peak hours  

---

# 4. Release Steps (Main Plan)

1. Prepare artifacts 2. Deploy to staging 3. Run staging smoke tests 4. Validate logs & metrics 5. Trigger approval step 6. Deploy to production using selected strategy 7. Verify deployment 8. Monitor for regression 9. Declare success or rollback

---

# 5. Deployment Instructions

## 5.1 Rolling Deployment

kubectl set image deployment/app app=registry/app:$VERSION kubectl rollout status deployment/app

Checklist:
- [ ] Readiness probe validated  
- [ ] No pod crash loops  
- [ ] Resource usage stable  

---

## 5.2 Blue/Green Deployment

blue = live green = new candidate

1. Deploy green 2. Smoke test green 3. Shift traffic to green 4. Keep blue as rollback target

Checklist:
- [ ] Health checks green  
- [ ] DB schema compatible with both versions  
- [ ] Traffic cutover logged  
- [ ] LB switch reversible  

---

## 5.3 Canary Deployment

weights: 1% → 5% → 20% → 50% → 100%

Monitoring:
- Error rate  
- P95/P99 latency  
- CPU/memory  
- Queue length  
- DB connections  

Rollback rules:
- > 5% error rate  
- P99 latency > threshold  
- SLO burn rate high  
- Saturation > safe level  

---

# 6. Automated Gating

## 6.1 Quality Gates

- [ ] Unit test coverage > X%  
- [ ] Zero critical vulns  
- [ ] Signed artifacts only  
- [ ] Image scanning passed  
- [ ] Static analysis passed  
- [ ] Integration tests passed  

## 6.2 Deployment Gates

- [ ] Canary performance green  
- [ ] Error rate < threshold  
- [ ] No new alerts triggered  
- [ ] Rollout step < 10 minutes  

---

# 7. Risk Assessment

## 7.1 Release Risk Scoring

Rate each item 1–5 (5 = high risk):

| Area | Score | Notes |
|------|--------|--------|
| Schema changes | | |
| Cross-service dependencies | | |
| External integrations | | |
| Large code diff | | |
| Release frequency | | |
| Operational history | | |

**Total Risk Score:** [sum]

Interpretation:  
- **< 10** → Low risk  
- **10–18** → Moderate risk  
- **18+** → High risk (extra approvals needed)  

---

# 8. Rollback Plan

**Rollback Method (choose one):**
- [ ] Revert deployment (Helm/K8s)  
- [ ] Redeploy previous artifact  
- [ ] GitOps revert  
- [ ] Blue/Green fallback  
- [ ] Disable feature flags  
- [ ] DB rollback via backups/PITR  

### Rollback Template

Rollback Trigger: Rollback Window: Rollback Steps: Verification Steps After Rollback: Communication Plan:

Rollback Checklist:
- [ ] DB schema compatible  
- [ ] Previous version tested  
- [ ] Rollback < 2 minutes  
- [ ] Observability dashboard ready  
- [ ] Post-rollback smoke tests  

---

# 9. Post-Deployment Verification

## 9.1 Technical Verification

- [ ] New pods healthy  
- [ ] No CrashLoopBackOff  
- [ ] Metrics stable (latency, errors, saturation)  
- [ ] No increase in p99 latency  
- [ ] No new alerts firing  
- [ ] Logs show expected patterns  

## 9.2 Functional Verification

- [ ] Critical endpoints healthy  
- [ ] User-facing flows validated  
- [ ] Background jobs running correctly  
- [ ] Scheduled tasks unaffected  

---

# 10. Communication Plan

Channels:
- Engineering  
- SRE  
- Product/Business  
- Customer-facing status page  

Templates:

### Start Notification

Deploying version X to production. Expected impact: none.

### Completion Notification

Deploy version X successful. All metrics normal.

### Rollback Notification

Rollback initiated for version X due to <reason>. Previous version restored.

---

# 11. Completed Example

**Service:** Payments API  
**Version:** v3.12.0  
**Strategy:** Canary (1%→5%→20%→100%)  
**Issues:** Slight latency spike at 5% stage  
**Rollback:** Not required  
**Post-Deploy:** Stable for 45 minutes, declared success  
**Next Steps:** Optimize DB connection pool  

---

# END

Cost Governance & Capacity Planning

Production-grade cost control for cloud infrastructure, Kubernetes, and DevOps platforms.

---

Cost Governance Framework

Cost Visibility Checklist

• [ ] Resource tagging strategy implemented
• [ ] Cost allocation by team/project/environment
• [ ] Budget alerts at 50%, 80%, 100%
• [ ] Anomaly detection enabled
• [ ] Monthly cost review meeting scheduled
• [ ] Chargeback/showback model defined
• [ ] Unused resource detection automated

Required Tags

Tagging Enforcement (Terraform)

# variables.tf - Required tags
variable "required_tags" {
  type = object({
    team        = string
    project     = string
    environment = string
    cost_center = string
    owner       = string
  })
}

# Validate tags exist
locals {
  common_tags = merge(var.required_tags, {
    managed_by = "terraform"
    created_at = timestamp()
  })
}

---

Cloud Cost Optimization

Compute Right-Sizing

AWS EC2:

# Find underutilized instances (AWS CLI + CloudWatch)
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-1234567890abcdef0 \
  --start-time $(date -d '7 days ago' --utc +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date --utc +%Y-%m-%dT%H:%M:%SZ) \
  --period 3600 \
  --statistics Average

# Target: <20% avg CPU = downsize candidate

GCP Compute:

# Recommender API for right-sizing
gcloud recommender recommendations list \
  --project=my-project \
  --location=us-central1 \
  --recommender=google.compute.instance.MachineTypeRecommender

Storage Optimization

# Find unattached EBS volumes (AWS)
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[*].[VolumeId,Size,CreateTime]' \
  --output table

Reserved/Committed Capacity

Coverage Target: 60-70% baseline on reservations, 30-40% on-demand/spot.

---

Kubernetes Cost Control

Resource Requests & Limits

# Good: Explicit requests and limits
resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

Golden Rules:

• Always set requests (enables scheduling efficiency)
• Set limits ≤ 2x requests (prevents runaway pods)
• Review and adjust quarterly based on actual usage

Namespace Resource Quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-backend
spec:
  hard:
    requests.cpu: "20"
    requests.memory: "40Gi"
    limits.cpu: "40"
    limits.memory: "80Gi"
    pods: "50"
    persistentvolumeclaims: "10"

Cluster Autoscaler Configuration

# Cluster autoscaler optimized for cost
apiVersion: autoscaling.k8s.io/v1
kind: ClusterAutoscaler
spec:
  scaleDownEnabled: true
  scaleDownDelayAfterAdd: 10m
  scaleDownUnneededTime: 10m
  scaleDownUtilizationThreshold: 0.5  # Scale down if <50% utilized
  maxNodeProvisionTime: 15m
  skipNodesWithLocalStorage: false
  expander: least-waste  # Choose node pool with least waste

Pod Disruption Budgets (for scale-down)

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2  # Or use maxUnavailable
  selector:
    matchLabels:
      app: api

---

Capacity Planning

Capacity Planning Checklist

• [ ] Current utilization baseline established
• [ ] Growth rate calculated (users, traffic, data)
• [ ] Lead time for scaling known (days/weeks)
• [ ] Peak load patterns documented
• [ ] Cost per unit capacity calculated
• [ ] Scaling triggers defined
• [ ] Quarterly capacity review scheduled

Capacity Model Template

## Capacity Model: [Service Name]

### Current State (as of YYYY-MM-DD)
- Active users: X
- Requests/second (p99): X
- CPU utilization (avg): X%
- Memory utilization (avg): X%
- Storage used: X GB
- Monthly cost: $X

### Growth Assumptions
- User growth: X% per month
- Traffic growth: X% per month
- Data growth: X GB/month

### Scaling Triggers
| Metric | Warning | Critical | Action |
|--------|---------|----------|--------|
| CPU | 60% | 80% | Add replicas |
| Memory | 70% | 85% | Increase limits |
| Storage | 70% | 85% | Expand volume |
| Latency p99 | 500ms | 1000ms | Scale out |

### 6-Month Projection
| Month | Users | RPS | Cost |
|-------|-------|-----|------|
| +1 | | | $ |
| +3 | | | $ |
| +6 | | | $ |

### Recommendations
1. [Action item]
2. [Action item]

---

FinOps Practices

Monthly Cost Review Agenda

1. Cost Summary (5 min)

• Total spend vs budget
• Month-over-month change
• Top 5 cost drivers

2. Anomalies (10 min)

• Unexpected cost spikes
• New resources without tags
• Orphaned resources

3. Optimization Opportunities (15 min)

• Right-sizing recommendations
• Reserved capacity gaps
• Unused resources to delete

4. Action Items (10 min)

• Assign optimization tasks
• Update budgets if needed
• Schedule follow-ups

Cost Optimization Workflow

Weekly:
├─ Review cost anomaly alerts
├─ Delete unused resources identified by automation
└─ Check for untagged resources

Monthly:
├─ Run right-sizing analysis
├─ Review reserved capacity utilization
├─ Update cost forecasts
└─ Present to stakeholders

Quarterly:
├─ Review tagging strategy
├─ Evaluate new pricing models
├─ Adjust reserved capacity
└─ Capacity planning refresh

---

Alert Configuration

Budget Alerts (AWS)

# Terraform - AWS Budget
resource "aws_budgets_budget" "monthly" {
  name              = "monthly-budget"
  budget_type       = "COST"
  limit_amount      = "10000"
  limit_unit        = "USD"
  time_period_start = "2025-01-01_00:00"
  time_unit         = "MONTHLY"

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 80
    threshold_type            = "PERCENTAGE"
    notification_type         = "ACTUAL"
    subscriber_email_addresses = ["finance@company.com"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 100
    threshold_type            = "PERCENTAGE"
    notification_type         = "FORECASTED"
    subscriber_email_addresses = ["cto@company.com"]
  }
}

Cost Anomaly Detection (AWS)

resource "aws_ce_anomaly_monitor" "service" {
  name              = "service-cost-monitor"
  monitor_type      = "DIMENSIONAL"
  monitor_dimension = "SERVICE"
}

resource "aws_ce_anomaly_subscription" "alert" {
  name      = "cost-anomaly-alerts"
  frequency = "DAILY"
  threshold_expression {
    dimension {
      key           = "ANOMALY_TOTAL_IMPACT_PERCENTAGE"
      match_options = ["GREATER_THAN_OR_EQUAL"]
      values        = ["10"]  # Alert if >10% above expected
    }
  }
  subscriber {
    type    = "EMAIL"
    address = "platform-team@company.com"
  }
}

---

Do / Avoid

GOOD: Do

• Tag all resources at creation time
• Set budget alerts before hitting limits
• Review right-sizing recommendations monthly
• Use spot/preemptible for fault-tolerant workloads
• Set Kubernetes resource requests on all pods
• Enable cluster autoscaler with scale-down
• Document capacity planning assumptions

BAD: Avoid

• Deploying without cost tags
• Running dev resources 24/7
• Over-provisioning "just in case"
• Ignoring reserved capacity opportunities
• Setting identical requests and limits (no burst)
• Disabling scale-down to "avoid disruption"
• Waiting for bill shock to investigate

---

Anti-Patterns

---

Optional: AI/Automation

Note: AI assists with analysis but cost decisions need human approval.

Automated Optimization

• Unused resource detection and notification
• Right-sizing recommendation generation
• Anomaly detection and alerting
• Reserved capacity recommendation

AI-Assisted Analysis

• Cost trend prediction
• Usage pattern identification
• Optimization prioritization

Bounded Claims

• AI recommendations need validation before action
• Automated deletions require approval workflow
• Cost predictions are estimates, not guarantees

---

Tools Reference

---

Related Templates

• template-aws-terraform.md — AWS infrastructure
• template-cost-optimization.md — AWS-specific optimization
• template-kubernetes-ops.md — K8s operations

---

Last Updated: December 2025

# Docker Operations Template (DevOps)

*Purpose: A complete operational template for building, securing, optimizing, distributing, running, and debugging Docker containers in production environments.*

---

# 1. Overview

**Service / Component:**  
[name]

**Environment:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  

**Task Type:**  
- [ ] Build image  
- [ ] Optimize Dockerfile  
- [ ] Scan / security harden  
- [ ] Publish to registry  
- [ ] Debug container  
- [ ] Multi-stage build  
- [ ] Runtime troubleshooting  

**Registry:**  
- [ ] Docker Hub  
- [ ] GHCR  
- [ ] ECR  
- [ ] GCR  
- [ ] ACR  
- [ ] Self-hosted  

---

# 2. Dockerfile Standards

## 2.1 Recommended Minimal Base Image

FROM alpine:3.20 RUN adduser -D appuser USER appuser

OR

FROM gcr.io/distroless/base

Checklist:
- [ ] Minimal base image  
- [ ] Multi-stage build used  
- [ ] Non-root user  
- [ ] No secrets copied into image  
- [ ] No `curl | bash`  
- [ ] Avoid `ADD` (use `COPY`)  
- [ ] Pin versions for deterministic builds  

---

## 2.2 Multi-Stage Build Template

Build Stage

FROM golang:1.22 AS builder WORKDIR /src COPY . . RUN go build -o app .

Runtime Stage

FROM alpine:3.20 RUN adduser -D appuser USER appuser COPY --from=builder /src/app /app ENTRYPOINT ["/app"]

Checklist:
- [ ] Build & runtime stages separate  
- [ ] No build tools in final image  
- [ ] Final image size small (<100MB preferred)  

---

# 3. Image Scanning & SBOM

## 3.1 Vulnerability Scan

trivy image registry/app:$VERSION

Checklist:
- [ ] No critical vulnerabilities  
- [ ] Medium vulns triaged  
- [ ] Image updated to latest patches  

## 3.2 SBOM (Software Bill of Materials)

syft registry/app:$VERSION -o json > sbom.json

## 3.3 Image Signing

cosign sign --key cosign.key registry/app:$VERSION

Checklist:
- [ ] Signature stored  
- [ ] Verification in CI/CD  
- [ ] Policy: unsigned images blocked (Kyverno/OPA)  

---

# 4. Build & Push Workflow

## 4.1 Build

docker build -t registry/app:$SHA .

## 4.2 Tag

docker tag registry/app:$SHA registry/app:latest

## 4.3 Push

docker push registry/app:$SHA docker push registry/app:latest

### Checklist

- [ ] Tag includes commit SHA  
- [ ] Immutable tags used for deploys  
- [ ] Avoid floating tags in prod  
- [ ] Use registry-side retention policies  

---

# 5. Local Development

## 5.1 Run Container

docker run -p 8080:8080 registry/app:$TAG

## 5.2 Mount Code for Live Reload

docker run -p 8080:8080 -v $(pwd):/workspace app-dev

Checklist:
- [ ] Environment parity maintained  
- [ ] App logs visible locally  

---

# 6. Runtime Operations (Production)

## 6.1 Health Check

Add to Dockerfile:

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s \ CMD curl -f <http://localhost:8080/healthz> || exit 1

Checklist:
- [ ] Healthcheck lightweight  
- [ ] App exposes proper health endpoint  

---

## 6.2 Resource Limits (Docker Compose)

deploy: resources: limits: cpus: "1" memory: 512M reservations: cpus: "0.25" memory: 256M

Checklist:
- [ ] Avoid unbounded resource usage  
- [ ] Reserve enough memory to avoid OOMKilled  

---

# 7. Debugging Containers

## 7.1 Check Logs

docker logs <container>

## 7.2 Exec In

docker exec -it <container> sh

## 7.3 Inspect Container Metadata

docker inspect <container>

## 7.4 Check Resource Usage

docker stats

Checklist:
- [ ] No CrashLoop events  
- [ ] EntryPoint correct  
- [ ] Ports exposed correctly  
- [ ] Env vars match expected config  

---

# 8. Networking

## 8.1 View Networks

docker network ls docker network inspect <network>

## 8.2 Connect Container to Network

docker network connect <network> <container>

Checklist:
- [ ] No open ports unintentionally exposed  
- [ ] Network separation applied (frontend/backend/db)  

---

# 9. Storage & Volumes

## 9.1 Create Volume

docker volume create app-data

## 9.2 Mount Volume

docker run -v app-data:/data registry/app

Checklist:
- [ ] Avoid ephemeral data for stateful apps  
- [ ] Volume permissions correct  
- [ ] Volume drivers documented  

---

# 10. Compose / Swarm / Local Orchestration

## 10.1 docker-compose Example

version: '3.9' services: app: image: registry/app:$TAG ports: ["8080:8080"] environment:

• ENV=prod

depends_on:

• db

Checklist:
- [ ] Services start in correct order  
- [ ] Health checks configured  
- [ ] Resource limits present  

---

# 11. Docker Registry Best Practices

Checklist:
- [ ] Use private registry for production  
- [ ] Enforce pull authentication  
- [ ] Enable vulnerability scans  
- [ ] Enforce retention policies  
- [ ] Delete old tags safely  
- [ ] Use digest pinning in K8s deploys  

---

# 12. Container Security Hardening

Checklist:
- [ ] Run as non-root  
- [ ] Read-only root filesystem  
- [ ] Drop all capabilities except required  
- [ ] No SSH inside container  
- [ ] No sensitive env vars (tokens/passwords)  
- [ ] Disable inter-container network if not needed  

---

# 13. Docker Troubleshooting Guide

## 13.1 Cannot Pull Image

- [ ] Check registry auth  
- [ ] Check tag exists  
- [ ] Check network/DNS  

## 13.2 App Crashing on Start

- [ ] Inspect logs  
- [ ] Validate ENTRYPOINT / CMD  
- [ ] Validate config files present  
- [ ] Validate required env vars  

## 13.3 High Memory Usage

- [ ] Memory leak in app  
- [ ] Missing resource limits  
- [ ] Large image causing startup overhead  

## 13.4 Permission Errors

- [ ] File ownership mismatch  
- [ ] Running as non-root missing permissions  
- [ ] Volume mounted with wrong uid/gid  

---

# 14. Final Operational Checklist

### Build
- [ ] Image reproducible  
- [ ] Multi-stage build  
- [ ] Minimal base image  
- [ ] No secrets in layers  

### Security
- [ ] Image scanned (Trivy/Grype)  
- [ ] SBOM generated  
- [ ] Image signed  
- [ ] Non-root user  
- [ ] Read-only FS  

### Deployment
- [ ] Immutable tag  
- [ ] Registry health checked  
- [ ] K8s digest pinned  
- [ ] Health checks enabled  

### Runtime
- [ ] Logs correct format  
- [ ] Metrics exported  
- [ ] Alerts configured  

---

# 15. Completed Example

**Service:** Orders API  
**Image:** `registry/orders-api:sha256:cb1a…`  
**Base Image:** Distroless  
**Security:**  
- No critical vulns  
- Signed with Cosign  
- SBOM generated  

**Dockerfile:** Multi-stage, non-root, minimal runtime  
**Deployment:** K8s using pinned digest  
**Outcome:** Fast startup, small image size (26MB), zero vulnerabilities.

---

# END

# Google Cloud Operations Template (DevOps)

*Purpose: A production-ready GCP operations template for compute, GKE, networking, IAM, Cloud Logging/Monitoring, scaling, and incident response.*

---

# 1. Overview

**Environment:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  

**Task Type:**  
- [ ] GKE deployment  
- [ ] IAM config  
- [ ] VPC networking  
- [ ] Cloud Run deployment  
- [ ] Cloud Functions  
- [ ] Load balancer config  
- [ ] Artifact Registry  
- [ ] Incident response  

---

# 2. Core GCP Architecture Patterns

## 2.1 VPC Design

Checklist:
- [ ] Regional VPC  
- [ ] Separate subnets per tier  
- [ ] Private service access for databases  
- [ ] Firewall rules least-privilege  
- [ ] Cloud NAT for egress  
- [ ] No public IPs unless needed  

---

# 3. GKE Operations

## 3.1 Deployment

kubectl apply -f deployment.yaml kubectl rollout status deployment/app

Checklist:
- [ ] Use Workload Identity  
- [ ] Autopilot vs Standard decision documented  
- [ ] Node pool autoscaling enabled  
- [ ] PodDisruptionBudgets for production  
- [ ] Cloud Logging + Cloud Monitoring configured  

---

## 3.2 GKE Node Pool Management

gcloud container node-pools upgrade <pool> \ --cluster <cluster> --region <region>

Checklist:
- [ ] Surge upgrades enabled  
- [ ] Rolling update tested  
- [ ] Version skew validated  

---

# 4. Cloud Run Operations

## 4.1 Deploy

gcloud run deploy <service> \ --image gcr.io/<project>/<image>:tag \ --region <region> \ --platform managed

Checklist:
- [ ] Concurrency set appropriately  
- [ ] VPC connector for private services  
- [ ] Min instances for warm starts  
- [ ] IAM: no unauthenticated access unless intended  

---

# 5. IAM Best Practices

Checklist:
- [ ] Use IAM roles, not primitive Owner/Editor  
- [ ] Service accounts for workloads  
- [ ] Workload Identity for GKE  
- [ ] Keyless auth preferred  
- [ ] No long-lived SA keys  
- [ ] IAM Recommender reviewed regularly  

---

# 6. Cloud Storage (GCS)

Checklist:
- [ ] Uniform bucket-level access enabled  
- [ ] Bucket encryption (KMS)  
- [ ] Retention policy enforced  
- [ ] Access logs enabled  
- [ ] Avoid publicly readable buckets  

---

# 7. Cloud Logging & Monitoring

## 7.1 Logging

gcloud logging read "resource.type=gke_container" --limit 50

Checklist:
- [ ] Logs routed to SIEM if required  
- [ ] Log-based metrics created  
- [ ] Retention configured  

---

## 7.2 Monitoring

Key metrics:
- L7 LB latency  
- GKE pod restarts  
- CPU, memory, throttling  
- Pub/Sub backlog  
- Cloud Run cold starts  

Checklist:
- [ ] Alerts on high latency & errors  
- [ ] Dashboards per microservice  
- [ ] SLOs implemented via Cloud Monitoring  

---

# 8. Pub/Sub Operations

gcloud pubsub subscriptions describe <sub>

Checklist:
- [ ] Backlog monitored  
- [ ] Dead-letter topics configured  
- [ ] Message retention configured  

---

# 9. SQL / Spanner / Firestore

## Cloud SQL
- [ ] Automated backups enabled  
- [ ] Failover replicas configured  
- [ ] High CPU & connection alerts  

## Spanner
- [ ] Multi-region if required  
- [ ] Workload isolation  

## Firestore
- [ ] Security rules validation  
- [ ] Indexes verified  

---

# 10. Incident Response (GCP)

## High Latency
- Check GKE workloads  
- Check load balancer capacity  
- Scale Cloud Run or GKE pods  

## GKE Pod Failures
- `kubectl describe pod`  
- Node pool issues  
- Resource limits too low  

## IAM Denied
- Check IAM role  
- Check service account  
- Check access boundary policies  

---

# 11. Final GCP Ops Checklist

- [ ] IAM least privilege  
- [ ] No public buckets  
- [ ] GKE Workload Identity  
- [ ] Autopilot/Standard documented  
- [ ] Monitoring + SLOs configured  
- [ ] Pub/Sub backlog safe  
- [ ] Backups tested  

---

# END

Google Cloud Terraform Module Template

Purpose: Safely provision and manage Google Cloud (GCP) resources with reusable Terraform/OpenTofu modules.

When to Use

• Provisioning GCP infrastructure (GCS, GKE, Cloud SQL, IAM, networking)
• Supporting environment promotion (dev → staging → prod)
• Enforcing consistent labels, IAM boundaries, and naming conventions

---

TEMPLATE STARTS HERE

versions.tf

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">= 5.0"
    }
    random = {
      source  = "hashicorp/random"
      version = ">= 3.0"
    }
  }
}

variables.tf

variable "project_id" {
  type        = string
  description = "GCP project ID"
}

variable "region" {
  type        = string
  description = "Default region for regional resources"
  default     = "us-central1"
}

variable "labels" {
  type        = map(string)
  description = "Resource labels"
  default     = {}
}

variable "bucket_name_prefix" {
  type        = string
  description = "Prefix used to build a globally-unique bucket name"
}

variable "force_destroy" {
  type        = bool
  description = "Whether to allow Terraform to delete non-empty buckets (AVOID true in prod)"
  default     = false
}

main.tf (example: GCS bucket)

resource "random_id" "suffix" {
  byte_length = 4
}

resource "google_storage_bucket" "main" {
  name          = "${var.bucket_name_prefix}-${random_id.suffix.hex}"
  location      = var.region
  labels        = var.labels
  force_destroy = var.force_destroy

  uniform_bucket_level_access = true
}

outputs.tf

output "bucket_name" {
  value       = google_storage_bucket.main.name
  description = "Created bucket name"
}

Provider Usage (root module)

provider "google" {
  project = var.project_id
  region  = var.region
}

README.md (snippet)

module "assets_bucket" {
  source            = "./modules/gcs-bucket"
  project_id        = "acme-prod"
  bucket_name_prefix = "acme-assets-prod"
  labels            = { env = "prod", owner = "team-x" }
}

Quality Checklist

• Bucket deletion is safe by default (force_destroy = false)
• All variables have descriptions and types
• terraform fmt + terraform validate run in CI
• Labels/tags are applied consistently for cost attribution and ownership
• IAM is least-privilege (prefer separate module for IAM bindings)

Incident Communication Template

Purpose: Communicate clearly and consistently during live incidents and recovery.

When to Use

• All major outages, SEV-1/SEV-2
• Status page, internal comms, stakeholder updates

---

TEMPLATE STARTS HERE

Initial Alert

Title: "Incident - [Service/Component] Impacting [User Segment]"

Summary:

• What’s broken?
• When did it start?
• Who is affected?

Current Status:

• Investigation/mitigation/monitoring
• ETA for next update

Actions Being Taken:

• [e.g., rolling back, failover, escalating]

Updates (every 30–60 min during major incidents)

Time:

• Progress since last update
• New findings
• Updated ETA/next step

Resolution

Summary of Fix:

• What was changed/fixed
• Is the system stable?
• Monitoring for recurrence

Quality Checklist

• [ ] Next update time set
• [ ] All comms archived
• [ ] Consistent template used

# Incident Response Template (DevOps)

*Purpose: A production-ready template for managing incidents from detection through resolution, including communications, triage, mitigation, escalation, and post-incident review.*

---

# 1. Incident Metadata

**Incident ID:**  
[IR-YYYYMMDD-001]

**Title:**  
[Short description: “Checkout API 500 errors”, “K8s cluster degraded”, etc.]

**Severity:**  
- [ ] SEV0 — Critical outage  
- [ ] SEV1 — Major degradation  
- [ ] SEV2 — Partial impairment  
- [ ] SEV3 — Minor operational issue  

**Start Time:**  
[UTC timestamp]

**Current Status:**  
- [ ] Investigating  
- [ ] Mitigating  
- [ ] Monitoring  
- [ ] Resolved  

**Reported By:**  
[Source: alert, user, SRE, etc.]

**Affected Systems:**  
[List services, clusters, DBs, regions]

---

# 2. Roles & Assignments

**Incident Commander (IC):**  
[Name]

**Communications Lead:**  
[Name]

**Technical Lead(s):**  
[Service/K8s/infra/DX/etc.]

**Scribe (Documentation):**  
[Name]

**Other SMEs:**  
[DBA, Network Eng, App Dev, SRE, etc.]

Checklist:
- [ ] IC assigned within 2 minutes  
- [ ] Comms channel opened (#incident-<id>)  
- [ ] Scribe logging major events  
- [ ] Escalation tree loaded  

---

# 3. Incident Summary (Live Updating)

**What is happening?**  
[Describe symptoms]

**Impact:**  
- [ ] Outage  
- [ ] High latency  
- [ ] Increased error rate  
- [ ] Partial regional effects  
- [ ] Data inconsistency  
- [ ] Degraded throughput  

**User Impact:**  
[What does a real user experience?]

---

# 4. Initial Triage

## 4.1 Is it real?

- [ ] False alert?  
- [ ] Partial outage or localized?  
- [ ] Monitoring gap?  

## 4.2 Immediate Observability Checks

### Metrics  
- Traffic  
- Latency (p95/p99)  
- Error rate  
- Saturation (CPU/mem)  

### Logs  
- Error spikes  
- Deployment events  
- Repeated exceptions  

### Traces  
- Slow spans  
- Failed downstream calls  

## 4.3 Quick Triage Checklist

- [ ] Check dashboards (Golden Signals)  
- [ ] Check last deploy  
- [ ] Check dependencies (DB/cache/queue)  
- [ ] Check resource exhaustion  
- [ ] Check infrastructure events (K8s/node/cloud)  

---

# 5. Mitigation Plan

**Current Hypothesis:**  
[Suspected root cause]

**Immediate Mitigation Actions:**  
(Choose those relevant)

- [ ] Rollback latest deployment  
- [ ] Scale pods/services  
- [ ] Recreate pods or nodes  
- [ ] Failover DB or region  
- [ ] Reduce traffic / enforce rate limiting  
- [ ] Disable heavy background jobs  
- [ ] Revert feature flag  
- [ ] Increase service quotas  
- [ ] Restart unhealthy workloads  

Mitigation Step: Reason: Command(s): Expected Outcome:

---

# 6. Communication

Announce updates every 10–15 minutes.

### Communication Template

[Time UTC] Status: Root cause hypothesis: Mitigation in progress: Next update in XX min.

Channels:
- #incident-<id>  
- Status page  
- Email to key stakeholders  

Checklist:
- [ ] Stakeholders notified  
- [ ] Customer-facing updates posted if required  
- [ ] No speculation  
- [ ] Clear next steps  

---

# 7. Investigative Actions

**Data Collected So Far:**
- Metrics snapshots  
- Logs with timestamps  
- Traces (long spans)  
- K8s events  
- DB metrics  
- Cloud provider alerts  

### Investigative Questions
- When did it start?  
- What changed (deploy, config, dependency)?  
- Which region(s) affected?  
- Can problem be reproduced?  
- Is it cascading from another service?  

### Safe Experiments

- [ ] Test rollback  
- [ ] Test endpoint with dev traffic  
- [ ] Disable/enable specific replica  
- [ ] Temporarily reroute traffic  

Never perform:
- Destructive K8s operations in prod  
- Full cluster/node delete  
- Unvalidated DB migrations  
- Forced failover without IC approval  

---

# 8. Resolution

**Resolution Time:**  
[Timestamp]

**Resolution Summary:**  
[What fixed the issue?]

Commands/steps executed:

• ...
• ...

Checklist:
- [ ] Systems stable for 30 min  
- [ ] Error budget updated  
- [ ] All rollbacks applied  
- [ ] Alerts firing as expected  
- [ ] Cluster/database fully healthy  

---

# 9. Recovery Actions

**After mitigation, apply:**

- [ ] Backfill dropped traffic if required  
- [ ] Sync caches or replicas  
- [ ] Re-enable paused jobs  
- [ ] Re-enable autoscaling  
- [ ] Re-deploy artifact if rollback used  
- [ ] Validate metrics for 1–2 hours  

---

# 10. Post-Incident Review (PIR)

Must be completed within 24–48 hours.

### PIR Template

Incident ID: Severity: Start / End Time: Duration: Services Impacted: User Impact:

Root Cause: Contributing Factors: Timeline: [time] event [time] event Mitigation: What Went Well: What Went Poorly: Where We Got Lucky: Action Items (with owners and due dates): Long-Term Fixes:

Checklist:
- [ ] No blame language  
- [ ] One action item per contributing factor  
- [ ] Business owner included  
- [ ] Regression tests added  

---

# 11. Severity Levels (Standard)

| Sev | Description | Response Time | Criteria |
|-----|-------------|----------------|----------|
| SEV0 | Full outage | Immediate | 100% down / major data loss |
| SEV1 | Major impact | < 15 min | Critical path degraded |
| SEV2 | Partial impact | < 1 hr | Non-critical slowdowns |
| SEV3 | Minor issue | < 1 day | No user-visible outage |

---

# 12. Incident Runbook Snippets

### Restart Pod Safely (K8s)

kubectl delete pod <pod> --grace-period=30 kubectl rollout status deployment/<app>

### Rollback Deployment

kubectl rollout undo deployment/<app>

### Restart Service (Systemd)

sudo systemctl restart <service>

### Rolling Restart

kubectl rollout restart deployment/<app>

### Verify Health

kubectl get pods kubectl logs <pod> curl -f https://<service>/healthz

---

# 13. Final Incident Checklist

### Before Declaring Resolved

- [ ] Systems stable 30+ minutes  
- [ ] No alert flapping  
- [ ] SLOs green  
- [ ] Runbooks updated if missing steps  
- [ ] Monitoring dashboards corrected  
- [ ] Root cause validated  
- [ ] Action items logged  

---

# 14. Completed Example

**Incident ID:** IR-20250310-001  
**Title:** Prod Checkout API 500 Spike  
**Severity:** SEV1  
**Impact:** Users unable to complete purchases  

**Cause:** A canary deployment introduced a latency regression → cascading DB saturation.

**Mitigation:**  
- Rolled back deployment  
- Added rate limiting  
- Increased DB read replicas  

**Resolution:** 17 minutes  
**Action Items:**  
- Add load tests to pipeline  
- Add DB connection pool alert  
- Strengthen canary gating  

---

# END

Incident Postmortem Template

Purpose: Capture facts, root cause, and corrective actions after SEV incidents (blameless).

When to Use

• Any production incident with customer impact or SLO burn
• SEV-1 / SEV-2 (required), SEV-3 (recommended)

---

Core

Template

Summary

• Incident ID:
• Start/End time (timezone):
• Detection source (alert, customer report, internal):
• Severity:
• Services/systems impacted:
• Customer impact summary:
• SLO/SLI impact (error budget burn, if applicable):
• Primary on-call / incident commander:

Timeline

Impact (What Users Experienced)

• User/business impact:
• Scope (tenants/regions/features):
• Duration and peak impact window:
• Data impact: none / delayed / incorrect / lost (explain):

Detection and Response

• Detection quality (actionable? noisy? missing signal?):
• Triage notes (first hypotheses and what was ruled out):
• Mitigation steps (what stopped the bleeding):
• Recovery steps (what restored normal service):

Root Cause (Why It Happened)

• Trigger event:
• Proximate cause:
• Contributing factors (tech/process/people/systemic):
• Why existing controls failed (tests, monitors, reviews, guardrails):

Remediation

• Immediate fix (already done):
• Long-term fix (planned):
• Rollback strategy (if fix causes issues):

Lessons Learned

• What worked:
• What didn’t:
• Documentation/process gaps:

Action Items

Evidence and References

• Dashboards:
• Logs/traces:
• Deployments/changes during window:
• Related incidents:

Quality Checklist (Gate to Close)

• [ ] Blameless review
• [ ] Action items have owners and due dates
• [ ] Customer communication completed (if applicable)
• [ ] Runbooks/docs updated
• [ ] Monitoring/alerting gaps addressed

---

Optional: AI/Automation

• Summarize timeline from incident channel and logs (human-verified)
• Cluster alerts and propose contributing factors (human-validated)
• Draft action items and owners (human-approved)

Bounded Claims

• Automation can miss context and nuance; humans own conclusions and commitments.

DevOps Platform Runbook Starter

Use this template to create consistent, actionable runbooks for on-call engineers.

---

Core

Service Overview

• Service/system name:
• Owner team:
• On-call rotation:
• Primary region(s)/environment(s):
• Dependencies (DB, cache, queue, third parties):
• Critical user journeys:

SLOs and Safety Limits

• SLO targets (latency, availability, freshness where applicable):
• Error budget policy (paging thresholds, burn-rate alerts):
• Data sensitivity (PII/PHI/PCI): yes/no (notes):

Standard Checks (First 5 Minutes)

• [ ] Confirm scope: single tenant vs all tenants, single region vs global
• [ ] Check recent changes (deploys/config/infra) in the last 60 minutes
• [ ] Check dashboards: latency, error rate, saturation, dependency health
• [ ] Check logs/traces with correlation IDs
• [ ] Verify if incident is ongoing or already recovering

Common Alerts

Alert: High Error Rate

• Trigger:
• Impact:
• Likely causes:
• Immediate mitigations (safe actions):
• Option A:
• Option B:
• Verification steps (how to confirm improvement):
• Escalation criteria:

Alert: High Latency

• Trigger:
• Impact:
• Likely causes:
• Immediate mitigations (safe actions):
• Verification steps:
• Escalation criteria:

Alert: Dependency Unavailable

• Trigger:
• Impact:
• Likely causes:
• Immediate mitigations (safe actions):
• Verification steps:
• Escalation criteria:

Safe Mitigations (Pre-Approved)

• Feature flags (names + effect):
• Rate limiting / circuit breaker actions:
• Traffic shedding / load shedding:
• Rollback procedure:
• Read-only mode procedure:

Escalation and Communication

• Incident commander criteria:
• Escalation contacts (primary/backup):
• Customer communication triggers:
• Status page ownership:

Post-Incident Follow-Up

• Link to postmortem template: assets/incident-response/template-postmortem.md
• Required updates after incident:
• [ ] Update this runbook with new learnings
• [ ] Add/adjust alerts if signal was missing or noisy
• [ ] Add regression tests or guardrails where applicable

---

Optional: AI/Automation

• Summarize current incident state from dashboards/logs (human-verified)
• Suggest likely dependency chain based on service graph (human-validated)
• Draft comms updates from structured incident fields (human-approved)

Bounded Claims

• Automation can be wrong; never execute mitigations without explicit approval.

# Kafka Operations Template (DevOps)

*Purpose: A complete operational template for running, scaling, securing, monitoring, and troubleshooting Apache Kafka clusters, topics, partitions, consumers, and brokers.*

---

# 1. Overview

**Cluster / Environment:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  
- [ ] multi-region  

**Kafka Distribution:**  
- [ ] Apache Kafka  
- [ ] Confluent  
- [ ] MSK (AWS Managed)  
- [ ] Strimzi / K8s operator  

**Purpose of Change / Task:**  
- [ ] Create topic  
- [ ] Update partitions  
- [ ] Add broker / scale cluster  
- [ ] Debug consumer lag  
- [ ] Fix under-replicated partitions  
- [ ] Rolling upgrade  
- [ ] ACL / security change  
- [ ] DR failover  

---

# 2. Kafka Topic Operations

## 2.1 Create Topic

kafka-topics.sh --create \ --bootstrap-server <brokers> \ --topic <name> \ --partitions <N> \ --replication-factor <R> \ --config retention.ms=604800000

Checklist:
- [ ] Partitions sized for throughput  
- [ ] Replication factor ≥ 3 in prod  
- [ ] Retention policies set  
- [ ] Cleanup policy = compact / delete validated  
- [ ] Topic naming convention followed  

---

## 2.2 Update Topic Partitions

kafka-topics.sh --alter \ --topic <name> \ --partitions <new_count> \ --bootstrap-server <brokers>

Checklist:
- [ ] Only increase allowed (never decrease)  
- [ ] Consumer group impact assessed  
- [ ] Rebalancing expected and monitored  

---

## 2.3 Retention Policies

retention.ms=604800000 cleanup.policy=delete

Checklist:
- [ ] Disk usage forecasted  
- [ ] Compaction required?  
- [ ] PII retention compliant?  

---

# 3. Consumer Group Operations

## 3.1 Check Consumer Lag

kafka-consumer-groups.sh \ --bootstrap-server <brokers> \ --describe \ --group <group-id>

Checklist:
- [ ] Lag increasing?  
- [ ] Consumer offline?  
- [ ] Partition imbalance?  
- [ ] Slow or stuck consumers traced?  
- [ ] Application logs inspected?  

---

## 3.2 Reset Offsets (Safe)

**Dry run:**

kafka-consumer-groups.sh --reset-offsets \ --to-earliest \ --group <group> \ --topic <topic> \ --dry-run \ --bootstrap-server <brokers>

**Apply:**

kafka-consumer-groups.sh --reset-offsets \ --to-latest \ --execute \ --group <group> \ --topic <topic> \ --bootstrap-server <brokers>

Checklist:
- [ ] Confirm NO other consumers running  
- [ ] Confirm business impact of resetting  
- [ ] Confirm correct direction (earliest/latest/timestamp)  

---

# 4. Broker & Cluster Operations

## 4.1 Rolling Restart

systemctl stop kafka systemctl start kafka

Checklist:
- [ ] One broker at a time  
- [ ] Controller stability monitored  
- [ ] No ISR shrinkage  
- [ ] Under-replicated partitions (URPs) = 0  

---

## 4.2 Add Broker to Cluster

Steps:

1. Provision node 2. Apply broker configs 3. Start broker 4. Rebalance partitions

Rebalance:

kafka-reassign-partitions.sh --execute --reassignment-json ...

Checklist:
- [ ] Disk/CPU/network sized  
- [ ] Inter-broker protocol version compatible  
- [ ] Auto-leader rebalance enabled  

---

## 4.3 Under-Replicated Partitions

Check:

kafka-topics.sh --describe --bootstrap-server <brokers>

Fix:
- [ ] Restart ISR follower  
- [ ] Verify network throughput  
- [ ] Reassign partition to healthy brokers  

---

# 5. Monitoring

## 5.1 Key Kafka Metrics (Prometheus / JMX)

### Broker
- UnderReplicatedPartitions  
- OfflinePartitions  
- ActiveControllerCount  
- ISR shrink count  
- RequestQueueSize  
- Network I/O  

### Consumer
- Consumer lag per partition  
- Commit latency  
- Rebalance activity  

### Producer
- Batch size  
- Request latency  
- Retries & errors  

Checklist:
- [ ] Alerts configured on URPs > 0  
- [ ] Alerts on controller election  
- [ ] Topic disk usage monitored  

---

# 6. Security & ACLs

## 6.1 Enable ACLs (if applicable)

kafka-acls.sh --add \ --allow-principal User:<user> \ --operation Read \ --topic <topic>

Checklist:
- [ ] No anonymous access in prod  
- [ ] SASL/SSL enabled  
- [ ] Cert rotation procedure in place  
- [ ] Use least privilege ACLs  

---

# 7. Disaster Recovery (DR)

## 7.1 MirrorMaker 2 Setup

connect-mirror-maker.sh mm2.properties

Checklist:
- [ ] Inter-region connection stable  
- [ ] Topic whitelist/blacklist correct  
- [ ] Replication lag monitored  

---

## 7.2 Region Failover

1. Verify source cluster offline 2. Promote DR cluster 3. Repoint producers/consumers 4. Update DNS/env vars 5. Monitor consumer offsets

Checklist:
- [ ] DR tested quarterly  
- [ ] Retention mirrors business RPO  
- [ ] ACLs synced between regions  

---

# 8. Troubleshooting

## 8.1 Slow Consumers

- [ ] Check processing time  
- [ ] Check downstream DB latency  
- [ ] Scale horizontally  
- [ ] Increase partitions  

---

## 8.2 Producer Timeouts

- [ ] Network issues  
- [ ] Broker overloaded  
- [ ] Batch size too small  
- [ ] acks=all slows producers if RF low  

---

## 8.3 Rebalance Storms

- [ ] Consumer group session timeouts too low  
- [ ] Uneven partitions  
- [ ] Overloaded brokers  

---

# 9. Final Checklist

- [ ] Topics configured correctly  
- [ ] Partition counts validated  
- [ ] Retention policies correct  
- [ ] ACLs enforced  
- [ ] Lag monitored  
- [ ] URPs = 0  
- [ ] DR working & tested  
- [ ] Cluster capacity within limits  

---

# END

# High Availability & Disaster Recovery (HA/DR) Template

*Purpose: A complete operational template for designing, validating, and executing high-availability (HA) and disaster recovery (DR) strategies across infrastructure, applications, Kubernetes clusters, and databases.*

---

# 1. Overview

**System / Service Name:**  
[name]

**Environment:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  
- [ ] multi-region  

**DR Tier:**  
- [ ] Tier 0 (mission critical)  
- [ ] Tier 1 (critical)  
- [ ] Tier 2 (important)  
- [ ] Tier 3 (non-critical)  

**Owner:**  
[Team/Engineer]

**Last DR Test:**  
[date]

---

# 2. Business Continuity Objectives

## 2.1 RTO (Recovery Time Objective)

Service must be fully restored within <N> minutes/hours after outage.

## 2.2 RPO (Recovery Point Objective)

Data loss must not exceed <N> minutes/hours.

Checklist:
- [ ] RTO < service tolerance  
- [ ] RPO aligned with backup/replication frequency  
- [ ] Business impact documented  

---

# 3. HA Architecture

## 3.1 Redundancy Model

- [ ] Multi-AZ  
- [ ] Multi-region  
- [ ] Active/active  
- [ ] Active/passive  
- [ ] N+1 redundancy  
- [ ] Self-healing Kubernetes workloads  
- [ ] DNS/LB-based failover  

## 3.2 HA Component Map  
(List each infra component)

| Component | HA Method | Notes |
|-----------|-----------|--------|
| API Service | Multi-AZ + HPA | |
| Database | Multi-AZ failover | |
| Storage | Replicated volumes | |
| Load Balancer | Multi-region | |

---

# 4. DR Architecture

## 4.1 DR Regions & Replication

| Region | Role | Replication Type | Notes |
|--------|-------|-------------------|--------|
| us-east-1 | primary | async/sync | |
| us-west-2 | failover | async | |

## 4.2 Replication Patterns

- **Synchronous replication:** zero data loss, higher latency  
- **Asynchronous replication:** low latency, possible data loss  
- **Log shipping:** WAL/binlog streaming  
- **Object storage replication:** S3/GCS cross-region  
- **Snapshot replication:** scheduled recovery points  

---

# 5. Backup & Restore Strategy

## 5.1 Backup Types

- [ ] Full backup (daily)  
- [ ] Incremental backups  
- [ ] WAL/Binlog streaming  
- [ ] Snapshot backups  
- [ ] Application-level backups  

## 5.2 Backup Locations

- [ ] Offsite region  
- [ ] Cross-account replication  
- [ ] Encrypted storage  
- [ ] Immutable/Write-once (WORM) backup  

## 5.3 Restore Procedures

1. Fetch backup artifact 2. Restore into isolated environment 3. Replay logs to target time (PITR) 4. Validate data integrity 5. Promote to primary if needed

Checklist:
- [ ] Restore tested recently  
- [ ] PITR validated  
- [ ] Backups encrypted  
- [ ] Access restrictions in place  

---

# 6. Failover Procedures

## 6.1 Application-Level Failover

1. Detect primary region outage 2. Freeze deploys 3. Shift traffic to standby 4. Update DNS/global load balancer 5. Validate service health 6. Notify on-call and engineering

Checklist:
- [ ] Failover < RTO  
- [ ] DR region warmed and ready  
- [ ] Secrets/configs replicated  
- [ ] Observability validated after failover  

---

## 6.2 Kubernetes Failover

### Option A — Multi-Cluster Active/Active

- [ ] Traffic split between clusters  
- [ ] Global load balancer configured  
- [ ] Shared service mesh (Istio/Linkerd)  
- [ ] Cross-region secrets sync  

### Option B — Active/Passive (DR cluster)

Failover steps:

1. Ensure cluster API available 2. Sync manifests via GitOps 3. Scale up workloads 4. Redirect traffic 5. Validate pods/services

Checklist:
- [ ] CI/CD supports region-aware deployments  
- [ ] etcd backups configured  
- [ ] Cluster version parity maintained  

---

## 6.3 Database Failover

### Automatic (recommended when safe)

- [ ] Synchronous commit for Tier 0  
- [ ] Health-based promotion  
- [ ] Application retries configured  

### Manual Failover

1. Promote replica 2. Repoint application connections 3. Rebuild old primary as replica

Checklist:
- [ ] Failover tested  
- [ ] Application uses retry logic  
- [ ] Read/write separation considered  

---

# 7. DR Test Procedure

## 7.1 Test Types

- [ ] Full failover drill  
- [ ] Partial component failure  
- [ ] Network cut test  
- [ ] Backup restore test  
- [ ] Data corruption simulation  
- [ ] Chaos test (pod/node failure)  

## 7.2 Standard DR Drill Template

1. Announce test 2. Disable production alerts 3. Trigger failover or restore 4. Measure time to recovery 5. Validate application behavior 6. Evaluate data correctness 7. Restore normal topology 8. Document results

Checklist:
- [ ] Test performed in isolated environment  
- [ ] Monitoring enabled  
- [ ] Logs collected  
- [ ] Action items created  

---

# 8. Resilience Engineering

## 8.1 Failure Scenarios

- [ ] Region outage  
- [ ] AZ failure  
- [ ] Network partition  
- [ ] Data corruption  
- [ ] DNS outage  
- [ ] Load balancer failure  
- [ ] Secrets manager outage  
- [ ] Container registry outage  

## 8.2 Mitigation Patterns

- [ ] Retry with exponential backoff  
- [ ] Circuit breakers  
- [ ] Timeouts  
- [ ] Bulkheads  
- [ ] Fallback logic  
- [ ] Idempotent operations  

---

# 9. Post-Failover Verification

## 9.1 Infrastructure Health

- [ ] Pods healthy  
- [ ] Nodes stable  
- [ ] Autoscaling resumed  
- [ ] Load balancer routes correct  
- [ ] K8s controllers active  

## 9.2 Application Health

- [ ] p95/p99 latency normal  
- [ ] Error rate stable  
- [ ] Background jobs operational  
- [ ] No data loss detected  

## 9.3 Data Validation

- [ ] Schema matches primary  
- [ ] Row counts match  
- [ ] Referential integrity intact  
- [ ] No orphaned data  

---

# 10. DR Readiness Checklist

### Must-Haves

- [ ] Backups tested quarterly  
- [ ] PITR validated  
- [ ] Failover scripts documented  
- [ ] DR cluster/region ready  
- [ ] Observability active in DR region  
- [ ] Secrets synchronized  
- [ ] Config stored in Git  
- [ ] Environment parity validated  
- [ ] RTO/RPO targets monitored  

### Nice-to-Have

- [ ] Automated failover  
- [ ] Self-healing K8s cluster  
- [ ] Multi-region service mesh  
- [ ] DB proxy with auto-retry  

---

# 11. Completed Example

**Service:** User Accounts API  
**RTO:** 15 minutes  
**RPO:** 5 minutes  
**Primary Region:** us-east-1  
**DR Region:** us-west-2  
**DB Replication:** Async cross-region  
**Cluster:** EKS multi-cluster active/passive  

**Last DR Drill Outcome:**  
- Failover time: 11 minutes  
- Data loss: 0 minutes (WAL replay successful)  
- Issues: Missing environment variable sync  
- Fixes: Implement secrets replication via SOPS + GitOps  

---

# END

# Kubernetes Operations Template (DevOps)

*Purpose: A practical template for day-to-day Kubernetes operations: deploying apps, scaling, debugging, performing maintenance, and validating production readiness.*

---

# 1. Overview

**Cluster Name / Context:**  
[e.g., prod-eu1]

**Environment:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  
- [ ] DR  

**Workload Type:**  
- [ ] Stateless app  
- [ ] Stateful app  
- [ ] CronJob  
- [ ] DaemonSet  
- [ ] Job  

**Change Type / Task:**  
- [ ] New deployment  
- [ ] Update deployment  
- [ ] Scale app  
- [ ] Debug incident  
- [ ] Node maintenance  
- [ ] Cluster upgrade  

---

# 2. Application Deployment

## 2.1 Deployment Manifest Skeleton

apiVersion: apps/v1 kind: Deployment metadata: name: <app-name> labels: app: <app-name> env: <env> spec: replicas: 3 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 selector: matchLabels: app: <app-name> template: metadata: labels: app: <app-name> env: <env> spec: containers:

• name: <app-name>

image: <registry>/<image>:<tag> ports:

• containerPort: 8080

envFrom:

• configMapRef:

name: <app-config>

• secretRef:

name: <app-secret> resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "300m" memory: "256Mi" livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 20 periodSeconds: 10 readinessProbe: httpGet: path: /ready port: 8080 initialDelaySeconds: 5 periodSeconds: 5

### Deployment Checklist

- [ ] Image pinned to digest or specific tag  
- [ ] Probes configured correctly  
- [ ] Resources set (requests/limits)  
- [ ] Config/Secret used (no inline secrets)  
- [ ] Labels and annotations set (tracing, version)  
- [ ] Rolling strategy defined  

---

# 3. Service & Ingress

## 3.1 Service

apiVersion: v1 kind: Service metadata: name: <app-name> spec: type: ClusterIP selector: app: <app-name> ports:

• name: http

port: 80 targetPort: 8080

## 3.2 Ingress (Example)

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: <app-name> annotations: cert-manager.io/cluster-issuer: letsencrypt spec: tls:

• hosts: [ "<host>" ]

secretName: <tls-secret> rules:

• host: <host>

http: paths:

• path: /

pathType: Prefix backend: service: name: <app-name> port: number: 80

### Exposure Checklist

- [ ] Service type appropriate (ClusterIP/NodePort/LoadBalancer)  
- [ ] Ingress host configured  
- [ ] TLS via cert-manager or cloud LB  
- [ ] NetworkPolicy restricts traffic where needed  

---

# 4. Scaling & Autoscaling

## 4.1 Manual Scaling

kubectl scale deployment/<app> --replicas=5

Checklist:
- [ ] Scale tested in staging  
- [ ] Resources support replicas  
- [ ] HPA limits adjusted accordingly  

---

## 4.2 Horizontal Pod Autoscaler (HPA)

apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: <app-name> spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: <app-name> minReplicas: 2 maxReplicas: 10 metrics:

• type: Resource

resource: name: cpu target: type: Utilization averageUtilization: 70

HPA Checklist:
- [ ] Metrics server installed  
- [ ] Reasonable min/max  
- [ ] Target utilization based on real data  
- [ ] Avoid flapping (cooldown configured via HPA/tuning)  

---

# 5. Operational Debugging

## 5.1 Basic Commands

kubectl get pods -n <ns> kubectl describe pod <pod> -n <ns> kubectl logs <pod> -n <ns> kubectl logs <pod> -n <ns> -c <container> kubectl exec -it <pod> -n <ns> -- sh kubectl get events -n <ns> --sort-by=.metadata.creationTimestamp

---

## 5.2 Common Issues & Checks

### CrashLoopBackOff

- [ ] Check `kubectl logs`  
- [ ] Check environment variables  
- [ ] Check config/secret mounts  
- [ ] Check liveness probe (too aggressive?)  
- [ ] Check image entrypoint  

---

### ImagePullBackOff

- [ ] Image name/tag correct  
- [ ] Registry credentials configured (imagePullSecrets)  
- [ ] Registry reachable  
- [ ] Rate limiting (DockerHub/others)  

---

### OOMKilled

- [ ] Check pod `status` and events  
- [ ] Increase memory requests/limits  
- [ ] Check memory leaks in app  
- [ ] Add limits gradually  

---

### Readiness/Liveness Failures

- [ ] Probe endpoints correct  
- [ ] Check app startup time  
- [ ] Increase `initialDelaySeconds`  
- [ ] Ensure dependency readiness (DB, cache)  

---

# 6. Node & Cluster Maintenance

## 6.1 Node Drain

kubectl cordon <node> kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

After maintenance:

kubectl uncordon <node>

Checklist:
- [ ] PodDisruptionBudgets checked  
- [ ] Critical pods tolerated elsewhere  
- [ ] StatefulSets drained carefully  

---

## 6.2 Cluster Upgrade Checklist

- [ ] Control plane upgraded first  
- [ ] Node pools upgraded gradually  
- [ ] API deprecation checked (kubectl convert / kube-no-trouble)  
- [ ] Admission controllers tested  
- [ ] Backup of etcd / state taken  
- [ ] DR plan validated  

---

# 7. Resource Management

## 7.1 Baseline Resource Template

resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "300m" memory: "256Mi"

Checklist:
- [ ] Based on real usage (metrics)  
- [ ] Avoid no-limit pods  
- [ ] Avoid requests >> limits  
- [ ] Watch for throttling  

---

## 7.2 Monitoring & Alerts

Key metrics:
- Pod restarts  
- CrashLoopBackOff events  
- CPU/memory usage  
- API server latency  
- Node memory/disk pressure  

Checklist:
- [ ] Dashboards exist per service and per cluster  
- [ ] Alerts actionable and non-noisy  
- [ ] Logs enriched with pod/namespace labels  

---

# 8. Security & Policies

## 8.1 Pod Security

- [ ] Run as non-root  
- [ ] Read-only root filesystem when possible  
- [ ] Drop unnecessary capabilities  
- [ ] Restrict host networking/paths  

---

## 8.2 Network Policies

Example:

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-app-to-db spec: podSelector: matchLabels: app: app ingress:

• from:
• podSelector:

matchLabels: app: db ports:

• protocol: TCP

port: 5432

Checklist:
- [ ] Default deny policies considered  
- [ ] Access granted only where needed  

---

# 9. Rollout & Rollback

## 9.1 Check Rollout

kubectl rollout status deployment/<app> -n <ns>

## 9.2 Rollback Deployment

kubectl rollout undo deployment/<app> -n <ns>

Checklist:
- [ ] Changed image tagged  
- [ ] Rollback tested in non-prod  
- [ ] Monitoring in place post-rollout  

---

# 10. Final Operational Readiness Checklist

- [ ] Deployment manifest reviewed  
- [ ] Resource sizing acceptable  
- [ ] Probes validated  
- [ ] Secrets & configs wired via K8s objects  
- [ ] SLOs & alerts defined  
- [ ] Runbook linked  
- [ ] CI/CD integrated with cluster (no manual `kubectl` in prod)  

---

# END

# Platform Team API / Self-Service Template

*Purpose: A template for defining an internal platform team’s API: services, self-service flows, SLAs, onboarding, and expectations between platform and product teams.*

---

# 1. Platform Overview

**Platform Name:**  
[e.g., “Internal Dev Platform”, “K8s Platform”, “Data Platform”]

**Owning Team:**  
[Platform team name]

**Supported Environments:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  

**Primary Customers:**  
- [ ] Product teams  
- [ ] Data teams  
- [ ] SRE  
- [ ] Other platform teams  

---

# 2. Platform Services Catalog

List platform capabilities as services.

| Service | Description | Interface | SLA | Owner |
|--------|-------------|-----------|-----|--------|
| app-runtime | run containerized services | GitOps / APIs | 99.9% | |
| ci-pipeline | standard CI templates | YAML/Actions | best effort | |
| db-provisioning | managed databases | ticket/API | 99.9% | |

---

# 3. Self-Service Flows

## 3.1 Service Onboarding Flow

Steps:
1. Product team submits service definition (name, team, runtime, SLOs)  
2. Platform team reviews and approves  
3. CI/CD template provisioned  
4. K8s namespace or tenancy created  
5. Observability baseline deployed  

Checklist:
- [ ] Minimal required info clear  
- [ ] Automated bootstrap where possible  
- [ ] Docs link for onboarding  

---

## 3.2 Standard “Golden Path” Pipeline

Templates provided:
- Build & test  
- Security scan  
- Deployment to K8s/ECS  
- Observability wiring (metrics/logs/traces)  

Checklist:
- [ ] Golden path documented  
- [ ] Deviations understood and approved  

---

# 4. Platform API Definition

## 4.1 Interface Types

- [ ] GitOps repo conventions  
- [ ] CLI  
- [ ] REST/GraphQL APIs  
- [ ] Service catalog UI  

For each API:

| Endpoint / Path | Method | Purpose | Auth |
|-----------------|--------|---------|------|
| `/services/register` | POST | register service | SSO/Token |

---

# 5. SLAs / SLOs for Platform

## Example SLOs:

- Control plane uptime: 99.9%  
- Build pipeline availability: 99.5%  
- New environment creation: < 1 hour  
- Incident response: < 15 minutes for P1  

Checklist:
- [ ] Platform SLOs defined  
- [ ] Error budgets in place  
- [ ] Communication when SLOs breached  

---

# 6. Responsibilities & Expectations

## 6.1 Platform Team Responsibilities

- Provide secure, reliable runtimes  
- Maintain tooling and workflows  
- Document usage and constraints  
- Offer enablement / consulting  

## 6.2 Product Team Responsibilities

- Build and own their services  
- Integrate with observability baselines  
- Use golden path where possible  
- Participate in incident resolution  

---

# 7. Onboarding & Documentation

Checklist:
- [ ] “Getting Started” guide  
- [ ] Service onboarding guide  
- [ ] CI/CD templates documented  
- [ ] Platform APIs documented  
- [ ] Runbooks provided  

---

# 8. Support & Escalation

Define channels:

- Slack: `#platform-support`  
- Ticket queue: [link]  
- Office hours: [times]  

Checklist:
- [ ] SLAs for response times  
- [ ] Routing of issues to correct team  

---

# 9. Operational Metrics

Track:

- Time to onboard new service  
- # services on golden path vs custom  
- Platform incidents / downtime  
- Feedback from product teams  

---

# 10. Completed Example

**Platform:** Cloud App Platform  
**Services:**  
- Runtime-as-a-Service (K8s)  
- CI/CD pipeline library  
- Database-as-a-Service  

**API:** GitOps + portal UI  
**SLO:** 99.9% platform availability  

---

# END

Prometheus Alert Rules Template

Purpose: Operational, actionable alert rules for critical metrics and SLOs.

groups:

• name: service-alerts

rules:

• alert: HighErrorRate

expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.001 for: 10m labels: severity: critical annotations: summary: "High error rate detected on main API" description: "More than 0.1% 5xx responses for 10min"

• alert: HighLatency

expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 0.8 for: 5m labels: severity: warning annotations: summary: "API latency 95th percentile above 800ms" description: "95% of requests slower than SLO for 5 minutes" ---

Add rules for uptime, resource exhaustion, burn rate, etc.

Checklist:

- [ ] All alerts have runbooks/playbooks

- [ ] Severity and routing correct

- [ ] Alert noise reviewed after incidents

# Load & Performance Testing Template (DevOps)

*Purpose: A template for designing, running, and analyzing load, stress, and performance tests for applications and infrastructure.*

---

# 1. Overview

**Service / Endpoint:**  
[name or URL]

**Environment:**  
- [ ] perf  
- [ ] staging  
- [ ] prod-like  

**Test Type:**  
- [ ] Load test  
- [ ] Stress test  
- [ ] Soak test  
- [ ] Spike test  

**Tooling:**  
- [ ] k6  
- [ ] JMeter  
- [ ] Locust  
- [ ] Gatling  
- [ ] Custom  

---

# 2. Test Objectives

- [ ] Validate SLOs under expected load  
- [ ] Identify bottlenecks  
- [ ] Validate autoscaling behavior  
- [ ] Validate DB/queue limits  
- [ ] Test caching efficacy  

**Success Criteria:**  
[Explicit metrics & thresholds]

---

# 3. Workload Model

## 3.1 Traffic Profile

- RPS: [e.g., 500 RPS average, 1000 RPS peak]  
- Concurrency: [users or VUs]  
- Test duration: [e.g., 30m / 2h / 24h]  
- Ramp-up/down strategy:  

---

## 3.2 Scenario Definitions

Describe each user flow:

| Scenario | Description | Weight | Notes |
|----------|-------------|--------|-------|
| Browse | GET /catalog | 60% | |
| View Item | GET /item/{id} | 30% | |
| Checkout | POST /checkout | 10% | |

---

# 4. Test Configuration

### Example (k6)

import http from 'k6/http'; import { sleep } from 'k6';

export let options = { stages: [ { duration: '5m', target: 100 }, { duration: '10m', target: 500 }, { duration: '5m', target: 0 } ], thresholds: { http_req_duration: ['p(95)<400'], http_req_failed: ['rate<0.01'] } };

export default function () { http.get('https://staging.example.com/catalog'); sleep(1); }

Checklist:

- [ ] Throttling / simulated think time included  
- [ ] Target env isolated (not shared with other tests)  
- [ ] Test data strategy defined  

---

# 5. Observability During Test

Monitor:

- Latency (p50/p95/p99)  
- Error rate  
- CPU & memory  
- DB connections and queries  
- Cache hit ratio  
- Queue length  
- Autoscaling events  

Checklist:

- [ ] Dedicated dashboards per test  
- [ ] Logs sampled and analyzed  
- [ ] Traces captured for slow outliers  

---

# 6. Execution Plan

Steps:

1. Announce test window  
2. Ensure monitoring and alerts ready  
3. Warm up environment  
4. Start test at low load  
5. Increase load gradually  
6. Observe autoscaling and resource usage  
7. Stop test and cool down  
8. Save logs and metrics  

---

# 7. Results & Analysis

## 7.1 Key Metrics

| Metric | Target | Observed |
|--------|--------|----------|
| p95 latency | < 400ms | |
| Error rate | < 1% | |
| Max RPS | | |
| CPU usage | | |
| DB CPU | | |
| Cache hit ratio | | |

---

## 7.2 Findings

- Bottlenecks:  
- Saturation points:  
- Autoscaling behavior:  
- Resource over/under-provisioning:  

---

## 7.3 Recommendations

- Increase CPU/memory for X  
- Add DB index on Y  
- Tweak autoscaling policies  
- Add cache for expensive queries  
- Optimize code paths  

---

# 8. Regression Strategy

- [ ] CI nightly load tests at smaller scale  
- [ ] Pre-release load test for major versions  
- [ ] Baseline comparison across versions  

---

# 9. Completed Example

**Service:** Checkout API  
**Test:** Load test, 15m ramp, 45m steady  
**Peak:** 800RPS  
**Results:**  

- p95 latency ~350ms (OK)  
- p99 latency ~600ms (marginal)  
- DB CPU 80% (acceptable)  

**Actions:**  

- Optimize DB queries  
- Add caching layer  

---

# END

# Observability & SLO Template (DevOps)

*Purpose: A complete template for designing, instrumenting, validating, and operating observability systems, including SLOs/SLIs, metrics, logs, traces, alerting, and dashboards.*

---

# 1. Overview

**Service Name:**  
[name]

**Environment:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  
- [ ] multi-region  

**Purpose of Observability Design:**  
- [ ] New service instrumentation  
- [ ] SLO creation  
- [ ] Alert tuning  
- [ ] Dashboard buildout  
- [ ] On-call readiness  
- [ ] Incident-driven improvement  

**Owner:**  
[Team/Engineer]  
**Date:**  
[YYYY-MM-DD]

---

# 2. SLO Definition

## 2.1 Summary

**SLI Type:**  
- [ ] Latency  
- [ ] Availability  
- [ ] Error rate  
- [ ] Throughput  
- [ ] Freshness (async jobs)  

**SLI Specification:**

SLI = % of requests < 300ms latency

**SLO Target:**

99.5% over 30 days

**Error Budget:**

0.5% of 30-day window

---

## 2.2 Detailed SLO Template

Service: <service-name> Customer Impact: <describe what failure looks like> SLI: <definition> SLO: <threshold and measurement window> Error Budget: <amount and burn alarms> Data Source: <Prometheus, ELK, Datadog, New Relic, OpenTelemetry>

Checklist:
- [ ] SLO tied to customer experience  
- [ ] SLI measurable in production  
- [ ] Multi-region aware  
- [ ] Retention aligns with SLO window  
- [ ] Can compute SLI retrospectively  

---

# 3. Error Budget Policy

## Burn Alerts

**Fast Burn:**  
Triggers when error budget depletes rapidly.

if error_budget_burn_rate > 5% / hour → alert P1

**Slow Burn:**  
Triggers when remaining budget trends downward.

if error_budget_burn_rate > 20% / 24h → alert P2

Checklist:
- [ ] Actions defined when burn starts  
- [ ] Deployment freeze criteria established  
- [ ] Escalation path documented  

---

# 4. Metrics (RED & Golden Signals)

## 4.1 Required Metrics

### Golden Signals
- **Latency** (p95, p99)
- **Error Rate**
- **Traffic**
- **Saturation**

### RED Method (for microservices)
- **Rate** (requests/sec)
- **Errors** (5xx + 4xx depending on SLO)
- **Duration** (latency across percentiles)

---

## 4.2 Metric Examples (Prometheus)

http_requests_total http_request_errors_total http_request_duration_seconds_bucket kube_pod_container_status_restarts_total node_cpu_seconds_total container_memory_working_set_bytes

Checklist:
- [ ] p50, p95, p99 latency tracked  
- [ ] Error rate measured over multiple windows  
- [ ] Separate metrics for success vs failure  
- [ ] Resource saturation metrics included  

---

# 5. Logging

## 5.1 Structured Logging Format

{ "ts": "2025-01-01T12:00:00Z", "level": "info", "msg": "request processed", "trace_id": "abc-123", "user_id": 42, "latency_ms": 85 }

Checklist:
- [ ] JSON only  
- [ ] No multi-line logs  
- [ ] No sensitive data (avoid: email, tokens, PII)  
- [ ] Include correlation IDs  
- [ ] Log levels appropriate  

---

# 6. Tracing

## 6.1 OpenTelemetry Standard

Required fields:
- `trace_id`
- `span_id`
- `parent_span_id`
- `service.name`
- `duration_ms`
- `status.code`
- `attributes.*`

## 6.2 Instrumentation Template

tracer.start_span("db.query", attributes={ "db.statement": "...", "db.table": "orders", "db.operation": "SELECT" })

Checklist:
- [ ] All inbound HTTP requests start a trace  
- [ ] All outbound calls propagate headers  
- [ ] DB calls wrapped in spans  
- [ ] Errors recorded with stack traces  
- [ ] Sampling strategy configured  

---

# 7. Alerting & Monitoring

## 7.1 Alert Design Rules

**Alerts MUST BE:**
- Actionable  
- Measurable  
- Urgent  
- Owned by a team  
- Have a runbook  

**Alerts MUST NOT BE:**
- Based on single data points  
- Flapping  
- Noisy  
- Without an owner  

---

## 7.2 Example Alerts (Prometheus)

### Latency Alert

ALERT HighLatencyP99 IF histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5 FOR 10m LABELS {severity="page"}

### Error Rate Alert

ALERT ErrorRateSpike IF rate(http_request_errors_total[5m]) / rate(http_requests_total[5m]) > 0.05 FOR 10m LABELS {severity="page"}

---

# 8. Dashboards

## Required Dashboard Panels

### Traffic
- RPS  
- Active sessions  

### Latency
- p50, p95, p99  
- Tail latencies  

### Errors
- 4xx vs 5xx split  
- Error budget burn chart  

### Saturation
- CPU/memory  
- Disk I/O  
- Queue depth  
- DB connection pools  

Dashboard Checklist:
- [ ] Real-time & historical views  
- [ ] Color-coded thresholds  
- [ ] Links to logs & traces  
- [ ] Summary and detail panels  

---

# 9. On-Call Readiness

## Runbook Template

Service: Alert Name: SLI: SLO: Symptoms: Immediate Actions: Mitigation: Escalation: Links:

---

## On-Call Checklist

- [ ] SLOs defined  
- [ ] Dashboards linked in alerts  
- [ ] Runbooks complete  
- [ ] Alerts tested  
- [ ] Escalation paths defined  
- [ ] No alert without remediation steps  

---

# 10. Observability Anti-Patterns

- AVOID: Logs without trace IDs  
- AVOID: Metrics without units  
- AVOID: Alerts without runbooks  
- AVOID: Dashboards that require tribal knowledge  
- AVOID: Noisy alerts ignored  
- AVOID: Reactive monitoring only  

---

# 11. Verification

### Observability Health Checks

- [ ] Tracing propagation verified  
- [ ] SLO reports generated correctly  
- [ ] Golden signals visible  
- [ ] Alerts fire correctly in intended scenarios  
- [ ] Logs searchable by trace_id  

---

# 12. Complete Example

**Service Name:** Checkout API  
**SLI:** `p95 latency < 400ms`  
**SLO:** 99% monthly  
**Error Budget:** 1%  
**Alerts:**  
- Fast burn (2%/h)  
- Slow burn (10%/24h)  
**Dashboards:**  
- Latency by endpoint  
- Error types  
- DB query latency  
- Queue backlog  

**Result:**  
- SLO tracking automated  
- Alerts actionable  
- On-call rotation effective  
- Latency regressions caught within minutes  

---

# END

SLO Definition Template

Purpose: Define, monitor, and review Service Level Objectives (SLOs) and error budgets.

When to Use

• All critical user-facing services
• New or existing SRE monitoring rollouts

---

TEMPLATE STARTS HERE

SLO Overview

• Service:
• Critical User Journey:
• SLO Owner:

SLI (Service Level Indicators)

Error Budget

• Calculation:

100% - SLO Target = Error Budget (e.g., 100% - 99.9% = 0.1% allowed failure per period)

• Burn Alerts:

Alert if > 25% of budget used in 24h

Monitoring/Alerting Config

• Prometheus rule:

  - alert: HighErrorRate
    expr: sum(rate(http_requests_total{status=~"5.."}[5m])) 
          / sum(rate(http_requests_total[5m])) > 0.001
    for: 10m
    labels:
      severity: critical

Quality Checklist

 SLO agreed with product/engineering
 SLI queries tested and automated
 Error budget policy documented
 SLO reviews scheduled quarterly

# Security Hardening Template (DevOps)

*Purpose: A comprehensive template for securing infrastructure, CI/CD, Kubernetes, containers, secrets, IAM, and runtime environments.*

---

# 1. Overview

**System / Component:**  
[name]

**Environment:**  
- [ ] dev  
- [ ] staging  
- [ ] prod  

**Security Context:**  
- [ ] New service onboarding  
- [ ] Hardening review  
- [ ] Incident-driven improvement  
- [ ] Compliance requirement (PCI/GDPR/SOC2/HIPAA)  

**Reviewer:**  
[name]  
**Date:**  
[YYYY-MM-DD]

---

# 2. Identity & Access Management (IAM)

## 2.1 IAM Checklist

- [ ] Principle of least privilege  
- [ ] No wildcard permissions (“\*”)  
- [ ] Role separation: admin vs deployer vs read-only  
- [ ] No long-lived credentials  
- [ ] Use IAM roles, not static keys  
- [ ] CI/CD uses OIDC + cloud IAM  
- [ ] MFA required for privileged accounts  
- [ ] Access logs enabled  

## 2.2 IAM Role Table

| Role | Purpose | Allowed Actions | Denied Actions | Notes |
|------|----------|------------------|-----------------|--------|
| | | | | |

---

# 3. Secrets Management

## 3.1 Approved Secret Storage

- AWS Secrets Manager  
- AWS SSM Parameter Store (SecureString)  
- HashiCorp Vault  
- GCP Secret Manager  
- Azure Key Vault  
- Kubernetes SealedSecrets/SOPS  

## 3.2 Checklist

- [ ] No plaintext secrets in repo  
- [ ] No secrets in Terraform vars/tfvars  
- [ ] Secrets encrypted at rest and in transit  
- [ ] Secret rotation configured  
- [ ] Access only for services that require it  
- [ ] K8s secrets encrypted using KMS  

---

# 4. Network Security

## 4.1 Ingress/Egress Rules

- [ ] Default deny egress  
- [ ] Only required ports allowed  
- [ ] Restrict public IP access  
- [ ] Enforce TLS everywhere  
- [ ] WAF enabled for web workloads  
- [ ] NetworkPolicies configured (K8s)  

## 4.2 Firewall Checklist

- [ ] No open ports to world  
- [ ] VPC/VNet segmentation enforced  
- [ ] Internal-only services protected  
- [ ] LB security groups tightened  
- [ ] Bastion host hardened or removed  

---

# 5. Container Security

## 5.1 Dockerfile Hardening

FROM alpine:3.19 RUN adduser -D appuser USER appuser ENTRYPOINT ["./app"]

Checklist:
- [ ] Use minimal base images (alpine/distroless)  
- [ ] Multi-stage builds  
- [ ] Run as non-root  
- [ ] No sensitive data copied into image  
- [ ] Pin image tags to digests  
- [ ] Avoid curl | bash  
- [ ] Avoid ADD (use COPY)  

## 5.2 Image Scanning

- Trivy  
- Grype  
- Docker Scout  
- Snyk  

Checklist:
- [ ] Critical vulnerabilities remediated  
- [ ] SBOM generated  
- [ ] Images signed (cosign/notary)  

---

# 6. Kubernetes Security

## 6.1 Pod Security

Checklist:
- [ ] runAsNonRoot: true  
- [ ] readOnlyRootFilesystem: true  
- [ ] drop capabilities (NET_RAW, etc.)  
- [ ] no host filesystem mounts  
- [ ] no privileged pods  
- [ ] limit memory/CPU to avoid DOS  
- [ ] use Pod Security Admission or Kyverno/OPA  

## 6.2 RBAC

Checklist:
- [ ] No cluster-admin usage  
- [ ] Namespace-scoped roles preferred  
- [ ] ServiceAccounts per workload  
- [ ] Token automount disabled  
- [ ] RoleBindings reviewed  

automountServiceAccountToken: false

---

# 7. CI/CD Security

## 7.1 Pipeline Hardening

- [ ] OIDC → cloud IAM, no static secrets  
- [ ] Repo secrets stored in encrypted vault  
- [ ] PR builds cannot access prod secrets  
- [ ] Artifact signing required  
- [ ] No untrusted code executed in privileged containers  

## 7.2 Required Security Scans

- [ ] SAST (static code analysis)  
- [ ] DAST (runtime scanning)  
- [ ] Dependency scanning (Snyk/Trivy)  
- [ ] IaC scanning (Checkov/Tfsec)  
- [ ] Secret scanning  

---

# 8. OS & Host Security (EC2/VM/Bare Metal)

Checklist:
- [ ] Patching automated  
- [ ] SSH disabled or keyless-only  
- [ ] Use SSM Session Manager  
- [ ] Filesystem encrypted  
- [ ] No root login  
- [ ] Audit logs enabled  
- [ ] Disk space monitoring  

---

# 9. Logging & Monitoring Security

Checklist:
- [ ] Logs do not contain secrets  
- [ ] Structured logs (JSON)  
- [ ] TLS for log ingestion  
- [ ] Alerting tied to SLO thresholds  
- [ ] SIEM integration (Splunk/ELK/Sentinel)  
- [ ] Access logs kept according to retention policy  

---

# 10. Disaster Recovery Security

Checklist:
- [ ] Backups encrypted in transit & at rest  
- [ ] Backups stored cross-region  
- [ ] RPO/RTO validated  
- [ ] Backup access restricted  
- [ ] DR failover tested quarterly  
- [ ] Snapshots immutable (WORM/S3 Object Lock)  

---

# 11. Compliance Alignment

Compliance Required:  
- [ ] SOC 2  
- [ ] PCI  
- [ ] GDPR  
- [ ] HIPAA  
- [ ] FedRAMP  

Checklist:
- [ ] Data classification completed  
- [ ] Access policies documented  
- [ ] Retention policies implemented  
- [ ] PII minimization confirmed  
- [ ] Encryption policies compliant  

---

# 12. Risk Assessment Table

| Risk | Severity | Probability | Mitigation | Owner |
|------|----------|-------------|------------|--------|
| | | | | |

---

# 13. Final Hardening Checklist

### Critical
- [ ] Least privilege everywhere  
- [ ] No plaintext secrets  
- [ ] No privileged containers  
- [ ] Images scanned & signed  
- [ ] TLS enforced  
- [ ] Backups validated  

### Recommended
- [ ] Runtime security (Falco, Cilium Tetragon)  
- [ ] eBPF-based monitoring  
- [ ] Automated SOAR playbooks  

---

# 14. Completed Example

**Service:** Payments API  
**Findings:**  
- Docker image running as root → fixed  
- Environment variables contained secret tokens → moved to SSM  
- IAM role overly permissive → tightened  
- TLS missing on staging ingress → added cert-manager  
- No IaC scanning → added tfsec + Checkov  

**Status:** Hardened  
**Next Review:** 90 days  

---

# END

Incident Postmortem Template

Purpose: Capture key facts, remediation, and action items after every SEV-1/SEV-2.

When to Use

• Every production incident, outage, or major bug

---

TEMPLATE STARTS HERE

Summary

• Incident ID:
• Date/Time:
• Reported By:
• Affected Systems:
• Severity:
• Duration:

Timeline

Impact

• User/business impact:
• Scope:

Root Cause

• Trigger event:
• Contributing factors:
• Why did existing controls fail?

Remediation

• Immediate fix:
• Long-term fix:

Lessons Learned

• What worked:
• What didn't:
• Documentation/process gaps:

Action Items

Quality Checklist

• [ ] Blameless review
• [ ] Action items tracked
• [ ] Docs/runbooks updated

Terraform/OpenTofu Module Template

Purpose: Standardize reusable, testable, and safe infrastructure modules across providers and environments.

When to Use

• Building cloud/network/database resources with Terraform/OpenTofu
• Enforcing DRY IaC (reusable patterns + consistent interfaces)
• Supporting multi-env or multi-region deployments with clear inputs/outputs

---

TEMPLATE STARTS HERE

Recommended Layout

modules/<module-name>/
  main.tf
  variables.tf
  outputs.tf
  versions.tf
  README.md

versions.tf (example)

terraform {
  required_version = ">= 1.5.0"
}

variables.tf (example)

variable "name" {
  type        = string
  description = "Resource name"
}

variable "tags" {
  type        = map(string)
  description = "Resource tags/labels"
  default     = {}
}

main.tf (example: AWS S3 bucket)

resource "aws_s3_bucket" "main" {
  bucket = var.name
  tags   = var.tags
}

outputs.tf (example)

output "bucket_arn" {
  value       = aws_s3_bucket.main.arn
  description = "Bucket ARN"
}

README.md (snippet)

module "bucket" {
  source = "./modules/s3-bucket"
  name   = "my-bucket"
  tags   = { environment = "prod", owner = "platform" }
}

Quality Checklist

• Inputs are minimal, typed, and documented (avoid giant any objects)
• Defaults are safe (no destructive deletes or public exposure by default)
• Output names are stable and documented (treat as API surface)
• CI runs terraform fmt + terraform validate (+ optional tflint/policy checks)
• Secrets are never inputs in plaintext (use secret managers; pass references/IDs)
• Providers are configured in the root module; modules stay provider-agnostic where possible

Related Templates

• template-env-promotion.md — Environment promotion workflow patterns
• ../cicd-pipelines/template-ci-cd.md — CI/CD gates (SAST/DAST/SCA)
• ../cicd-pipelines/template-github-actions.md — GitHub Actions workflow template