Devops Iac Engineer

DevOps IaC Engineer

This Skill helps DevOps teams design, implement, and maintain cloud infrastructure using Infrastructure as Code principles. Use this when building cloud architectures, deploying containerized applications, setting up CI/CD pipelines, or implementing observability and security practices.

Quick Navigation

• Terraform & IaC: See terraform.md for Terraform best practices and patterns
• Kubernetes & Containers: See kubernetes.md for container orchestration
• Cloud Platforms: See cloud_platforms.md for AWS, Azure, GCP guidance
• CI/CD Pipelines: See cicd.md for pipeline design and GitOps
• Observability: See observability.md for monitoring and logging
• Security: See security.md for DevSecOps practices
• Templates & Tools: See templates.md for ready-to-use templates

Core Principles

Key DevOps Terminology (Consistent Throughout)

• Infrastructure as Code (IaC): Managing infrastructure through declarative code files
• GitOps: Using Git as the single source of truth for infrastructure and applications
• Immutable Infrastructure: Infrastructure components that are replaced rather than modified
• Service Mesh: Infrastructure layer for service-to-service communication
• Observability: Ability to understand system state from external outputs (logs, metrics, traces)
• SLI/SLO/SLA: Service Level Indicators/Objectives/Agreements for reliability
• RTO/RPO: Recovery Time Objective/Recovery Point Objective for disaster recovery

Workflow: Infrastructure Implementation

When implementing infrastructure, follow this structured approach:

1. Understand Requirements

• What is the business need? (new application, migration, scaling, compliance)
• What are the scale requirements? (traffic, data, geographic distribution)
• What are the constraints? (budget, timeline, regulatory)
• What are the dependencies? (existing systems, data sources)

2. Design Architecture

• Choose appropriate cloud platform(s) and services
• Design for high availability and fault tolerance
• Plan network topology and security boundaries
• Identify data flows and storage requirements
• Document architecture with diagrams

3. Select IaC Tools

• Terraform for multi-cloud infrastructure provisioning
• Kubernetes manifests/Helm for container orchestration
• CI/CD tool selection based on team and requirements
• Configuration management tools if needed

4. Implement Infrastructure

• Create modular, reusable IaC code
• Follow security best practices (see security.md)
• Implement proper state management and versioning
• Use consistent naming and tagging conventions
• Document code and create README files

5. Set Up Observability

• Define SLIs and SLOs for critical services
• Implement logging, metrics, and tracing
• Create dashboards and alerts
• Set up log aggregation and analysis
• Plan on-call rotation and runbooks

6. Implement CI/CD

• Design deployment pipeline stages
• Implement automated testing (unit, integration, e2e)
• Set up GitOps workflows
• Configure deployment strategies (blue/green, canary)
• Implement rollback procedures

7. Test & Validate

• Run infrastructure tests (security, compliance, cost)
• Perform disaster recovery drills
• Load testing and performance validation
• Security scanning and penetration testing
• Document test results and improvements

8. Deploy & Monitor

• Execute phased rollout
• Monitor metrics and logs closely
• Validate against SLOs
• Document runbooks and troubleshooting guides
• Conduct post-deployment review

Decision Framework: Tool Selection

Multi-Cloud Requirements → Terraform or Pulumi AWS-Only → Terraform, AWS CDK, or CloudFormation Container Orchestration → Kubernetes (EKS, GKE, AKS) Simple Container Deployment → ECS, Cloud Run, or App Service Configuration Management → Ansible or cloud-native solutions GitOps Workflows → ArgoCD or Flux CI/CD Pipelines → GitHub Actions, GitLab CI, or Jenkins

Common Challenges & Solutions

Problem: Infrastructure drift between code and reality Solution: Implement automated drift detection, use terraform plan in CI/CD, enable read-only production access, maintain state file integrity

Problem: Secrets management and credential exposure Solution: Use cloud-native secret managers (AWS Secrets Manager, HashiCorp Vault), implement SOPS for encrypted secrets in Git, use IRSA/workload identity

Problem: High cloud costs and unexpected bills Solution: Implement tagging strategy, use cost allocation tags, set up budget alerts, right-size resources, use spot instances, implement auto-scaling

Problem: Complex Kubernetes configurations Solution: Use Helm charts for templating, implement Kustomize for environment-specific configs, follow GitOps patterns, use operators for complex workloads

Collaboration Tips

• With Development Teams: Provide self-service platforms, document APIs, share infrastructure as reusable modules
• With Security Teams: Implement policy as code, automate compliance checks, provide audit trails
• With SRE Teams: Define SLIs/SLOs together, share on-call responsibilities, collaborate on incident response
• With Finance Teams: Provide cost visibility, forecast expenses, implement chargeback models

---

Next Steps

1. Start with terraform.md if you're implementing infrastructure as code 2. Use kubernetes.md for container orchestration 3. Reference templates.md for ready-to-use configurations 4. Check observability.md to set up monitoring

Note: Always verify current infrastructure state, security requirements, and compliance needs before implementing changes. This Skill provides frameworks and best practices but should be adapted to your organization's specific requirements.

DevOps IaC Engineer Skill - Directory Structure & Quick Reference

Complete Directory Structure

DevOps IaC Engineer/
│
├── SKILL.md                                    ← Main skill file (START HERE)
├── README.md                                   ← Overview and quick start guide
├── DIRECTORY_STRUCTURE.md                      ← This file
│
├── reference/                                  ← Core reference materials
│   ├── terraform.md                           ← Terraform best practices & patterns
│   ├── kubernetes.md                          ← Kubernetes & container orchestration
│   ├── cloud_platforms.md                     ← AWS, Azure, GCP guidance (to be created)
│   ├── cicd.md                                ← CI/CD pipelines & GitOps (to be created)
│   ├── observability.md                       ← Monitoring, logging, tracing (to be created)
│   ├── security.md                            ← DevSecOps practices (to be created)
│   └── templates.md                           ← Ready-to-use templates
│
├── scripts/                                    ← Executable utility scripts
│   └── devops_utils.py                        ← Common DevOps automation tasks
│
├── examples/                                   ← Sample files for reference
│   ├── terraform/                             ← Example Terraform configurations
│   ├── kubernetes/                            ← Example Kubernetes manifests
│   └── pipelines/                             ← Example CI/CD pipelines
│
└── [After customization, you may add]:
    ├── company-standards/                     ← Your organization's standards
    ├── runbooks/                              ← Operational runbooks
    └── architecture/                          ← Architecture diagrams and docs

File Relationships

SKILL.md (Entry point)
  │
  ├─→ reference/terraform.md
  │   ├─ For: Infrastructure as Code, Terraform modules
  │   ├─ Use when: Provisioning cloud infrastructure
  │   └─ Contains: Best practices, module templates, state management
  │
  ├─→ reference/kubernetes.md
  │   ├─ For: Container orchestration, deployments
  │   ├─ Use when: Deploying containerized applications
  │   └─ Contains: Production-ready manifests, Helm charts, RBAC
  │
  ├─→ reference/cloud_platforms.md
  │   ├─ For: AWS, Azure, GCP architecture
  │   ├─ Use when: Designing cloud solutions
  │   └─ Contains: Well-Architected patterns, service selection
  │
  ├─→ reference/cicd.md
  │   ├─ For: CI/CD pipelines, GitOps workflows
  │   ├─ Use when: Automating deployments
  │   └─ Contains: Pipeline templates, testing strategies
  │
  ├─→ reference/observability.md
  │   ├─ For: Monitoring, logging, tracing
  │   ├─ Use when: Implementing observability
  │   └─ Contains: SLI/SLO frameworks, alert rules
  │
  ├─→ reference/security.md
  │   ├─ For: DevSecOps, compliance, secrets management
  │   ├─ Use when: Securing infrastructure and applications
  │   └─ Contains: Security best practices, compliance automation
  │
  ├─→ reference/templates.md
  │   ├─ For: Ready-to-use configurations
  │   ├─ Use when: Need quick-start templates
  │   └─ Contains: Complete application stacks, pipelines
  │
  └─→ scripts/devops_utils.py
      ├─ For: Automated DevOps tasks
      ├─ Use when: Generating boilerplate, validation
      └─ Contains: Terraform init, K8s validation, secret scanning

---

Quick Reference By Use Case

"I need to provision cloud infrastructure"

1. Read: SKILL.md → Section "Workflow: Infrastructure Implementation" 2. Use: reference/terraform.md → Module templates 3. Reference: reference/cloud_platforms.md → Architecture patterns 4. Tool: python scripts/devops_utils.py terraform init-project

"I need to deploy a containerized application"

1. Read: reference/kubernetes.md → Production-ready deployment 2. Use: reference/templates.md → Complete application stack 3. Validate: python scripts/devops_utils.py k8s validate --file deployment.yaml

"I need to set up a CI/CD pipeline"

1. Read: reference/cicd.md → Pipeline design patterns 2. Choose: reference/templates.md → GitHub Actions, GitLab CI 3. Implement: GitOps with ArgoCD or Flux

"I need to implement monitoring"

1. Read: reference/observability.md → SLI/SLO framework 2. Use: reference/templates.md → Prometheus rules 3. Set up: Dashboards and alerting

"I need to secure my infrastructure"

1. Read: reference/security.md → DevSecOps practices 2. Scan: python scripts/devops_utils.py security scan-secrets 3. Implement: Policy as code, secrets management

---

Navigation Tips

For DevOps Beginners

Start with: README.md → SKILL.md → One reference file based on your task

Suggested First Tasks: 1. Read SKILL.md core principles 2. Initialize Terraform project with utility script 3. Deploy sample application to Kubernetes 4. Set up basic monitoring

For Experienced Engineers

Start with: Specific reference file based on current task

Advanced Features: 1. Multi-cloud Terraform modules 2. Advanced Kubernetes patterns (StatefulSets, Operators) 3. GitOps workflows with progressive delivery 4. Custom observability with distributed tracing

For Platform Teams

Focus on: 1. Creating reusable Terraform modules 2. Building self-service platforms 3. Implementing policy as code 4. Establishing SRE practices

---

File Dependencies & Links

No file depends on other files being loaded - This is by design.

• Each reference file is self-contained
• No circular references between files
• All links point FROM reference files, not between them
• SKILL.md is the only entry point

This architecture ensures:

• ✓ Efficient context usage
• ✓ Parallel loading of information
• ✓ No broken reference chains
• ✓ Easy file updates without cascading changes

---

Tool Usage

DevOps Utilities Script

# Initialize Terraform project
python scripts/devops_utils.py terraform init-project \
  --name myproject --cloud aws --region us-east-1

# Validate Terraform files
python scripts/devops_utils.py terraform validate --file main.tf

# Validate Kubernetes manifests
python scripts/devops_utils.py k8s validate --file deployment.yaml

# Generate Kubernetes deployment
python scripts/devops_utils.py k8s generate \
  --name myapp --image myapp:1.0.0 --namespace production

# Initialize GitOps structure
python scripts/devops_utils.py gitops init \
  --tool argocd --environments dev,staging,prod

# Scan for secrets
python scripts/devops_utils.py security scan-secrets --directory .

---

Best Practices Summary

Infrastructure as Code

1. Version everything in Git 2. Use modules for reusability 3. Implement state locking (S3 + DynamoDB) 4. Never hardcode secrets 5. Tag all resources consistently

Kubernetes

1. Set resource limits on all containers 2. Implement health checks (liveness, readiness) 3. Use namespaces for isolation 4. Enable RBAC and network policies 5. Run as non-root user

CI/CD

1. Automate everything possible 2. Test before deploy (lint, security scan, unit tests) 3. Use GitOps for declarative deployments 4. Implement rollback strategies 5. Monitor deployments closely

Security

1. Least privilege access 2. Encrypt data at rest and in transit 3. Scan for vulnerabilities regularly 4. Rotate secrets automatically 5. Audit all changes

---

Storage & Sharing

The complete skill folder can be:

• Shared - Copy entire folder to team members
• Backed up - Store in version control (git)
• Customized - Modify files for your organization
• Extended - Add new files while maintaining structure

Recommended: Store in git repository for version tracking and collaboration.

---

Last Updated: October 2025 Version: 1.0 Compatible Models: Claude 4 family (Opus 4.1, Sonnet 4.5, Haiku 4.5)

DevOps IaC Engineer Skill - Implementation Guide

Deployment Checklist

Use this checklist to ensure the DevOps IaC Engineer Skill is properly deployed and ready for your team.

Phase 1: Preparation (Before Deployment)

File Structure Validation

• [ ] All files created in correct locations with proper structure
• [ ] SKILL.md exists in root directory with proper YAML frontmatter
• [ ] All reference files in /reference/ directory
• [ ] Python script in /scripts/ directory
• [ ] Example files in /examples/ directory
• [ ] README.md in root directory

File Content Verification

• [ ] SKILL.md has required frontmatter (name: 64 chars max, description: 1024 chars max)
• [ ] All reference files use markdown format (.md)
• [ ] Python script is executable and tested
• [ ] Links between files use correct relative paths (forward slashes)

Quality Review

• [ ] No typos in SKILL.md or reference files
• [ ] Terminology is consistent throughout
• [ ] All internal links tested and working
• [ ] Code in scripts directory is well-commented
• [ ] Examples folder contains working sample configurations

Phase 2: Customization (Modify for Your Organization)

Cloud Provider Standards

• [ ] Update cloud provider preferences (AWS/Azure/GCP)
• [ ] Add your organization's cloud account structure
• [ ] Include your naming conventions
• [ ] Define your tagging strategy
• [ ] Add your cost allocation approach

Security & Compliance

• [ ] Update reference/security.md:
• [ ] Add your compliance requirements (HIPAA, PCI-DSS, SOC2, etc.)
• [ ] Specify your secrets management solution
• [ ] Include your organization's security policies
• [ ] Add your vulnerability scanning tools

Infrastructure Standards

• [ ] Define your Terraform module standards
• [ ] Specify your Kubernetes cluster configurations
• [ ] Add your CI/CD tool preferences
• [ ] Include your observability stack

Naming Conventions

• [ ] Review consistent terminology throughout (search and replace if needed)
• [ ] Update environment names (dev, staging, prod, etc.)
• [ ] Add your resource naming patterns
• [ ] Include your label/tag standards

Phase 3: Testing (Verify Skill Works)

Functionality Tests

Test 1: Terraform Project Initialization

• [ ] Run: python scripts/devops_utils.py terraform init-project --name test --cloud aws --region us-east-1
• [ ] Verify directory structure is created correctly
• [ ] Check that all template files are generated
• [ ] Validate Terraform syntax

Test 2: Kubernetes Validation

• [ ] Run: python scripts/devops_utils.py k8s validate --file examples/kubernetes/complete-app.yaml
• [ ] Verify validation passes
• [ ] Check all manifests are recognized
• [ ] Test with invalid YAML to ensure error detection

Test 3: Security Scanning

• [ ] Run: python scripts/devops_utils.py security scan-secrets --directory .
• [ ] Verify no secrets are detected in examples
• [ ] Test with a file containing a fake secret
• [ ] Ensure detection works correctly

Test 4: Template Usage

• [ ] Deploy example Kubernetes application
• [ ] Verify all resources are created
• [ ] Check health checks work
• [ ] Validate autoscaling configuration

Test 5: CI/CD Pipeline

• [ ] Use GitHub Actions example pipeline
• [ ] Test build and push stages
• [ ] Verify deployment logic
• [ ] Check rollback procedures

Performance Tests

Token Usage

• [ ] SKILL.md body is under 500 lines ✓
• [ ] Reference files are at appropriate length
• [ ] No unnecessary repetition between files
• [ ] Files load efficiently without excessive context

Load Tests

• [ ] Test with Haiku model (fast, economical)
• [ ] Test with Sonnet model (balanced)
• [ ] Test with Opus model (powerful reasoning)
• [ ] Verify consistent behavior across models

Phase 4: Deployment

Make Skill Available

• [ ] Upload all files to skill repository/system
• [ ] Verify file structure matches expected format
• [ ] Test that Claude can access all files
• [ ] Confirm skill appears in available skills list

Team Onboarding

• [ ] Share README.md with team
• [ ] Provide quick start guide (specific to your organization)
• [ ] Schedule skill training/demo session
• [ ] Create internal documentation on how to use

Process Documentation

• [ ] Document your standard infrastructure workflow
• [ ] Create quick reference guides for common tasks
• [ ] Establish when to use each reference file
• [ ] Document any customizations you made

Phase 5: Optimization (After Initial Use)

Monitor Usage

• [ ] Track which parts of the skill are used most
• [ ] Note any confusion or questions from team
• [ ] Collect feedback on missing information
• [ ] Track time saved vs. previous approach

Iterative Improvements

• [ ] Update based on team feedback
• [ ] Add organization-specific examples
• [ ] Refine terminology based on actual usage
• [ ] Improve unclear sections
• [ ] Add new templates based on common needs

Knowledge Capture

• [ ] Document successful infrastructure patterns discovered
• [ ] Add new modules as they're created
• [ ] Update with actual performance data
• [ ] Share lessons learned with team

---

Customization Template

When customizing the skill for your organization, use this template:

1. Organization Information

• Company Name: [Your Company]
• Industry: [Your Industry]
• Cloud Provider(s): [AWS/Azure/GCP]
• Primary Services: [List key applications/services]
• Team Size: [Number of engineers]

2. Infrastructure Standards

• IaC Tool: [Terraform/Pulumi/CloudFormation]
• Container Orchestration: [Kubernetes/ECS/Other]
• CI/CD Platform: [GitHub Actions/GitLab CI/Jenkins]
• GitOps Tool: [ArgoCD/Flux/Other]

3. Cloud Architecture

• Primary Cloud: [AWS/Azure/GCP]
• Multi-Cloud: [Yes/No]
• Account Structure: [Multi-account strategy]
• Regions: [List primary regions]

4. Security & Compliance

• Compliance Requirements: [HIPAA/PCI-DSS/SOC2/etc.]
• Secrets Manager: [AWS Secrets Manager/Vault/etc.]
• Security Scanning: [Trivy/Snyk/Checkov/etc.]
• SIEM/Logging: [Splunk/ELK/CloudWatch/etc.]

5. Observability Stack

• Metrics: [Prometheus/Datadog/New Relic/etc.]
• Logging: [ELK/Splunk/CloudWatch/etc.]
• Tracing: [Jaeger/Zipkin/X-Ray/etc.]
• Alerting: [PagerDuty/Opsgenie/etc.]

6. Naming Conventions

• Resource Naming: [Pattern: {env}-{app}-{resource}]
• Tags Required: [Environment, Owner, CostCenter, etc.]
• Git Branching: [GitFlow/Trunk-based/etc.]

7. Deployment Strategy

• Environments: [dev, staging, prod, etc.]
• Deployment Method: [Rolling/Blue-Green/Canary]
• Approval Process: [Manual gates, auto-deploy rules]
• Rollback Strategy: [Automatic/Manual]

---

Common Customization Updates

Update 1: Cloud Provider Configuration

Files: reference/terraform.md, reference/cloud_platforms.md

Replace generic cloud provider examples with your specific configuration:

# Your organization's Terraform backend configuration
terraform {
  backend "s3" {
    bucket         = "your-company-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "your-company-terraform-lock"
  }
}

Update 2: Kubernetes Cluster Configuration

File: reference/kubernetes.md

Add your organization's cluster configuration:

# Your organization's namespace structure
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    environment: production
    team: platform
    cost-center: engineering

Update 3: CI/CD Pipeline Standards

File: reference/cicd.md, examples/pipelines/

Update with your CI/CD platform and standards:

# Your organization's pipeline template
name: Standard Deploy Pipeline
# Include your specific stages, approval gates, etc.

Update 4: Tagging Strategy

File: reference/terraform.md

Replace generic tags with your organization's required tags:

locals {
  common_tags = {
    Environment   = var.environment
    Project       = var.project_name
    ManagedBy     = "Terraform"
    CostCenter    = var.cost_center
    Owner         = var.owner_email
    Compliance    = var.compliance_level
    DataClass     = var.data_classification
  }
}

---

Troubleshooting Deployment Issues

Issue: Files not loading Solution: Verify file paths use forward slashes and are relative to SKILL.md location

Issue: Links between files broken Solution: Check that all links use proper markdown format: [text](path/to/file.md)

Issue: Skill not appearing Solution: Verify SKILL.md frontmatter has exactly name and description fields

Issue: Python script not executing Solution: Verify script has execute permissions and Python 3 is available in environment

chmod +x scripts/devops_utils.py
python3 --version  # Should be 3.8 or higher

Issue: Terraform examples don't work Solution: Update provider versions and check AWS credentials configuration

Issue: Kubernetes manifests fail validation Solution: Check apiVersion compatibility with your cluster version

---

Maintenance Schedule

Monthly

• [ ] Review team feedback and usage patterns
• [ ] Update templates based on recent successful deployments
• [ ] Refresh examples with current configurations
• [ ] Verify all links are still working
• [ ] Update provider versions in examples

Quarterly

• [ ] Update Kubernetes version compatibility
• [ ] Review and update security best practices
• [ ] Add new infrastructure patterns discovered
• [ ] Update cost optimization recommendations
• [ ] Refresh CI/CD pipeline templates

Annually

• [ ] Major review of all reference files
• [ ] Update with new cloud services and features
• [ ] Refresh all examples to reflect current state
• [ ] Conduct team training refresher
• [ ] Review and update compliance requirements

---

Success Metrics for Skill Implementation

Track these to measure skill effectiveness:

Adoption Metrics

• % of team using the skill regularly
• Number of infrastructure projects using skill templates
• Frequency of skill usage per engineer

Efficiency Metrics

• Time to provision new infrastructure (before vs. after)
• Time to deploy new application (before vs. after)
• Reduction in infrastructure errors
• Consistency of team output

Quality Metrics

• Infrastructure security score improvement
• Compliance audit results
• Infrastructure drift reduction
• Incident rate decrease

Business Impact

• Infrastructure provisioning time reduction
• Team productivity increase
• Cost optimization achieved
• Mean time to recovery (MTTR) improvement

---

Script Dependencies

The utility script requires:

# Python 3.8 or higher
python3 --version

# Install dependencies
pip install pyyaml

# Make script executable
chmod +x scripts/devops_utils.py

---

Questions or Issues?

• Content questions: See relevant reference file
• Technical issues: Check file structure and verify all files uploaded
• Customization help: Use the template in "Customization Template" section
• New feature requests: Document in quarterly review and add to next update

---

Integration with Existing Tools

Integrate with Git

cd "DevOps IaC Engineer"
git init
git add .
git commit -m "Initial DevOps IaC Engineer skill"

Integrate with VS Code

• Open folder in VS Code
• Install Terraform and YAML extensions
• Configure workspace settings for linting

Integrate with CI/CD

• Add skill files to your infrastructure repository
• Reference templates in your pipelines
• Use utility scripts in automation

---

Last Updated: October 2025 Version: 1.0

DevOps IaC Engineer Skill for Claude

A comprehensive Claude skill designed to help DevOps teams implement infrastructure as code, manage cloud platforms, deploy containerized applications, and maintain reliable production systems.

Overview

This skill provides frameworks, templates, and best practices for:

• Terraform & IaC - Infrastructure as Code implementation patterns
• Kubernetes & Containers - Container orchestration and deployment
• Cloud Platforms - AWS, Azure, GCP architecture and services
• CI/CD Pipelines - GitOps, automated testing, and deployment
• Observability - Monitoring, logging, and tracing strategies
• Security - DevSecOps practices and compliance automation
• Ready-to-Use Templates - Production-ready configurations

File Structure

DevOps IaC Engineer/
├── SKILL.md                          # Main skill file with overview & core workflows
├── reference/
│   ├── terraform.md                  # Terraform best practices and patterns
│   ├── kubernetes.md                 # Kubernetes and container orchestration
│   ├── cloud_platforms.md            # AWS, Azure, GCP guidance
│   ├── cicd.md                       # CI/CD pipelines and GitOps
│   ├── observability.md              # Monitoring, logging, and tracing
│   ├── security.md                   # DevSecOps and compliance
│   └── templates.md                  # Ready-to-use templates
├── scripts/
│   └── devops_utils.py               # Utility scripts for common DevOps tasks
├── examples/
│   ├── terraform/                    # Example Terraform configurations
│   ├── kubernetes/                   # Example Kubernetes manifests
│   └── pipelines/                    # Example CI/CD pipelines
└── README.md                         # This file

Quick Start

1. For Infrastructure as Code

Start with SKILL.md → reference/terraform.md

• Use Terraform module templates for reusable infrastructure
• Implement remote state management with S3 + DynamoDB
• Follow security best practices for secrets and credentials
• Apply cost optimization patterns

2. For Kubernetes Deployments

reference/kubernetes.md → reference/templates.md

• Deploy production-ready workloads with health checks
• Implement autoscaling (HPA/VPA)
• Configure RBAC and network policies
• Use Helm charts for application packaging

3. For Cloud Architecture

reference/cloud_platforms.md

• Design multi-tier architectures on AWS/Azure/GCP
• Implement high availability and disaster recovery
• Optimize costs with right-sizing and spot instances
• Follow Well-Architected Framework principles

4. For CI/CD Pipelines

reference/cicd.md

• Implement GitOps workflows with ArgoCD or Flux
• Create automated testing pipelines
• Configure deployment strategies (blue/green, canary)
• Set up infrastructure validation and security scanning

5. For Observability

reference/observability.md

• Define SLIs, SLOs, and SLAs
• Implement structured logging with correlation IDs
• Set up distributed tracing
• Create actionable alerts and runbooks

6. For Security

reference/security.md

• Implement least privilege access
• Automate security scanning and compliance checks
• Manage secrets with cloud-native solutions
• Apply zero-trust networking principles

Using the Utility Scripts

Generate Terraform Boilerplate

python scripts/devops_utils.py terraform init-project \
  --name myproject \
  --cloud aws \
  --region us-east-1

Validate Kubernetes Manifests

python scripts/devops_utils.py k8s validate \
  --file deployment.yaml \
  --schema-version 1.28

Generate GitOps Structure

python scripts/devops_utils.py gitops init \
  --tool argocd \
  --environments dev,staging,prod

Key Concepts

Infrastructure as Code Principles

All infrastructure should be:

• Version Controlled: Track all changes in Git
• Declarative: Define desired state, not imperative steps
• Idempotent: Multiple runs produce same result
• Modular: Reusable components with clear interfaces
• Tested: Automated validation and security scanning
• Documented: Clear README and inline comments

DevOps Best Practices

This skill follows the DevOps best practices:

✓ Security First - Never hardcode credentials, use least privilege ✓ Immutable Infrastructure - Replace rather than modify ✓ GitOps Workflows - Git as single source of truth ✓ Observability Built-In - Logs, metrics, and traces from day one ✓ Cost Optimization - Right-size resources, use spot instances ✓ Disaster Recovery - Automated backups and tested failover ✓ CI/CD Automation - Automated testing and deployment ✓ Policy as Code - Automated compliance and governance

Customization Guide

For Your Organization

Before using this skill, customize:

1. Cloud Provider Standards (reference/cloud_platforms.md)

• Update with your organization's cloud accounts
• Add your naming conventions and tagging standards
• Include your cost allocation and budgeting approach

2. Security Requirements (reference/security.md)

• Define your compliance requirements (HIPAA, PCI-DSS, SOC2)
• Specify your secrets management solution
• Add your organization's security policies

3. Naming Conventions (Throughout all files)

• Replace generic names with your naming standards
• Use consistent environment names (dev, staging, prod)
• Include your resource tagging strategy

4. Observability Tools (reference/observability.md)

• Specify your monitoring stack (Prometheus, Datadog, New Relic)
• Define your logging solution (ELK, Splunk, CloudWatch)
• Add your alerting and on-call system

Common DevOps Workflows

Workflow: Deploy New Application

1. Design architecture using reference/cloud_platforms.md 2. Create infrastructure with reference/terraform.md 3. Containerize application and deploy with reference/kubernetes.md 4. Set up CI/CD pipeline using reference/cicd.md 5. Implement monitoring with reference/observability.md 6. Apply security controls from reference/security.md

Workflow: Cloud Migration

1. Assess current architecture and dependencies 2. Design target cloud architecture 3. Create infrastructure as code modules 4. Implement migration pipeline with testing 5. Execute phased migration 6. Validate and optimize

Workflow: Implement GitOps

1. Set up Git repository structure 2. Install ArgoCD or Flux in cluster 3. Create application manifests and Helm charts 4. Configure automated sync policies 5. Implement progressive delivery 6. Set up monitoring and alerts

Troubleshooting

Issue: Terraform state conflicts Solution: Use remote state with locking (S3 + DynamoDB), implement state file backups, check for concurrent runs

Issue: Kubernetes pods failing health checks Solution: Review pod logs, check resource limits, verify network policies, validate environment variables

Issue: High cloud costs Solution: Implement tagging strategy, use cost allocation tags, right-size resources, use spot instances, enable auto-scaling

Issue: Secrets exposed in Git Solution: Use git-secrets or trufflehog to scan, rotate exposed secrets immediately, implement SOPS or Sealed Secrets

Support & Questions

Each reference file includes:

• Best practices and proven patterns
• Common mistakes to avoid
• Production-ready examples
• Implementation guidance

For questions about specific topics:

• Terraform & IaC: See reference/terraform.md
• Kubernetes: See reference/kubernetes.md
• Cloud platforms: See reference/cloud_platforms.md
• CI/CD: See reference/cicd.md
• Observability: See reference/observability.md
• Security: See reference/security.md

Version History

• v1.0 - Initial release with core DevOps functions
• Terraform best practices and module patterns
• Kubernetes production-ready configurations
• Cloud platform guidance (AWS, Azure, GCP)
• CI/CD pipeline templates
• Observability frameworks
• DevSecOps practices
• Utility scripts for common tasks

License & Usage

This skill is designed for use with Claude AI. Adapt and customize for your organization's specific needs.

---

Last Updated: October 2025 Skill Version: 1.0 For Claude Models: Claude Opus 4.1, Claude Sonnet 4.5, Claude Haiku 4.5

CI/CD Pipelines & GitOps

CI/CD Pipeline Stages

1. Source → 2. Build → 3. Test → 4. Deploy → 5. Monitor

┌──────────┐   ┌───────┐   ┌──────┐   ┌────────┐   ┌─────────┐
│  Source  │──▶│ Build │──▶│ Test │──▶│ Deploy │──▶│ Monitor │
│   (Git)  │   │       │   │      │   │        │   │         │
└──────────┘   └───────┘   └──────┘   └────────┘   └─────────┘

GitOps Principles

Core Concepts

1. Git as Single Source of Truth: All configuration in Git 2. Declarative: Desired state defined, not imperative steps 3. Automated: Continuous reconciliation of desired vs actual state 4. Auditable: All changes tracked in Git history

GitOps Workflow

Developer ──▶ Git Push ──▶ GitOps Controller ──▶ Kubernetes Cluster
                             (ArgoCD/Flux)
                                  │
                                  ▼
                            Continuous Sync

ArgoCD Setup

Installation

# Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Access ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Get initial password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

ArgoCD Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-production
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default

  source:
    repoURL: https://github.com/myorg/myapp-gitops.git
    targetRevision: main
    path: kubernetes/overlays/production

  destination:
    server: https://kubernetes.default.svc
    namespace: production

  syncPolicy:
    automated:
      prune: true      # Delete resources not in Git
      selfHeal: true   # Sync if cluster state drifts
      allowEmpty: false

    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true

    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas  # Ignore HPA-managed replicas

ArgoCD ApplicationSet (Multi-Environment)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - env: dev
        cluster: https://kubernetes.default.svc
      - env: staging
        cluster: https://kubernetes.default.svc
      - env: production
        cluster: https://prod-cluster-api.example.com

  template:
    metadata:
      name: 'myapp-{{env}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/myapp-gitops.git
        targetRevision: main
        path: 'kubernetes/overlays/{{env}}'
      destination:
        server: '{{cluster}}'
        namespace: '{{env}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Flux CD Setup

Installation

# Install Flux CLI
brew install fluxcd/tap/flux

# Check prerequisites
flux check --pre

# Bootstrap Flux
flux bootstrap github \
  --owner=myorg \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/production \
  --personal

Flux GitRepository

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/myapp-gitops
  ref:
    branch: main
  secretRef:
    name: git-credentials

Flux Kustomization

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp-production
  namespace: flux-system
spec:
  interval: 5m
  path: ./kubernetes/overlays/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: myapp
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: myapp
      namespace: production
  timeout: 2m

Progressive Delivery

Canary Deployment with Flagger

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp

  service:
    port: 80
    targetPort: 8080

  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10

    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m

    - name: request-duration
      thresholdRange:
        max: 500
      interval: 1m

    webhooks:
    - name: load-test
      url: http://flagger-loadtester/
      timeout: 5s
      metadata:
        cmd: "hey -z 1m -q 10 -c 2 http://myapp-canary.production/"

    - name: smoke-test
      url: http://flagger-loadtester/
      timeout: 5s
      metadata:
        type: smoke
        cmd: "curl -s http://myapp-canary.production/healthz | grep ok"

Blue/Green Deployment

apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: production
spec:
  selector:
    app: myapp
    version: blue  # Switch to 'green' for deployment
  ports:
  - port: 80
    targetPort: 8080

---
# Blue Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-blue
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: myapp
        image: myapp:v1.0.0

---
# Green Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-green
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: green
  template:
    metadata:
      labels:
        app: myapp
        version: green
    spec:
      containers:
      - name: myapp
        image: myapp:v2.0.0

CI/CD Pipeline Examples

GitHub Actions - Complete Pipeline

See templates.md for full GitHub Actions pipeline

GitLab CI Pipeline

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: "/certs"
  IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

stages:
  - lint
  - build
  - test
  - security
  - deploy

# Lint stage
lint:code:
  stage: lint
  image: node:20-alpine
  script:
    - npm ci
    - npm run lint
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/

lint:dockerfile:
  stage: lint
  image: hadolint/hadolint:latest-alpine
  script:
    - hadolint Dockerfile

# Build stage
build:image:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build --cache-from $CI_REGISTRY_IMAGE:latest -t $IMAGE_TAG .
    - docker tag $IMAGE_TAG $CI_REGISTRY_IMAGE:latest
    - docker push $IMAGE_TAG
    - docker push $CI_REGISTRY_IMAGE:latest
  only:
    - main
    - develop

# Test stage
test:unit:
  stage: test
  image: node:20-alpine
  script:
    - npm ci
    - npm run test:unit
  coverage: '/Statements\s*:\s*(\d+\.\d+)%/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

test:integration:
  stage: test
  image: $IMAGE_TAG
  services:
    - postgres:15-alpine
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: testuser
    POSTGRES_PASSWORD: testpass
  script:
    - npm run test:integration

# Security stage
security:trivy:
  stage: security
  image: aquasec/trivy:latest
  script:
    - trivy image --exit-code 1 --severity HIGH,CRITICAL $IMAGE_TAG
  allow_failure: true

security:secrets:
  stage: security
  image: trufflesecurity/trufflehog:latest
  script:
    - trufflehog filesystem . --fail

# Deploy stage
deploy:dev:
  stage: deploy
  image: bitnami/kubectl:latest
  script:
    - kubectl config use-context dev-cluster
    - kubectl set image deployment/myapp myapp=$IMAGE_TAG -n development
    - kubectl rollout status deployment/myapp -n development
  environment:
    name: development
  only:
    - develop

deploy:staging:
  stage: deploy
  image: bitnami/kubectl:latest
  script:
    - kubectl config use-context staging-cluster
    - kubectl set image deployment/myapp myapp=$IMAGE_TAG -n staging
    - kubectl rollout status deployment/myapp -n staging
  environment:
    name: staging
  only:
    - main

deploy:production:
  stage: deploy
  image: bitnami/kubectl:latest
  script:
    - kubectl config use-context prod-cluster
    - kubectl set image deployment/myapp myapp=$IMAGE_TAG -n production
    - kubectl rollout status deployment/myapp -n production
  environment:
    name: production
  when: manual
  only:
    - main

Jenkins Pipeline (Jenkinsfile)

pipeline {
    agent any

    environment {
        DOCKER_REGISTRY = 'ghcr.io'
        IMAGE_NAME = 'myorg/myapp'
        IMAGE_TAG = "${env.GIT_COMMIT.take(7)}"
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Lint') {
            parallel {
                stage('Lint Code') {
                    steps {
                        sh 'npm ci'
                        sh 'npm run lint'
                    }
                }
                stage('Lint Dockerfile') {
                    steps {
                        sh 'hadolint Dockerfile'
                    }
                }
            }
        }

        stage('Build') {
            steps {
                script {
                    docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-credentials') {
                        def app = docker.build("${IMAGE_NAME}:${IMAGE_TAG}")
                        app.push()
                        app.push('latest')
                    }
                }
            }
        }

        stage('Test') {
            parallel {
                stage('Unit Tests') {
                    steps {
                        sh 'npm run test:unit'
                    }
                }
                stage('Integration Tests') {
                    steps {
                        sh 'npm run test:integration'
                    }
                }
            }
        }

        stage('Security Scan') {
            steps {
                sh "trivy image --exit-code 1 --severity HIGH,CRITICAL ${IMAGE_NAME}:${IMAGE_TAG}"
            }
        }

        stage('Deploy to Dev') {
            when {
                branch 'develop'
            }
            steps {
                kubernetesDeploy(
                    configs: 'kubernetes/overlays/dev',
                    kubeconfigId: 'dev-kubeconfig'
                )
            }
        }

        stage('Deploy to Production') {
            when {
                branch 'main'
            }
            steps {
                input message: 'Deploy to production?', ok: 'Deploy'
                kubernetesDeploy(
                    configs: 'kubernetes/overlays/prod',
                    kubeconfigId: 'prod-kubeconfig'
                )
            }
        }
    }

    post {
        success {
            slackSend(
                color: 'good',
                message: "✅ Build ${env.BUILD_NUMBER} succeeded: ${env.JOB_NAME}"
            )
        }
        failure {
            slackSend(
                color: 'danger',
                message: "❌ Build ${env.BUILD_NUMBER} failed: ${env.JOB_NAME}"
            )
        }
    }
}

Infrastructure Testing

Terratest (Go)

package test

import (
    "testing"
    "github.com/gruntwork-io/terratest/modules/terraform"
    "github.com/stretchr/testify/assert"
)

func TestTerraformVPCModule(t *testing.T) {
    t.Parallel()

    terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
        TerraformDir: "../modules/vpc",
        Vars: map[string]interface{}{
            "name_prefix":   "test",
            "vpc_cidr":      "10.0.0.0/16",
        },
    })

    defer terraform.Destroy(t, terraformOptions)

    terraform.InitAndApply(t, terraformOptions)

    vpcID := terraform.Output(t, terraformOptions, "vpc_id")
    assert.NotEmpty(t, vpcID)
}

Checkov (Infrastructure Security)

# Scan Terraform
checkov -d terraform/ --framework terraform

# Scan Kubernetes
checkov -d kubernetes/ --framework kubernetes

# Output JSON
checkov -d terraform/ --framework terraform -o json > results.json

# Skip specific checks
checkov -d terraform/ --skip-check CKV_AWS_23

Kitchen-Terraform

# kitchen.yml
driver:
  name: terraform

provisioner:
  name: terraform

verifier:
  name: terraform
  systems:
    - name: default
      backend: aws
      controls:
        - vpc_test

platforms:
  - name: aws

suites:
  - name: default
    driver:
      variables:
        region: us-east-1

Deployment Strategies

Rolling Update (Default in Kubernetes)

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # Max pods above desired during update
      maxUnavailable: 0  # Zero-downtime deployment

Recreate (Downtime Acceptable)

spec:
  strategy:
    type: Recreate  # Kill all pods, then create new ones

Canary (Gradual Rollout)

• Deploy new version alongside old
• Route small % of traffic to new version
• Monitor metrics
• Gradually increase traffic
• Rollback if issues detected

Blue/Green (Instant Switch)

• Deploy new version (green) alongside old (blue)
• Test green environment
• Switch traffic from blue to green
• Keep blue for quick rollback

Rollback Procedures

Kubernetes Rollback

# View rollout history
kubectl rollout history deployment/myapp -n production

# Rollback to previous version
kubectl rollout undo deployment/myapp -n production

# Rollback to specific revision
kubectl rollout undo deployment/myapp -n production --to-revision=3

# Check rollout status
kubectl rollout status deployment/myapp -n production

ArgoCD Rollback

# Rollback application
argocd app rollback myapp-production

# Rollback to specific revision
argocd app rollback myapp-production 123

CI/CD Best Practices

1. Automate Everything: From code commit to production 2. Fast Feedback: Fail fast, fix fast 3. Test in Production-like: Staging mirrors production 4. Gradual Rollout: Canary or blue/green for production 5. Easy Rollback: One-click rollback capability 6. Security Scanning: Every stage, every pipeline 7. Infrastructure as Code: No manual changes 8. GitOps: Git as single source of truth 9. Observability: Monitor every deployment 10. Post-Deployment Tests: Smoke tests after deploy

---

Pipeline Performance Optimization

• Caching: Cache dependencies between runs
• Parallel Execution: Run independent stages in parallel
• Docker Layer Caching: Use BuildKit and layer caching
• Artifacts: Share build artifacts between stages
• Resource Limits: Optimize CI runner resources
• Skip Unnecessary: Only run tests for changed code

Cloud Platforms: AWS, Azure, GCP

Cloud Platform Selection

AWS (Amazon Web Services)

• Strengths: Market leader, most services, mature ecosystem
• Best for: Enterprise, startups, wide service selection
• Key Services: EC2, EKS, RDS, S3, Lambda, CloudFormation

Azure (Microsoft Azure)

• Strengths: Enterprise integration, hybrid cloud, Microsoft stack
• Best for: Windows workloads, hybrid scenarios, Microsoft shops
• Key Services: VMs, AKS, SQL Database, Blob Storage, ARM Templates

GCP (Google Cloud Platform)

• Strengths: Kubernetes-native, ML/AI, data analytics
• Best for: Kubernetes, data processing, ML workloads
• Key Services: Compute Engine, GKE, Cloud SQL, Cloud Storage, Deployment Manager

AWS Architecture Patterns

Multi-Tier Web Application

Internet
    │
    ▼
┌─────────────────┐
│  CloudFront     │ CDN
│  (Global)       │
└────────┬────────┘
         │
    ┌────▼────┐
    │   ALB   │ Load Balancer
    └────┬────┘
         │
    ┌────▼──────────┐
    │   ECS/EKS     │ Application Layer
    │  (Multi-AZ)   │
    └────┬──────────┘
         │
    ┌────▼────┐
    │   RDS   │ Database Layer
    │ (Multi-AZ)│
    └─────────┘

AWS Well-Architected Framework Pillars

1. Operational Excellence

• IaC (Terraform/CloudFormation)
• CI/CD automation
• Monitoring and observability

2. Security

• IAM least privilege
• Encryption at rest and in transit
• Network segmentation (VPC, Security Groups)

3. Reliability

• Multi-AZ deployment
• Auto Scaling
• Backup and disaster recovery

4. Performance Efficiency

• Right-sizing instances
• CloudFront for content delivery
• ElastiCache for caching

5. Cost Optimization

• Reserved Instances
• Spot Instances
• Auto Scaling based on demand

6. Sustainability

• Region selection for renewable energy
• Right-sizing to minimize waste

AWS Core Services

Compute

# EC2 Instance
resource "aws_instance" "app" {
  ami           = data.aws_ami.amazon_linux_2.id
  instance_type = "t3.medium"

  vpc_security_group_ids = [aws_security_group.app.id]
  subnet_id              = aws_subnet.private[0].id

  iam_instance_profile = aws_iam_instance_profile.app.name

  user_data = <<-EOF
              #!/bin/bash
              yum update -y
              yum install -y docker
              systemctl start docker
              EOF

  tags = {
    Name = "${var.name_prefix}-app-server"
  }
}

# ECS Fargate
resource "aws_ecs_service" "app" {
  name            = "${var.name_prefix}-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 3

  launch_type = "FARGATE"

  network_configuration {
    subnets          = aws_subnet.private[*].id
    security_groups  = [aws_security_group.app.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.app.arn
    container_name   = "app"
    container_port   = 8080
  }
}

# Lambda Function
resource "aws_lambda_function" "processor" {
  filename      = "lambda.zip"
  function_name = "${var.name_prefix}-processor"
  role          = aws_iam_role.lambda.arn
  handler       = "index.handler"
  runtime       = "nodejs20.x"

  environment {
    variables = {
      ENV = var.environment
    }
  }
}

Storage

# S3 Bucket
resource "aws_s3_bucket" "data" {
  bucket = "${var.name_prefix}-data-bucket"
}

resource "aws_s3_bucket_versioning" "data" {
  bucket = aws_s3_bucket.data.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "data" {
  bucket = aws_s3_bucket.data.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.s3.arn
    }
  }
}

# EBS Volume
resource "aws_ebs_volume" "data" {
  availability_zone = var.availability_zone
  size              = 100
  type              = "gp3"
  encrypted         = true
  kms_key_id        = aws_kms_key.ebs.arn

  tags = {
    Name = "${var.name_prefix}-data-volume"
  }
}

# EFS File System
resource "aws_efs_file_system" "shared" {
  encrypted = true
  kms_key_id = aws_kms_key.efs.arn

  lifecycle_policy {
    transition_to_ia = "AFTER_30_DAYS"
  }

  tags = {
    Name = "${var.name_prefix}-efs"
  }
}

Database

# RDS PostgreSQL
resource "aws_db_instance" "main" {
  identifier = "${var.name_prefix}-db"

  engine         = "postgres"
  engine_version = "15.4"
  instance_class = "db.t3.medium"

  allocated_storage     = 100
  max_allocated_storage = 1000
  storage_type          = "gp3"
  storage_encrypted     = true

  multi_az               = true
  db_subnet_group_name   = aws_db_subnet_group.main.name
  vpc_security_group_ids = [aws_security_group.db.id]

  backup_retention_period = 7
  backup_window          = "03:00-04:00"
  maintenance_window     = "sun:04:00-sun:05:00"

  enabled_cloudwatch_logs_exports = ["postgresql"]

  deletion_protection = var.environment == "prod"
}

# DynamoDB Table
resource "aws_dynamodb_table" "sessions" {
  name           = "${var.name_prefix}-sessions"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "session_id"

  attribute {
    name = "session_id"
    type = "S"
  }

  ttl {
    attribute_name = "ttl"
    enabled        = true
  }

  point_in_time_recovery {
    enabled = true
  }

  server_side_encryption {
    enabled = true
  }
}

Azure Architecture Patterns

Multi-Tier Application on Azure

Internet
    │
    ▼
┌─────────────────┐
│  Azure Front    │ CDN + WAF
│     Door        │
└────────┬────────┘
         │
    ┌────▼────────┐
    │   App GW    │ Load Balancer
    └────┬────────┘
         │
    ┌────▼──────────┐
    │     AKS       │ Application Layer
    │ (Multi-Zone)  │
    └────┬──────────┘
         │
    ┌────▼──────────┐
    │ Azure SQL DB  │ Database Layer
    │ (Geo-Replica) │
    └───────────────┘

Azure Core Services

Compute

# Virtual Machine
resource "azurerm_linux_virtual_machine" "app" {
  name                = "${var.name_prefix}-vm"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  size                = "Standard_D2s_v3"

  network_interface_ids = [
    azurerm_network_interface.app.id,
  ]

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Premium_LRS"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-focal"
    sku       = "20_04-lts"
    version   = "latest"
  }

  admin_username = "azureuser"
  admin_ssh_key {
    username   = "azureuser"
    public_key = file("~/.ssh/id_rsa.pub")
  }
}

# AKS Cluster
resource "azurerm_kubernetes_cluster" "main" {
  name                = "${var.name_prefix}-aks"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  dns_prefix          = var.name_prefix

  default_node_pool {
    name            = "default"
    node_count      = 3
    vm_size         = "Standard_D2s_v3"
    os_disk_size_gb = 100
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin = "azure"
    network_policy = "calico"
  }
}

# Azure Functions
resource "azurerm_linux_function_app" "processor" {
  name                = "${var.name_prefix}-func"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location

  storage_account_name       = azurerm_storage_account.func.name
  storage_account_access_key = azurerm_storage_account.func.primary_access_key
  service_plan_id            = azurerm_service_plan.func.id

  site_config {
    application_stack {
      node_version = "20"
    }
  }
}

GCP Architecture Patterns

Multi-Tier Application on GCP

Internet
    │
    ▼
┌─────────────────┐
│  Cloud CDN      │
└────────┬────────┘
         │
    ┌────▼────────┐
    │  Cloud LB   │ Global Load Balancer
    └────┬────────┘
         │
    ┌────▼──────────┐
    │     GKE       │ Application Layer
    │ (Multi-Zone)  │
    └────┬──────────┘
         │
    ┌────▼──────────┐
    │  Cloud SQL    │ Database Layer
    │   (HA)        │
    └───────────────┘

GCP Core Services

Compute

# Compute Engine Instance
resource "google_compute_instance" "app" {
  name         = "${var.name_prefix}-vm"
  machine_type = "e2-medium"
  zone         = var.zone

  boot_disk {
    initialize_params {
      image = "ubuntu-os-cloud/ubuntu-2004-lts"
      size  = 50
      type  = "pd-ssd"
    }
  }

  network_interface {
    network    = google_compute_network.vpc.name
    subnetwork = google_compute_subnetwork.private.name

    access_config {
      // Ephemeral public IP
    }
  }

  metadata_startup_script = file("startup.sh")

  service_account {
    scopes = ["cloud-platform"]
  }
}

# GKE Cluster
resource "google_container_cluster" "main" {
  name     = "${var.name_prefix}-gke"
  location = var.region

  # Remove default node pool
  remove_default_node_pool = true
  initial_node_count       = 1

  network    = google_compute_network.vpc.name
  subnetwork = google_compute_subnetwork.private.name

  ip_allocation_policy {
    cluster_ipv4_cidr_block  = "/16"
    services_ipv4_cidr_block = "/22"
  }

  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }
}

resource "google_container_node_pool" "main" {
  name       = "main-pool"
  location   = var.region
  cluster    = google_container_cluster.main.name
  node_count = 1

  autoscaling {
    min_node_count = 1
    max_node_count = 10
  }

  node_config {
    machine_type = "e2-medium"
    disk_size_gb = 100
    disk_type    = "pd-standard"

    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]

    workload_metadata_config {
      mode = "GKE_METADATA"
    }
  }
}

Multi-Cloud Strategy

When to Use Multi-Cloud

✓ Avoid vendor lock-in ✓ Leverage best-of-breed services ✓ Geographic requirements ✓ Disaster recovery

Multi-Cloud Challenges

✗ Increased complexity ✗ Higher operational overhead ✗ Different APIs and tools ✗ Data transfer costs

Multi-Cloud Tools

• Terraform: Unified IaC across clouds
• Kubernetes: Consistent compute layer
• Service Mesh: Unified networking
• OpenTelemetry: Unified observability

Cost Optimization Strategies

AWS Cost Optimization

# Savings Plans / Reserved Instances
# Purchase via AWS Console or API

# Spot Instances
resource "aws_autoscaling_group" "app" {
  mixed_instances_policy {
    instances_distribution {
      on_demand_base_capacity                  = 1
      on_demand_percentage_above_base_capacity = 20
      spot_allocation_strategy                 = "capacity-optimized"
    }

    launch_template {
      launch_template_specification {
        launch_template_id = aws_launch_template.app.id
      }

      override {
        instance_type = "t3.medium"
      }
      override {
        instance_type = "t3a.medium"
      }
    }
  }
}

# S3 Lifecycle Policy
resource "aws_s3_bucket_lifecycle_configuration" "data" {
  bucket = aws_s3_bucket.data.id

  rule {
    id     = "archive-old-data"
    status = "Enabled"

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 90
      storage_class = "GLACIER"
    }

    expiration {
      days = 365
    }
  }
}

Cost Monitoring

# AWS Budget
resource "aws_budgets_budget" "monthly" {
  name         = "monthly-budget"
  budget_type  = "COST"
  limit_amount = "1000"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 80
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = ["devops@example.com"]
  }
}

Disaster Recovery

RTO/RPO Targets

• RTO (Recovery Time Objective): Maximum acceptable downtime
• RPO (Recovery Point Objective): Maximum acceptable data loss

DR Strategies (Lowest to Highest Cost)

1. Backup & Restore (RPO: hours, RTO: hours)

• Regular backups to cloud storage
• Restore when needed

2. Pilot Light (RPO: minutes, RTO: hours)

• Minimal infrastructure always running
• Scale up when needed

3. Warm Standby (RPO: seconds, RTO: minutes)

• Scaled-down version running
• Scale up for failover

4. Multi-Site Active/Active (RPO: none, RTO: none)

• Full capacity in multiple regions
• Traffic distributed across sites

Multi-Region Setup (AWS)

provider "aws" {
  alias  = "primary"
  region = "us-east-1"
}

provider "aws" {
  alias  = "dr"
  region = "us-west-2"
}

# Primary region resources
module "vpc_primary" {
  source = "./modules/vpc"
  providers = {
    aws = aws.primary
  }
}

# DR region resources
module "vpc_dr" {
  source = "./modules/vpc"
  providers = {
    aws = aws.dr
  }
}

# Route53 health check and failover
resource "aws_route53_health_check" "primary" {
  fqdn              = aws_lb.primary.dns_name
  port              = 443
  type              = "HTTPS"
  resource_path     = "/health"
  failure_threshold = "3"
  request_interval  = "30"
}

resource "aws_route53_record" "app" {
  zone_id = aws_route53_zone.main.id
  name    = "app.example.com"
  type    = "A"

  set_identifier = "primary"
  failover_routing_policy {
    type = "PRIMARY"
  }

  alias {
    name                   = aws_lb.primary.dns_name
    zone_id                = aws_lb.primary.zone_id
    evaluate_target_health = true
  }

  health_check_id = aws_route53_health_check.primary.id
}

resource "aws_route53_record" "app_dr" {
  zone_id = aws_route53_zone.main.id
  name    = "app.example.com"
  type    = "A"

  set_identifier = "secondary"
  failover_routing_policy {
    type = "SECONDARY"
  }

  alias {
    name                   = aws_lb.dr.dns_name
    zone_id                = aws_lb.dr.zone_id
    evaluate_target_health = true
  }
}

Best Practices Summary

AWS

• Use IAM roles, not access keys
• Enable CloudTrail in all regions
• Encrypt everything (S3, EBS, RDS)
• Use VPC for network isolation
• Tag all resources for cost allocation

Azure

• Use Managed Identities
• Enable Azure Policy for governance
• Use Azure Key Vault for secrets
• Implement RBAC
• Use Resource Groups for organization

GCP

• Use Service Accounts with least privilege
• Enable Cloud Audit Logs
• Use VPC Service Controls
• Implement Organization Policies
• Use Labels for resource management

---

Cloud Service Comparison

Kubernetes & Container Orchestration

Kubernetes Architecture Essentials

Core Components

• Control Plane: API Server, Scheduler, Controller Manager, etcd
• Worker Nodes: Kubelet, Kube-proxy, Container Runtime
• Add-ons: CoreDNS, Metrics Server, Ingress Controller

Key Kubernetes Resources

• Workloads: Pods, Deployments, StatefulSets, DaemonSets, Jobs, CronJobs
• Networking: Services, Ingress, NetworkPolicies
• Configuration: ConfigMaps, Secrets
• Storage: PersistentVolumes, PersistentVolumeClaims, StorageClasses
• Access Control: ServiceAccounts, Roles, RoleBindings, ClusterRoles, ClusterRoleBindings

Production-Ready Deployment Pattern

Deployment with Best Practices

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: production
  labels:
    app: myapp
    version: v1.0.0
    environment: production
spec:
  replicas: 3
  revisionHistoryLimit: 10

  # Deployment strategy
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0  # Zero-downtime deployment

  selector:
    matchLabels:
      app: myapp

  template:
    metadata:
      labels:
        app: myapp
        version: v1.0.0
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"

    spec:
      # Security context at pod level
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000
        seccompProfile:
          type: RuntimeDefault

      # Service account for pod identity
      serviceAccountName: myapp

      # Init container for setup tasks
      initContainers:
      - name: init-config
        image: busybox:1.36
        command: ['sh', '-c', 'echo Initializing... && sleep 2']
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          runAsUser: 1000
          capabilities:
            drop:
              - ALL

      containers:
      - name: myapp
        image: myapp:1.0.0
        imagePullPolicy: IfNotPresent

        # Resource limits and requests
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"

        # Container security
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
          capabilities:
            drop:
              - ALL

        # Health checks
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3

        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 3
          successThreshold: 1
          failureThreshold: 3

        startupProbe:
          httpGet:
            path: /startup
            port: 8080
          initialDelaySeconds: 0
          periodSeconds: 10
          failureThreshold: 30  # 5 minutes max startup time

        # Environment variables
        env:
        - name: ENV
          value: "production"
        - name: LOG_LEVEL
          value: "info"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP

        # Environment from ConfigMap
        envFrom:
        - configMapRef:
            name: myapp-config
        - secretRef:
            name: myapp-secrets

        # Container ports
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        - name: metrics
          containerPort: 9090
          protocol: TCP

        # Volume mounts
        volumeMounts:
        - name: config
          mountPath: /etc/myapp
          readOnly: true
        - name: secrets
          mountPath: /etc/secrets
          readOnly: true
        - name: tmp
          mountPath: /tmp
        - name: cache
          mountPath: /var/cache

      # Volumes
      volumes:
      - name: config
        configMap:
          name: myapp-config
      - name: secrets
        secret:
          secretName: myapp-secrets
          defaultMode: 0400
      - name: tmp
        emptyDir: {}
      - name: cache
        emptyDir: {}

      # Pod scheduling
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - myapp
              topologyKey: kubernetes.io/hostname

      # Tolerations for node taints
      tolerations:
      - key: "node-role.kubernetes.io/spot"
        operator: "Exists"
        effect: "NoSchedule"

Service Configuration

apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: production
  labels:
    app: myapp
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  selector:
    app: myapp
  ports:
  - name: http
    port: 80
    targetPort: 8080
    protocol: TCP
  - name: https
    port: 443
    targetPort: 8443
    protocol: TCP

  # Session affinity (optional)
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

---
apiVersion: v1
kind: Service
metadata:
  name: myapp-headless
  namespace: production
spec:
  clusterIP: None  # Headless service for StatefulSets
  selector:
    app: myapp
  ports:
  - name: http
    port: 8080
    targetPort: 8080

Ingress with TLS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  namespace: production
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rate-limit: "100"
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
  tls:
  - hosts:
    - myapp.example.com
    secretName: myapp-tls

  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp
            port:
              number: 8080

Configuration Management

ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
  namespace: production
data:
  # Simple key-value pairs
  app.env: "production"
  log.level: "info"

  # Multi-line configuration files
  application.yaml: |
    server:
      port: 8080
      host: 0.0.0.0
    database:
      max_connections: 100
      timeout: 30s
    cache:
      ttl: 3600
      max_size: 1000

Secrets Management

# ❌ NOT RECOMMENDED - Base64 is not encryption!
apiVersion: v1
kind: Secret
metadata:
  name: myapp-secrets
  namespace: production
type: Opaque
data:
  db-password: cGFzc3dvcmQxMjM=  # base64 encoded

---
# ✅ BETTER - Use Sealed Secrets
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: myapp-secrets
  namespace: production
spec:
  encryptedData:
    db-password: AgBqV7zJ8...  # Encrypted with controller's public key

---
# ✅ BEST - Use External Secrets Operator with AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore

  target:
    name: myapp-secrets
    creationPolicy: Owner

  data:
  - secretKey: db-password
    remoteRef:
      key: prod/myapp/db-password
  - secretKey: api-key
    remoteRef:
      key: prod/myapp/api-key

StatefulSets for Stateful Applications

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: production
spec:
  serviceName: postgres-headless
  replicas: 3

  selector:
    matchLabels:
      app: postgres

  template:
    metadata:
      labels:
        app: postgres
    spec:
      securityContext:
        fsGroup: 999
        runAsUser: 999

      containers:
      - name: postgres
        image: postgres:15-alpine

        env:
        - name: POSTGRES_DB
          value: myapp
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: postgres-secrets
              key: username
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secrets
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata

        ports:
        - name: postgres
          containerPort: 5432

        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"

        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - pg_isready -U $POSTGRES_USER -d $POSTGRES_DB
          initialDelaySeconds: 30
          periodSeconds: 10

        readinessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - pg_isready -U $POSTGRES_USER -d $POSTGRES_DB
          initialDelaySeconds: 5
          periodSeconds: 5

        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data

  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3
      resources:
        requests:
          storage: 100Gi

Autoscaling

Horizontal Pod Autoscaler (HPA)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp

  minReplicas: 3
  maxReplicas: 20

  metrics:
  # CPU-based scaling
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

  # Memory-based scaling
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

  # Custom metrics (requires metrics adapter)
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"

  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 4
        periodSeconds: 15
      selectPolicy: Max

Vertical Pod Autoscaler (VPA)

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: myapp-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp

  updatePolicy:
    updateMode: "Auto"  # "Off", "Initial", "Recreate", or "Auto"

  resourcePolicy:
    containerPolicies:
    - containerName: myapp
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: 2000m
        memory: 2Gi
      controlledResources:
      - cpu
      - memory

RBAC (Role-Based Access Control)

ServiceAccount, Role, and RoleBinding

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp
  namespace: production

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: myapp-role
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps", "secrets"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-rolebinding
  namespace: production
subjects:
- kind: ServiceAccount
  name: myapp
  namespace: production
roleRef:
  kind: Role
  name: myapp-role
  apiGroup: rbac.authorization.k8s.io

ClusterRole for Cluster-Wide Permissions

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-pods-global
subjects:
- kind: ServiceAccount
  name: myapp
  namespace: production
roleRef:
  kind: ClusterRole
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Network Policies

Restrict Ingress Traffic

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: myapp-netpol
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: myapp

  policyTypes:
  - Ingress
  - Egress

  ingress:
  # Allow traffic from nginx ingress controller
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    - podSelector:
        matchLabels:
          app: nginx-ingress
    ports:
    - protocol: TCP
      port: 8080

  # Allow traffic from prometheus for metrics
  - from:
    - namespaceSelector:
        matchLabels:
          name: monitoring
    - podSelector:
        matchLabels:
          app: prometheus
    ports:
    - protocol: TCP
      port: 9090

  egress:
  # Allow DNS
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    - podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53

  # Allow database access
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432

  # Allow external HTTPS
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: TCP
      port: 443

Jobs and CronJobs

Job for One-Time Task

apiVersion: batch/v1
kind: Job
metadata:
  name: database-migration
  namespace: production
spec:
  backoffLimit: 3
  activeDeadlineSeconds: 600  # 10 minutes timeout

  template:
    metadata:
      labels:
        app: migration
    spec:
      restartPolicy: OnFailure

      containers:
      - name: migrate
        image: myapp:1.0.0
        command: ["/app/migrate"]
        args: ["--direction", "up"]

        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: myapp-secrets
              key: database-url

        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"

CronJob for Scheduled Tasks

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-database
  namespace: production
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  timeZone: "America/New_York"
  concurrencyPolicy: Forbid  # Don't allow concurrent runs
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1

  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 3600  # 1 hour timeout

      template:
        spec:
          restartPolicy: OnFailure

          containers:
          - name: backup
            image: postgres:15-alpine
            command:
            - /bin/sh
            - -c
            - |
              pg_dump -h $DB_HOST -U $DB_USER -d $DB_NAME | \
              gzip > /backup/backup-$(date +%Y%m%d-%H%M%S).sql.gz

            envFrom:
            - secretRef:
                name: postgres-secrets

            volumeMounts:
            - name: backup
              mountPath: /backup

          volumes:
          - name: backup
            persistentVolumeClaim:
              claimName: backup-pvc

Helm Charts

Chart Structure

myapp-chart/
├── Chart.yaml
├── values.yaml
├── values-dev.yaml
├── values-prod.yaml
├── templates/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── configmap.yaml
│   ├── secret.yaml
│   ├── hpa.yaml
│   ├── serviceaccount.yaml
│   ├── NOTES.txt
│   └── _helpers.tpl
└── README.md

Chart.yaml

apiVersion: v2
name: myapp
description: A Helm chart for MyApp
type: application
version: 1.0.0
appVersion: "1.0.0"
keywords:
  - myapp
  - web
maintainers:
  - name: DevOps Team
    email: devops@example.com
dependencies:
  - name: postgresql
    version: "12.x.x"
    repository: https://charts.bitnami.com/bitnami
    condition: postgresql.enabled

values.yaml

replicaCount: 3

image:
  repository: myapp
  pullPolicy: IfNotPresent
  tag: ""  # Defaults to chart appVersion

imagePullSecrets: []

serviceAccount:
  create: true
  annotations: {}
  name: ""

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 2000

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000
  capabilities:
    drop:
    - ALL

service:
  type: ClusterIP
  port: 80
  targetPort: 8080

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: myapp.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: myapp-tls
      hosts:
        - myapp.example.com

resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 20
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80

nodeSelector: {}

tolerations: []

affinity: {}

postgresql:
  enabled: true
  auth:
    username: myapp
    database: myapp

Helm Commands

# Install chart
helm install myapp ./myapp-chart -n production

# Install with custom values
helm install myapp ./myapp-chart -n production -f values-prod.yaml

# Upgrade release
helm upgrade myapp ./myapp-chart -n production

# Rollback
helm rollback myapp 1 -n production

# Uninstall
helm uninstall myapp -n production

# Template rendering (dry-run)
helm template myapp ./myapp-chart -f values-prod.yaml

# Lint chart
helm lint ./myapp-chart

kubectl Command Reference

# Get resources
kubectl get pods -n production
kubectl get deployments -n production -o wide
kubectl get svc -n production

# Describe resources
kubectl describe pod myapp-123 -n production
kubectl describe deployment myapp -n production

# Logs
kubectl logs myapp-123 -n production
kubectl logs -f myapp-123 -n production  # Follow
kubectl logs myapp-123 -n production --previous  # Previous container

# Execute commands in pod
kubectl exec -it myapp-123 -n production -- /bin/sh
kubectl exec myapp-123 -n production -- env

# Port forwarding
kubectl port-forward svc/myapp 8080:80 -n production

# Copy files
kubectl cp myapp-123:/tmp/file.txt ./file.txt -n production

# Scale deployment
kubectl scale deployment myapp --replicas=5 -n production

# Rollout management
kubectl rollout status deployment/myapp -n production
kubectl rollout history deployment/myapp -n production
kubectl rollout undo deployment/myapp -n production

# Apply/Delete manifests
kubectl apply -f deployment.yaml
kubectl delete -f deployment.yaml

# Resource usage
kubectl top nodes
kubectl top pods -n production

# Debug
kubectl run debug --image=busybox:1.36 -it --rm --restart=Never -- sh

---

Best Practices Summary

1. Always set resource requests and limits 2. Implement proper health checks (liveness, readiness, startup) 3. Use non-root containers with security contexts 4. Enable RBAC and use service accounts 5. Implement network policies for zero-trust networking 6. Use namespaces for isolation 7. Tag everything with consistent labels 8. Use ConfigMaps and Secrets (never hardcode) 9. Implement HPA for auto-scaling 10. Use readOnlyRootFilesystem when possible

Observability: Monitoring, Logging & Tracing

The Three Pillars of Observability

1. Metrics (What is happening?)

• Definition: Numeric measurements over time
• Examples: CPU usage, request rate, error rate, latency
• Tools: Prometheus, Datadog, CloudWatch, New Relic

2. Logs (Why is it happening?)

• Definition: Timestamped event records
• Examples: Application logs, access logs, error logs
• Tools: ELK Stack, Splunk, CloudWatch Logs, Loki

3. Traces (Where is it happening?)

• Definition: Request journey through distributed system
• Examples: Service call chains, database queries, external API calls
• Tools: Jaeger, Zipkin, AWS X-Ray, Datadog APM

SLI/SLO/SLA Framework

Service Level Indicators (SLIs)

Quantitative measurements of service quality

# Common SLIs
availability:
  definition: "Percentage of successful requests"
  measurement: "(successful_requests / total_requests) * 100"

latency:
  definition: "Time to process request"
  measurement: "p95 response time < 200ms"

error_rate:
  definition: "Percentage of failed requests"
  measurement: "(failed_requests / total_requests) * 100"

throughput:
  definition: "Requests processed per second"
  measurement: "requests_per_second"

Service Level Objectives (SLOs)

Target values for SLIs

# Example SLOs
availability_slo:
  target: 99.9%
  measurement_window: 30 days
  error_budget: 0.1% (43 minutes per month)

latency_slo:
  target: "95% of requests < 200ms"
  measurement_window: 7 days

error_rate_slo:
  target: "< 0.1%"
  measurement_window: 24 hours

Service Level Agreements (SLAs)

Business contracts with consequences

# Example SLA
web_application_sla:
  availability: 99.9%
  latency_p95: 300ms
  consequences:
    - availability < 99.9%: 10% service credit
    - availability < 99.0%: 25% service credit
    - availability < 95.0%: 50% service credit

Prometheus Setup

Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    environment: 'prod'

# Alert manager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

# Load rules
rule_files:
  - "/etc/prometheus/rules/*.yml"

# Scrape configurations
scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Kubernetes pods
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

  # Node exporter
  - job_name: 'node-exporter'
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

Alert Rules

# alert-rules.yml
groups:
  - name: application_alerts
    interval: 30s
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }} for {{ $labels.job }}"
          runbook: "https://wiki.example.com/runbooks/high-error-rate"

      # High latency
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 0.5
        for: 10m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "High latency detected"
          description: "P95 latency is {{ $value | humanizeDuration }} for {{ $labels.job }}"

      # Low availability
      - alert: ServiceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
          team: sre
        annotations:
          summary: "Service is down"
          description: "{{ $labels.job }} has been down for more than 2 minutes"

  - name: kubernetes_alerts
    interval: 30s
    rules:
      # Pod crash looping
      - alert: PodCrashLooping
        expr: |
          rate(kube_pod_container_status_restarts_total[15m]) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Pod crash looping"
          description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"

      # High memory usage
      - alert: HighMemoryUsage
        expr: |
          (container_memory_usage_bytes / container_spec_memory_limit_bytes) > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage"
          description: "Container {{ $labels.container }} in pod {{ $labels.pod }} is using {{ $value | humanizePercentage }} of memory"

      # Node disk space
      - alert: NodeDiskSpaceLow
        expr: |
          (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Node disk space low"
          description: "Node {{ $labels.node }} has less than 10% disk space available"

Structured Logging

Best Practices

{
  "timestamp": "2025-10-17T10:30:45.123Z",
  "level": "ERROR",
  "service": "api-gateway",
  "version": "v1.2.3",
  "trace_id": "abc123def456",
  "span_id": "789ghi012jkl",
  "user_id": "user-12345",
  "request_id": "req-67890",
  "method": "POST",
  "path": "/api/v1/orders",
  "status_code": 500,
  "duration_ms": 245,
  "error": {
    "type": "DatabaseConnectionError",
    "message": "Failed to connect to database",
    "stack_trace": "..."
  },
  "context": {
    "order_id": "order-98765",
    "customer_id": "cust-54321"
  }
}

Logging Configuration (Node.js Example)

const winston = require('winston');

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.errors({ stack: true }),
    winston.format.json()
  ),
  defaultMeta: {
    service: process.env.SERVICE_NAME,
    version: process.env.SERVICE_VERSION,
    environment: process.env.ENVIRONMENT
  },
  transports: [
    new winston.transports.Console(),
    new winston.transports.File({
      filename: 'error.log',
      level: 'error'
    }),
    new winston.transports.File({
      filename: 'combined.log'
    })
  ]
});

// Usage with correlation ID
app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] || uuidv4();
  req.logger = logger.child({
    request_id: req.id,
    trace_id: req.headers['x-trace-id']
  });
  next();
});

app.post('/api/orders', async (req, res) => {
  req.logger.info('Creating order', {
    customer_id: req.body.customer_id
  });

  try {
    const order = await createOrder(req.body);
    req.logger.info('Order created successfully', {
      order_id: order.id
    });
    res.json(order);
  } catch (error) {
    req.logger.error('Failed to create order', {
      error: error.message,
      stack: error.stack
    });
    res.status(500).json({ error: 'Internal server error' });
  }
});

Distributed Tracing

OpenTelemetry Configuration

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024

  memory_limiter:
    check_interval: 1s
    limit_mib: 512

  resource:
    attributes:
      - key: environment
        value: production
        action: insert

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  prometheus:
    endpoint: 0.0.0.0:8889

  logging:
    loglevel: info

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [jaeger, logging]

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [prometheus, logging]

Application Instrumentation (Python Example)

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

# Set up tracing
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

# Configure OTLP exporter
otlp_exporter = OTLPSpanExporter(
    endpoint="otel-collector:4317",
    insecure=True
)

# Add span processor
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

# Instrument Flask and requests library
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

# Manual span creation
@app.route('/api/order/<order_id>')
def get_order(order_id):
    with tracer.start_as_current_span("get_order") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("user.id", request.headers.get('X-User-ID'))

        # Add events
        span.add_event("Fetching order from database")
        order = fetch_order_from_db(order_id)

        if not order:
            span.set_status(Status(StatusCode.ERROR, "Order not found"))
            return {"error": "Order not found"}, 404

        span.add_event("Order retrieved successfully")
        return order

Dashboards & Visualization

Grafana Dashboard JSON (Example)

{
  "dashboard": {
    "title": "Application Performance",
    "panels": [
      {
        "title": "Request Rate",
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])",
            "legendFormat": "{{method}} {{status}}"
          }
        ],
        "type": "graph"
      },
      {
        "title": "Error Rate",
        "targets": [
          {
            "expr": "rate(http_requests_total{status=~\"5..\"}[5m]) / rate(http_requests_total[5m])",
            "legendFormat": "Error Rate"
          }
        ],
        "type": "graph"
      },
      {
        "title": "P95 Latency",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "P95 Latency"
          }
        ],
        "type": "graph"
      },
      {
        "title": "Active Connections",
        "targets": [
          {
            "expr": "sum(up{job=\"myapp\"})",
            "legendFormat": "Active Instances"
          }
        ],
        "type": "stat"
      }
    ]
  }
}

On-Call & Incident Response

Runbook Template

# Runbook: High Error Rate Alert

## Alert Details
- **Alert Name**: HighErrorRate
- **Severity**: Critical
- **Team**: Backend Engineering
- **On-Call**: See PagerDuty schedule

## Symptoms
- Error rate exceeds 5% for 5 minutes
- Users experiencing 5xx errors
- Elevated p95 latency

## Investigation Steps

1. **Check service health**

kubectl get pods -n production -l app=myapp kubectl logs -n production -l app=myapp --tail=100

2. **Review error logs**
   - Check Grafana dashboard
   - Review application logs in Kibana
   - Check CloudWatch metrics

3. **Identify error patterns**
   - What endpoints are failing?
   - Are errors consistent across all pods?
   - Is there a pattern in timing?

4. **Check dependencies**
   - Database connectivity
   - External API availability
   - Redis/cache status

## Common Causes & Solutions

### Database Connection Issues
- **Symptoms**: Connection timeout errors
- **Solution**:

Check database connectivity

kubectl exec -it <pod-name> -- nc -zv database-host 5432

Check connection pool

kubectl logs <pod-name> | grep "connection pool"

### Memory Leaks
- **Symptoms**: Increasing memory usage, OOM kills
- **Solution**: Restart affected pods, investigate memory usage

### Deployment Issues
- **Symptoms**: Errors started after deployment
- **Solution**: Rollback deployment

kubectl rollout undo deployment/myapp -n production

## Escalation
- If unresolved after 15 minutes, escalate to Senior Engineer
- If service degradation > 30 minutes, notify VP Engineering

## Post-Incident
- Create incident report
- Schedule post-mortem
- Update runbook with findings

Observability Best Practices

1. Use consistent naming: Follow naming conventions for metrics, logs, traces 2. Add context: Include correlation IDs in logs and traces 3. Set meaningful alerts: Avoid alert fatigue with actionable alerts 4. Define SLOs: Measure what matters to users 5. Practice incident response: Regular game days and fire drills 6. Automate runbooks: Convert manual steps to automated remediation 7. Monitor the monitors: Ensure observability stack is reliable 8. Continuous improvement: Review and refine based on incidents

---

Tools Comparison

DevSecOps & Security Best Practices

Security Principles

Defense in Depth

• Multiple layers of security controls
• Network security, application security, data security
• No single point of failure

Least Privilege

• Minimum permissions necessary
• Regular access reviews
• Time-bound elevated access

Zero Trust

• Never trust, always verify
• Verify every request regardless of origin
• Micro-segmentation and strict access controls

Secrets Management

❌ Never Do This

# NEVER hardcode secrets!
apiVersion: v1
kind: Pod
metadata:
  name: bad-example
spec:
  containers:
  - name: app
    env:
    - name: DATABASE_PASSWORD
      value: "SuperSecret123!"  # 🚨 NEVER DO THIS
    - name: API_KEY
      value: "sk-abc123def456"  # 🚨 NEVER DO THIS

✅ Use External Secrets Operator

# External Secrets with AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: production
spec:
  refreshInterval: 1h

  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore

  target:
    name: app-secrets
    creationPolicy: Owner

  data:
  - secretKey: database-password
    remoteRef:
      key: prod/myapp/database-password

  - secretKey: api-key
    remoteRef:
      key: prod/myapp/api-key

✅ Use Sealed Secrets

# Install kubeseal
brew install kubeseal

# Create sealed secret
kubectl create secret generic myapp-secrets \
  --from-literal=db-password='secret123' \
  --dry-run=client -o yaml | \
  kubeseal -o yaml > sealed-secret.yaml

# Apply sealed secret (safe to commit to Git)
kubectl apply -f sealed-secret.yaml

AWS Secrets Manager with IRSA

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp
  namespace: production
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/myapp-secrets-role

---
# Application retrieves secrets using AWS SDK
# No credentials in code or config!

Container Security

Secure Dockerfile

# Use specific version tags, not 'latest'
FROM node:20.10.0-alpine3.19 AS builder

# Run as non-root user
RUN addgroup -g 1000 nodejs && \
    adduser -u 1000 -G nodejs -s /bin/sh -D nodejs

WORKDIR /app

# Copy and install dependencies
COPY --chown=nodejs:nodejs package*.json ./
RUN npm ci --only=production && \
    npm cache clean --force

# Copy application code
COPY --chown=nodejs:nodejs . .

# Build application
RUN npm run build

# Production stage
FROM node:20.10.0-alpine3.19

# Install dumb-init for signal handling
RUN apk add --no-cache dumb-init

# Create non-root user
RUN addgroup -g 1000 nodejs && \
    adduser -u 1000 -G nodejs -s /bin/sh -D nodejs

WORKDIR /app

# Copy from builder
COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist
COPY --from=builder --chown=nodejs:nodejs /app/node_modules ./node_modules
COPY --from=builder --chown=nodejs:nodejs /app/package*.json ./

# Switch to non-root user
USER nodejs

# Use dumb-init
ENTRYPOINT ["dumb-init", "--"]

# Run application
CMD ["node", "dist/index.js"]

# Expose port
EXPOSE 8080

# Add labels
LABEL org.opencontainers.image.source="https://github.com/myorg/myapp" \
      org.opencontainers.image.version="1.0.0" \
      org.opencontainers.image.vendor="MyOrg"

Security Scanning

Trivy (Container Vulnerability Scanner)

# Scan Docker image
trivy image myapp:latest

# Scan with severity filter
trivy image --severity HIGH,CRITICAL myapp:latest

# Scan filesystem
trivy fs --security-checks vuln,config /path/to/project

# Output to JSON
trivy image -f json -o results.json myapp:latest

Grype (Vulnerability Scanner)

# Scan image
grype myapp:latest

# Scan with specific severity
grype myapp:latest --fail-on high

# Output SARIF for GitHub
grype myapp:latest -o sarif > results.sarif

Pod Security Standards

# Pod Security Admission
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

---
# Secure Pod Configuration
apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
  namespace: production
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 2000
    seccompProfile:
      type: RuntimeDefault

  containers:
  - name: app
    image: myapp:1.0.0

    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
      capabilities:
        drop:
        - ALL
        add:
        - NET_BIND_SERVICE  # Only if needed

    resources:
      limits:
        memory: "512Mi"
        cpu: "500m"
      requests:
        memory: "256Mi"
        cpu: "250m"

    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: cache
      mountPath: /var/cache

  volumes:
  - name: tmp
    emptyDir: {}
  - name: cache
    emptyDir: {}

Network Security

Network Policies

# Default deny all ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress

---
# Allow specific ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: app-allow-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: myapp

  policyTypes:
  - Ingress
  - Egress

  ingress:
  # Allow from ingress controller
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080

  egress:
  # Allow DNS
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    - podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53

  # Allow database
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432

  # Allow HTTPS egress
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: TCP
      port: 443

AWS Security Groups (Terraform)

# Application Security Group
resource "aws_security_group" "app" {
  name        = "${var.name_prefix}-app-sg"
  description = "Security group for application"
  vpc_id      = var.vpc_id

  # Allow inbound from ALB only
  ingress {
    from_port       = 8080
    to_port         = 8080
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
    description     = "HTTP from ALB"
  }

  # Allow outbound to database
  egress {
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.database.id]
    description     = "PostgreSQL to database"
  }

  # Allow HTTPS egress
  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "HTTPS to internet"
  }

  tags = var.tags
}

IAM & RBAC

Kubernetes RBAC

# Least privilege service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp
  namespace: production

---
# Role with minimal permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: myapp-role
  namespace: production
rules:
# Only read ConfigMaps and Secrets
- apiGroups: [""]
  resources: ["configmaps", "secrets"]
  verbs: ["get", "list"]
  resourceNames: ["myapp-config", "myapp-secrets"]

# Read own pod information
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]

---
# Bind role to service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-rolebinding
  namespace: production
subjects:
- kind: ServiceAccount
  name: myapp
  namespace: production
roleRef:
  kind: Role
  name: myapp-role
  apiGroup: rbac.authorization.k8s.io

AWS IAM Policy (Terraform)

# Least privilege IAM policy
data "aws_iam_policy_document" "app" {
  # Allow reading specific secrets
  statement {
    actions = [
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret",
    ]
    resources = [
      "arn:aws:secretsmanager:${var.region}:${data.aws_caller_identity.current.account_id}:secret:prod/myapp/*"
    ]
  }

  # Allow writing to specific S3 bucket
  statement {
    actions = [
      "s3:PutObject",
      "s3:GetObject",
    ]
    resources = [
      "${aws_s3_bucket.app.arn}/*"
    ]
  }

  # Allow publishing to specific SNS topic
  statement {
    actions = [
      "sns:Publish",
    ]
    resources = [
      aws_sns_topic.app_notifications.arn
    ]
  }
}

resource "aws_iam_policy" "app" {
  name   = "${var.name_prefix}-app-policy"
  policy = data.aws_iam_policy_document.app.json
}

Compliance & Policy as Code

OPA (Open Policy Agent)

# Deny pods running as root
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Pod"
  input.request.object.spec.securityContext.runAsNonRoot != true
  msg := "Pods must not run as root"
}

# Require resource limits
deny[msg] {
  input.request.kind.kind == "Pod"
  container := input.request.object.spec.containers[_]
  not container.resources.limits
  msg := sprintf("Container %v must specify resource limits", [container.name])
}

# Deny latest tag
deny[msg] {
  input.request.kind.kind == "Pod"
  container := input.request.object.spec.containers[_]
  endswith(container.image, ":latest")
  msg := sprintf("Container %v uses 'latest' tag", [container.name])
}

# Require specific labels
deny[msg] {
  input.request.kind.kind == "Pod"
  required_labels := ["app", "version", "environment"]
  label := required_labels[_]
  not input.request.object.metadata.labels[label]
  msg := sprintf("Missing required label: %v", [label])
}

Terraform Sentinel Policy

# sentinel.hcl
policy "require-tags" {
  enforcement_level = "hard-mandatory"
}

policy "restrict-instance-type" {
  enforcement_level = "soft-mandatory"
}

# require-tags.sentinel
import "tfplan/v2" as tfplan

required_tags = ["Environment", "Owner", "CostCenter"]

main = rule {
  all tfplan.resource_changes as _, rc {
    rc.mode is "managed" and
    rc.type in ["aws_instance", "aws_db_instance", "aws_s3_bucket"] and
    rc.change.actions contains "create"
    implies
    all required_tags as tag {
      rc.change.after.tags contains tag
    }
  }
}

Security Scanning in CI/CD

GitHub Actions Security Workflow

name: Security Scan

on: [push, pull_request]

jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: TruffleHog Secret Scan
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.repository.default_branch }}
          head: HEAD

  container-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Run Trivy scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:${{ github.sha }}
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'

      - name: Upload to GitHub Security
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

  terraform-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Checkov
        uses: bridgecrewio/checkov-action@master
        with:
          directory: terraform/
          framework: terraform
          soft_fail: false

  sast-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Semgrep
        uses: returntocorp/semgrep-action@v1
        with:
          config: auto

Encryption

At Rest

# S3 bucket encryption
resource "aws_s3_bucket_server_side_encryption_configuration" "app" {
  bucket = aws_s3_bucket.app.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.s3.arn
    }
    bucket_key_enabled = true
  }
}

# RDS encryption
resource "aws_db_instance" "app" {
  storage_encrypted = true
  kms_key_id        = aws_kms_key.rds.arn
  # ... other configuration
}

# EBS encryption
resource "aws_ebs_volume" "app" {
  encrypted  = true
  kms_key_id = aws_kms_key.ebs.arn
  # ... other configuration
}

In Transit

# Enforce TLS in Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  tls:
  - hosts:
    - app.example.com
    secretName: app-tls

Security Best Practices Checklist

• [ ] Never hardcode secrets - Use secret managers
• [ ] Run containers as non-root - Set securityContext
• [ ] Use specific image tags - Never use 'latest'
• [ ] Scan images for vulnerabilities - Use Trivy/Grype
• [ ] Enable Pod Security Standards - Use restricted profile
• [ ] Implement network policies - Default deny, allow specific
• [ ] Use RBAC - Least privilege access
• [ ] Encrypt data at rest - KMS for all data stores
• [ ] Enforce TLS - All traffic encrypted in transit
• [ ] Scan IaC for issues - Checkov, tfsec in CI/CD
• [ ] Rotate credentials regularly - Automate rotation
• [ ] Audit and log everything - CloudTrail, audit logs
• [ ] Implement policy as code - OPA, Sentinel
• [ ] Regular security reviews - Penetration testing, audits
• [ ] Keep dependencies updated - Renovate, Dependabot

---

Security Incident Response

1. Detect: Automated alerting on security events 2. Contain: Isolate affected systems 3. Eradicate: Remove threat and vulnerabilities 4. Recover: Restore services securely 5. Learn: Post-incident review and improvements