Infrastructure Engineer

Infrastructure Engineer

Overview

Design and implement cloud infrastructure. This skill covers IaC (Terraform, Pulumi), CI/CD pipelines, container orchestration (Kubernetes), networking, observability, and security hardening.

Features

• Infrastructure as Code: Terraform modules, Pulumi stacks, state management, drift detection
• CI/CD pipelines: pipeline essentials, deployment strategies, environment promotion, rollback procedures
• Container orchestration: Kubernetes architecture, Helm charts, service mesh, autoscaling
• Networking: VPC design, load balancers, DNS, CDN, private endpoints
• Observability: metrics, logs, traces, alerting, dashboards
• Security hardening: IAM policies, encryption, network security, compliance controls

Usage

1. Identify the user's infrastructure need (IaC, CI/CD, Kubernetes, networking, observability, or security) 2. Follow the corresponding workflow below 3. Produce structured outputs: Terraform configs, pipeline definitions, Kubernetes manifests, or architecture diagrams

Examples

• User: "Set up Terraform for AWS"

Agent: Runs IaC workflow, creates module structure, configures state backend, produces VPC, EC2, and RDS resources

• User: "Build a CI/CD pipeline"

Agent: Runs CI/CD workflow, designs pipeline with build/test/deploy stages, adds environment promotion and rollback

• User: "Configure Kubernetes autoscaling"

Agent: Runs Kubernetes workflow, sets up HPA based on CPU/memory, configures cluster autoscaler, tests scaling behavior

When to Use

• Designing cloud or hybrid networks, compute, storage, and IAM patterns
• Building CI/CD, GitOps, Kubernetes, or IaC (Terraform/Pulumi/CloudFormation)
• Implementing observability, SRE practices, and infrastructure incident response
• Hardening infrastructure for security and compliance (SOC 2, ISO 27001, etc.)

For security-control ownership (IdP, KMS, SIEM integration, PAM, guardrails as primary deliverable), prefer information-security-engineer.

For authorized network/AD/segmentation pentest validation (not IaC design), prefer network-pentester.

For internal developer platform, golden paths, Backstage/catalog, and platform-as-product work, prefer platform-engineer.

When NOT to Use

• Data warehouse modeling, ETL, or analytics pipeline SLAs → use data-warehouse-engineer or data-system-ops-lead
• Data governance catalogs, quality SLAs, or steward workflows → use data-architect or data-manager
• LLM prompt/agent design or production guardrails → use prompt-engineer
• Technical documentation or research synthesis deliverables → use tech-writer-researcher
• Cross-service solution architecture before build → use senior-system-architecture
• Cloud reference architecture, landing zone, migration design → use cloud-architect
• Day-to-day managed cloud services (networking, RDS, serverless, cloud IAM) → use cloud-engineer
• K8s cluster workload deploy, upgrades, and in-cluster troubleshooting → use cluster-deployment-engineer
• Data center design, MEP, colo build, and facility commissioning → use data-center-design-execution-lead
• Physical compute utilization, supply forecast, consolidation, refresh → use data-center-compute-supply-efficiency
• Infrastructure org strategy, portfolio prioritization, board/CFO narratives → use vp-of-infrastructure
• Cloud program strategy, migration portfolio, CCoE, EA governance → use vp-of-cloud
• Customer RFP, discovery, PoC charter, solution handoff (not production build) → use solutions-architect

Core Workflows

1. Infrastructure Design & Provisioning

Design checklist:

1. Define requirements

• Traffic patterns (steady, bursty, batch)
• Compliance needs (data residency, encryption)
• RTO/RPO targets
• Budget constraints

2. Choose compute model

3. Design network

• VPC/VNet with public/private subnets
• NAT Gateway for outbound-only workloads
• Transit Gateway for multi-VPC/VNet
• PrivateLink/Private Service Connect for SaaS

4. Provision with IaC

• Terraform for multi-cloud
• CloudFormation for AWS-only
• Pulumi for programmatic (Python/TS)
• Ansible for configuration management

2. CI/CD & Platform Automation

Pipeline essentials:

• Build → Test → Security scan → Deploy → Verify
• GitOps with ArgoCD/Flux for K8s
• Feature flags for progressive rollout
• Automated rollback on failure

3. Monitoring & Reliability

Observability stack:

• Metrics: Prometheus + Grafana, Datadog, CloudWatch
• Logs: ELK, Loki, Splunk
• Traces: Jaeger, Zipkin, AWS X-Ray
• Alerts: PagerDuty, Opsgenie, Alertmanager

Reliability patterns:

• Health checks: liveness, readiness, startup probes
• Circuit breakers: fail fast, degrade gracefully
• Rate limiting: token bucket, leaky bucket
• Bulkheads: isolate failure domains

4. Security & Compliance

Security by layer:

• Network: Security groups, NACLs, WAF, DDoS protection
• Identity: RBAC, least privilege, MFA, service accounts
• Data: Encryption at rest (KMS) and in transit (TLS 1.3)
• Application: Secrets management, vulnerability scanning
• Compliance: SOC 2, ISO 27001, GDPR, HIPAA controls

Cloud Infrastructure

VPC / VNet Design

Multi-AZ Architecture

VPC (10.0.0.0/16)
├── Public Subnet A (10.0.1.0/24) — AZ-a
│   ├── ALB/NLB
│   ├── NAT Gateway A
│   └── Bastion host
├── Public Subnet B (10.0.2.0/24) — AZ-b
│   └── NAT Gateway B
├── Private Subnet A (10.0.3.0/24) — AZ-a
│   ├── App tier (ECS/EKS/EC2)
│   └── RDS read replica
├── Private Subnet B (10.0.4.0/24) — AZ-b
│   ├── App tier
│   └── RDS primary
└── Database Subnet (10.0.5.0/24) — AZ-a,b
    └── RDS, ElastiCache, MSK

Traffic Flow Patterns

Compute

AWS

GCP

Azure

Storage

Object Storage

Block Storage

IAM & Identity

AWS IAM Best Practices

• Use IAM roles, not long-term access keys
• Apply least privilege with conditions
• Enable MFA for console access
• Use service-linked roles where available
• Regular access review (quarterly)

Cross-Cloud Identity

• OIDC: Federate identity across clouds (e.g., GitHub Actions → AWS via OIDC)
• Azure AD / Google Workspace: SSO across SaaS and cloud
• HashiCorp Vault: Dynamic secrets across clouds

Policy Structure

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject"],
    "Resource": "arn:aws:s3:::my-bucket/*",
    "Condition": {
      "StringEquals": {"aws:RequestedRegion": "us-east-1"},
      "Bool": {"aws:MultiFactorAuthPresent": "true"}
    }
  }]
}

Networking

Load Balancers

DNS & Traffic Management

• Route 53: Weighted, latency-based, geo, failover routing
• Cloud DNS: Global anycast, low TTL for failover
• Azure DNS: Integrated with Traffic Manager

CDN

• CloudFront: Edge caching, signed URLs, origin shield
• Cloud CDN: GCP-native, integrates with GCLB
• Azure CDN / Front Door: Rules engine, WAF

Serverless Patterns

Event-Driven Architecture

Event Source → Event Bus → Lambda/Function → Target

Event sources:

• S3, DynamoDB Streams, Kinesis, SNS, SQS, EventBridge
• API Gateway, ALB, CloudWatch Events
• Third-party SaaS (via EventBridge)

Cold Start Mitigation

• Provisioned concurrency (Lambda)
• Minimum instances (Cloud Run)
• Always-ready plans (Azure Functions Premium)
• Keep-alive pings (less reliable)

Cost Optimization

Disaster Recovery

RTO/RPO Targets by Tier

Multi-Region Patterns

• Read replica: Async replication, promote on failover
• Global database: Aurora Global, Cosmos DB, Spanner
• Active-active: Multi-master writes, conflict resolution needed

DevOps & Platform Engineering

CI/CD Pipeline Design

Pipeline Stages

Code Commit → Build → Unit Test → Integration Test → Security Scan → Deploy Staging → E2E Test → Deploy Prod → Verify

Git Branching Strategies

Pipeline as Code Examples

GitHub Actions:

name: CI/CD
on:
  push:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npm test
      - run: npm run build
  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to AWS
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.DEPLOY_ROLE_ARN }}
      - run: aws ecs update-service --cluster prod --service app --force-new-deployment

GitLab CI:

stages: [build, test, deploy]

build:
  stage: build
  script: docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .

test:
  stage: test
  script: pytest
  coverage: '/TOTAL.+?([0-9]{1,3}%)/'

deploy_prod:
  stage: deploy
  script: helm upgrade --install app ./chart
  environment:
    name: production
  when: manual

Containerization

Dockerfile Best Practices

# Multi-stage build
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build

FROM node:20-alpine
RUN apk add --no-cache dumb-init
USER node
WORKDIR /app
COPY --from=builder --chown=node:node /app/dist ./dist
COPY --from=builder --chown=node:node /app/node_modules ./node_modules
EXPOSE 3000
ENTRYPOINT ["dumb-init", "node", "dist/main.js"]

Kubernetes Patterns

Deployment with HPA:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
        - name: app
          image: app:v1.2.0
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

ConfigMap + Secret:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DB_HOST: "postgres"
  LOG_LEVEL: "info"
---
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  DB_PASSWORD: "<encrypted>"
  API_KEY: "<encrypted>"

Infrastructure as Code

Terraform Patterns

Module structure:

modules/
├── vpc/
│   ├── main.tf
│   ├── variables.tf
│   └── outputs.tf
├── ecs/
├── rds/
└── s3/

State management:

• Use remote backend (S3 + DynamoDB, GCS, Azure Storage)
• Enable state locking
• Separate state per environment

Workspace strategy:

terraform workspace new prod
terraform workspace select prod
terraform apply -var-file=prod.tfvars

Pulumi Example

import pulumi
from pulumi_aws import s3

bucket = s3.Bucket("my-bucket",
    versioning=s3.BucketVersioningArgs(
        enabled=True
    ))

pulumi.export("bucket_name", bucket.id)

GitOps

ArgoCD Setup

# Application manifest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/gitops-repo
    targetRevision: HEAD
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Flux Setup

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/org/gitops-repo
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 10m
  path: ./overlays/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: my-app

SRE Practices

SLI / SLO / SLA

Error Budget

Error Budget = 100% - SLO

Example: SLO = 99.9% → Error Budget = 0.1% = 43.8 min/month

Policy: If error budget is consumed, freeze feature launches until recovered.

Toil Reduction

• Automate repeated manual tasks
• Self-service platforms for developers
• Standardized templates and modules
• Target: <50% of time on toil

Monitoring & Alerting

Metrics Stack

Key Metrics

Alerting Rules

# Prometheus alert rule
groups:
  - name: app_alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"

Incident Response

On-Call Setup

• Primary + secondary rotation
• Escalation: 15 min → 30 min → manager
• Runbook required for every alert
• Blameless post-mortems within 48 hours

Post-Mortem Template

## Incident [ID] — [Date]

### Summary
What happened in 2 sentences.

### Timeline
| Time | Event |
|---|---|
| 10:00 | Alert fired |
| 10:15 | On-call acknowledged |
| 10:30 | Mitigation applied |
| 11:00 | Service restored |

### Root Cause
5 Whys analysis.

### Impact
- Duration: 1 hour
- Affected users: ~500
- Data loss: None

### Action Items
| Action | Owner | Due |
|---|---|---|
| Fix X | @alice | Friday |
| Add monitoring for Y | @bob | Next week |

Hybrid & On-Premise Infrastructure

Virtualization

VMware vSphere

KVM / QEMU

• Open source alternative to VMware
• Proxmox VE: Web UI for KVM + LXC
• oVirt: Enterprise virtualization management

Hyper-V (Microsoft)

• Integrated with Windows Server
• Hyper-V Replica for DR
• Nested virtualization support

Container on Bare Metal

Networking (On-Premise)

Core Concepts

• VLANs: Logical network segmentation (L2)
• Subnets: IP range allocation (L3)
• Routing: Static vs dynamic (OSPF, BGP)
• Switching: Access, trunk, L3 switches

Network Topology

Internet
  └── Firewall (Palo Alto, Fortinet, pfSense)
      └── Core Switch (L3)
          ├── DMZ VLAN
          ├── Production VLAN
          ├── Management VLAN
          └── Storage VLAN (iSCSI/NFS)

SD-WAN

• Replace MPLS with internet + overlay
• Cloud on-ramps for SaaS optimization
• Vendors: Cisco Meraki, Fortinet, VMware VeloCloud

Storage

SAN (Storage Area Network)

• Protocol: Fibre Channel (FC), iSCSI, FCoE
• Use case: Databases, high-performance shared storage
• Vendors: Dell EMC, NetApp, HPE, Pure Storage

NAS (Network Attached Storage)

• Protocol: NFS, SMB/CIFS
• Use case: File shares, home directories
• Vendors: NetApp, QNAP, Synology

Object Storage (On-Premise)

• MinIO: S3-compatible, cloud-native
• Ceph: Distributed, unified block/file/object
• Swift: OpenStack object storage

Storage Tiers

Hybrid Cloud Connectivity

VPN

• Site-to-site: IPsec VPN between on-prem and cloud
• Client VPN: Remote access for users
• Limitation: Bandwidth and latency constraints

Direct Connect / ExpressRoute / Cloud Interconnect

• Dedicated: Private fiber connection to cloud
• Benefits: Lower latency, higher bandwidth, predictable pricing
• Use case: Hybrid databases, replication, large data transfers

Hybrid Patterns

Physical Data Center

Design Considerations

• Power: UPS, generators, dual feeds, PDU redundancy
• Cooling: Hot/cold aisle containment, CRAC/CRAH units
• Racking: 42U-52U racks, weight limits, cable management
• Fire suppression: FM-200, inert gas (not water)
• Security: Biometric access, mantraps, CCTV

Tier Classification (Uptime Institute)

Hardware Lifecycle

Procurement to Decommission

1. Planning: Capacity forecast, budget cycle 2. Procurement: Vendor selection, lead times 3. Deployment: Racking, cabling, imaging 4. Operation: Monitoring, maintenance 5. Refresh: 3-5 year replacement cycle 6. Decommission: Data wipe, disposal, asset recovery

Asset Management

• Track: serial numbers, warranty, location, owner
• Tools: ServiceNow, Snipe-IT, GLPI
• Audit: Annual physical verification

Migration Strategies

Cloud Migration (7 Rs)

Database Migration

1. Assessment: Schema compatibility, feature gaps 2. Replication: Set up ongoing sync (DMS, Striim, Attunity) 3. Cutover: Brief downtime or blue-green switch 4. Validation: Data consistency checks, performance 5. Decommission: Stop replication, shut down source

Security & Compliance

Security Hardening

Compute Hardening

Linux (CIS Benchmarks):

• Disable root SSH login
• Enforce password complexity (PAM)
• Configure AIDE for file integrity monitoring
• Enable auditd for system call logging
• Apply security updates within 30 days
• Disable unnecessary services

Windows (CIS/Security Baselines):

• Enable Windows Defender + ASR rules
• Configure AppLocker / WDAC
• Disable SMBv1, LLMNR, NetBIOS
• Enforce Credential Guard
• Apply Microsoft Security Baselines via GPO

Containers:

• Run as non-root user
• Read-only root filesystem
• Drop unnecessary capabilities
• No privileged containers in production
• Scan images for CVEs (Trivy, Clair, Snyk)
• Distroless or minimal base images

Network Security

VPC Security:

• Default deny all inbound
• Explicit allow rules by service/port
• No 0.0.0.0/0 except for public-facing LB
• VPC Flow Logs for traffic analysis
• Private subnets for workloads

DDoS Protection:

• AWS Shield Advanced / Cloud Armor / Azure DDoS
• CDN for absorption
• Rate limiting at edge
• Anycast for distribution

Identity & Access Management

RBAC Design

# Kubernetes RBAC example
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-developer
  namespace: app
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "configmaps"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "update", "patch"]

Zero Trust Principles

1. Verify explicitly: Authenticate and authorize every access request 2. Least privilege: Minimum permissions for minimum time 3. Assume breach: Micro-segmentation, continuous monitoring

Secrets Management

Vault policy example:

path "database/creds/app" {
  capabilities = ["read"]
}

path "secret/data/app/*" {
  capabilities = ["read"]
  allowed_parameters = {
    "environment" = ["prod", "staging"]
  }
}

Vulnerability Management

Scanning Pipeline

Source Code → SAST (Semgrep, SonarQube)
Dependencies → SCA (Snyk, Dependabot)
Container Image → Image Scan (Trivy, Clair)
Infrastructure → IaC Scan (Checkov, tfsec)
Runtime → RASP / CSPM (Wiz, Orca)

Patch Management

Compliance Frameworks

SOC 2 Type II

Trust Services Criteria:

• CC1: Control environment
• CC2: Communication and information
• CC3: Risk assessment
• CC4: Monitoring activities
• CC5: Control activities
• CC6: Logical and physical access controls
• CC7: System operations
• CC8: Change management
• CC9: Risk mitigation

ISO 27001

Key domains:

• Information security policies
• Organization of information security
• Human resource security
• Asset management
• Access control
• Cryptography
• Physical security
• Operations security
• Communications security
• System acquisition and development
• Supplier relationships
• Incident management
• Business continuity
• Compliance

GDPR (if applicable)

• Data inventory and classification
• Privacy by design
• Consent management
• Right to erasure
• Data portability
• Breach notification (72 hours)
• DPO appointment (if required)

Encryption

At Rest

• Database: Transparent Data Encryption (TDE)
• Storage: AES-256-GCM (managed keys or customer-managed)
• Backups: Encrypted before leaving production
• Key rotation: Annual or on suspected compromise

In Transit

• TLS 1.3 minimum for all external traffic
• mTLS for service-to-service in zero-trust environments
• Certificate management: Auto-renewal, monitoring expiry

Key Management

• Use cloud KMS or HashiCorp Vault
• Separate keys by environment
• Envelope encryption: DEK encrypted by KEK
• Audit all key usage

Incident Response

Security Incident Severity

Response Playbook

1. Contain: Isolate affected systems 2. Eradicate: Remove threat actor access 3. Recover: Restore from clean backups 4. Post-incident: Forensics, lessons learned

Compliance Automation

Policy as Code

# Terraform Sentinel / OPA Rego
package terraform.aws.security

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_security_group"
  resource.change.after.ingress[_].cidr_blocks[_] == "0.0.0.0/0"
  msg := sprintf("Security group %s allows unrestricted ingress", [resource.name])
}

Continuous Compliance

• Terraform plan scanning (Checkov, tfsec, terrascan)
• K8s admission controllers (OPA Gatekeeper, Kyverno)
• Cloud posture management (Wiz, Orca, Prisma Cloud)
• Scheduled compliance audits ( ScoutSuite, Prowler, Steampipe)

Audit Evidence

• Maintain evidence repository (S3, SharePoint)
• Automate evidence collection where possible
• Review access logs quarterly
• Document exceptions with justification