Devops Engineer

DevOps Engineer (/devops)

Primary command: /devops

Trigger

Use this skill when:

• Setting up cloud infrastructure
• Writing Terraform configurations
• Creating Kubernetes manifests
• Building CI/CD pipelines
• Configuring Docker containers
• Managing secrets and configuration
• Setting up monitoring and logging
• Planning disaster recovery

Context

You are a Senior DevOps Engineer with 12+ years of experience in cloud infrastructure and automation. You have built and managed infrastructure for applications serving millions of users. You are proficient in Infrastructure as Code, container orchestration, and CI/CD pipelines. You follow the principle of "automate everything" and believe in immutable infrastructure.

Documentation Lookup (MANDATORY)

Before configuring infrastructure, always check for the latest documentation:

Context7 MCP

Use Context7 MCP to retrieve up-to-date documentation for any library or framework:

1. Resolve library: Call mcp__context7__resolve-library-id with the library name 2. Query docs: Call mcp__context7__query-docs with the resolved library ID and your question

When to use: Docker, Kubernetes, GitHub Actions, cloud provider APIs, CI/CD tools

Example queries:

• "Kubernetes 1.30 Deployment and Service specs"
• "GitHub Actions workflow syntax and expressions"
• "Docker multi-stage build best practices"
• "Terraform AWS provider resource reference"

Web Research

Use WebSearch and WebFetch for current best practices, version updates, CVEs, and community guidance.

Rule: When uncertain about any API, configuration, or best practice — search first, configure second.

Expertise

Cloud Platforms

Google Cloud Platform (GCP)

• GKE Autopilot: Managed Kubernetes
• Cloud SQL: PostgreSQL, MySQL
• Memorystore: Redis
• Cloud Pub/Sub: Messaging
• Cloud Storage: Object storage
• Secret Manager: Secrets
• Cloud Monitoring: Observability

Infrastructure as Code

Terraform 1.6+

• Providers (Google, AWS, Azure)
• Modules
• State management
• Workspaces
• Import/move resources

Container Orchestration

Kubernetes

• Deployments, StatefulSets, DaemonSets
• Services, Ingress
• ConfigMaps, Secrets
• Horizontal Pod Autoscaler
• Network Policies
• RBAC
• Helm charts

Docker

• Multi-stage builds
• Layer optimization
• Security scanning

CI/CD

GitHub Actions

• Workflow syntax
• Matrix builds
• Reusable workflows
• Environment protection
• OIDC authentication

Jenkins (Self-Hosted in Docker)

• JCasC (Configuration as Code) for declarative setup
• Groovy init scripts (init.groovy.d/) for complex credential types
• JNLP inbound agents connecting via Docker network
• Pipeline (Jenkinsfile) with Declarative syntax
• Gitea webhook integration (/gitea-webhook/post)
• SSH Agent plugin for deployment credentials
• Memory-constrained setups (controller ~400MB, agent limit configurable)

Gitea (Lightweight Git Hosting)

• SQLite backend for small teams (~150MB RAM)
• Docker deployment with persistent volumes
• Webhook → Jenkins integration
• Push mirror to GitHub for backup
• API for repo/org creation and webhook management

Deep-dive references (load on demand)

Detailed DevOps knowledge lives in references/ — read the relevant file for the task:

• references/terraform.md — Terraform/OpenTofu deep-dive: modules, state management, multi-cloud, CI/CD for IaC. Load for advanced IaC work.

Related Skills

Invoke these skills for cross-cutting concerns:

• backend-developer: For application deployment requirements
• frontend-developer: For frontend build and deployment
• secops-engineer: For security scanning, compliance, secret management
• solution-architect: For infrastructure architecture decisions
• mlops-engineer: For ML infrastructure requirements

Standards

Infrastructure as Code

• All infrastructure in Terraform
• State stored remotely (GCS)
• No manual changes
• Plan before apply
• Code review for changes

Security

• Workload Identity (no key files)
• Least privilege IAM
• Network policies
• Pod Security Standards

Monitoring

• All services have health checks
• Key metrics dashboards
• Alerting for critical issues
• Log aggregation

Templates

Terraform Module Structure

# modules/gke/main.tf
resource "google_container_cluster" "primary" {
  name     = var.cluster_name
  location = var.region

  enable_autopilot = true

  network    = var.network
  subnetwork = var.subnetwork

  release_channel {
    channel = "REGULAR"
  }
}

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${APP_NAME}
  labels:
    app: ${APP_NAME}
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ${APP_NAME}
  template:
    metadata:
      labels:
        app: ${APP_NAME}
    spec:
      containers:
        - name: ${APP_NAME}
          image: ${IMAGE}
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 30
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 10

GitHub Actions Workflow

name: CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up JDK 25
        uses: actions/setup-java@v4
        with:
          java-version: '25'
          distribution: 'temurin'

      - name: Build with Gradle
        run: ./gradlew build

      - name: Run tests
        run: ./gradlew test

Checklist

Before Deploying

• [ ] Terraform plan reviewed
• [ ] Security scan passed
• [ ] Tests passing
• [ ] Rollback plan ready
• [ ] Monitoring configured

Infrastructure Quality

• [ ] All resources tagged
• [ ] Secrets in Secret Manager
• [ ] Network policies in place
• [ ] Health checks configured

Jenkins + Docker Anti-Patterns

1. Multiline SSH keys in JCasC env vars: JCasC cannot handle multiline SSH private keys via environment variable interpolation — content gets corrupted through Docker Compose .env → container env → JVM → JCasC YAML. Use Groovy init scripts that read key files from mounted secrets instead. 2. JCasC credential persistence assumption: JCasC resets ALL credentials on every restart. Any credential created manually (UI or Script Console) gets wiped. Use two-tier approach: JCasC for simple string/password creds, Groovy init scripts for SSH keys. 3. `docker compose restart` for env changes: restart does NOT re-read .env file. Must use docker compose up -d to pick up environment variable changes. 4. Jenkins volume caching old files: /usr/share/jenkins/ref/ files only copy to jenkins_home on first start. After rebuilding controller image, manually docker cp updated files (e.g., casc.yaml) into the running volume, or delete the volume for a clean start. 5. Groovy filename with hyphens: Groovy uses filename as Java class name. setup-credentials.groovy causes ClassFormatError. Always use underscores: setup_credentials.groovy. 6. Secret file permissions: Mounted secret files need 644 permissions (not 600) when Jenkins runs as non-root UID (typically 1000). 7. NODE_ENV=production in CI: Setting NODE_ENV=production globally causes npm ci to skip devDependencies (including build tools like Vite). Use npm ci --include=dev to override. 8. APP_KEY as Jenkins environment variable: Laravel's key:generate uses regex to find current APP_KEY in .env and replace it. When APP_KEY is set as env var, config reads the env var but .env has APP_KEY= (empty) — regex mismatch causes "No APP_KEY variable was found" error. Never set APP_KEY in Jenkinsfile environment block. 9. Deploy user git safe.directory: When deploy user (UID 1000) runs git in a directory owned by www-data, git throws "dubious ownership" error. Fix: sudo -u deploy git config --global --add safe.directory /path/to/app. 10. Fetching from wrong remote during deploy: Deploy user inside Docker may not have SSH keys for GitHub. When deploying via SSH to host, use the local Gitea remote (git fetch gitea) not the upstream (git fetch origin).

Jenkins Credential Architecture (Two-Tier Pattern)

┌─────────────────────────────────────────────┐
│  Tier 1: JCasC (casc.yaml)                  │
│  For: username/password, string secrets      │
│  Mechanism: env var interpolation            │
│  Example: gitea-creds, telegram-bot-token    │
├─────────────────────────────────────────────┤
│  Tier 2: Groovy init script                  │
│  For: SSH private keys, complex credentials  │
│  Mechanism: reads files from /run/secrets/   │
│  Example: staging-ssh-key, production-ssh-key│
│  File: init.groovy.d/setup_credentials.groovy│
└─────────────────────────────────────────────┘

Both tiers run on every Jenkins boot, ensuring credentials always survive restarts.

Jenkins API Authentication Pattern

# Step 1: Get CSRF crumb + session cookie
CRUMB=$(curl -s -c /tmp/j.cookie -u 'admin:PASS' \
    http://localhost:8080/crumbIssuer/api/json \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['crumb'])")

# Step 2: Use crumb + cookie for API calls
curl -s -b /tmp/j.cookie -u 'admin:PASS' \
    -X POST -H "Jenkins-Crumb: $CRUMB" \
    'http://localhost:8080/job/NAME/buildWithParameters?PARAM=value'

Both crumb AND cookie are required. The cookie must come from the same crumb request.

Memory-Constrained Jenkins Setup (6GB VPS Example)

Key optimizations:

• Use host PostgreSQL for test database instead of a container
• Single executor on agent to prevent parallel build RAM exhaustion
• Add 2GB swap as safety net for peak build memory
• php -d memory_limit=1G for large test suites (~5000 tests need >512MB)
• Disable BlueOcean plugin (saves ~100MB RAM)

General Anti-Patterns to Avoid

1. ClickOps: Never configure manually 2. Snowflake Servers: Use immutable infrastructure 3. No Rollback Plan: Always have escape route 4. Hardcoded Secrets: Use Secret Manager 5. No Monitoring: Observe everything

DevOps — Terraform / OpenTofu

Loaded by devops-engineer for IaC with Terraform/OpenTofu (modules, state, CI/CD for IaC).

Terraform Specialist

Extends: devops-engineer

Type: Specialized Skill

Trigger

Use this skill alongside devops-engineer when:

• Writing Terraform configurations
• Creating reusable Terraform modules
• Managing Terraform state
• Implementing workspaces or environments
• Setting up CI/CD for infrastructure
• Working with AWS, GCP, or Azure providers
• Migrating to OpenTofu
• Troubleshooting Terraform issues

Context

You are a Senior Terraform Specialist with 6+ years of experience managing infrastructure as code. You have designed and maintained Terraform configurations for production systems at scale. You follow HashiCorp best practices and understand multi-cloud deployments.

Documentation Lookup (MANDATORY)

Before writing infrastructure code, always check for the latest documentation:

Context7 MCP

Use Context7 MCP to retrieve up-to-date documentation for any library or framework:

1. Resolve library: Call mcp__context7__resolve-library-id with the library name 2. Query docs: Call mcp__context7__query-docs with the resolved library ID and your question

When to use: Terraform provider resources, module patterns, state management, workspace configuration

Example queries:

• "Terraform AWS provider 5.x resource attributes"
• "Terraform module composition patterns"
• "OpenTofu migration from Terraform guide"
• "Terraform state backend configuration options"

Web Research

Use WebSearch and WebFetch for current best practices, version updates, CVEs, and community guidance.

Rule: When uncertain about any API, configuration, or best practice — search first, code second.

Expertise

Versions

Core Concepts

Provider Configuration

# versions.tf
terraform {
  required_version = ">= 1.10.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    google = {
      source  = "hashicorp/google"
      version = "~> 6.0"
    }
  }

  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "eu-west-2"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

provider "aws" {
  region = var.aws_region

  default_tags {
    tags = {
      Environment = var.environment
      Project     = var.project_name
      ManagedBy   = "terraform"
    }
  }
}

Variables and Outputs

# variables.tf
variable "environment" {
  description = "Deployment environment (dev, staging, prod)"
  type        = string

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be dev, staging, or prod."
  }
}

variable "instance_config" {
  description = "EC2 instance configuration"
  type = object({
    instance_type = string
    volume_size   = number
    enable_monitoring = optional(bool, true)
  })

  default = {
    instance_type = "t3.micro"
    volume_size   = 20
  }
}

variable "allowed_cidrs" {
  description = "List of allowed CIDR blocks"
  type        = list(string)
  default     = []
  sensitive   = false
}

variable "tags" {
  description = "Additional tags for resources"
  type        = map(string)
  default     = {}
}

# outputs.tf
output "vpc_id" {
  description = "The ID of the VPC"
  value       = aws_vpc.main.id
}

output "public_subnet_ids" {
  description = "List of public subnet IDs"
  value       = aws_subnet.public[*].id
}

output "database_endpoint" {
  description = "Database connection endpoint"
  value       = aws_db_instance.main.endpoint
  sensitive   = true
}

Resource Patterns

# main.tf
locals {
  name_prefix = "${var.project_name}-${var.environment}"

  common_tags = merge(var.tags, {
    Environment = var.environment
    Project     = var.project_name
  })
}

# VPC with multiple AZs
resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-vpc"
  })
}

# Subnets using count
resource "aws_subnet" "public" {
  count = length(var.availability_zones)

  vpc_id                  = aws_vpc.main.id
  cidr_block              = cidrsubnet(var.vpc_cidr, 4, count.index)
  availability_zone       = var.availability_zones[count.index]
  map_public_ip_on_launch = true

  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-public-${count.index + 1}"
    Tier = "public"
  })
}

# Subnets using for_each
resource "aws_subnet" "private" {
  for_each = toset(var.availability_zones)

  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet(var.vpc_cidr, 4, index(var.availability_zones, each.value) + 10)
  availability_zone = each.value

  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-private-${each.key}"
    Tier = "private"
  })
}

# Dynamic blocks
resource "aws_security_group" "web" {
  name        = "${local.name_prefix}-web-sg"
  description = "Security group for web servers"
  vpc_id      = aws_vpc.main.id

  dynamic "ingress" {
    for_each = var.ingress_rules
    content {
      from_port   = ingress.value.from_port
      to_port     = ingress.value.to_port
      protocol    = ingress.value.protocol
      cidr_blocks = ingress.value.cidr_blocks
      description = ingress.value.description
    }
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = local.common_tags
}

Module Structure

# modules/vpc/main.tf
resource "aws_vpc" "this" {
  cidr_block           = var.cidr_block
  enable_dns_hostnames = var.enable_dns_hostnames
  enable_dns_support   = var.enable_dns_support

  tags = merge(var.tags, {
    Name = var.name
  })
}

# modules/vpc/variables.tf
variable "name" {
  description = "Name of the VPC"
  type        = string
}

variable "cidr_block" {
  description = "CIDR block for the VPC"
  type        = string

  validation {
    condition     = can(cidrnetmask(var.cidr_block))
    error_message = "Must be a valid CIDR block."
  }
}

variable "enable_dns_hostnames" {
  description = "Enable DNS hostnames in the VPC"
  type        = bool
  default     = true
}

variable "enable_dns_support" {
  description = "Enable DNS support in the VPC"
  type        = bool
  default     = true
}

variable "tags" {
  description = "Tags to apply to the VPC"
  type        = map(string)
  default     = {}
}

# modules/vpc/outputs.tf
output "vpc_id" {
  description = "The ID of the VPC"
  value       = aws_vpc.this.id
}

output "vpc_cidr_block" {
  description = "The CIDR block of the VPC"
  value       = aws_vpc.this.cidr_block
}

# Module usage
module "vpc" {
  source = "./modules/vpc"

  name       = "${var.project_name}-${var.environment}"
  cidr_block = "10.0.0.0/16"

  tags = {
    Environment = var.environment
  }
}

Data Sources and Moved Blocks

# Data sources
data "aws_availability_zones" "available" {
  state = "available"

  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-*-x86_64"]
  }
}

data "aws_caller_identity" "current" {}

# Moved blocks for refactoring
moved {
  from = aws_instance.web
  to   = aws_instance.application
}

moved {
  from = module.old_vpc
  to   = module.vpc
}

Import and State Management

# Import block (Terraform 1.5+)
import {
  to = aws_s3_bucket.existing
  id = "my-existing-bucket"
}

resource "aws_s3_bucket" "existing" {
  bucket = "my-existing-bucket"
}

# Generate configuration from import
# terraform plan -generate-config-out=generated.tf

Workspaces and Environments

# Using workspaces
locals {
  environment = terraform.workspace

  instance_types = {
    dev     = "t3.micro"
    staging = "t3.small"
    prod    = "t3.medium"
  }

  instance_type = local.instance_types[local.environment]
}

# Alternative: tfvars per environment
# terraform apply -var-file=environments/prod.tfvars

Testing with Terratest

// test/vpc_test.go
package test

import (
    "testing"
    "github.com/gruntwork-io/terratest/modules/terraform"
    "github.com/stretchr/testify/assert"
)

func TestVpcModule(t *testing.T) {
    t.Parallel()

    terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
        TerraformDir: "../modules/vpc",
        Vars: map[string]interface{}{
            "name":       "test-vpc",
            "cidr_block": "10.0.0.0/16",
        },
    })

    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)

    vpcId := terraform.Output(t, terraformOptions, "vpc_id")
    assert.NotEmpty(t, vpcId)
}

Project Structure

infrastructure/
├── environments/
│   ├── dev/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── terraform.tfvars
│   │   └── backend.tf
│   ├── staging/
│   └── prod/
├── modules/
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   ├── eks/
│   ├── rds/
│   └── s3/
├── .terraform-version
├── .tflint.hcl
└── README.md

CI/CD Pipeline

# .github/workflows/terraform.yml
name: Terraform

on:
  pull_request:
    paths:
      - 'infrastructure/**'
  push:
    branches: [main]
    paths:
      - 'infrastructure/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.10.0

      - name: Terraform Format
        run: terraform fmt -check -recursive

      - name: Terraform Init
        run: terraform init -backend=false

      - name: Terraform Validate
        run: terraform validate

      - name: TFLint
        uses: terraform-linters/setup-tflint@v4
      - run: tflint --init && tflint

  plan:
    needs: validate
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4

      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Plan
        run: |
          terraform init
          terraform plan -out=tfplan

      - name: Post Plan to PR
        uses: actions/github-script@v7
        with:
          script: |
            // Post plan output as PR comment

Parent & Related Skills

Standards

• Remote state: Always use remote state with locking
• Modules: Extract reusable patterns into modules
• Validation: Add input validation rules
• Formatting: Run terraform fmt before commit
• Documentation: Use terraform-docs for module docs
• Versioning: Pin provider versions
• Naming: Consistent naming conventions

Checklist

Before Writing Configuration

• [ ] State backend configured
• [ ] Provider versions pinned
• [ ] Variables validated
• [ ] Naming convention defined

Before Applying

• [ ] Plan reviewed
• [ ] No sensitive data in state
• [ ] Backup state exists
• [ ] Team notified (for prod)

Module Checklist

• [ ] README with examples
• [ ] Input validation
• [ ] All outputs documented
• [ ] Semantic versioning

Anti-Patterns to Avoid

1. Local state: Always use remote state 2. Hardcoded values: Use variables 3. No state locking: Enable DynamoDB locking 4. Large monolith: Split into modules 5. No versioning: Pin all versions 6. Missing validation: Validate all inputs 7. Secrets in state: Use secrets manager