Runbook Generator

Runbook Generator

Generate a comprehensive operational runbook.md from a system's codebase, infrastructure config, and deployment scripts -- written for on-call engineers working incidents, deployments, and routine operations at 3am.

Contents

• references/discovery-patterns.md -- Glob/Grep patterns and config files to inspect during discovery.
• references/output-template.md -- The full 9-section runbook.md structure to populate.
• references/writing-style.md -- Clarity, urgency, completeness, formatting, and accuracy rules plus output requirements.

Workflow

1. Discover the system. Run the Glob patterns in references/discovery-patterns.md (Step 1) to identify the stack and infrastructure. 2. Read key configuration. Open and analyze the config files listed in references/discovery-patterns.md (Step 2): IaC, containers, CI/CD, app config, deploy scripts, monitoring, migrations, load balancers, docs. 3. Identify operational behavior. Run the Grep patterns in references/discovery-patterns.md (Step 3) to surface health checks, metrics, caching, jobs, migrations, rollback, scaling, backups, TLS, secrets, alerting, and more. 4. Generate the runbook. Produce runbook.md following the structure in references/output-template.md. Cover all 9 sections; adapt content to what discovery found; never include purely speculative sections. 5. Apply writing standards. Follow every rule in references/writing-style.md -- copy-pasteable commands, expected outputs, sequential steps, P1-first ordering, tables for structured data, and the accuracy rules. 6. Verify and flag gaps. Confirm referenced paths and scripts exist. Mark unverifiable details with [VERIFY] and missing sections with [ACTION REQUIRED]: ... so the runbook doubles as a gap analysis.

Output

Write runbook.md to the project root (or a directory the user specifies). It must be 500+ lines, cover all 9 sections, contain real commands derived from the codebase, include at least one architecture diagram, carry a complete table of contents, and never fabricate infrastructure details. See references/writing-style.md for the full output checklist.

Discovery Patterns

Use these patterns during the discovery phase to map a system's structure, configuration, and operational behavior before generating a runbook.

Step 1: Glob patterns for project structure

Run these Glob patterns to identify what kind of system this is:

**/*.tf                    # Terraform infrastructure
**/*.yaml, **/*.yml        # Kubernetes manifests, CI/CD configs, docker-compose
**/Dockerfile*             # Container definitions
**/docker-compose*         # Multi-container orchestration
**/*.toml                  # Rust/Python config files
**/package.json            # Node.js projects
**/go.mod                  # Go projects
**/requirements.txt        # Python projects
**/Cargo.toml              # Rust projects
**/pom.xml                 # Java/Maven projects
**/build.gradle*           # Java/Gradle projects
**/Gemfile                 # Ruby projects
**/.github/workflows/*     # GitHub Actions CI/CD
**/.gitlab-ci.yml          # GitLab CI/CD
**/Jenkinsfile             # Jenkins pipelines
**/Makefile                # Build automation
**/Procfile                # Heroku-style process definitions
**/serverless.yml          # Serverless Framework
**/sam-template.yaml       # AWS SAM
**/cdk.json                # AWS CDK
**/pulumi.*                # Pulumi infrastructure
**/ansible/**              # Ansible playbooks
**/helm/**                 # Helm charts
**/.env.example            # Environment variable templates

Step 2: Key configuration files to read

Read and analyze these files when found:

• Infrastructure as Code: all Terraform files, CloudFormation templates, Pulumi programs, CDK constructs
• Container configs: Dockerfiles, docker-compose files, Kubernetes manifests
• CI/CD pipelines: GitHub Actions workflows, GitLab CI, Jenkinsfiles, CircleCI configs
• Application config: environment variable templates, config files, secrets references
• Deployment scripts: any scripts in scripts/, deploy/, bin/, or ops/ directories
• Monitoring config: Datadog, Prometheus, Grafana, PagerDuty, OpsGenie configurations
• Database migrations: migration files, schema definitions, seed data scripts
• Load balancer config: Nginx, HAProxy, ALB/NLB, Traefik configurations
• README and docs: existing documentation for context

Step 3: Grep patterns for operational behavior

Search the codebase for these operational patterns:

"healthcheck|health_check|health-check"    # Health endpoints
"readiness|liveness|startup"               # Kubernetes probes
"metric|prometheus|statsd|datadog"         # Metrics instrumentation
"sentry|bugsnag|rollbar|error.track"       # Error tracking
"redis|memcache|cache"                     # Caching layers
"queue|worker|job|sidekiq|celery|bull"     # Background job processing
"migrate|migration"                        # Database migrations
"rollback|revert"                          # Rollback mechanisms
"scale|autoscal|replica"                   # Scaling configuration
"backup|snapshot|dump"                     # Backup procedures
"ssl|tls|cert|certificate"                 # TLS/certificate management
"cron|schedule|periodic"                   # Scheduled tasks
"rate.limit|throttle"                      # Rate limiting
"circuit.break|retry|timeout"              # Resilience patterns
"log.level|LOG_LEVEL|debug|verbose"        # Log level configuration
"feature.flag|toggle|flipper|launchdarkly" # Feature flags
"cdn|cloudfront|fastly|cloudflare"         # CDN configuration
"dns|route53|domain"                       # DNS management
"secret|vault|ssm|kms"                     # Secrets management
"alert|alarm|notification|pagerduty"       # Alerting rules

Runbook Output Template

Generate runbook.md using the structure below. Adapt content to what discovery actually found -- do not include sections that are entirely speculative with no basis in the codebase. The runbook should be 500+ lines and cover all 9 major sections.

# [System Name] Operational Runbook

**Last Updated**: [date]
**Maintained By**: [team/owner from codebase]
**On-Call Rotation**: [link or description if found]
**Escalation Contact**: [if found in config]

---

## Table of Contents

[Auto-generated TOC with all sections]

---

## 1. System Overview

### 1.1 Purpose
[What this system does, derived from README and code analysis]

### 1.2 Architecture Diagram
[ASCII or Mermaid diagram showing components and data flow]

### 1.3 Service Inventory
| Service | Language/Runtime | Port | Purpose |
|---------|-----------------|------|---------|
[Populated from discovery]

### 1.4 Dependencies
#### Internal Dependencies
[Other internal services this system depends on]

#### External Dependencies
[Third-party services, APIs, databases]

### 1.5 Data Flow
[How data moves through the system, request lifecycle]

### 1.6 Environment Matrix
| Environment | URL/Endpoint | Cluster/Region | Notes |
|-------------|-------------|----------------|-------|
[Populated from config files]

---

## 2. Access and Authentication

### 2.1 Required Access
[Cloud provider accounts, VPN, SSH keys, kubectl contexts]

### 2.2 Service Accounts
[Service account details found in config]

### 2.3 Secrets Management
[How secrets are stored and rotated -- Vault, AWS SSM, etc.]

### 2.4 Common Access Commands
[kubectl config, AWS profile switching, VPN connection]

---

## 3. Common Operations

### 3.1 Deployment

#### Standard Deployment

Step-by-step deployment commands derived from CI/CD config

**Pre-deployment Checklist:**
- [ ] [Items derived from pipeline gates and checks]

**Post-deployment Verification:**
- [ ] [Health checks, smoke tests, metric verification]

#### Canary Deployment
[If canary/progressive deployment is configured]

#### Hotfix Deployment

Emergency deployment bypassing normal gates

### 3.2 Rollback

#### Automated Rollback

Commands to trigger automated rollback

#### Manual Rollback

Step-by-step manual rollback procedure

#### Database Rollback

How to revert database migrations

**Rollback Decision Matrix:**
| Symptom | Action | Rollback? |
|---------|--------|-----------|
[Common scenarios and whether to rollback]

### 3.3 Scaling

#### Horizontal Scaling

Commands to scale service instances

#### Vertical Scaling
[Procedure for increasing resource limits]

#### Auto-scaling Configuration
[Current auto-scaling rules and how to modify them]

#### Scaling Decision Guide
| Metric | Threshold | Action |
|--------|-----------|--------|
[CPU, memory, request rate thresholds]

### 3.4 Restart Procedures

#### Graceful Restart

Commands for graceful restart with zero downtime

#### Hard Restart

Commands for forced restart when graceful fails

#### Restart Individual Components
[Per-service restart commands]

### 3.5 Database Operations

#### Run Migrations

Migration commands

#### Connection Management

Check active connections, kill stuck queries

#### Emergency Read-Only Mode

How to switch to read-only if needed

### 3.6 Cache Operations

#### Cache Flush

Commands to flush cache safely

#### Cache Warmup

Commands to warm cache after flush

### 3.7 Log Management

#### Viewing Logs

Commands to tail/search logs per service

#### Log Level Changes

How to change log levels at runtime

#### Log Retention
[Current retention policies and how to retrieve archived logs]

### 3.8 Configuration Changes

#### Feature Flags
[How to toggle feature flags]

#### Environment Variable Updates
[Procedure for updating env vars without full redeploy]

#### Config Reload

Hot-reload config without restart if supported

---

## 4. Monitoring and Alerts

### 4.1 Dashboards
| Dashboard | URL | Purpose |
|-----------|-----|---------|
[Populated from monitoring config]

### 4.2 Key Metrics
| Metric | Normal Range | Warning | Critical |
|--------|-------------|---------|----------|
[Derived from alerting config and application metrics]

### 4.3 Health Checks
| Endpoint | Expected Response | Check Interval |
|----------|------------------|----------------|
[From health check configuration]

### 4.4 Alert Response Procedures

For each alert discovered in the codebase, provide:

#### ALERT: [Alert Name]
- **Severity**: P1/P2/P3/P4
- **Meaning**: What this alert indicates
- **Impact**: User-facing impact
- **Diagnosis**:
  1. [Step-by-step diagnosis commands]
- **Resolution**:
  1. [Step-by-step fix]
- **Escalation**: When and who to escalate to

---

## 5. Troubleshooting Guide

### 5.1 Symptom-Based Troubleshooting

For each common failure mode, provide a structured diagnosis flow:

#### Symptom: [Description]
**Possible Causes (check in order):**

1. **[Most likely cause]**
   - Diagnosis:

diagnostic command

   - Expected output: [what healthy looks like]
   - Fix:

fix command

2. **[Next likely cause]**
   - Diagnosis: ...
   - Fix: ...

3. **[Less common cause]**
   - Diagnosis: ...
   - Fix: ...

Common symptom categories to cover:
- High latency / slow responses
- 5xx errors / service unavailable
- Connection timeouts
- Memory pressure / OOM kills
- CPU saturation
- Disk space exhaustion
- Database connection pool exhaustion
- Queue backup / consumer lag
- Certificate expiration
- DNS resolution failures
- Authentication / authorization failures
- Data inconsistency
- Deployment failures
- Pod crash loops (Kubernetes)
- Network connectivity issues

### 5.2 Dependency Failure Modes
[What happens when each dependency fails and how to mitigate]

### 5.3 Known Issues and Workarounds
[Document any known issues found in code comments, TODOs, or issue trackers]

---

## 6. Escalation Procedures

### 6.1 Severity Definitions
| Severity | Definition | Response Time | Examples |
|----------|-----------|---------------|----------|
| P1 - Critical | Complete service outage | 15 min | [specific examples] |
| P2 - High | Major feature degraded | 30 min | [specific examples] |
| P3 - Medium | Minor feature impacted | 4 hours | [specific examples] |
| P4 - Low | Cosmetic / non-urgent | Next business day | [specific examples] |

### 6.2 Escalation Matrix
| Level | Who | When | Contact |
|-------|-----|------|---------|
[Derived from config or templated for completion]

### 6.3 Communication Templates

#### Internal Status Update

Subject: [P1/P2] [Service] - [Brief Description] Status: Investigating / Identified / Monitoring / Resolved Impact: [User-facing impact] Current Actions: [What is being done] Next Update: [Time of next update]

#### External Customer Communication

We are aware of an issue affecting [feature/service]. Our team is actively investigating. We will provide an update by [time].

### 6.4 Incident Management Process
1. **Detect**: Alert fires or user report received
2. **Triage**: Assess severity using definitions above
3. **Assemble**: Page appropriate responders
4. **Diagnose**: Use troubleshooting guide section 5
5. **Mitigate**: Apply fix or rollback
6. **Resolve**: Confirm service restoration
7. **Communicate**: Send resolution notice
8. **Review**: Schedule post-incident review within 48 hours

---

## 7. Disaster Recovery

### 7.1 Backup Inventory
| Data Store | Backup Method | Frequency | Retention | Location |
|-----------|--------------|-----------|-----------|----------|
[Derived from backup configuration]

### 7.2 Recovery Point Objective (RPO)
[Maximum acceptable data loss, derived from backup frequency]

### 7.3 Recovery Time Objective (RTO)
[Maximum acceptable downtime]

### 7.4 Recovery Procedures

#### Database Recovery

Step-by-step database restore from backup

#### Full Service Recovery

Steps to rebuild the entire service from scratch

#### Partial Recovery
[Procedures for recovering individual components]

### 7.5 Failover Procedures
[If multi-region or HA is configured]

#### Automatic Failover
[How automatic failover works and when it triggers]

#### Manual Failover

Commands to manually trigger failover

#### Failback

Commands to return to primary after failover

### 7.6 DR Testing Schedule
[Recommended DR test cadence and procedure]

---

## 8. Scheduled Maintenance

### 8.1 Recurring Tasks
| Task | Schedule | Procedure | Owner |
|------|----------|-----------|-------|
[Derived from cron jobs, scheduled tasks]

### 8.2 Certificate Rotation

Certificate renewal procedure

### 8.3 Secret Rotation

Secret rotation procedure

### 8.4 Dependency Updates
[Procedure for updating dependencies safely]

### 8.5 Capacity Review
[Monthly/quarterly capacity planning checklist]

---

## 9. Reference

### 9.1 Glossary
[System-specific terminology]

### 9.2 Architecture Decision Records
[Key architectural decisions that affect operations]

### 9.3 Related Runbooks
[Links to dependent service runbooks]

### 9.4 External Documentation
[Links to cloud provider docs, framework docs, vendor docs]

### 9.5 Change Log
| Date | Author | Change |
|------|--------|--------|
[Runbook revision history]

Writing Style and Output Requirements

Apply these rules strictly when writing the runbook.

Clarity

• Write for an engineer who has never seen this system before.
• Make every command copy-pasteable -- no placeholder values without clear labels.
• Use <PLACEHOLDER> format for values the engineer must fill in.
• Include expected output for diagnostic commands so engineers know what "healthy" looks like.
• Number all steps sequentially -- never use ambiguous ordering.

Urgency-Appropriate

• Put P1 procedures first in each section.
• Mark time-sensitive steps clearly: "MUST complete within 5 minutes".
• Separate "do this now" from "do this after incident".
• Include estimated time for each major procedure.

Completeness

• Include the full flags needed on every kubectl, aws, gcloud, docker, or CLI command.
• Include both the "happy path" and what to do when a step fails.
• Document prerequisites for each procedure (access, tools, permissions).
• Cross-reference related sections.

Formatting

• Use tables for structured data (metrics, thresholds, contacts).
• Use code blocks for all commands with language hints for syntax highlighting.
• Use bold for warnings and critical notes.
• Use checklists for multi-step procedures.
• Never use emojis anywhere in the document.
• Keep lines under 120 characters where possible.

Output Requirements

Generate the runbook as runbook.md in the project root directory (or the directory the user specifies). The file MUST:

1. Be 500+ lines. 2. Cover all 9 major sections from the template. 3. Contain actual commands and configuration derived from the codebase (not just generic placeholders). 4. Include at least one ASCII or Mermaid architecture diagram. 5. Have a complete table of contents. 6. Be immediately useful to an on-call engineer.

If the codebase lacks information for certain sections (for example, no monitoring config found), still include the section with a clear note: [ACTION REQUIRED]: No monitoring configuration found in codebase. Complete this section with your monitoring setup. This ensures the runbook serves as both documentation and a gap analysis.

Accuracy Rules

• Never fabricate infrastructure details -- only document what is verifiable from the codebase.
• When uncertain about a detail, mark it clearly with [VERIFY] so the team can confirm.
• Prefer specificity over generality -- a runbook with real commands is worth ten with generic advice.
• Confirm that referenced file paths and scripts actually exist in the codebase.
• If the system uses multiple environments (dev/staging/prod), document differences between them.
• Include version numbers for all tools and dependencies where visible in config files.