Runbooks Troubleshooting Guides

Runbooks - Troubleshooting Guides

Creating effective troubleshooting guides for diagnosing and resolving operational issues.

Troubleshooting Framework

The 5-Step Method

1. Observe - Gather symptoms and data 2. Hypothesize - Form theories about root cause 3. Test - Validate hypotheses with experiments 4. Fix - Apply solution 5. Verify - Confirm resolution

Basic Troubleshooting Guide

# Troubleshooting: [Problem Statement]

## Symptoms

What the user/system is experiencing:
- API returning 503 errors
- Response time > 10 seconds
- High CPU usage alerts

## Quick Checks (< 2 minutes)

### 1. Is the service running?

kubectl get pods -n production | grep api-server

**Expected:** STATUS = Running

### 2. Are recent deploys the cause?

kubectl rollout history deployment/api-server

**Check:** Did we deploy in the last 30 minutes?

### 3. Is this affecting all users?

Check error rate in Datadog:

- If < 5%: Isolated issue, may be client-specific
- If > 50%: Widespread issue, likely infrastructure

## Common Causes

| Symptom | Likely Cause | Quick Fix |
|---------|-------------|-----------|
| 503 errors | Pod crashlooping | Restart deployment |
| Slow responses | Database connection pool | Increase pool size |
| High memory | Memory leak | Restart pods |

## Detailed Diagnosis

### Hypothesis 1: Database Connection Issues

**Test:**

Check database connections

kubectl exec -it api-server-abc -- psql -h $DB_HOST -c "SELECT count(*) FROM pg_stat_activity"

**If connections > 90:** Pool is saturated.
**Next step:** Increase pool size or investigate slow queries.

### Hypothesis 2: High Traffic Spike

**Test:**

Check request rate

curl -H "Authorization: Bearer $DD_API_KEY" \ "https://api.datadoghq.com/api/v1/query?query=sum:nginx.requests{*}"

**If requests 3x normal:** Traffic spike.
**Next step:** Scale up pods or enable rate limiting.

### Hypothesis 3: External Service Degradation

**Test:**

Check third-party API

curl -w "@curl-format.txt" https://api.stripe.com/v1/charges

**If response time > 2s:** External service slow.
**Next step:** Implement circuit breaker or increase timeouts.

## Resolution Steps

### Solution A: Immediate (< 5 minutes)

Restart affected pods:

kubectl rollout restart deployment/api-server -n production

**When to use:** Quick mitigation while investigating root cause.

### Solution B: Short-term (< 30 minutes)

Scale up resources:

kubectl scale deployment/api-server --replicas=10 -n production

**When to use:** Traffic spike or resource exhaustion.

### Solution C: Long-term (< 2 hours)

Fix root cause:

1. Identify slow database query
2. Add database index
3. Deploy code optimization

**When to use:** After immediate pressure is relieved.

## Validation

- [ ] Error rate < 1%
- [ ] Response time p95 < 200ms
- [ ] CPU usage < 70%
- [ ] No active alerts

## Prevention

How to prevent this issue in the future:

- Add monitoring alert for connection pool saturation
- Implement auto-scaling based on request rate
- Set up load testing to find capacity limits

Decision Tree Format

# Troubleshooting: Slow API Responses

## Start Here

Check response time | ┌──────────────┴──────────────┐ │ │ < 500ms > 500ms │ │ NOT THIS RUNBOOK Continue below

## Step 1: Locate the Slowness

Check which service is slow

curl -w "@timing.txt" https://api.example.com/users

**Decision:**

- Time to first byte > 2s → Database slow (go to Step 2)
- Time to first byte < 100ms → Network slow (go to Step 3)
- Timeout → Service down (go to Step 4)

## Step 2: Database Diagnosis

Check active queries

psql -c "SELECT query, state, query_start FROM pg_stat_activity WHERE state != 'idle'"

**Decision:**

- Query running > 5s → Slow query (Solution A)
- Many idle in transaction → Connection leak (Solution B)
- High connection count → Pool exhausted (Solution C)

### Solution A: Optimize Slow Query

1. Identify slow query from above
2. Run EXPLAIN ANALYZE
3. Add missing index or optimize query

### Solution B: Fix Connection Leak

1. Restart application pods
2. Review code for unclosed connections
3. Add connection timeout

### Solution C: Increase Connection Pool

1. Edit database config
2. Increase max_connections
3. Update application pool size

## Step 3: Network Diagnosis

... (continue with network troubleshooting)

Layered Troubleshooting

Layer 1: Application

## Application Layer Issues

### Check Application Health

1. **Health endpoint:**

curl https://api.example.com/health

1. **Application logs:**

kubectl logs deployment/api-server --tail=100 | grep ERROR

2. **Application metrics:**
   - Request rate
   - Error rate
   - Response time percentiles

### Common Application Issues

**Memory Leak**

- **Symptom:** Memory usage climbing over time
- **Test:** Check memory metrics in Datadog
- **Fix:** Restart pods, investigate with heap dump

**Thread Starvation**

- **Symptom:** Slow responses, high CPU
- **Test:** Thread dump analysis
- **Fix:** Increase thread pool size

**Code Bug**

- **Symptom:** Specific endpoints fail
- **Test:** Review recent deploys
- **Fix:** Rollback or hotfix

Layer 2: Infrastructure

## Infrastructure Layer Issues

### Check Infrastructure Health

1. **Node resources:**

kubectl top nodes

1. **Pod resources:**

kubectl top pods -n production

2. **Network connectivity:**

kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- ping database.internal

### Common Infrastructure Issues

**Node Under Pressure**

- **Symptom:** Pods evicted, slow scheduling
- **Test:** `kubectl describe node` for pressure conditions
- **Fix:** Scale node pool or add nodes

**Network Partition**

- **Symptom:** Intermittent timeouts
- **Test:** MTR between pods and destination
- **Fix:** Check security groups, routing tables

**Disk I/O Saturation**

- **Symptom:** Slow database, high latency
- **Test:** Check IOPS metrics in CloudWatch
- **Fix:** Increase provisioned IOPS

Layer 3: External Dependencies

## External Dependencies Issues

### Check External Services

1. **Third-party APIs:**

curl -w "@timing.txt" https://api.stripe.com/health

1. **Status pages:**
   - Check status.stripe.com
   - Check status.aws.amazon.com

2. **DNS resolution:**

nslookup api.stripe.com dig api.stripe.com

### Common External Issues

**API Rate Limiting**

- **Symptom:** 429 responses from external service
- **Test:** Check rate limit headers
- **Fix:** Implement backoff, cache responses

**Service Degradation**

- **Symptom:** Slow external API responses
- **Test:** Check their status page
- **Fix:** Implement circuit breaker, use fallback

**DNS Failure**

- **Symptom:** Cannot resolve hostname
- **Test:** DNS queries
- **Fix:** Check DNS config, try alternative resolver

Systematic Debugging

Use the Scientific Method

# Debugging: Database Connection Failures

## 1. Observation

**What we know:**
- Error: "connection refused" in logs
- Started: 2025-01-15 14:30 UTC
- Frequency: Every database query fails
- Scope: All pods affected

## 2. Hypothesis

**Possible causes:**
1. Database instance is down
2. Security group blocking traffic
3. Network partition
4. Wrong credentials

## 3. Test Each Hypothesis

### Test 1: Database instance status

aws rds describe-db-instances --db-instance-identifier prod-db | jq '.DBInstances[0].DBInstanceStatus'

**Result:** "available"
**Conclusion:** Database is running ✗ Hypothesis 1 rejected

### Test 2: Security group rules

aws ec2 describe-security-groups --group-ids sg-abc123 | jq '.SecurityGroups[0].IpPermissions'

**Result:** Port 5432 open only to 10.0.0.0/16
**Pod IP:** 10.1.0.5
**Conclusion:** Pod IP not in allowed range ✓ **ROOT CAUSE FOUND**

## 4. Fix

Update security group:

aws ec2 authorize-security-group-ingress \ --group-id sg-abc123 \ --protocol tcp \ --port 5432 \ --cidr 10.1.0.0/16

## 5. Verify

Test connection from pod:

kubectl exec -it api-server-abc -- psql -h prod-db.rds.amazonaws.com -c "SELECT 1"

**Result:** Success ✓

Time-Boxed Investigation

# Troubleshooting: Production Outage

**Time Box:** Spend MAX 15 minutes investigating before escalating.

## First 5 Minutes: Quick Wins

- [ ] Check pod status
- [ ] Check recent deploys
- [ ] Check external status pages
- [ ] Review monitoring dashboards

**If issue persists:** Continue to next phase.

## Minutes 5-10: Common Causes

- [ ] Restart pods (quick mitigation)
- [ ] Check database connectivity
- [ ] Review application logs
- [ ] Check resource limits

**If issue persists:** Continue to next phase.

## Minutes 10-15: Deep Dive

- [ ] Enable debug logging
- [ ] Capture thread dump
- [ ] Check for memory leaks
- [ ] Review network traces

**If issue persists:** ESCALATE to senior engineer.

## Escalation

**Escalate to:** Platform Team Lead
**Provide:**
- Timeline of issue
- Tests performed
- Current error rate
- Mitigation attempts

Common Troubleshooting Patterns

Binary Search

## Finding Which Service is Slow

Using binary search to narrow down the problem:

1. **Check full request:** 5000ms total
2. **Check first half (API → Database):** 4900ms
   → Problem is in database query
3. **Check database:** Query takes 4800ms
4. **Check query plan:** Sequential scan on large table
5. **Root cause:** Missing index

**Fix:** Add index on frequently queried column.

Correlation Analysis

## Finding Related Events

Look for patterns and correlations:

**Timeline:**
- 14:25 - Deploy completed
- 14:30 - Error rate spike
- 14:35 - Database CPU at 100%
- 14:40 - Requests timing out

**Correlation:** Deploy introduced N+1 query.

**Evidence:**
- No config changes
- No infrastructure changes
- Only code deploy
- Error coincides with deploy

**Action:** Rollback deploy.

Anti-Patterns

Don't Skip Obvious Checks

# Bad: Jump to complex solutions
## Database Slow

Must be a query optimization issue. Let's analyze query plans...

# Good: Check basics first
## Database Slow

1. Is the database actually running?
2. Can we connect to it?
3. Are there any locks?
4. What does the slow query log show?

Don't Guess Randomly

# Bad: Random changes
## API Errors

Let's try:
- Restarting the database
- Scaling to 100 pods
- Changing the load balancer config
- Updating the kernel

# Good: Systematic approach
## API Errors

1. What is the actual error message?
2. When did it start?
3. What changed before it started?
4. Can we reproduce it?

Don't Skip Documentation

# Bad: No notes
## Fixed It

I restarted some pods and now it works.

# Good: Document findings
## Resolution

**Root Cause:** Memory leak in worker process
**Evidence:** Pod memory climbing linearly over 6 hours
**Temporary Fix:** Restarted pods
**Long-term Fix:** PR #1234 fixes memory leak
**Prevention:** Added memory usage alerts

Related Skills

• runbook-structure: Organizing operational documentation
• incident-response: Handling production incidents