
Incident Response
- 107 installs
- 44 repo stars
- Updated May 22, 2026
- bagelhole/devops-security-agent-skills
incident-response is a Claude Code skill for handling security incidents with IR playbooks covering detection, containment, eradication, recovery, and forensic evidence collection.
About
A Claude skill for handling security incidents with structured response procedures. A security engineer uses it to build IR playbooks, run detection, containment, eradication, and recovery, and collect forensic evidence during a breach or intrusion. It includes ready-to-run Linux evidence-collection scripts and severity classification.
- Structures security incident response across detection, containment, eradication, and recovery
- Ships Linux forensic evidence-collection scripts with chain-of-custody logging
- Provides IR playbooks, severity classification, and tabletop exercise guidance
Incident Response by the numbers
- 107 all-time installs (skills.sh)
- Ranked #990 of 2,203 Security skills by installs in the Skillselion catalog
- Data as of Jul 28, 2026 (Skillselion catalog sync)
incident-response capabilities & compatibility
- Capabilities
- security audit · debugging
- Works with
- docker · kubernetes
- Use cases
- security audit
- Platforms
- Linux
- Pricing
- Free
What incident-response says it does
Handle security incidents with IR playbooks and procedures. Implement detection, containment, eradication, and recovery processes.
linux-evidence-collect.sh - Collect forensic evidence from a Linux host
npx skills add https://github.com/bagelhole/devops-security-agent-skills --skill incident-responseAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 107 |
|---|---|
| repo stars | ★ 44 |
| Last updated | May 22, 2026 |
| Repository | bagelhole/devops-security-agent-skills ↗ |
What it does
Respond to security breaches with structured IR playbooks and forensic evidence collection.
Who is it for?
Security teams responding to breaches, malware, or unauthorized access.
Skip if: General production outage management (use incident-management instead).
When should I use this skill?
Responding to a security breach, building IR playbooks, or collecting forensic evidence.
What you get
A structured IR capability with playbooks, severity classification, and forensic evidence collection.
- IR playbooks and runbooks
- Forensic evidence-collection scripts
- Severity classification and post-incident review
By the numbers
- Defines 6 IR phases (preparation through lessons learned)
- Uses 4 severity levels (P1 critical through P4 low)
Files
Incident Response
Handle security incidents effectively with structured response procedures.
When to Use This Skill
Use this skill when:
- Responding to an active security incident (breach, malware, unauthorized access)
- Building incident response playbooks and runbooks
- Conducting IR tabletop exercises and drills
- Setting up evidence collection and forensic capabilities
- Establishing communication protocols for security events
- Performing post-incident reviews and process improvements
Prerequisites
- IR team roster with on-call rotation and escalation paths
- Secure communication channel (separate from production systems)
- Forensic workstation with analysis tools installed
- Evidence storage with chain-of-custody controls
- Legal counsel contact information
- Pre-authorized incident response actions documented
Incident Response Phases
phases:
1_preparation:
- IR team roster and 24/7 contact info
- Tools and privileged access ready
- Playbooks documented and tested
- Evidence collection kit prepared
- Communication templates drafted
2_detection:
- Alert triage and validation
- Initial assessment and scoping
- Severity classification
- Incident ticket creation
3_containment:
- Short-term containment (stop bleeding)
- Evidence preservation (before changes)
- System isolation (network/host level)
- Credential rotation if needed
4_eradication:
- Root cause analysis
- Remove threat actor access
- Patch exploited vulnerabilities
- Clean compromised systems
5_recovery:
- System restoration from clean backups
- Enhanced monitoring deployment
- Phased return to production
- Business continuity verification
6_lessons_learned:
- Post-incident review (within 72 hours)
- Timeline reconstruction
- Documentation update
- Process and detection improvementsSeverity Classification
| Level | Impact | Response Time | Examples |
|---|---|---|---|
| Critical (P1) | Active data breach, full outage, ransomware | Immediate (< 15 min) | Data exfiltration in progress, ransomware spreading |
| High (P2) | Service degraded, potential breach | < 1 hour | Unauthorized admin access, malware detected |
| Medium (P3) | Limited impact, contained | < 4 hours | Phishing compromise (single user), policy violation |
| Low (P4) | Minimal impact | Next business day | Failed brute force, blocked scanning activity |
Evidence Collection Scripts
Linux Evidence Collection
#!/bin/bash
# linux-evidence-collect.sh - Collect forensic evidence from a Linux host
# Run with sudo. Preserves evidence with timestamps and hashes.
set -euo pipefail
EVIDENCE_DIR="/evidence/$(hostname)-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$EVIDENCE_DIR"
LOGFILE="$EVIDENCE_DIR/collection.log"
log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" | tee -a "$LOGFILE"; }
log "Starting evidence collection on $(hostname)"
log "Collector: $(whoami)"
log "System time: $(date -u)"
# System information
log "Collecting system information..."
uname -a > "$EVIDENCE_DIR/uname.txt"
cat /etc/os-release > "$EVIDENCE_DIR/os-release.txt"
uptime > "$EVIDENCE_DIR/uptime.txt"
date -u > "$EVIDENCE_DIR/system-time.txt"
# Running processes (full command line)
log "Collecting process list..."
ps auxwwf > "$EVIDENCE_DIR/processes.txt"
ps -eo pid,ppid,user,args --sort=-pcpu > "$EVIDENCE_DIR/processes-by-cpu.txt"
# Network connections
log "Collecting network state..."
ss -tulnp > "$EVIDENCE_DIR/listening-ports.txt"
ss -anp > "$EVIDENCE_DIR/all-connections.txt"
ip addr show > "$EVIDENCE_DIR/ip-addresses.txt"
ip route show > "$EVIDENCE_DIR/routes.txt"
iptables -L -n -v > "$EVIDENCE_DIR/iptables.txt" 2>&1 || true
cat /etc/resolv.conf > "$EVIDENCE_DIR/dns-config.txt"
# User activity
log "Collecting user activity..."
last -a > "$EVIDENCE_DIR/login-history.txt"
lastb > "$EVIDENCE_DIR/failed-logins.txt" 2>&1 || true
who > "$EVIDENCE_DIR/currently-logged-in.txt"
w > "$EVIDENCE_DIR/user-activity.txt"
cat /etc/passwd > "$EVIDENCE_DIR/passwd.txt"
cat /etc/shadow > "$EVIDENCE_DIR/shadow.txt" 2>/dev/null || true
cat /etc/group > "$EVIDENCE_DIR/group.txt"
# Scheduled tasks
log "Collecting scheduled tasks..."
for user in $(cut -f1 -d: /etc/passwd); do
crontab -u "$user" -l 2>/dev/null >> "$EVIDENCE_DIR/crontabs.txt" && \
echo "--- $user ---" >> "$EVIDENCE_DIR/crontabs.txt"
done
ls -la /etc/cron.* > "$EVIDENCE_DIR/cron-dirs.txt" 2>&1
# File system state
log "Collecting filesystem state..."
find /tmp /var/tmp /dev/shm -type f -ls > "$EVIDENCE_DIR/temp-files.txt" 2>/dev/null
find / -name "*.sh" -mtime -7 -ls > "$EVIDENCE_DIR/recent-scripts.txt" 2>/dev/null
find / -perm -4000 -type f -ls > "$EVIDENCE_DIR/suid-files.txt" 2>/dev/null
find /home -name ".*history" -ls > "$EVIDENCE_DIR/history-files.txt" 2>/dev/null
# Loaded kernel modules
log "Collecting kernel modules..."
lsmod > "$EVIDENCE_DIR/kernel-modules.txt"
# Open files
log "Collecting open files..."
lsof -n > "$EVIDENCE_DIR/open-files.txt" 2>/dev/null
# Systemd services
log "Collecting service state..."
systemctl list-units --type=service --all > "$EVIDENCE_DIR/services.txt"
systemctl list-timers --all > "$EVIDENCE_DIR/timers.txt"
# Log preservation
log "Preserving system logs..."
tar czf "$EVIDENCE_DIR/var-log.tar.gz" /var/log/ 2>/dev/null
# Docker containers (if present)
if command -v docker &>/dev/null; then
log "Collecting Docker state..."
docker ps -a > "$EVIDENCE_DIR/docker-containers.txt"
docker images > "$EVIDENCE_DIR/docker-images.txt"
docker network ls > "$EVIDENCE_DIR/docker-networks.txt"
fi
# Kubernetes (if kubectl available)
if command -v kubectl &>/dev/null; then
log "Collecting Kubernetes state..."
kubectl get pods --all-namespaces > "$EVIDENCE_DIR/k8s-pods.txt" 2>/dev/null
kubectl get events --all-namespaces --sort-by=.lastTimestamp > "$EVIDENCE_DIR/k8s-events.txt" 2>/dev/null
fi
# Hash all evidence files
log "Computing evidence hashes..."
find "$EVIDENCE_DIR" -type f ! -name "checksums.sha256" -exec sha256sum {} \; > "$EVIDENCE_DIR/checksums.sha256"
log "Evidence collection complete: $EVIDENCE_DIR"
echo "Total files collected: $(find "$EVIDENCE_DIR" -type f | wc -l)"Memory Acquisition
#!/bin/bash
# memory-capture.sh - Capture volatile memory for forensic analysis
EVIDENCE_DIR="/evidence/memory-$(hostname)-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$EVIDENCE_DIR"
# Using LiME (Linux Memory Extractor)
if [ -f /lib/modules/$(uname -r)/extra/lime.ko ]; then
insmod /lib/modules/$(uname -r)/extra/lime.ko "path=$EVIDENCE_DIR/memory.lime format=lime"
echo "Memory captured with LiME"
fi
# Alternative: /proc/kcore (partial, but always available)
cp /proc/kcore "$EVIDENCE_DIR/kcore" 2>/dev/null
# Capture /proc/meminfo for context
cat /proc/meminfo > "$EVIDENCE_DIR/meminfo.txt"
# Hash the memory dump
sha256sum "$EVIDENCE_DIR"/* > "$EVIDENCE_DIR/checksums.sha256"AWS Evidence Collection
#!/bin/bash
# aws-evidence-collect.sh - Collect evidence from compromised AWS resources
INCIDENT_ID="${1:?Usage: $0 <incident-id>}"
INSTANCE_ID="${2:?Usage: $0 <incident-id> <instance-id>}"
EVIDENCE_BUCKET="s3://incident-evidence-${AWS_ACCOUNT_ID}"
EVIDENCE_PREFIX="${INCIDENT_ID}/$(date +%Y%m%d-%H%M%S)"
echo "=== AWS Evidence Collection ==="
echo "Incident: $INCIDENT_ID"
echo "Instance: $INSTANCE_ID"
# Snapshot EBS volumes
echo "Creating EBS snapshots..."
VOLUMES=$(aws ec2 describe-volumes \
--filters "Name=attachment.instance-id,Values=${INSTANCE_ID}" \
--query 'Volumes[].VolumeId' --output text)
for vol in $VOLUMES; do
SNAP_ID=$(aws ec2 create-snapshot \
--volume-id "$vol" \
--description "IR Evidence - ${INCIDENT_ID} - ${vol}" \
--tag-specifications "ResourceType=snapshot,Tags=[{Key=IncidentId,Value=${INCIDENT_ID}},{Key=Purpose,Value=forensic-evidence}]" \
--query 'SnapshotId' --output text)
echo " Snapshot created: $SNAP_ID for volume $vol"
done
# Capture instance metadata
echo "Capturing instance metadata..."
aws ec2 describe-instances --instance-ids "$INSTANCE_ID" \
> "/tmp/${INCIDENT_ID}-instance-describe.json"
aws s3 cp "/tmp/${INCIDENT_ID}-instance-describe.json" \
"${EVIDENCE_BUCKET}/${EVIDENCE_PREFIX}/instance-describe.json"
# Capture security group rules
SG_IDS=$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" \
--query 'Reservations[].Instances[].SecurityGroups[].GroupId' --output text)
for sg in $SG_IDS; do
aws ec2 describe-security-group-rules --filters "Name=group-id,Values=${sg}" \
> "/tmp/${INCIDENT_ID}-sg-${sg}.json"
aws s3 cp "/tmp/${INCIDENT_ID}-sg-${sg}.json" \
"${EVIDENCE_BUCKET}/${EVIDENCE_PREFIX}/sg-${sg}.json"
done
# Collect CloudTrail events for the instance
echo "Collecting CloudTrail events..."
aws cloudtrail lookup-events \
--lookup-attributes "AttributeKey=ResourceName,AttributeValue=${INSTANCE_ID}" \
--start-time "$(date -d '7 days ago' -u +%Y-%m-%dT%H:%M:%SZ)" \
> "/tmp/${INCIDENT_ID}-cloudtrail.json"
aws s3 cp "/tmp/${INCIDENT_ID}-cloudtrail.json" \
"${EVIDENCE_BUCKET}/${EVIDENCE_PREFIX}/cloudtrail.json"
# Collect VPC flow logs
echo "Collecting VPC flow logs..."
ENI_ID=$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" \
--query 'Reservations[].Instances[].NetworkInterfaces[0].NetworkInterfaceId' --output text)
aws ec2 describe-flow-logs --filter "Name=resource-id,Values=${ENI_ID}" \
> "/tmp/${INCIDENT_ID}-flow-logs.json"
aws s3 cp "/tmp/${INCIDENT_ID}-flow-logs.json" \
"${EVIDENCE_BUCKET}/${EVIDENCE_PREFIX}/flow-logs-config.json"
# Isolate the instance (move to quarantine security group)
echo "Isolating instance..."
QUARANTINE_SG=$(aws ec2 create-security-group \
--group-name "quarantine-${INCIDENT_ID}" \
--description "Quarantine SG for incident ${INCIDENT_ID}" \
--vpc-id "$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" \
--query 'Reservations[].Instances[].VpcId' --output text)" \
--query 'GroupId' --output text)
# Quarantine SG: deny all inbound, allow outbound only to evidence bucket
aws ec2 modify-instance-attribute \
--instance-id "$INSTANCE_ID" \
--groups "$QUARANTINE_SG"
echo "Instance isolated with quarantine SG: $QUARANTINE_SG"
echo "Evidence stored at: ${EVIDENCE_BUCKET}/${EVIDENCE_PREFIX}/"Forensics Commands Reference
# --- Disk forensics ---
# Create forensic image of a disk
dd if=/dev/sda of=/evidence/disk.img bs=4M status=progress
sha256sum /evidence/disk.img > /evidence/disk.img.sha256
# Mount forensic image read-only
mount -o ro,loop,noexec /evidence/disk.img /mnt/forensic
# Find recently modified files
find /mnt/forensic -type f -mtime -3 -ls | sort -k11
# Find files by owner
find /mnt/forensic -user www-data -type f -newer /tmp/reference-time -ls
# --- Log analysis ---
# Search auth logs for brute force
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn | head -20
# Search for privilege escalation
grep -E "(sudo|su\[)" /var/log/auth.log | grep -v "session opened"
# Search web logs for attack patterns
grep -iE "(union.*select|<script|\.\.\/|%00)" /var/log/nginx/access.log
# Timeline analysis with find
find / -newermt "2025-01-15 00:00" ! -newermt "2025-01-16 00:00" -ls 2>/dev/null | sort -k9
# --- Network forensics ---
# Capture network traffic
tcpdump -i eth0 -w /evidence/capture.pcap -c 100000
# Analyze pcap for suspicious connections
tcpdump -r /evidence/capture.pcap -nn 'dst port 4444 or dst port 8888 or dst port 1337'
# Check for DNS tunneling
tcpdump -r /evidence/capture.pcap -nn 'udp port 53' | awk '{print $NF}' | sort | uniq -c | sort -rn | head -20
# --- Malware analysis ---
# Check file for known malware hashes
sha256sum suspicious_file
# Compare against VirusTotal: https://www.virustotal.com
# Strings analysis
strings suspicious_file | grep -iE "(http|ftp|ssh|password|key|token)"
# Check for packed/obfuscated binaries
file suspicious_file
readelf -h suspicious_file 2>/dev/nullCommunication Templates
Initial Notification (Internal)
## Security Incident Notification
**Incident ID:** INC-YYYY-NNNN
**Severity:** [Critical/High/Medium/Low]
**Status:** Active - Investigating
**Time Detected:** YYYY-MM-DD HH:MM UTC
**Reported By:** [Name/System]
### Summary
[1-2 sentence description of what was detected]
### Impact Assessment
- **Systems affected:** [list]
- **Data at risk:** [type and scope]
- **Users impacted:** [count/scope]
- **Business impact:** [description]
### Current Actions
- [ ] Evidence preservation in progress
- [ ] Containment measures being applied
- [ ] IR team assembled
### Next Update
Expected at: YYYY-MM-DD HH:MM UTC
### Incident Commander
[Name] - [Contact info]Stakeholder Update
## Incident Update - INC-YYYY-NNNN
**Update #:** N
**Time:** YYYY-MM-DD HH:MM UTC
**Severity:** [unchanged/upgraded/downgraded]
**Status:** [Investigating/Contained/Eradicating/Recovering/Resolved]
### Progress Since Last Update
- [Bullet points of actions taken]
### Current Understanding
- **Root cause:** [Known/Under investigation]
- **Scope:** [Expanded/Unchanged/Reduced]
- **Threat actor:** [If applicable]
### Active Containment Measures
- [List of measures in place]
### Next Steps
- [Planned actions with ETA]
### Decisions Needed
- [If any decisions required from leadership]External Breach Notification (if required)
## Notice of Data Security Incident
Dear [Customer/Partner],
We are writing to inform you of a security incident that we detected on
[date]. Upon discovery, we immediately activated our incident response
procedures and engaged external cybersecurity experts.
### What Happened
[Brief, factual description]
### What Information Was Involved
[Types of data affected]
### What We Are Doing
[Remediation steps taken and planned]
### What You Can Do
[Recommended actions for affected parties]
### Contact Information
For questions, please contact: [dedicated contact/hotline]
[Company Name]
[Date]IR Playbook: Compromised Credentials
playbook: compromised-credentials
trigger: "Alert indicating credential theft, brute force success, or credential dump"
steps:
1_validate:
- Confirm the alert is not a false positive
- Identify which credentials are compromised
- Determine scope (single user, service account, API key)
2_contain:
- Disable compromised accounts immediately
- Revoke active sessions and tokens
- Rotate API keys and service account credentials
- Block source IP if identified
commands:
- "aws iam update-login-profile --user-name USER --password-reset-required"
- "aws iam delete-access-key --user-name USER --access-key-id AKIAXXXX"
- "aws iam deactivate-mfa-device --user-name USER --serial-number ARN"
- "kubectl delete secret compromised-secret -n NAMESPACE"
3_investigate:
- Review CloudTrail/audit logs for the compromised identity
- Identify all actions taken with compromised credentials
- Check for persistence (new keys, roles, backdoors)
- Determine initial compromise vector (phishing, leak, breach)
4_eradicate:
- Remove any backdoors or persistence mechanisms
- Rotate all credentials that may have been exposed
- Update access policies to enforce MFA
- Patch credential storage if vault/secret manager was compromised
5_recover:
- Issue new credentials with MFA enforced
- Restore access with least-privilege review
- Monitor new credentials for abnormal usage
6_improve:
- Add detection for initial compromise vector
- Review credential management policies
- Update security awareness training if phishing was involvedTroubleshooting
| Problem | Cause | Solution |
|---|---|---|
| Evidence collection script fails | Insufficient permissions | Run with sudo/root; pre-authorize IR accounts |
| Cannot access compromised system | System encrypted by ransomware | Use offline disk imaging; restore from backups |
| Logs are missing or tampered | Attacker cleared logs | Check centralized log aggregator; restore from log backups |
| Cannot determine incident scope | Insufficient logging | Enable CloudTrail, VPC flow logs, audit logging for future |
| Stakeholders demanding immediate answers | Pressure to resolve quickly | Follow IR process; provide regular updates; avoid speculation |
| False positive triggered full IR | Detection rules too sensitive | Tune alerting thresholds; add validation step before escalation |
| Evidence integrity questioned | No chain of custody | Hash all evidence immediately; document who accessed what and when |
Best Practices
- Pre-define and practice playbooks with tabletop exercises quarterly
- Maintain separate, secure communication channels for IR (not email or Slack on corporate infra)
- Always preserve evidence before making changes to compromised systems
- Establish chain of custody for all collected evidence
- Engage legal counsel early in any potential data breach
- Conduct blameless post-incident reviews within 72 hours
- Update detection rules and playbooks based on lessons learned
- Pre-authorize common IR actions so responders can act without delay
- Keep an IR "go bag" with tools, credentials, and documentation ready
- Test backup restoration procedures regularly (not just backup creation)
Related Skills
- audit-logging - Log analysis
- alerting-oncall - Alert management
- security-automation - Automated response workflows
- threat-modeling - Proactive threat identification
Incident Report: [INCIDENT-ID]
Executive Summary
| Field | Value |
|---|---|
| Incident ID | INC-YYYY-MMDD-XXX |
| Status | Open / Contained / Resolved |
| Severity | SEV1 / SEV2 / SEV3 / SEV4 |
| Incident Commander | [Name] |
| Detection Time | YYYY-MM-DD HH:MM UTC |
| Resolution Time | YYYY-MM-DD HH:MM UTC |
| Duration | X hours Y minutes |
Summary: [1-2 sentence description of the incident]
---
Timeline
| Time (UTC) | Event |
|---|---|
| YYYY-MM-DD HH:MM | [Event description] |
| YYYY-MM-DD HH:MM | [Event description] |
| YYYY-MM-DD HH:MM | [Event description] |
---
Impact Assessment
Systems Affected
- [ ] System 1 - [Impact description]
- [ ] System 2 - [Impact description]
Data Affected
- [ ] Type of data
- [ ] Volume
- [ ] Sensitivity classification
Users Affected
- Number of users: [X]
- User groups: [Groups]
Business Impact
- [ ] Service downtime: [Duration]
- [ ] Financial impact: [Estimate]
- [ ] Reputation impact: [Assessment]
---
Root Cause Analysis
Attack Vector
[Description of how the incident occurred]
Contributing Factors
1. [Factor 1] 2. [Factor 2] 3. [Factor 3]
Root Cause
[Description of the underlying cause]
---
Indicators of Compromise (IOCs)
IP Addresses
X.X.X.X - [Description]Domains
malicious.domain.com - [Description]File Hashes
SHA256: [hash] - [Filename]Other IOCs
[Any other relevant indicators]
---
Response Actions
Containment
- [x] Action 1
- [x] Action 2
- [ ] Action 3 (in progress)
Eradication
- [ ] Action 1
- [ ] Action 2
Recovery
- [ ] Action 1
- [ ] Action 2
---
Lessons Learned
What Went Well
1. [Item 1] 2. [Item 2]
What Could Be Improved
1. [Item 1] 2. [Item 2]
---
Action Items
| ID | Action | Owner | Due Date | Status |
|---|---|---|---|---|
| 1 | [Action description] | [Name] | YYYY-MM-DD | Open |
| 2 | [Action description] | [Name] | YYYY-MM-DD | Open |
---
Notifications
Internal
- [ ] Security Team
- [ ] Engineering Team
- [ ] Executive Team
- [ ] Legal/Compliance
External
- [ ] Affected customers
- [ ] Regulatory bodies
- [ ] Law enforcement
---
Appendix
Evidence Files
- [Link to evidence archive]
- [Link to log exports]
Related Documents
- [Link to runbook used]
- [Link to previous incidents]
---
Report Author: [Name] Report Date: YYYY-MM-DD Last Updated: YYYY-MM-DD
Incident Response Playbook
Incident Severity Levels
| Level | Name | Description | Response Time | Example |
|---|---|---|---|---|
| SEV1 | Critical | Active breach, data exfiltration | Immediate | Ransomware, active attacker |
| SEV2 | High | Confirmed compromise, contained | 1 hour | Malware, credential theft |
| SEV3 | Medium | Suspicious activity, potential threat | 4 hours | Phishing success, anomaly |
| SEV4 | Low | Minor security event | 24 hours | Policy violation |
Response Phases
1. Detection & Triage (0-15 minutes)
□ Confirm the incident is real (not false positive)
□ Assess initial scope and severity
□ Assign Incident Commander
□ Open incident channel (#incident-YYYY-MM-DD)
□ Start incident timeline documentationKey Questions:
- What systems are affected?
- Is the threat active?
- What data may be compromised?
- Is it contained or spreading?
2. Containment (15-60 minutes)
Short-term Containment:
□ Isolate affected systems (network/firewall)
□ Block malicious IPs/domains
□ Disable compromised accounts
□ Preserve evidence before changesCommands:
# Network isolation
iptables -I INPUT -s <malicious-ip> -j DROP
iptables -I OUTPUT -d <malicious-ip> -j DROP
# Account disable
usermod -L <username>
passwd -l <username>
# Service isolation
systemctl stop <compromised-service>3. Investigation (1-4 hours)
□ Collect evidence (logs, memory, disk)
□ Identify attack vector
□ Determine scope of compromise
□ Document findings in timelineLog Sources:
- Authentication: /var/log/auth.log, CloudTrail
- Application: Application logs, APM
- Network: Firewall logs, VPC Flow Logs
- System: syslog, journald
4. Eradication (1-24 hours)
□ Remove malware/backdoors
□ Patch vulnerabilities
□ Reset compromised credentials
□ Update security controls
□ Verify complete removal5. Recovery (1-48 hours)
□ Restore systems from clean backups
□ Validate system integrity
□ Monitor for re-infection
□ Gradually restore services
□ Communicate status updates6. Post-Incident (1-2 weeks)
□ Conduct blameless post-mortem
□ Document lessons learned
□ Create action items
□ Update runbooks and detection
□ Report to stakeholders
□ File regulatory notifications (if required)Communication Templates
Internal Notification
SECURITY INCIDENT - [SEV LEVEL]
Status: Active/Contained/Resolved
Incident Commander: [Name]
Channel: #incident-YYYY-MM-DD
Summary: [Brief description]
Impact:
- Systems: [List]
- Data: [Type if applicable]
- Users: [Count/scope]
Current Actions:
- [Action 1]
- [Action 2]
Next Update: [Time]External Notification (if required)
Subject: Security Incident Notification
We are writing to inform you of a security incident
that occurred on [DATE].
What Happened: [Description]
Data Involved: [Types]
Actions Taken: [Response measures]
What You Can Do: [Recommendations]
Contact: [Security team contact]Escalation Contacts
| Role | Primary | Secondary |
|---|---|---|
| Incident Commander | [Name] | [Name] |
| Security Lead | [Name] | [Name] |
| Engineering Lead | [Name] | [Name] |
| Legal/Compliance | [Name] | [Name] |
| Communications | [Name] | [Name] |
| Executive Sponsor | [Name] | [Name] |
Indicator of Compromise (IOC) Hunting Guide
Common IOC Types
| Type | Description | Example |
|---|---|---|
| IP Address | Malicious source/destination | 192.168.1.100 |
| Domain | C2 or phishing domain | malware.evil.com |
| File Hash | Malware signature | SHA256:abc123... |
| File Path | Suspicious file location | /tmp/.hidden |
| Process | Malicious process name | cryptominer |
| User | Compromised account | admin |
Log Hunting Queries
SSH Brute Force Detection
# Failed SSH attempts
grep "Failed password" /var/log/auth.log | \
awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head
# Successful logins after failures
grep -E "Accepted|Failed" /var/log/auth.log | \
grep -B5 "Accepted" | grep "Failed"Suspicious Process Activity
# Processes running from /tmp
ps aux | grep -E "^.*/tmp/|^.*/dev/shm/"
# Hidden processes
ps aux | awk '$11 ~ /^\./'
# Processes with deleted binaries
ls -la /proc/*/exe 2>/dev/null | grep deleted
# Unusual parent-child relationships
ps -eo pid,ppid,cmd | grep -E "bash.*-c|sh.*-c"Network IOCs
# Connections to known bad ports
ss -anp | grep -E ":4444|:5555|:6666|:31337"
# Outbound connections from unusual processes
ss -anp | grep -v -E "chrome|firefox|curl|wget" | grep ESTAB
# DNS queries to suspicious domains
grep -E "query.*\.(tk|ml|ga|cf|gq)$" /var/log/syslog
# Large outbound transfers
ss -anp | awk '$3 > 1000000'File System IOCs
# Recently modified files in sensitive locations
find /etc /usr/bin /usr/sbin -mtime -1 -ls 2>/dev/null
# Files with suspicious permissions
find / -perm -4000 -o -perm -2000 -ls 2>/dev/null
# Hidden files
find / -name ".*" -type f -ls 2>/dev/null | head -50
# World-writable files
find / -perm -002 -type f -ls 2>/dev/nullUser Activity IOCs
# Recent sudo usage
grep sudo /var/log/auth.log | tail -50
# Users logged in from multiple IPs
last | awk '{print $1, $3}' | sort | uniq -c | sort -rn
# SSH keys added recently
find /home -name "authorized_keys" -mtime -7 -ls
# Unusual cron jobs
for user in $(cut -d: -f1 /etc/passwd); do
crontab -l -u $user 2>/dev/null | grep -v "^#"
doneAWS CloudTrail Hunting
# Console logins from unusual locations
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=ConsoleLogin \
--query 'Events[*].[CloudTrailEvent]' --output text | jq '.'
# Root account usage
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=Username,AttributeValue=root
# Security group changes
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=AuthorizeSecurityGroupIngressYARA Rule Example
rule Suspicious_Shell_Script {
meta:
description = "Detects suspicious shell scripts"
severity = "medium"
strings:
$s1 = "curl" ascii
$s2 = "wget" ascii
$s3 = "/dev/tcp/" ascii
$s4 = "base64 -d" ascii
$s5 = "chmod +x" ascii
condition:
3 of them
}Response Actions
Block IOC
# Block IP
iptables -I INPUT -s <IP> -j DROP
iptables -I OUTPUT -d <IP> -j DROP
# Block domain (via hosts)
echo "127.0.0.1 malicious.domain.com" >> /etc/hosts
# Kill process
kill -9 <PID>Preserve Evidence
# Capture process memory
gcore <PID>
# Copy suspicious file
cp --preserve=all /path/to/file /evidence/
# Capture network traffic
tcpdump -i any -w /evidence/capture.pcap &#!/bin/bash
# Security Incident Evidence Collection Script
# Usage: ./collect-evidence.sh [incident-id]
set -euo pipefail
INCIDENT_ID="${1:-incident-$(date +%Y%m%d-%H%M%S)}"
EVIDENCE_DIR="/tmp/evidence-$INCIDENT_ID"
HOSTNAME=$(hostname)
mkdir -p "$EVIDENCE_DIR"
echo "========================================="
echo "Security Incident Evidence Collection"
echo "Incident ID: $INCIDENT_ID"
echo "Host: $HOSTNAME"
echo "Time: $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
echo "Output: $EVIDENCE_DIR"
echo "========================================="
echo ""
# Create metadata file
cat > "$EVIDENCE_DIR/metadata.txt" << EOF
Incident ID: $INCIDENT_ID
Collection Time: $(date -u '+%Y-%m-%d %H:%M:%S UTC')
Hostname: $HOSTNAME
Kernel: $(uname -a)
Collector: $(whoami)
EOF
# System information
echo "Collecting system information..."
mkdir -p "$EVIDENCE_DIR/system"
uname -a > "$EVIDENCE_DIR/system/uname.txt"
cat /etc/os-release > "$EVIDENCE_DIR/system/os-release.txt" 2>/dev/null || true
uptime > "$EVIDENCE_DIR/system/uptime.txt"
date -u > "$EVIDENCE_DIR/system/date.txt"
# Running processes
echo "Collecting process information..."
mkdir -p "$EVIDENCE_DIR/processes"
ps auxf > "$EVIDENCE_DIR/processes/ps-auxf.txt"
ps -eo pid,ppid,user,cmd --sort=-pid > "$EVIDENCE_DIR/processes/ps-sorted.txt"
pstree -p > "$EVIDENCE_DIR/processes/pstree.txt" 2>/dev/null || true
# Network connections
echo "Collecting network information..."
mkdir -p "$EVIDENCE_DIR/network"
ss -tlnp > "$EVIDENCE_DIR/network/listening-tcp.txt"
ss -ulnp > "$EVIDENCE_DIR/network/listening-udp.txt"
ss -anp > "$EVIDENCE_DIR/network/all-connections.txt"
ip addr > "$EVIDENCE_DIR/network/ip-addr.txt"
ip route > "$EVIDENCE_DIR/network/ip-route.txt"
iptables -L -n -v > "$EVIDENCE_DIR/network/iptables.txt" 2>/dev/null || true
cat /etc/hosts > "$EVIDENCE_DIR/network/hosts.txt"
# User information
echo "Collecting user information..."
mkdir -p "$EVIDENCE_DIR/users"
cat /etc/passwd > "$EVIDENCE_DIR/users/passwd.txt"
cat /etc/group > "$EVIDENCE_DIR/users/group.txt"
who > "$EVIDENCE_DIR/users/who.txt"
w > "$EVIDENCE_DIR/users/w.txt"
last -100 > "$EVIDENCE_DIR/users/last.txt"
lastlog > "$EVIDENCE_DIR/users/lastlog.txt" 2>/dev/null || true
# Authentication logs
echo "Collecting authentication logs..."
mkdir -p "$EVIDENCE_DIR/logs"
tail -1000 /var/log/auth.log > "$EVIDENCE_DIR/logs/auth.log" 2>/dev/null || true
tail -1000 /var/log/secure > "$EVIDENCE_DIR/logs/secure.log" 2>/dev/null || true
tail -1000 /var/log/syslog > "$EVIDENCE_DIR/logs/syslog.txt" 2>/dev/null || true
journalctl -u sshd --since "1 day ago" > "$EVIDENCE_DIR/logs/sshd.log" 2>/dev/null || true
# Scheduled tasks
echo "Collecting scheduled tasks..."
mkdir -p "$EVIDENCE_DIR/scheduled"
crontab -l > "$EVIDENCE_DIR/scheduled/crontab-current.txt" 2>/dev/null || true
ls -la /etc/cron.* > "$EVIDENCE_DIR/scheduled/cron-dirs.txt" 2>/dev/null || true
cat /etc/crontab > "$EVIDENCE_DIR/scheduled/etc-crontab.txt" 2>/dev/null || true
systemctl list-timers > "$EVIDENCE_DIR/scheduled/systemd-timers.txt" 2>/dev/null || true
# File system
echo "Collecting filesystem information..."
mkdir -p "$EVIDENCE_DIR/filesystem"
df -h > "$EVIDENCE_DIR/filesystem/df.txt"
mount > "$EVIDENCE_DIR/filesystem/mounts.txt"
find /tmp /var/tmp -type f -mtime -1 -ls > "$EVIDENCE_DIR/filesystem/recent-tmp.txt" 2>/dev/null || true
# Package hashes
echo "Collecting hash information..."
if command -v sha256sum &>/dev/null; then
find /usr/bin /usr/sbin -type f -executable 2>/dev/null | head -100 | xargs sha256sum > "$EVIDENCE_DIR/filesystem/binary-hashes.txt" 2>/dev/null || true
fi
# Create archive
echo ""
echo "Creating evidence archive..."
ARCHIVE="/tmp/$INCIDENT_ID-evidence.tar.gz"
tar -czf "$ARCHIVE" -C /tmp "evidence-$INCIDENT_ID"
echo ""
echo "========================================="
echo "Evidence collection complete"
echo "Archive: $ARCHIVE"
echo "Size: $(du -h "$ARCHIVE" | cut -f1)"
echo ""
echo "SHA256: $(sha256sum "$ARCHIVE" | cut -d' ' -f1)"
echo "========================================="
Related skills
FAQ
What phases does the IR process follow?
Preparation, detection, containment, eradication, recovery, and lessons learned.
Does it include forensic tooling?
Yes, it ships a Linux evidence-collection script that preserves timestamps, hashes, and chain-of-custody logs.