Ctf Reverse

CTF Reverse Engineering

Quick reference for RE challenges. For detailed techniques, see supporting files.

Prerequisites

Python packages (all platforms):

pip install frida-tools angr qiling uncompyle6 capstone lief z3-solver
# For Python 3.9+ bytecode: build pycdc from source
git clone https://github.com/zrax/pycdc && cd pycdc && cmake . && make

Linux (apt):

apt install gdb radare2 binutils strace ltrace apktool upx

macOS (Homebrew):

brew install gdb radare2 binutils apktool upx ghidra

radare2 plugins:

r2pm -ci r2ghidra   # Native Ghidra decompiler for radare2

Manual install:

• pwndbg — Linux: GitHub, macOS: brew install pwndbg/tap/pwndbg-gdb

Additional Resources

• tools.md - Static analysis tools (GDB, Ghidra, radare2, IDA, Binary Ninja, dogbolt.org, RISC-V with Capstone, Unicorn emulation, Python bytecode, WASM, Android APK, .NET, packed binaries)
• tools-dynamic.md - Dynamic analysis tools: Frida (hooking, anti-debug bypass, memory scanning, Android/iOS), angr symbolic execution (path exploration, constraints, CFG), lldb (macOS/LLVM debugger), x64dbg (Windows)
• tools-emulation.md - Emulation frameworks and side-channel tooling: Qiling (cross-platform OS-level emulation), Triton (DSE), Intel Pin instruction-counting + genetic algorithm side channel, opcode-only trace reconstruction, LD_PRELOAD time freeze and memcmp side-channel for byte-by-byte bruteforce
• tools-advanced.md - Advanced tools (Part 1): VMProtect/Themida analysis, binary diffing (BinDiff, Diaphora), deobfuscation frameworks (D-810, GOOMBA, Miasm), Qiling framework, Triton DSE, Manticore, Rizin/Cutter, RetDec, custom VM bytecode lifting to LLVM IR
• tools-advanced-2.md - Advanced tools (Part 2): advanced GDB (Python scripting, brute-force, conditional breakpoints, watchpoints, reverse debugging with rr, pwndbg/GEF), advanced Ghidra scripting, patching (Binary Ninja API, LIEF), GDB constraint extraction + ILP solver (BackdoorCTF 2017), GDB position-encoded input zero flag monitoring (EKOPARTY 2017), LD_PRELOAD execute-only binary dump (BackdoorCTF 2017), PEDA current_inst bit-by-bit flag scraper (CONFidence CTF 2019 Teaser)
• anti-analysis.md - Anti-analysis taxonomy: Linux anti-debug (ptrace, /proc, timing, signals, direct syscalls), Windows anti-debug (PEB, NtQueryInformationProcess, heap flags, TLS callbacks, HW/SW breakpoint detection, exception-based, thread hiding), anti-VM/sandbox (CPUID, MAC, timing, artifacts, resources), anti-DBI (Frida detection/bypass), code integrity/self-hashing, anti-disassembly (opaque predicates, junk bytes), MBA identification/simplification, comprehensive bypass strategies
• anti-analysis-ctf.md - CTF writeup techniques: SIGILL handler for execution mode switching (Hack.lu 2015), SIGFPE signal handler side-channel via strace counting (PlaidCTF 2017), instruction trace inversion with Keystone and Unicorn (MeePwn 2017), call-less function chaining via stack frame manipulation (THC 2018), parent-patched child binary dump via process_vm_writev (Google CTF Quals 2018)
• patterns.md - Foundational binary patterns: custom VMs, anti-debugging, nanomites, self-modifying code, XOR ciphers, mixed-mode stagers, LLVM obfuscation, S-box/keystream, SECCOMP/BPF, exception handlers, memory dumps, byte-wise transforms, x86-64 gotchas, custom mangle reversing, position-based transforms, hex-encoded string comparison, signal-based binary exploration
• patterns-runtime.md - Runtime patching and oracle techniques: malware anti-analysis bypass, multi-stage shellcode loaders, timing side-channel attacks, multi-thread anti-debug with decoy + signal handler MBA (ApoorvCTF 2026), INT3 patch + coredump brute-force oracle (Pwn2Win 2016), signal handler chain + LD_PRELOAD oracle (Nuit du Hack 2016), printf format string VM decompilation to Z3 (SECCON 2017), quadtree recursive image format parser (Google CTF Quals 2018)
• patterns-ctf.md - Competition-specific patterns (Part 1): hidden emulator opcodes, LD_PRELOAD key extraction, SPN static extraction, image XOR smoothness, byte-at-a-time cipher, mathematical convergence bitmap, Windows PE XOR bitmap OCR, two-stage RC4+VM loaders, GBA ROM meet-in-the-middle, Sprague-Grundy game theory, kernel module maze solving, multi-threaded VM channels, backdoored shared library detection via string diffing, custom binfmt kernel module with RC4 flat binaries, hash-resolved imports / no-import ransomware, ELF section header corruption for anti-analysis
• patterns-ctf-2.md - Competition-specific patterns (Part 2): multi-layer self-decrypting brute-force, embedded ZIP+XOR license, stack string deobfuscation, prefix hash brute-force, CVP/LLL lattice for integer validation, decision tree function obfuscation, GF(2^8) Gaussian elimination, ROP chain obfuscation analysis (ROPfuscation)
• patterns-ctf-3.md - Competition-specific patterns (Part 3): Z3 single-line Python circuit, sliding window popcount, keyboard LED Morse code via ioctl, C++ destructor-hidden validation, syscall side-effect memory corruption, MFC dialog event handlers, VM sequential key-chain brute-force, Burrows-Wheeler transform inversion, OpenType font ligature exploitation, GLSL shader VM with self-modifying code, instruction counter as cryptographic state, batch crackme automation via objdump, fork+pipe+dead branch anti-analysis, TensorFlow DNN inversion via sigmoid layer inversion, BPF filter analysis via kernel JIT to x64 assembly
• languages.md - Language-specific: Python bytecode & opcode remapping, Python version-specific bytecode, Pyarmor static unpack, DOS stubs, Unity IL2CPP, HarmonyOS HAP/ABC, Brainfuck/esolangs (+ BF character-by-character static analysis, BF side-channel read count oracle, BF comparison idiom detection), UEFI, transpilation to C, code coverage side-channel, OPAL functional reversing, non-bijective substitution, FRACTRAN program inversion
• languages-platforms.md - Platform/framework-specific: Roblox place file analysis, Godot game asset extraction, Rust serde_json schema recovery, Android JNI RegisterNatives obfuscation, Android DEX runtime bytecode patching via /proc/self/maps, Android native .so loading bypass via new project, Frida Firebase Cloud Functions bypass, Verilog/hardware RE, prefix-by-prefix hash reversal, Ruby/Perl polyglot constraint satisfaction, Electron ASAR extraction + native binary analysis, Node.js npm runtime introspection
• languages-compiled.md - Go binary reversing (GoReSym, goroutines, memory layout, channel ops, embed.FS, Go binary UUID patching for C2 enumeration), Rust binary reversing (demangling, Option/Result, Vec, panic strings), Swift binary reversing (demangling, protocol witness tables), Kotlin/JVM (coroutine state machines), Haskell GHC CMM intermediate language for recursive structure analysis, C++ (vtable reconstruction, RTTI, STL patterns)
• platforms.md - Platform-specific RE: macOS/iOS (Mach-O, code signing, Objective-C runtime, Swift, dyld, jailbreak bypass), embedded/IoT firmware (binwalk, UART/JTAG/SPI extraction, ARM/MIPS, RTOS), kernel drivers (Linux .ko, eBPF, Windows .sys), game engines (Unreal Engine, Unity, anti-cheat, Lua), automotive CAN bus
• platforms-hardware.md - Hardware and advanced architecture RE: HD44780 LCD controller GPIO reconstruction, RISC-V advanced (custom extensions, privileged modes, debugging), ARM64/AArch64 reversing and exploitation (calling convention, ROP gadgets, qemu-aarch64-static emulation)
• field-notes.md - Quick reference notes: binary types, anti-debugging bypass, specialized patterns, CTF case notes

---

When to Pivot

• If you already understand the binary and now need heap, ROP, or kernel exploitation, switch to /ctf-pwn.
• If the challenge is really about recovering deleted files, PCAP data, or disk artifacts, switch to /ctf-forensics.
• If the target is a web app and you are only reversing a small client-side helper script, switch to /ctf-web.
• If the binary implements a machine learning model and the challenge is about model attacks or adversarial inputs, switch to /ctf-ai-ml.
• If the reversed binary's core logic is a cryptographic algorithm or math problem, switch to /ctf-crypto.
• If the binary is a real malware sample with C2, packing, or evasion behavior, switch to /ctf-malware.
• If the challenge is a toy VM, encoding puzzle, or pyjail rather than a real binary, switch to /ctf-misc.

Problem-Solving Workflow

1. Start with strings extraction - many easy challenges have plaintext flags 2. Try ltrace/strace - dynamic analysis often reveals flags without reversing 3. Try Frida hooking - hook strcmp/memcmp to capture expected values without reversing 4. Try angr - symbolic execution solves many flag-checkers automatically 5. Try Qiling - emulate foreign-arch binaries or bypass heavy anti-debug without artifacts 6. Map control flow before modifying execution 7. Automate manual processes via scripting (r2pipe, Frida, angr, Python) 8. Validate assumptions by comparing decompiler outputs (dogbolt.org for side-by-side)

Quick Wins (Try First!)

# Plaintext flag extraction
strings binary | grep -E "flag\{|CTF\{|pico"
strings binary | grep -iE "flag|secret|password"
rabin2 -z binary | grep -i "flag"

# Dynamic analysis - often captures flag directly
ltrace ./binary
strace -f -s 500 ./binary

# Hex dump search
xxd binary | grep -i flag

# Run with test inputs
./binary AAAA
echo "test" | ./binary

Initial Analysis

file binary           # Type, architecture
checksec --file=binary # Security features (for pwn)
chmod +x binary       # Make executable

Memory Dumping Strategy

Key insight: Let the program compute the answer, then dump it. Break at final comparison (b *main+OFFSET), enter any input of correct length, then x/s $rsi to dump computed flag.

Decoy Flag Detection

Pattern: Multiple fake targets before real check. Look for multiple comparison targets in sequence with different success messages. Set breakpoint at FINAL comparison, not earlier ones.

GDB PIE Debugging

PIE binaries randomize base address. Use relative breakpoints:

gdb ./binary
start                    # Forces PIE base resolution
b *main+0xca            # Relative to main
run

Comparison Direction (Critical!)

Two patterns: (1) transform(flag) == stored_target — reverse the transform. (2) transform(stored_target) == flag — flag IS the transformed data, just apply transform to stored target.

Common Encryption Patterns

• XOR with single byte - try all 256 values
• XOR with known plaintext (flag{, CTF{)
• RC4 with hardcoded key
• Custom permutation + XOR
• XOR with position index (^ i or ^ (i & 0xff)) layered with a repeating key

Quick Tool Reference

# Radare2
r2 -d ./binary     # Debug mode
aaa                # Analyze
afl                # List functions
pdf @ main         # Disassemble main

# Ghidra (headless)
analyzeHeadless project/ tmp -import binary -postScript script.py

# IDA
ida64 binary       # Open in IDA64

Deep-Dive Notes

Use field-notes.md after the first round of triage when you know what kind of target you have.

• Target formats: Python bytecode, WASM, Android, Flutter, .NET, UPX, Tauri
• Technique notes: anti-debug bypass, VM analysis, x86-64 gotchas, iterative solvers, Unicorn, timing side channels
• Platform notes: Godot, Roblox, macOS/iOS, embedded firmware, kernel drivers, game engines, Swift, Kotlin, Go, Rust, D
• Case notes: modern CTF-specific reversing patterns and older classic challenge patterns

CTF Reverse - Anti-Analysis CTF Writeups

CTF-specific anti-analysis techniques: signal-handler tricks, instruction-trace inversion, call-less function chaining, parent-patched child binary dumping. For the core anti-analysis taxonomy (Linux/Windows anti-debug, anti-VM, anti-DBI, code integrity, anti-disassembly), see anti-analysis.md.

Table of Contents

• SIGILL Handler for Execution Mode Switching (Hack.lu 2015)
• SIGFPE Signal Handler Side-Channel via strace Counting (PlaidCTF 2017)
• Instruction Trace Inversion with Keystone and Unicorn (MeePwn CTF 2017)
• Call-less Function Chaining via Stack Frame Manipulation (THC CTF 2018)
• Parent-Patched Child Binary Dump via strace process_vm_writev (Google CTF Quals 2018)
• ConfuserEx Dynamic Module Dump via Constructor Breakpoint (Kaspersky 2018)

---

SIGILL Handler for Execution Mode Switching (Hack.lu 2015)

Binaries may install SIGILL (illegal instruction) handlers to switch between x86 and x86-64 execution modes or implement custom opcode dispatch:

1. Signal registration: signal(SIGILL, handler) installs a callback for illegal instruction exceptions 2. Mode switching: The handler modifies the saved instruction pointer or segment registers to switch between 32-bit and 64-bit code 3. Custom opcodes: Invalid x86 instructions trigger the handler, which interprets operand bytes as custom VM opcodes

// Signal handler decodes "illegal" instructions as custom opcodes
void sigill_handler(int sig, siginfo_t *info, void *ucontext) {
    ucontext_t *ctx = (ucontext_t *)ucontext;
    unsigned char *pc = (unsigned char *)ctx->uc_mcontext.gregs[REG_RIP];
    // Decode custom opcode from bytes at PC
    // Advance PC past the custom instruction
    ctx->uc_mcontext.gregs[REG_RIP] += opcode_length;
}

Key insight: If a binary installs signal handlers for SIGILL/SIGSEGV/SIGTRAP early in execution, suspect custom instruction dispatch. Trace signal deliveries with strace -e signal or set GDB to not intercept: handle SIGILL nostop pass.

---

SIGFPE Signal Handler Side-Channel via strace Counting (PlaidCTF 2017)

Binary uses SIGFPE signal handlers for control flow, making static analysis unreliable. Brute-force by counting SIGFPE signals via strace — correct input characters produce more signals.

# Count SIGFPE signals per input character guess
for c in {a..z} {A..Z} {0..9}; do
    count=$(echo -n "${c}AAAAAAA" | strace -e signal=SIGFPE ./binary 2>&1 | grep -c SIGFPE)
    echo "$c: $count"
done
# Character producing the most SIGFPEs is correct
# Repeat for each position, extending the known prefix

Key insight: Signal handlers (SIGFPE, SIGSEGV, SIGILL) create implicit control flow invisible to static analysis. The number of signals raised correlates with validation progress. Counting signals via strace -e signal=SIGFPE turns opaque signal-based validation into a measurable side-channel for character-by-character brute-force.

---

Instruction Trace Inversion with Keystone and Unicorn (MeePwn CTF 2017)

UPX-packed binary applies a sequence of arithmetic-only transforms (sub, add, xor, rol, ror) to the flag. No memory side-effects — purely register arithmetic. IDAPython traces non-jump instructions, the sequence is then inverted to recover the flag.

Inversion rules:

• Reverse the instruction sequence (last instruction first)
• Swap inverse pairs: add ↔ sub, rol ↔ ror, xor is self-inverse

# IDAPython: collect non-jump instructions in the obfuscated routine
import idaapi, idc

def trace_transforms(start_ea, end_ea):
    instructions = []
    ea = start_ea
    while ea < end_ea:
        mnem = idc.print_insn_mnem(ea)
        if mnem not in ('jmp', 'je', 'jne', 'call', 'ret'):
            instructions.append((ea, mnem, idc.print_operands(ea)))
        ea = idc.next_head(ea)
    return instructions

transforms = trace_transforms(0x401000, 0x401200)

# Invert: reverse order, swap add/sub and rol/ror
inverse_map = {'add': 'sub', 'sub': 'add', 'rol': 'ror', 'ror': 'rol', 'xor': 'xor'}
inverted = [(mnem, op) for (_, mnem, op) in reversed(transforms)]
inverted = [(inverse_map.get(m, m), op) for m, op in inverted]

# Assemble inverted instructions with Keystone, emulate with Unicorn
from keystone import *
from unicorn import *
from unicorn.x86_const import *

ks = Ks(KS_ARCH_X86, KS_MODE_64)
uc = Uc(UC_ARCH_X86, UC_MODE_64)

asm_src = '\n'.join(f'{mnem} {op}' for mnem, op in inverted)
encoding, _ = ks.asm(asm_src)

CODE_BASE = 0x400000
uc.mem_map(CODE_BASE, 0x10000)
uc.mem_write(CODE_BASE, bytes(encoding))

# Set initial register state to the observed output value
uc.reg_write(UC_X86_REG_RAX, known_output)
uc.emu_start(CODE_BASE, CODE_BASE + len(encoding))
flag_bytes = uc.reg_read(UC_X86_REG_RAX).to_bytes(8, 'little')

PEB anti-debug note: If the binary reads PEB.BeingDebugged and uses it to select between two comparison target values, the traced instructions under IDAPython may use the debug-mode target. Patch BeingDebugged to 0 before tracing, or identify both branches and use the non-debug target value.

Key insight: Arithmetic-only obfuscation (no memory writes) is fully reversible by tracing, inverting the instruction sequence, and swapping inverse operations. PEB anti-debug can silently change comparison targets — always verify which branch is taken.

References: MeePwn CTF 2017

---

Call-less Function Chaining via Stack Frame Manipulation (THC CTF 2018)

Pattern: Binary hides function calls by building a linked list of function pointers on the stack, then modifying saved RBP and return addresses so leave; ret instructions chain through the list without any explicit CALL instructions. IDA fails to decompile because push/pop are unbalanced and function boundaries cannot be determined.

Each function in the chain: 1. Pushes operands and the next function's address onto the stack 2. Sets saved RBP to point to the next stack frame 3. Sets the return address to the next function 4. leave restores RSP from RBP (moving to next frame), ret jumps to the next function

# Reversed processing chain (each function applied via leave/ret):
def reverse_processing(byte):
    res = byte | 0x80       # OR 0x80
    res = res ^ 0xCA        # XOR 0xCA
    res = (res + 66) & 0xFF # ADD 66
    res = res ^ 0xCA        # XOR 0xCA (repeated)
    res = (res + 66) & 0xFF
    res = res ^ 0xCA
    res = (res + 66) & 0xFF
    res = res ^ 0xFE        # XOR 0xFE (final)
    return res
# Apply in reverse order, then reverse the character sequence

Key insight: By manipulating saved RBP to point to the next stack frame and saved RIP to the next function, leave; ret chains through functions without any call instructions. Disassemblers that track call/ret balance fail to identify function boundaries. Patch each function body individually for IDA to handle them.

Detection: Binary with many small code blocks ending in leave; ret but no corresponding call instructions. Stack contains interleaved function pointers and data. IDA shows "stack frame is too big" or fails to create functions.

References: THC CTF 2018

---

Parent-Patched Child Binary Dump via strace process_vm_writev (Google CTF Quals 2018)

Pattern (Keygenme): The binary forks. The child is stub code full of int3 (0xcc) traps. The parent uses ptrace + process_vm_writev to write the real instructions into the child right before each trap fires, then stepping continues. Static analysis of the child sees only junk; dynamic analysis in a single-process debugger misses the parent's writes.

Bypass — let strace do the work:

# Record every process_vm_writev the parent performs, including full iov contents.
strace -f -e trace=process_vm_writev -e write=all -o trace.log ./keygenme

# Each entry looks like:
#   process_vm_writev(child_pid, [{iov_base="\x48\x89\xe5...", iov_len=12}], 1,
#                     [{iov_base=0x400c80, iov_len=12}], 1, 0) = 12

Parse the log to extract (remote_addr, bytes) pairs and emit an IDA patch_bytes script:

import re, pathlib
patches = []
pattern = re.compile(
    r'process_vm_writev\(\d+, \[{iov_base="([^"]+)", iov_len=(\d+)}\].*?\[{iov_base=(0x[0-9a-f]+)',
)
for m in pattern.finditer(pathlib.Path('trace.log').read_text()):
    data = m.group(1).encode('latin1').decode('unicode_escape').encode('latin1')
    addr = int(m.group(3), 16)
    patches.append((addr, data))

with open('patch.py', 'w') as fh:
    for addr, data in patches:
        for i, b in enumerate(data):
            fh.write(f'patch_byte({addr + i:#x}, {b:#x})\n')

Load patch.py in IDA (File → Script file) to apply every parent-written instruction, turning the trap-riddled child into a fully readable binary. With the patched binary, the crypto routine is a plain loop — black-box the irreversible portion and replace the final strcmp with a leak of the expected value.

Key insight: Any anti-analysis scheme that uses a ptracer to rewrite the tracee's text is transparent to strace on the parent. process_vm_writev calls carry both the target address and the bytes, so a one-pass strace run is enough to dump the real code. The same trick works for self-modifying packers that use ptrace(PTRACE_POKEDATA) or write() into /proc/<pid>/mem.

References: Google CTF Quals 2018 — writeup 10330

---

ConfuserEx Dynamic Module Dump via Constructor Breakpoint (Kaspersky 2018)

Pattern: ConfuserEx (.NET protector) encrypts method bodies and decrypts them at runtime from a <Module> constructor. Break on the constructor in dnSpy, step until the dynamic module is fully built in memory, right-click → Save Module to dump the decrypted assembly with tokens intact. Run de4dot over the dump to rename obfuscated symbols.

dnSpy:
  File → Open → target.exe
  Assembly Explorer → <Module> .cctor → F9 (breakpoint)
  F5 to run; wait until loaded
  Right-click assembly → Save Module → out.exe
$ de4dot out.exe        # symbol cleanup

Key insight: ConfuserEx protects on-disk code but not the runtime representation. Any time a .NET protector ships a compiled constructor that performs decryption, the dumped post-constructor module is the cleartext binary. Chain with de4dot to undo the follow-up symbol obfuscation.

References: Kaspersky Industrial CTF 2018 — glardomos, writeup 12325

CTF Reverse - Anti-Analysis Techniques & Bypasses

Comprehensive reference for anti-debugging, anti-VM, anti-DBI, and integrity-check techniques encountered in CTF challenges, with practical bypasses.

Table of Contents

• Linux Anti-Debug (Advanced)
• ptrace-Based
• /proc Filesystem Checks
• Timing-Based Detection
• Signal-Based Anti-Debug
• Syscall-Level Evasion
• Trap-Flag Self-Check with cmovz Patcher (Hack.lu 2018)
• SIGFPE Handler for mprotect Code Mutation (Hack.lu 2018)
• Windows Anti-Debug (Advanced)
• PEB (Process Environment Block) Checks
• NtQueryInformationProcess
• Heap Flags
• TLS Callbacks
• Hardware Breakpoint Detection
• Software Breakpoint Detection (INT3 Scanning)
• Exception-Based Anti-Debug
• NtSetInformationThread (Thread Hiding)
• Anti-VM / Anti-Sandbox
• CPUID Hypervisor Bit
• MAC Address / Hardware Fingerprinting
• Timing-Based VM Detection
• File / Registry Artifacts
• Resource Checks (CPU Count, RAM, Disk)
• Anti-DBI (Dynamic Binary Instrumentation)
• Frida Detection
• Pin/DynamoRIO Detection
• Code Integrity / Self-Hashing
• Anti-Disassembly Techniques
• Opaque Predicates
• Junk Bytes / Overlapping Instructions
• Jump-in-the-Middle
• Function Chunking / Scattered Code
• Control Flow Flattening (Advanced)
• Mixed Boolean-Arithmetic (MBA) Identification & Simplification
• Comprehensive Bypass Strategies

For CTF writeup techniques (SIGILL handler, SIGFPE strace side-channel, instruction trace inversion, call-less function chaining, parent-patched child binary dump), see anti-analysis-ctf.md.

• Universal Bypass Checklist
• Layered Anti-Debug (Real-World Pattern)
• Quick Reference: Check to Bypass

---

Linux Anti-Debug (Advanced)

ptrace-Based

Self-ptrace (most common):

if (ptrace(PTRACE_TRACEME, 0, 0, 0) == -1) exit(1); // Already traced = debugger attached

Bypasses:

# 1. LD_PRELOAD (see patterns.md for full hook)
LD_PRELOAD=./hook.so ./binary

# 2. Patch with pwntools
python3 -c "
from pwn import *
elf = ELF('./binary', checksec=False)
elf.asm(elf.symbols.ptrace, 'xor eax, eax; ret')
elf.save('patched')
"

# 3. GDB: catch the syscall
gdb ./binary
(gdb) catch syscall ptrace
(gdb) run
# When it stops at ptrace:
(gdb) set $rax = 0
(gdb) continue

# 4. Kernel config (requires root)
echo 0 > /proc/sys/kernel/yama/ptrace_scope

Double-ptrace pattern:

// Fork child to ptrace parent — blocks all other debuggers
pid_t child = fork();
if (child == 0) {
    ptrace(PTRACE_ATTACH, getppid(), 0, 0);
    // Child sits in waitpid loop, keeping parent traced
} else {
    // Parent continues with real logic
}

Bypass: Kill the watchdog child process, then attach debugger.

/proc Filesystem Checks

// TracerPid check
FILE *f = fopen("/proc/self/status", "r");
// Looks for "TracerPid:\t0" — non-zero means debugger

// /proc/self/exe link check (some debuggers change this)
readlink("/proc/self/exe", buf, sizeof(buf));

// /proc/self/maps — check for debugger libraries
grep("frida", "/proc/self/maps");

Bypasses:

# 1. LD_PRELOAD fopen/fread to fake /proc contents
# 2. Mount namespace isolation
unshare -m bash -c 'mount --bind /dev/null /proc/self/status && ./binary'

# 3. GDB: set breakpoint at fopen, change filename argument
(gdb) b fopen
(gdb) run
(gdb) set {char[20]} $rdi = "/dev/null"
(gdb) continue

Timing-Based Detection

// rdtsc (CPU timestamp counter)
uint64_t start = __rdtsc();
// ... code ...
uint64_t delta = __rdtsc() - start;
if (delta > THRESHOLD) exit(1);  // too slow = debugger

// clock_gettime
struct timespec ts1, ts2;
clock_gettime(CLOCK_MONOTONIC, &ts1);
// ... code ...
clock_gettime(CLOCK_MONOTONIC, &ts2);

// gettimeofday
struct timeval tv1, tv2;
gettimeofday(&tv1, NULL);

Bypasses:

# 1. Frida hook (see tools-dynamic.md for clock_gettime hook)

# 2. GDB: skip rdtsc by patching with constant
(gdb) set {unsigned char[2]} 0x401234 = {0x90, 0x90}  # NOP the rdtsc

# 3. Pin tool to fix TSC reads
# 4. faketime library
LD_PRELOAD=/usr/lib/faketime/libfaketime.so.1 FAKETIME="2024-01-01" ./binary

Signal-Based Anti-Debug

// SIGTRAP handler — INT3 under debugger is caught by debugger, not handler
signal(SIGTRAP, handler);
__asm__("int3");
// If handler runs: no debugger. If debugger catches: debugged.

// SIGALRM timeout — kill self if analysis takes too long
signal(SIGALRM, kill_handler);
alarm(5);

// SIGSEGV handler that does real work (see patterns.md for MBA pattern)
signal(SIGSEGV, real_logic_handler);
*(int*)0 = 0;  // deliberate crash → handler runs real code

Bypasses:

# GDB: pass signals to program instead of handling them
(gdb) handle SIGTRAP nostop pass
(gdb) handle SIGALRM ignore
(gdb) handle SIGSEGV nostop pass

# For alarm-based: patch alarm() to return immediately

Syscall-Level Evasion

// Direct syscall instead of libc — bypasses LD_PRELOAD hooks
long ret;
asm volatile("syscall" : "=a"(ret) : "a"(101), "D"(0), "S"(0), "d"(0), "r"(0));
// Syscall 101 = ptrace on x86_64

Bypass: Must patch the binary itself or use ptrace to intercept at syscall level.

# GDB: catch syscall
(gdb) catch syscall 101
(gdb) commands
> set $rax = 0
> continue
> end

---

Windows Anti-Debug (Advanced)

PEB (Process Environment Block) Checks

// BeingDebugged flag (offset 0x2 in PEB)
bool debugged = NtCurrentPeb()->BeingDebugged;

// NtGlobalFlag (offset 0x68/0xBC in PEB)
// When debugger: FLG_HEAP_ENABLE_TAIL_CHECK | FLG_HEAP_ENABLE_FREE_CHECK | FLG_HEAP_VALIDATE_PARAMETERS = 0x70
DWORD flags = *(DWORD*)((BYTE*)NtCurrentPeb() + 0xBC); // 64-bit offset
if (flags & 0x70) exit(1);

Bypass (x64dbg):

# ScyllaHide plugin auto-patches PEB fields
# Manual: dump PEB, zero BeingDebugged and NtGlobalFlag

NtQueryInformationProcess

// ProcessDebugPort (0x7)
DWORD_PTR debugPort = 0;
NtQueryInformationProcess(GetCurrentProcess(), 7, &debugPort, sizeof(debugPort), NULL);
if (debugPort != 0) exit(1);

// ProcessDebugObjectHandle (0x1E)
HANDLE debugObj = NULL;
NTSTATUS status = NtQueryInformationProcess(GetCurrentProcess(), 0x1E, &debugObj, sizeof(debugObj), NULL);
if (status == 0) exit(1); // STATUS_SUCCESS means debugger present

// ProcessDebugFlags (0x1F) — returns inverse: 0 = debugger present
DWORD noDebug = 0;
NtQueryInformationProcess(GetCurrentProcess(), 0x1F, &noDebug, sizeof(noDebug), NULL);
if (noDebug == 0) exit(1);

Bypass: Hook NtQueryInformationProcess to return fake values, or use ScyllaHide.

Heap Flags

// Process heap has debug flags when debugger attached
PHEAP heap = (PHEAP)GetProcessHeap();
// Flags at offset 0x70 (64-bit): should be HEAP_GROWABLE (0x2)
// ForceFlags at offset 0x74: should be 0
if (heap->Flags != 0x2 || heap->ForceFlags != 0) exit(1);

TLS Callbacks

Key technique: TLS (Thread Local Storage) callbacks execute BEFORE main() / entry point.

// Registered in PE header's TLS directory
void NTAPI TlsCallback(PVOID DllHandle, DWORD Reason, PVOID Reserved) {
    if (Reason == DLL_PROCESS_ATTACH) {
        if (IsDebuggerPresent()) {
            ExitProcess(1);  // Kills process before main runs
        }
    }
}

#pragma comment(linker, "/INCLUDE:_tls_used")
#pragma data_seg(".CRT$XLB")
PIMAGE_TLS_CALLBACK callbacks[] = { TlsCallback, NULL };

Detection in IDA/Ghidra: Check PE TLS Directory → AddressOfCallBacks. Functions listed there run before EP.

Bypass: Set breakpoint on TLS callback in x64dbg (Options → Events → TLS Callbacks), or patch the TLS directory entry.

Hardware Breakpoint Detection

// Read debug registers via GetThreadContext
CONTEXT ctx;
ctx.ContextFlags = CONTEXT_DEBUG_REGISTERS;
GetThreadContext(GetCurrentThread(), &ctx);
if (ctx.Dr0 || ctx.Dr1 || ctx.Dr2 || ctx.Dr3) exit(1);

// Also via exception handler: deliberate exception, check DR regs in handler

Bypass:

# x64dbg: use software breakpoints instead, or hook GetThreadContext
# Frida: hook GetThreadContext to zero DR registers

Software Breakpoint Detection (INT3 Scanning)

// CRC / hash check over code section
unsigned char *code = (unsigned char*)function_addr;
uint32_t checksum = 0;
for (int i = 0; i < code_size; i++) {
    checksum += code[i];
    if (code[i] == 0xCC) exit(1);  // INT3 = software breakpoint
}
if (checksum != EXPECTED_CHECKSUM) exit(1);

Bypass: Use hardware breakpoints (DR0-DR3) instead of software breakpoints. Or hook the scanning function.

Exception-Based Anti-Debug

// UnhandledExceptionFilter — under debugger, filter is NOT called
SetUnhandledExceptionFilter(handler);
RaiseException(EXCEPTION_ACCESS_VIOLATION, 0, 0, NULL);
// If handler runs: no debugger
// If debugger catches: debugger present

// INT 2D — debugger single-step anomaly
__asm { int 2dh }  // Debugger silently consumes the exception
// If execution continues: debugger present

NtSetInformationThread (Thread Hiding)

// Hide thread from debugger — stops all debug events
typedef NTSTATUS(NTAPI *pNtSIT)(HANDLE, ULONG, PVOID, ULONG);
pNtSIT NtSIT = (pNtSIT)GetProcAddress(GetModuleHandle("ntdll"), "NtSetInformationThread");
NtSIT(GetCurrentThread(), 0x11 /*ThreadHideFromDebugger*/, NULL, 0);
// After this, debugger won't see breakpoints or exceptions from this thread

Bypass: Hook NtSetInformationThread to ignore class 0x11, or patch the call.

---

Anti-VM / Anti-Sandbox

CPUID Hypervisor Bit

int regs[4];
__cpuid(regs, 1);
if (regs[2] & (1 << 31)) {  // ECX bit 31 = hypervisor present
    exit(1);
}

// Hypervisor brand string
__cpuid(regs, 0x40000000);
char brand[13] = {0};
memcpy(brand, &regs[1], 12);
// "VMwareVMware", "Microsoft Hv", "KVMKVMKVM", "XenVMMXenVMM"

Bypass: Patch cpuid results or use LD_PRELOAD to hook wrapper functions.

MAC Address / Hardware Fingerprinting

Known VM MAC prefixes:
  VMware:     00:0C:29, 00:50:56
  VirtualBox: 08:00:27
  Hyper-V:    00:15:5D
  Parallels:  00:1C:42
  QEMU:       52:54:00

Timing-Based VM Detection

// VM exits on privileged instructions are measurably slower
uint64_t start = __rdtsc();
__cpuid(regs, 0);  // Forces VM exit
uint64_t delta = __rdtsc() - start;
if (delta > 500) { /* likely VM */ }

File / Registry Artifacts

Files: C:\Windows\System32\drivers\vm*.sys, vbox*.dll, VBoxService.exe
Registry: HKLM\SOFTWARE\VMware, Inc.\VMware Tools
Services: VMTools, VBoxService
Processes: vmtoolsd.exe, VBoxTray.exe, qemu-ga.exe
Linux: /sys/class/dmi/id/product_name contains "VirtualBox"|"VMware"
       dmesg | grep -i "hypervisor detected"

Resource Checks (CPU Count, RAM, Disk)

// Sandboxes typically have minimal resources
SYSTEM_INFO si;
GetSystemInfo(&si);
if (si.dwNumberOfProcessors < 2) exit(1);

MEMORYSTATUSEX ms;
ms.dwLength = sizeof(ms);
GlobalMemoryStatusEx(&ms);
if (ms.ullTotalPhys < 2ULL * 1024 * 1024 * 1024) exit(1); // < 2GB RAM

// Disk size check (< 60GB = sandbox)
GetDiskFreeSpaceEx("C:\\", NULL, &total, NULL);

Bypass: Use a VM configured with adequate resources (4+ CPUs, 8GB+ RAM, 100GB+ disk).

---

Anti-DBI (Dynamic Binary Instrumentation)

Frida Detection

// 1. Check /proc/self/maps for frida-agent
FILE *f = fopen("/proc/self/maps", "r");
while (fgets(line, sizeof(line), f)) {
    if (strstr(line, "frida") || strstr(line, "gadget")) exit(1);
}

// 2. Check for Frida's default port (27042)
int sock = socket(AF_INET, SOCK_STREAM, 0);
struct sockaddr_in addr = {.sin_family=AF_INET, .sin_port=htons(27042), .sin_addr.s_addr=inet_addr("127.0.0.1")};
if (connect(sock, (struct sockaddr*)&addr, sizeof(addr)) == 0) exit(1);

// 3. Check for inline hooks (function prologue modification)
// Compare first bytes of libc functions against expected values
unsigned char *strcmp_bytes = (unsigned char *)strcmp;
if (strcmp_bytes[0] == 0xE9 || strcmp_bytes[0] == 0xFF) exit(1); // JMP = hooked

// 4. Thread name check
// Frida creates threads with names like "gmain", "gdbus", "frida-*"
DIR *dir = opendir("/proc/self/task");
while ((entry = readdir(dir))) {
    char comm_path[256];
    snprintf(comm_path, sizeof(comm_path), "/proc/self/task/%s/comm", entry->d_name);
    // Read comm and check for "gmain", "gdbus"
}

// 5. Named pipe detection (Windows)
// Frida creates \\.\pipe\frida-* named pipes

Frida bypass of Frida detection:

// Hook the detection functions themselves
Interceptor.attach(Module.findExportByName(null, "strstr"), {
    onEnter(args) {
        this.haystack = Memory.readUtf8String(args[0]);
        this.needle = Memory.readUtf8String(args[1]);
    },
    onLeave(retval) {
        if (this.needle && (this.needle.includes("frida") || this.needle.includes("gadget"))) {
            retval.replace(ptr(0)); // Not found
        }
    }
});

// Early Frida load (before anti-DBI runs)
// Use frida-gadget as early-init shared library

Pin/DynamoRIO Detection

// Check for instrumentation libraries in /proc/self/maps
// Pin: "pin-", "pinbin", "pinatrace"
// DynamoRIO: "dynamorio", "drcov", "drrun"

// Instruction count timing — DBI adds overhead
// Execute known instruction sequence, compare execution time

---

Code Integrity / Self-Hashing

// CRC32 over .text section
uint32_t crc = compute_crc32(text_start, text_size);
if (crc != EXPECTED_CRC) exit(1);  // Code was modified (breakpoints, patches)

// MD5/SHA256 of function bodies
unsigned char hash[32];
SHA256(function_addr, function_size, hash);
if (memcmp(hash, expected_hash, 32) != 0) exit(1);

Bypasses: 1. Hardware breakpoints (don't modify code, DR0-DR3) 2. Patch the comparison to always succeed 3. Hook the hash function to return expected value 4. Emulate instead of debug (Unicorn/Qiling — no code modification) 5. Snapshot + restore: dump memory before and after, diff to find checks

Self-checksumming in loops:

// Continuous integrity check in separate thread
void *watchdog(void *arg) {
    while (1) {
        if (compute_crc32(text_start, text_end - text_start) != saved_crc) {
            memset(flag_buffer, 0, flag_len);  // Destroy flag
            exit(1);
        }
        usleep(100000);
    }
}

Bypass: Kill the watchdog thread or patch its sleep to infinite.

---

Anti-Disassembly Techniques

Opaque Predicates

; Condition that always evaluates the same way but looks data-dependent
mov eax, [some_memory]
imul eax, eax          ; x^2
and eax, 1             ; x^2 mod 2 is always 0 for any x
jnz fake_branch        ; Never taken, but disassembler doesn't know
; real code here

Identification: Z3/SMT can prove branch is always/never taken.

Junk Bytes / Overlapping Instructions

jmp real_code
db 0xE8           ; Looks like start of CALL to linear disassembler
real_code:
mov eax, 1        ; Real code — disassembler may misalign here

Fix: Switch to graph-mode disassembly (Ghidra/IDA handle this well). Manual: undefine and re-analyze from correct offset.

Jump-in-the-Middle

; Jumps into the middle of a multi-byte instruction
eb 01          ; jmp +1 (skip next byte)
e8             ; fake CALL opcode — disassembler tries to decode as call
90             ; real: NOP (landed here from jmp)

Function Chunking / Scattered Code

Functions split into non-contiguous chunks connected by unconditional jumps. Defeats linear function boundary detection.

Tool: IDA's "Append function tail" or Ghidra's "Create function" at each chunk.

Control Flow Flattening (Advanced)

Beyond basic switch-case (see patterns.md): modern OLLVM variants use:

• Bogus control flow: Fake branches with opaque predicates
• Instruction substitution: a + b → a - (-b), a ^ b → (a | b) & ~(a & b)
• String encryption: Strings decrypted at runtime, cleared after use

Deobfuscation tools:

• D-810 (IDA plugin): Pattern-based deobfuscation, MBA simplification
• GOOMBA (Ghidra): Automated deobfuscation for OLLVM
• Miasm: Symbolic execution for deobfuscation
• Arybo / SiMBA: MBA expression simplification

# D-810: install in IDA plugins directory, Edit → Plugins → D-810
# Simplifies MBA expressions: (a | b) & ~(a & b) → a ^ b
# Removes opaque predicates via pattern matching

Mixed Boolean-Arithmetic (MBA) Identification & Simplification

# Common MBA patterns and their simplified forms:
# (x & y) + (x | y) == x + y
# (x ^ y) + 2*(x & y) == x + y
# (x | y) - (x & ~y) == y
# ~(~x & ~y) == x | y (De Morgan's)
# (x | y) & ~(x & y) == x ^ y

# SiMBA tool for automated simplification:
# pip install simba-simplifier
from simba import simplify_mba
expr = "(a | b) + (a & b) - (~a & b)"
print(simplify_mba(expr))  # → a

See anti-analysis-ctf.md for CTF writeup techniques: SIGILL handler for mode switching (Hack.lu 2015), SIGFPE strace side-channel (PlaidCTF 2017), instruction trace inversion (MeePwn 2017), call-less function chaining (THC 2018), and parent-patched child binary dump via process_vm_writev (Google CTF Quals 2018).

---

Comprehensive Bypass Strategies

Universal Bypass Checklist

1. Identify all anti-analysis checks — search for: ptrace, IsDebuggerPresent, rdtsc, cpuid, NtQuery, GetTickCount, CheckRemoteDebuggerPresent, /proc/self, SIGTRAP, alarm 2. Static patching — NOP/patch checks with pwntools or Ghidra before running 3. LD_PRELOAD (Linux) — hook libc functions returning fake values 4. ScyllaHide (Windows x64dbg) — patches PEB, hooks NT functions automatically 5. Emulation (Unicorn/Qiling) — no debugger artifacts to detect 6. Kernel-level bypass — modify /proc/sys/kernel/yama/ptrace_scope, use prctl

Layered Anti-Debug (Real-World Pattern)

Many CTF challenges stack multiple checks:

1. TLS callback → IsDebuggerPresent (before main)
2. main() → ptrace(TRACEME)
3. Watchdog thread → timing check + /proc scan
4. Code section → self-CRC32 integrity
5. Signal handler → real logic in SIGSEGV handler

Approach: Identify ALL checks before patching. Patch or hook each one systematically. Run under emulator if too many to patch individually.

Quick Reference: Check to Bypass

---

Trap-Flag Self-Check with cmovz Patcher (Hack.lu 2018)

Pattern: The binary checks EFLAGS by doing pushf; pop edx; and edx, 0x100 (isolating the single-step Trap Flag) and using the result inside a cmovz so the correct instruction is overwritten only when the TF bit is clear. Single-stepping in gdb leaves TF set, the cmovz never fires, and the program silently runs the wrong code path without crashing.

check_debugger:
    pushf
    pop   edx
    and   edx, 0x100          ; Trap Flag only
    test  edx, edx
    cmovz eax, ebx            ; overwrite `eax` only when NOT single-stepping
    mov   [rip+target], eax

Bypass with hardware breakpoints:

(gdb) hbreak *0x56557267       # hardware BP, no INT3, no TF side effect
(gdb) run
(gdb) # inspect EAX at the hbreak — the patched value is now written

Key insight: pushf; pop reg; and reg, 0x100 is the cleanest way to check TF without triggering a trap. Single-stepping changes the visible EFLAGS and can also perturb the instruction pipeline, so software breakpoints plus stepi both poison the check. Hardware breakpoints (hbreak) run the instruction normally and halt afterwards, so the check fires in "not debugging" mode. Apply the same fix to any anti-debug that reads EFLAGS, RFLAGS, or DR6.

References: Hack.lu CTF 2018 — Forgetful Commander, writeup 11858

---

SIGFPE Handler for mprotect Code Mutation (Hack.lu 2018)

Pattern: The binary installs a custom SIGFPE handler with sys_sigaction and arranges for an arithmetic instruction to trap (e.g., div by zero). The handler runs with kernel-delivered context, calls mprotect on the .text page to make it writable, and mutates code that would otherwise stay constant. Standard static analysis misses the mutation because the FPE never fires in the normal path, and standard dynamic analysis misses it because most debuggers intercept SIGFPE before delivery.

// Handler installed at startup
void on_fpe(int sig, siginfo_t *info, void *uap) {
    ucontext_t *ctx = uap;
    void *page = (void *)((uintptr_t)ctx->uc_mcontext.gregs[REG_RIP] & ~0xfff);
    mprotect(page, 0x1000, PROT_READ | PROT_WRITE | PROT_EXEC);
    // Patch a constant used by the next check
    *((uint32_t *)(page + 0x42)) = 0xDEADBEEF;
}

Bypass:

# Let SIGFPE reach the program, not the debugger
gdb ./challenge
(gdb) handle SIGFPE nostop noprint pass
(gdb) break *on_fpe
(gdb) run

Key insight: Signal handlers that mprotect + mutate code are cross-delimited in a way decompilers cannot model. SIGFPE is particularly effective because it is rarely raised during normal execution, so the mutation stays dormant until the attacker hits the crafted input. When you see sys_sigaction(SIGFPE,...) or signal(SIGFPE,...) in a binary that also calls mprotect, trace the handler with strace -e signal=SIGFPE and annotate the mutated region by diffing the .text pages before and after the first FPE.

References: Hack.lu CTF 2018 — Cheat Console, writeup 11868

Reverse Engineering Field Notes

Detailed quick notes that support `SKILL.md`. Read this file after triage, not before.

Table of Contents

• Binary Types
• Python .pyc
• WASM
• Android APK
• Flutter APK (Dart AOT)
• .NET
• Packed (UPX)
• Tauri Packed Desktop Apps
• Anti-Debugging Bypass
• Specialized Patterns
• S-Box / Keystream Patterns
• Custom VM Analysis
• Python Bytecode Reversing
• Signal-Based Binary Exploration
• Malware Anti-Analysis Bypass via Patching
• Expected Values Tables
• x86-64 Gotchas
• Iterative Solver Pattern
• Unicorn Emulation (Complex State)
• Multi-Stage Shellcode Loaders
• Timing Side-Channel Attack
• Godot Game Asset Extraction
• Roblox Place File Analysis
• Unstripped Binary Information Leaks
• Custom Mangle Function Reversing
• Rust serde_json Schema Recovery
• Position-Based Transformation Reversing
• Hex-Encoded String Comparison
• CTF Case Notes
• Embedded ZIP + XOR License Decryption
• Stack String Deobfuscation (.rodata XOR Blob)
• Prefix Hash Brute-Force
• Mathematical Convergence Bitmap
• RISC-V Binary Analysis
• Sprague-Grundy Game Theory Binary
• Kernel Module Maze Solving
• Multi-Threaded VM with Channels
• CVP/LLL Lattice for Constrained Integer Validation
• Decision Tree Function Obfuscation
• Android JNI RegisterNatives Obfuscation
• Multi-Layer Self-Decrypting Binary
• GLSL Shader VM with Self-Modifying Code
• GF(2^8) Gaussian Elimination for Flag Recovery
• Z3 for Single-Line Python Boolean Circuit
• Sliding Window Popcount Differential Propagation
• Ruby/Perl Polyglot Constraint Satisfaction
• Verilog/Hardware RE
• Custom binfmt Kernel Module with RC4 Flat Binaries
• Hash-Resolved Imports / No-Import Ransomware
• ELF Section Header Corruption for Anti-Analysis
• Brainfuck Character-by-Character Static Analysis
• Brainfuck Side-Channel via Read Count Oracle
• Brainfuck Comparison Idiom Detection
• Backdoored Shared Library Detection
• Go Binary Reversing
• Go Binary UUID Patching for C2 Enumeration
• D Language Binary Reversing
• Rust Binary Reversing
• Frida Dynamic Instrumentation
• Frida Firebase Cloud Functions Bypass
• angr Symbolic Execution
• Qiling Emulation
• VMProtect / Themida Analysis
• Binary Diffing
• Advanced GDB (pwndbg, rr)
• macOS / iOS Reversing
• Embedded / IoT Firmware RE
• Kernel Driver Reversing
• Game Engine Reversing
• Swift / Kotlin Binary Reversing
• INT3 Patch + Coredump Brute-Force Oracle
• Signal Handler Chain + LD_PRELOAD Oracle
• Font Ligature Exploitation
• Instruction Counter as Cryptographic State
• Burrows-Wheeler Transform Inversion
• FRACTRAN Program Inversion
• Opcode-Only Trace Reconstruction
• Thread Race Signed Integer Overflow
• ESP32/Xtensa Firmware Reversing
• Custom VM Bytecode Lifting to LLVM IR
• SIGFPE Signal Handler Side-Channel
• Batch Crackme Automation via objdump
• Android DEX Runtime Bytecode Patching
• Fork + Pipe + Dead Branch Anti-Analysis

Binary Types

Python .pyc

Disassemble with marshal.load() + dis.dis(). Header: 8 bytes (2.x), 12 (3.0-3.6), 16 (3.7+). See languages.md.

WASM

wasm2c checker.wasm -o checker.c
gcc -O3 checker.c wasm-rt-impl.c -o checker

# WASM patching (game challenges):
wasm2wat main.wasm -o main.wat    # Binary → text
# Edit WAT: flip comparisons, change constants
wat2wasm main.wat -o patched.wasm # Text → binary

WASM game patching (Tac Tic Toe, Pragyan 2026): If proof generation is independent of move quality, patch minimax (flip i64.lt_s → i64.gt_s, change bestScore sign) to make AI play badly while proofs remain valid. Invoke /ctf-misc for full game patching patterns (games-and-vms).

Android APK

apktool d app.apk -o decoded/ for resources; jadx app.apk for Java decompilation. Check decoded/res/values/strings.xml for flags. See tools.md.

Flutter APK (Dart AOT)

If lib/arm64-v8a/libapp.so + libflutter.so present, use Blutter: python3 blutter.py path/to/app/lib/arm64-v8a out_dir. Outputs reconstructed Dart symbols + Frida script. See tools.md.

.NET

• dnSpy - debugging + decompilation
• ILSpy - decompiler

Packed (UPX)

upx -d packed -o unpacked

If unpacking fails, inspect UPX metadata first: verify UPX section names, header fields, and version markers are intact. If metadata looks tampered or uncertain, review UPX source on GitHub to identify likely modification points.

Tauri Packed Desktop Apps

Tauri embeds Brotli-compressed frontend assets in the executable. Find index.html xrefs to locate asset index table, dump blobs, Brotli decompress. Reference: tauri-codegen/src/embedded_assets.rs.

Anti-Debugging Bypass

Common checks:

• IsDebuggerPresent() / PEB.BeingDebugged / NtQueryInformationProcess (Windows)
• ptrace(PTRACE_TRACEME) / /proc/self/status TracerPid (Linux)
• TLS callbacks (run before main — check PE TLS Directory)
• Timing checks (rdtsc, clock_gettime, GetTickCount)
• Hardware breakpoint detection (DR0-DR3 via GetThreadContext)
• INT3 scanning / code self-hashing (CRC over .text section)
• Signal-based: SIGTRAP handler, SIGALRM timeout, SIGSEGV for real logic
• Frida/DBI detection: /proc/self/maps scan, port 27042, inline hook checks

Bypass: Set breakpoint at check, modify register to bypass conditional. pwntools patch: elf.asm(elf.symbols.ptrace, 'ret') to replace function with immediate return. See patterns.md.

For comprehensive anti-analysis techniques and bypasses (30+ methods with code), see anti-analysis.md.

Specialized Patterns

S-Box / Keystream Patterns

Xorshift32: Shifts 13, 17, 5 Xorshift64: Shifts 12, 25, 27 Magic constants: 0x2545f4914f6cdd1d, 0x9e3779b97f4a7c15

Custom VM Analysis

1. Identify structure: registers, memory, IP 2. Reverse executeIns for opcode meanings 3. Write disassembler mapping opcodes to mnemonics 4. Often easier to bruteforce than fully reverse 5. Look for the bytecode file loaded via command-line arg

See patterns.md for VM workflow, opcode tables, and state machine BFS.

Sequential key-chain brute-force: When a VM validates input in small blocks (e.g., 3 bytes = 2^24 candidates) with each block's output key feeding the next, brute-force each block sequentially with OpenMP parallelization. Compile solver with gcc -O3 -march=native -fopenmp. See patterns-ctf-3.md.

Python Bytecode Reversing

XOR flag checkers with interleaved even/odd tables are common. See languages.md for bytecode analysis tips and reversing patterns.

Signal-Based Binary Exploration

Binary uses UNIX signals as binary tree navigation; hook sigaction via LD_PRELOAD, DFS by sending signals. See patterns.md.

Malware Anti-Analysis Bypass via Patching

Flip JNZ/JZ (0x75/0x74), change sleep values, patch environment checks in Ghidra (Ctrl+Shift+G). See patterns-runtime.md.

Expected Values Tables

Locate with objdump -s -j .rodata binary | less — look near comparison instructions, size matches flag length.

x86-64 Gotchas

Sign extension and 32-bit truncation pitfalls. See patterns.md for details and code examples.

Iterative Solver Pattern

Try each byte (0-255) per position, match against expected output. Uniform transform shortcut: if one input byte only changes one output byte, build 0..255 mapping then invert. See patterns.md for full implementation.

Unicorn Emulation (Complex State)

from unicorn import * -- map segments, set up stack, hook to trace. Mixed-mode pitfall: 64-bit stub jumping to 32-bit via retf requires switching to UC_MODE_32 and copying GPRs + EFLAGS + XMM regs. See tools.md.

Multi-Stage Shellcode Loaders

Nested shellcode with XOR decode loops; break at call rax, bypass ptrace with set $rax=0, extract flag from mov instructions. See patterns-runtime.md.

Timing Side-Channel Attack

Validation time varies per correct character; measure elapsed time per candidate to recover flag byte-by-byte. See patterns-runtime.md.

Godot Game Asset Extraction

Use KeyDot to extract encryption key from executable, then gdsdecomp to extract .pck package. See languages-platforms.md.

Roblox Place File Analysis

Query Asset Delivery API for version history; parse .rbxlbin chunks (INST/PROP/PRNT) to diff script sources across versions. See languages-platforms.md.

Unstripped Binary Information Leaks

Pattern: Debug info and file paths leak author identity. Quick checks: strings binary | grep "/home/" (home dirs), file binary (stripped?), readelf -S binary | grep debug (debug sections).

Custom Mangle Function Reversing

Binary mangles input 2 bytes at a time with running state; extract target from .rodata, write inverse function. See patterns.md.

Rust serde_json Schema Recovery

Disassemble serde Visitor implementations to recover expected JSON schema; field names in order reveal flag. See languages-platforms.md.

Position-Based Transformation Reversing

Binary adds/subtracts position index; reverse by undoing per-index offset. See patterns.md.

Hex-Encoded String Comparison

Input converted to hex, compared against constant. Decode with xxd -r -p. See patterns.md.

CTF Case Notes

Embedded ZIP + XOR License Decryption

Binary with named symbols (EMBEDDED_ZIP, ENCRYPTED_MESSAGE) in .rodata → extract ZIP containing license, XOR encrypted message with license bytes to recover flag. No execution needed. See patterns-ctf-2.md.

Stack String Deobfuscation (.rodata XOR Blob)

Binary mmaps .rodata blob, XOR-deobfuscates, uses it to validate input. Reimplement verification loop with pyelftools to extract blob. Look for 0x9E3779B9, 0x85EBCA6B constants and rol32(). See patterns-ctf-2.md.

Prefix Hash Brute-Force

Binary hashes every prefix independently. Recover one character at a time by matching prefix hashes. See patterns-ctf-2.md.

Mathematical Convergence Bitmap

Pattern: Binary classifies coordinate pairs by Newton's method convergence (e.g., z^3-1=0). Grid of pass/fail results renders ASCII art flag. Key: the binary is a classifier, not a checker — reverse the math and visualize. See patterns-ctf.md.

RISC-V Binary Analysis

Statically linked, stripped RISC-V ELF. Use Capstone with CS_MODE_RISCVC | CS_MODE_RISCV64 for mixed compressed instructions. Emulate with qemu-riscv64. Watch for fake flags and XOR decryption with incremental keys. See tools.md.

Sprague-Grundy Game Theory Binary

Game binary plays bounded Nim with PRNG for losing-position moves. Identify game framework (Grundy values = pile % (k+1), XOR determines position), track PRNG state evolution through user input feedback. See patterns-ctf.md.

Kernel Module Maze Solving

Rust kernel module implements maze via device ioctls. Enumerate commands dynamically, build DFS solver with decoy avoidance, deploy as minimal static binary (raw syscalls, no libc). See patterns-ctf.md.

Multi-Threaded VM with Channels

Custom VM with 16+ threads communicating via futex channels. Trace data flow across thread boundaries, extract constants from GDB, watch for inverted validity logic, solve via BFS state space search. See patterns-ctf.md.

CVP/LLL Lattice for Constrained Integer Validation

Binary validates flag via matrix multiplication with 64-bit coefficients; solutions must be printable ASCII. Use LLL reduction + CVP in SageMath to find nearest lattice point in the constrained range. Two-phase pattern: Phase 1 recovers AES key, Phase 2 decrypts custom VM bytecode with another linear system (mod 2^32). See patterns-ctf-2.md.

Decision Tree Function Obfuscation

~200+ auto-generated functions routing input through polynomial comparisons. Script extraction via Ghidra headless rather than reversing each function manually. Constraint propagation from known output format cascades through arithmetic constraints. See patterns-ctf-2.md.

Android JNI RegisterNatives Obfuscation

RegisterNatives in JNI_OnLoad hides which C++ function handles each Java native method (no standard Java_com_pkg_Class_method symbol). Find the real handler by tracing JNI_OnLoad → RegisterNatives → fnPtr. Use x86_64 .so from APK for best Ghidra decompilation. See languages-platforms.md.

Multi-Layer Self-Decrypting Binary

N-layer binary where each layer decrypts the next using user-provided key bytes + SHA-NI. Use oracle (correct key → valid code with expected pattern). JIT execution with fork-per-candidate COW isolation for speed. See patterns-ctf-2.md.

GLSL Shader VM with Self-Modifying Code

Pattern: WebGL2 fragment shader implements Turing-complete VM on a 256x256 RGBA texture (program memory + VRAM). Self-modifying code (STORE opcode) patches drawing instructions. GPU parallelism causes write conflicts — emulate sequentially in Python to recover full output. See patterns-ctf-3.md.

GF(2^8) Gaussian Elimination for Flag Recovery

Pattern: Binary performs Gaussian elimination over GF(2^8) with the AES polynomial (0x11b). Matrix + augmentation vector in .rodata; solution vector is the flag. Look for constant 0x1b in disassembly. Addition is XOR, multiplication uses polynomial reduction. See patterns-ctf-2.md.

Z3 for Single-Line Python Boolean Circuit

Pattern: Single-line Python (2000+ semicolons) with walrus operator chains validates flag as big-endian integer via boolean circuit. Obfuscated XOR (a | b) & ~(a & b). Split on semicolons, translate to Z3 symbolically, solve in under a second. See patterns-ctf-3.md.

Sliding Window Popcount Differential Propagation

Pattern: Binary validates input via expected popcount for each position of a 16-bit sliding window. Popcount differences create a recurrence: bit[i+16] = bit[i] + (data[i+1] - data[i]). Brute-force ~4000-8000 valid initial 16-bit windows; each determines the entire bit sequence. See patterns-ctf-3.md.

Ruby/Perl Polyglot Constraint Satisfaction

Pattern: Single file valid in both Ruby and Perl, each imposing different constraints on a key. Exploits =begin/=end (Ruby block comment) vs =begin/=cut (Perl POD) to run different code per interpreter. Intersect constraints from both languages to recover the unique key. See languages-platforms.md.

Verilog/Hardware RE

Pattern: Verilog HDL source for state machines with hidden conditions gated on shift register history. Analyze always @(posedge clk) blocks and case statements to find correct input sequences. See languages-platforms.md.

Custom binfmt Kernel Module with RC4 Flat Binaries

Pattern: Kernel module registers binfmt handler for encrypted flat binaries. Reverse the .ko to find RC4 key (in movabs immediates), decrypt the flat binary, import at the fixed virtual address from the module's vm_mmap call. See patterns-ctf.md.

Hash-Resolved Imports / No-Import Ransomware

Pattern: Binary with zero visible imports resolves APIs via symbol name hashing at runtime. Skip the hash reversing — hook OpenSSL functions via LD_PRELOAD in Docker to capture AES keys directly. See patterns-ctf.md.

ELF Section Header Corruption for Anti-Analysis

Pattern: Corrupted section headers crash analysis tools but program headers are intact so binary runs normally. Patch e_shoff to zero or use readelf -l (program headers only). Flag hidden after corrupted sections with magic marker + XOR. See patterns-ctf.md.

Brainfuck Character-by-Character Static Analysis

Pattern: BF programs validating input have , (read char) followed by + operations whose count = expected ASCII value. Extract increment counts per input position to recover expected input without execution. See languages.md.

Brainfuck Side-Channel via Read Count Oracle

Pattern: BF input validators read more bytes when a character is correct. Count , operations per candidate — highest read count = correct byte. Character-by-character recovery. See languages.md.

Brainfuck Comparison Idiom Detection

Pattern: Compiled BF uses fixed idioms for equality checks (<[-<->] +<[>-<[-]]>[-<+>]). Instrument interpreter to detect patterns and extract comparison operands (expected flag bytes). See languages.md.

Backdoored Shared Library Detection

Binary works in GDB but fails when run normally (suid)? Check ldd for non-standard libc paths, then strings | diff the suspicious vs. system library to find injected code/passwords. See patterns-ctf.md.

Go Binary Reversing

Large static binary with go.buildid? Use GoReSym to recover function names (works even on stripped binaries). Go strings are {ptr, len} pairs — not null-terminated. Look for main.main, runtime.gopanic, channel ops (runtime.chansend1/chanrecv1). Use Ghidra golang-loader plugin for best results. See languages-compiled.md.

Go Binary UUID Patching for C2 Enumeration

Pattern: Go C2 client with UUID from -ldflags -X. Binary-patch UUID bytes (same length), register with C2, enumerate clients/files via API. See languages-compiled.md.

D Language Binary Reversing

D language binaries have unique symbol mangling (not C++ style). Template-heavy, many function variants. Look for _D prefix in symbols. See languages-compiled.md.

Rust Binary Reversing

Binary with core::panicking strings and _ZN mangled symbols? Use rustfilt for demangling. Panic messages contain source paths and line numbers — strings binary | grep "panicked" is the fastest approach. Option/Result enums use discriminant byte (0=None/Err, 1=Some/Ok). See languages-compiled.md.

Frida Dynamic Instrumentation

Hook runtime functions without modifying binary. frida -f ./binary -l hook.js to spawn with instrumentation. Hook strcmp/memcmp to capture expected values, bypass anti-debug by replacing ptrace return value, scan memory for flag patterns, replace validation functions. See tools-dynamic.md.

Frida Firebase Cloud Functions Bypass

Pattern: Android app validates via Firebase Cloud Functions. Post-login Frida hook constructs valid payload (UID + value + timestamp) and calls Cloud Function directly, bypassing QR/payment validation. See languages-platforms.md.

angr Symbolic Execution

Automatic path exploration to find inputs satisfying constraints. Load binary with angr.Project, set find/avoid addresses, call simgr.explore(). Constrain input to printable ASCII and known prefix for faster solving. Hook expensive functions (crypto, I/O) to prevent path explosion. See tools-dynamic.md.

Qiling Emulation

Cross-platform binary emulation with OS-level support (syscalls, filesystem). Emulate Linux/Windows/ARM/MIPS binaries on any host. No debugger artifacts — bypasses all anti-debug by default. Hook syscalls and addresses with Python API. See tools-dynamic.md.

VMProtect / Themida Analysis

VMProtect virtualizes code into custom bytecode. Identify VM entry (pushad-like), find handler table (large indirect jump), trace handlers dynamically. For CTF, focus on tracing operations on input rather than full devirtualization. Themida: dump at OEP with ScyllaHide + Scylla. See tools-advanced.md.

Binary Diffing

BinDiff and Diaphora compare two binaries to highlight changes. Essential when challenge provides patched/original versions. Export from IDA/Ghidra, diff to find vulnerability or hidden functionality. See tools-advanced.md.

Advanced GDB (pwndbg, rr)

pwndbg: context, vmmap, search -s "flag{", telescope $rsp. GEF alternative. Reverse debugging with rr record/rr replay — step backward through execution. Python scripting for brute-force and automated tracing. See tools-advanced-2.md.

macOS / iOS Reversing

Mach-O binaries: otool -l for load commands, class-dump for Objective-C headers. Swift: swift demangle for symbols. iOS apps: decrypt FairPlay DRM with frida-ios-dump, bypass jailbreak detection with Frida hooks. Re-sign patched binaries with codesign -f -s -. See platforms.md.

Embedded / IoT Firmware RE

binwalk -Me firmware.bin for recursive extraction. Hardware: UART/JTAG/SPI flash for firmware dumps. Filesystems: SquashFS (unsquashfs), JFFS2, UBI. Emulate with QEMU: qemu-arm -L /usr/arm-linux-gnueabihf/ ./binary. See platforms.md.

Kernel Driver Reversing

Linux .ko: find ioctl handler via file_operations struct, trace copy_from_user/copy_to_user. Debug with QEMU+GDB (-s -S). eBPF: bpftool prog dump xlated. Windows .sys: find DriverEntry → IoCreateDevice → IRP handlers. See platforms.md.

Game Engine Reversing

Unreal: extract .pak with UnrealPakTool, reverse Blueprint bytecode with FModel. Unity Mono: decompile Assembly-CSharp.dll with dnSpy. Anti-cheat (EAC, BattlEye, VAC): identify system, bypass specific check. Lua games: luadec/unluac for bytecode. See platforms.md.

Swift / Kotlin Binary Reversing

Swift: swift demangle symbols, protocol witness tables for dispatch, __swift5_* sections. Kotlin/JVM: coroutines compile to state machines in invokeSuspend, jadx with Kotlin mode for best decompilation. Kotlin/Native: LLVM backend, looks like C++ in disassembly. See languages-compiled.md.

INT3 Patch + Coredump Brute-Force Oracle

Patch 0xCC (INT3) after transform output, enable core dumps, brute-force each input character by extracting computed state from coredump via strings. Avoids full reverse of transformation. See patterns.md.

Signal Handler Chain + LD_PRELOAD Oracle

Binary uses signal handler chains for per-character password validation. Hook signal() via LD_PRELOAD -- the call to install the next handler confirms the current character is correct. See patterns.md.

Font Ligature Exploitation

Custom OpenType font maps multi-character ligature sequences to single glyphs; reverse the GSUB table to decode hidden messages. See patterns-ctf-3.md.

Instruction Counter as Cryptographic State

Pattern: Hand-written assembly uses a dedicated register (e.g., r12) as an instruction counter incremented after nearly every instruction. The counter feeds into XOR/ROL/multiply transformations on input bytes, making transformation path-dependent. Byte-by-byte brute force with Unicorn emulation recovers the flag. See patterns-ctf-3.md.

Burrows-Wheeler Transform Inversion

Invert BWT without terminator character by trying all possible row indices. Standard bwtool or manual column-sorting reconstruction. See patterns-ctf-3.md.

FRACTRAN Program Inversion

Esoteric language using iterated fraction multiplication. Invert by swapping numerator/denominator in fraction table, run output backward. I/O encoded as prime factorization exponents. See languages.md.

Opcode-Only Trace Reconstruction

Execution traces with only opcodes (no data) still leak info through branch decisions. Sorting algorithm comparisons reveal element ordering. Reconstruct by deduplicating trace, splitting into basic blocks. See tools-dynamic.md.

Thread Race Signed Integer Overflow

Game binary with thread-unsafe skill lock. Race between skill selection and damage calculation; cdqe sign-extends 0xFFFFFFFF to -1 (signed), causing HP overflow on subtraction. See patterns-ctf-3.md.

ESP32/Xtensa Firmware Reversing

No IDA support — use radare2 + ESP-IDF ROM linker script (esp32.rom.ld) for symbol resolution. Cross-reference with public ESP-IDF HTTP server examples to identify app logic. See patterns-ctf-3.md.

Custom VM Bytecode Lifting to LLVM IR

Transpile custom VM bytecode to LLVM IR, then use opt -O3 to simplify (inlining, constant folding, dead code elimination). Reduces 1300 lines to ~150 lines, revealing the underlying algorithm. See tools-advanced.md.

SIGFPE Signal Handler Side-Channel

SIGFPE signal handlers create implicit control flow invisible to static analysis. Count SIGFPE signals via strace -e signal=SIGFPE per candidate character -- correct characters produce more signals. See anti-analysis.md.

Batch Crackme Automation via objdump

Mass crackme challenges (100s of binaries) with identical structure: script objdump to extract CMP immediates and add/sub arithmetic sequences, then reverse-compute keys algebraically without execution. See patterns-ctf-3.md.

Android DEX Runtime Bytecode Patching

Native JNI library patches Dalvik bytecode in memory via /proc/self/maps + mprotect + XOR. Static APK analysis alone is insufficient -- extract XOR key and offsets from the native .so to reconstruct the runtime DEX. See languages-platforms.md.

Fork + Pipe + Dead Branch Anti-Analysis

Fork/pipe IPC where parent writes data and exits, child reads and continues. Real validation hidden in a dead branch (always-false comparison). strace reveals the fork/pipe pattern; patch the comparison constant to reach hidden code. See patterns-ctf-3.md.

CTF Reverse - Compiled Language Reversing (Go, Rust)

Table of Contents

• Go Binary Reversing
• Recognition
• Symbol Recovery
• Go Memory Layout
• Goroutine and Concurrency Analysis
• Common Go Patterns in Decompilation
• Go Binary Reversing Workflow
• Go Binary UUID Patching for C2 Client Enumeration (BSidesSF 2026)
• Rust Binary Reversing
• Rust Recognition
• Symbol Demangling
• Common Rust Patterns in Decompilation
• Rust-Specific Analysis Tools
• Rust Lifetime Escape via Compiler Bug #25860 (Hack.lu 2018)
• [Rust #[no_mangle] libc Override for seccomp Bypass (Hack.lu 2018)](#rust-no_mangle-libc-override-for-seccomp-bypass-hacklu-2018)
• Rust xmmword Constant Extraction via IDAPython (Insomnihack 2019)
• Nuitka-Compiled Python — Module Stub Injection (X-MAS CTF 2018)
• Swift Binary Reversing
• Kotlin / JVM Binary Reversing
• JVM Bytecode (Android/Server)
• Kotlin/Native
• D Language Binary Reversing (CSAW CTF 2016)
• Haskell Binary Reversing via STG Closures and hsdecomp (hxp CTF 2017, Codegate 2018)
• Haskell Binary RE via GHC CMM Intermediate Language (N1CTF 2018)
• C++ Binary Reversing (Quick Reference)
• vtable Reconstruction
• RTTI (Run-Time Type Information)
• Standard Library Patterns

---

Go Binary Reversing

Go binaries are increasingly common in CTF challenges due to Go's popularity for CLI tools, network services, and malware.

Recognition

# Detect Go binary
file binary | grep -i "go"
strings binary | grep "go.buildid"
strings binary | grep "runtime.gopanic"

# Go version embedded in binary
strings binary | grep "^go1\."

Key indicators:

• Very large static binary (even "hello world" is ~2MB)
• Embedded go.buildid string
• runtime.* symbols (even in stripped binaries, some remain)
• main.main as entry point (not main)
• Strings like GOROOT, GOPATH, /usr/local/go/src/

Symbol Recovery

Go embeds rich type and function information even in stripped binaries:

# GoReSym - recovers function names, types, interfaces from Go binaries
# https://github.com/mandiant/GoReSym
./GoReSym -d binary > symbols.json

# Parse output
python3 -c "
import json
with open('symbols.json') as f:
    data = json.load(f)
for fn in data.get('UserFunctions', []):
    print(f\"{fn['Start']:#x}  {fn['FullName']}\")
"

Ghidra with golang-loader:

# Install: Ghidra → Window → Script Manager → search "golang"
# Or use: https://github.com/getCUJO/ThreatFox/tree/main/ghidra-golang
# Recovers function names, string references, interface tables

redress (Go binary analysis):

# https://github.com/goretk/redress
redress -src binary         # Reconstruct source tree
redress -pkg binary         # List packages
redress -type binary        # List types and methods
redress -interface binary   # List interfaces

Go Memory Layout

Understanding Go's data structures in decompilation:

# String: {pointer, length} (16 bytes on 64-bit)
# NOT null-terminated! Length field is critical.
struct GoString {
    char *ptr;    // pointer to UTF-8 data
    int64 len;    // byte length
};

# Slice: {pointer, length, capacity} (24 bytes on 64-bit)
struct GoSlice {
    void *ptr;    // pointer to backing array
    int64 len;    // current length
    int64 cap;    // allocated capacity
};

# Interface: {type_descriptor, data_pointer} (16 bytes)
struct GoInterface {
    void *type;   // points to type metadata (itab for non-empty interface)
    void *data;   // points to actual value
};

# Map: pointer to runtime.hmap struct
# Channel: pointer to runtime.hchan struct

In Ghidra/IDA: When you see a function taking (ptr, int64) — it's likely a Go string. Three-field (ptr, int64, int64) is a slice.

Goroutine and Concurrency Analysis

# Identify goroutine spawns in disassembly
strings binary | grep "runtime.newproc"
# newproc1 is the internal goroutine creation function

# In GDB with Go support:
gdb ./binary
(gdb) source /usr/local/go/src/runtime/runtime-gdb.py
(gdb) info goroutines          # List all goroutines
(gdb) goroutine 1 bt          # Backtrace for goroutine 1

Channel operations in disassembly:

• runtime.chansend1 → ch <- value
• runtime.chanrecv1 → value = <-ch
• runtime.selectgo → select { case ... }
• runtime.closechan → close(ch)

Common Go Patterns in Decompilation

Defer mechanism:

• runtime.deferproc → registers deferred function
• runtime.deferreturn → executes deferred functions at function exit
• Deferred calls execute in LIFO order — relevant for cleanup/crypto key wiping

Error handling (the `if err != nil` pattern):

# In disassembly, this appears as:
# call some_function        → returns (result, error) as two values
# test rax, rax             → check if error (second return value) is nil
# jne error_handler

String concatenation:

• runtime.concatstrings → s1 + s2 + s3
• fmt.Sprintf → formatted string building
• Look for format strings in .rodata: "%s%d", "%x"

Common stdlib patterns in CTF:

// Crypto operations → look for these in strings/imports:
// "crypto/aes", "crypto/cipher", "crypto/sha256", "encoding/hex", "encoding/base64"

// Network operations:
// "net/http", "net.Dial", "bufio.NewReader"

// File operations:
// "os.Open", "io.ReadAll", "os.ReadFile"

Go Binary Reversing Workflow

1. file binary                          # Confirm Go, get arch
2. GoReSym -d binary > syms.json       # Recover symbols
3. strings binary | grep -i flag        # Quick win check
4. Load in Ghidra with golang-loader    # Apply recovered symbols
5. Find main.main                       # Entry point
6. Identify string comparisons          # GoString {ptr, len} pairs
7. Trace crypto operations              # crypto/* package usage
8. Check for embedded resources         # embed.FS in Go 1.16+

Go embed.FS (Go 1.16+): Binaries can embed files at compile time:

# Look for embedded file data
strings binary | grep "embed"
# Embedded files appear as raw data in the binary
# Search for known file signatures (PK for zip, PNG header, etc.)

Key insight: Go's runtime embeds extensive metadata even in stripped binaries. Use GoReSym before any manual analysis — it often recovers 90%+ of function names, making decompilation dramatically easier. Go strings are {ptr, len} tuples, not null-terminated — Ghidra's default string analysis will miss them without the golang-loader plugin.

Detection: Large static binary (2MB+ for simple programs), go.buildid, runtime.gopanic, source paths like /home/user/go/src/.

Go Binary UUID Patching for C2 Client Enumeration (BSidesSF 2026)

Pattern (see-two): A Go-compiled C2 client has a UUID embedded via -ldflags -X. The C2 server uses mTLS for authentication. To enumerate other clients and their files, patch the UUID to register as a new client, then use the C2 API to list all clients and download their exfiltrated files.

Approach: 1. Extract embedded UUID from Go build metadata: go version -m client_binary 2. Binary-patch the UUID (simple byte replacement — Go strings have fixed-length backing arrays) 3. Register with the C2 server using the patched binary (mTLS certs are embedded or in distfiles) 4. Enumerate clients via API: GET /api/clients or iterate known endpoints 5. List and download files from each client's GCS bucket or file store 6. Grep downloaded files for the flag

# Extract Go build info
go version -m ./client_binary | grep ldflags
# Output shows: -X main.clientUUID=<uuid>

# Patch UUID in binary (replace old UUID bytes with new UUID)
python3 -c "
import sys
data = open('client_binary', 'rb').read()
old_uuid = b'original-uuid-value-here'
new_uuid = b'attacker-uuid-value-here'
patched = data.replace(old_uuid, new_uuid)
open('client_patched', 'wb').write(patched)
"
chmod +x client_patched
./client_patched --register

Key insight: Go binaries embed string values from -ldflags -X directly in the binary data section. Since Go strings are {ptr, len} pairs pointing to backing byte arrays, replacing the UUID bytes (same length) produces a valid patched binary. The mTLS certificates authenticate the client to the server but don't bind to a specific UUID.

References: BSidesSF 2026 "see-two"

---

Rust Binary Reversing

Rust binaries are common in modern CTFs, especially for crypto, systems, and security tooling challenges.

Rust Recognition

# Detect Rust binary
strings binary | grep -c "rust"
strings binary | grep "rustc"             # Compiler version
strings binary | grep "/rustc/"           # Source paths
strings binary | grep "core::panicking"   # Panic infrastructure

Key indicators:

• core::panicking::panic in strings
• Mangled symbols starting with _ZN (Itanium ABI) — e.g., _ZN4main4main17h...
• .rustc section in ELF
• References to /rustc/<commit_hash>/library/
• Large binary size (Rust statically links by default)

Symbol Demangling

# Rust uses Itanium ABI mangling (same as C++)
# rustfilt demangles Rust-specific symbols
cargo install rustfilt
nm binary | rustfilt | grep "main"

# Or use c++filt (works for most Rust symbols)
nm binary | c++filt | grep "main"

# In Ghidra: Window → Script Manager → search "Demangler"
# Enable "DemangleAllScript" for automatic demangling

Common Rust Patterns in Decompilation

Option/Result enum:

# Option<T> in memory: {discriminant (0=None, 1=Some), value}
# Result<T, E>: {discriminant (0=Ok, 1=Err), union{ok_val, err_val}}

# In disassembly:
# cmp byte [rbp-0x10], 0    → check if None/Err
# je handle_none_case

Vec<T> (same as Go slice):

struct RustVec {
    void *ptr;      // heap pointer
    uint64 cap;     // capacity
    uint64 len;     // length
};

String / &str:

# String (owned): {ptr, capacity, length} — 24 bytes, heap-allocated
# &str (borrowed): {ptr, length} — 16 bytes, can point anywhere

# In decompilation, look for:
# alloc::string::String::from    → String creation
# core::str::from_utf8           → byte slice to str

Iterator chains:

# .iter().map().filter().collect() compiles to loop fusion
# In disassembly: tight loop with inlined closures
# Look for: core::iter::adapters::map, filter, etc.

Panic unwinding:

# Panic strings reveal source locations and error messages
strings binary | grep "panicked at"
strings binary | grep "called .unwrap().. on"
# These often contain file paths, line numbers, and variable names

Rust-Specific Analysis Tools

# cargo-bloat: analyze binary size by function
cargo install cargo-bloat
cargo bloat --release -n 50

# Ghidra Rust helper scripts
# https://github.com/AmateursCTF/ghidra-rust (community scripts for Rust RE)

Key insight: Rust panic messages are goldmines — they contain source file paths, line numbers, and descriptive error strings even in release builds. Always strings binary | grep "panicked" first. Rust's monomorphization means generic functions get duplicated per type — expect many similar-looking functions.

Detection: core::panicking, .rustc section, /rustc/ paths, _ZN mangled symbols with Rust-style module paths.

---

Swift Binary Reversing

See platforms.md for full Swift reversing guide including demangling, runtime structures, and Ghidra integration. Key quick reference:

# Detect Swift binary
strings binary | grep "swift"
otool -l binary | grep "swift"

# Demangle Swift symbols
swift demangle 's14MyApp0A8ClassC10checkInput6resultSbSS_tF'
# → MyApp.MyAppClass.checkInput(result: String) -> Bool

# Key runtime functions: swift_allocObject, swift_release, swift_once
# String: small (≤15 bytes inline) or large (heap pointer + length)
# Protocol witness tables = dynamic dispatch (like vtables)

Detection: __swift5_* sections in Mach-O, swift_ runtime symbols, s prefix in mangled names.

---

Kotlin / JVM Binary Reversing

Kotlin compiles to JVM bytecode or native (via Kotlin/Native). Common in Android and server-side CTF.

JVM Bytecode (Android/Server)

# Detect Kotlin
strings classes.dex | grep "kotlin"
# Look for: kotlin.Metadata annotation, kotlin/jvm/internal/*

# Decompile
jadx classes.dex                     # Best for Kotlin bytecode
cfr classes.jar --kotlin             # CFR with Kotlin mode
fernflower classes.jar output/       # IntelliJ's decompiler

# Kotlin-specific patterns in decompiled output:
# - Companion objects: ClassName$Companion
# - Data classes: copy(), component1(), component2(), toString()
# - Coroutines: ContinuationImpl, invokeSuspend, state machine
# - Null checks: Intrinsics.checkNotNull() everywhere
# - When expression: compiled as tableswitch/lookupswitch
# - Sealed classes: instanceof checks in chain

Kotlin coroutines in disassembly:

# Coroutines compile to state machines:
# invokeSuspend(result) {
#     switch (this.label) {
#         case 0: this.label = 1; return suspendFunction();
#         case 1: processResult(result); return Unit;
#     }
# }
# Each suspend point becomes a state in the switch.
# Follow the state machine to understand async flow.

Kotlin/Native

# Kotlin/Native produces platform binaries (no JVM)
# Recognize by: konan, kotlin.native strings
strings binary | grep "konan"

# Much harder to reverse — no reflection metadata
# Uses LLVM backend, looks similar to C/C++ in disassembly
# Key functions: InitRuntime, DeinitRuntime, CreateStablePointer
# Memory management: automatic reference counting (not GC)

Detection: kotlin.Metadata annotations (JVM), konan strings (Native), kotlin/ package paths.

---

D Language Binary Reversing (CSAW CTF 2016)

D language binaries have unique symbol mangling different from C++. Template instantiation at compile-time produces many function variants.

# Recognition: D binaries use different mangling than C++
# Symbols contain "_D" prefix and numeric length-prefixed names
# Example: _D4mainQaFNaNbNfZv

# Symbol demangling:
# GDB: set language d
# Radare2: export names show demangled D symbols
# Online: dlang.org/phobos/core_demangle.html

# Common D binary patterns:
# - Templates instantiated at compile-time: enc!("111"), enc!("222"), ...
# - Garbage collector references (GC.malloc, GC.free)
# - Phobos standard library functions (_D3std...)
# - String processing: std.string, std.conv.to

# Reversing a D cipher (XOR with cycling key):
def reverse_d_cipher(encrypted, num_functions=500):
    """D binaries may chain multiple transformation functions.
    Each function XORs with key character, then XORs with key length.
    Process in reverse order."""
    result = encrypted[:]
    for i in range(num_functions - 1, -1, -1):
        key = str(i) * 3  # e.g., "499499499" for function enc!("499")
        key_len = len(key)
        for j in range(len(result)):
            result[j] ^= key_len
            result[j] ^= ord(key[j % key_len])
    return bytes(result)

Key insight: D binaries are rare in CTFs but identifiable by _D symbol prefixes and Phobos library references. The compile-time template system means D functions may be duplicated hundreds of times with different parameters — look for patterns like enc!("N") where N varies.

---

Haskell Binary Reversing via STG Closures and hsdecomp (hxp CTF 2017, Codegate 2018)

GHC-compiled Haskell binaries use the STG (Spineless Tagless G-machine) execution model, making them notoriously difficult to reverse due to lazy evaluation, closures, and thunks. The STG machine turns everything into closure calls rather than direct function calls.

Recognition:

• Shared libraries: libHSbase-*, libHSrts-*
• Entry symbol: hs_main (replaces standard main)
• Mangled symbols use Z-encoding: z = prefix, Z = uppercase, zd = ., zi = $
• GHC calling convention register mapping: rbx = R1, r14 = R2

Closure structure: Closures are structs where the first qword points to the info table/code. The info table precedes the code pointer and contains metadata (closure type, layout info, SRT).

# Identify Haskell binary
ldd ./binary | grep libHS
readelf -s ./binary | grep hs_main

# Decompile with hsdecomp (github.com/gereeter/hsdecomp)
# Recovers closure structure and pattern matching into pseudo-Haskell
python2 hsdecomp ./binary

# Compile reference for monkey-patching
ghc -O0 reference.hs -o reference
objcopy --dump-section .text=main_code reference

Monkey-patching technique: When decompilation fails or closures are opaque, compile a minimal Haskell program with the same GHC version, extract the compiled Main_main_info closure code, and patch it into the challenge binary. This forces evaluation of hidden closures and prints their results by replacing the main entry point with a known evaluator.

-- reference.hs: minimal program that evaluates and prints the target closure
module Main where
main :: IO ()
main = print targetClosure  -- replace with the closure you want to evaluate

Key insight: Haskell binaries are notoriously hard to reverse due to lazy evaluation, closures, and thunks. The STG machine turns everything into closure calls rather than direct function calls. hsdecomp recovers the closure structure and pattern matching. When decompilation fails, monkey-patching a known Main_main_info from a reference binary forces evaluation of hidden closures and prints results.

Detection: libHSbase-* shared libraries, hs_main entry, Z-encoded symbols (e.g., MainZCmain), GHC version strings.

References: hxp CTF 2017, Codegate 2018

---

Haskell Binary RE via GHC CMM Intermediate Language (N1CTF 2018)

GHC-compiled Haskell binaries are nearly impossible to decompile with IDA due to the STG execution model. When a .cmm (C-- intermediate) file is available or recoverable, read it to understand thunks, closures, and lazy evaluation semantics. For exponentially-growing recursive structures, compute segment sizes with memoization and use binary search instead of materializing the full string.

Pattern: The binary builds a recursive string structure where f(n) = s1 + f(n-1) + s2 + f(n-1) + s3. Direct evaluation is O(2^n) in both time and space. Instead, compute the size of each recursion level with memoization, then binary-search for the target character index by walking the segment boundaries.

# Haskell recursive string: f(n) = s1 + f(n-1) + s2 + f(n-1) + s3
# Direct evaluation is O(2^n) -- use size memoization:
from functools import lru_cache

@lru_cache(maxsize=None)
def fsize(n):
    if n == 0: return len(s0)
    return len(s1) + fsize(n-1) + len(s2) + fsize(n-1) + len(s3)

def char_at(n, offset):
    if n == 0: return s0[offset]
    if offset < len(s1): return s1[offset]
    offset -= len(s1)
    if offset < fsize(n-1): return char_at(n-1, offset)
    offset -= fsize(n-1)
    if offset < len(s2): return s2[offset]
    offset -= len(s2)
    return char_at(n-1, offset)

Key insight: GHC's CMM (C minus minus) intermediate representation preserves enough structure to identify algorithms. For recursive string constructions that double in size each level, compute segment sizes with memoization and binary-search for target indices instead of materializing the exponentially-growing string.

Detection: Haskell binary (see recognition above) with a .cmm file included in the challenge distribution. Look for recursive closure applications that produce string-like data with exponential growth.

References: N1CTF 2018

---

C++ Binary Reversing (Quick Reference)

While C++ RE is well-covered by general tools, these patterns are CTF-specific:

vtable Reconstruction

# Virtual function tables (vtables):
# First 8 bytes of object → pointer to vtable
# vtable entries: [typeinfo_ptr, destructor, method1, method2, ...]
# In Ghidra: Data → Create Pointer at vtable address

# Identify polymorphic dispatch:
# mov rax, [rdi]           # Load vtable from this pointer
# call [rax + 0x18]        # Call 4th virtual method (0x18/8 = 3rd after typeinfo+dtor)

RTTI (Run-Time Type Information)

# If not stripped, RTTI reveals class hierarchy
strings binary | grep -E "^[0-9]+[A-Z]"   # Mangled type names
c++filt _ZTI7MyClass                        # → typeinfo for MyClass

# In Ghidra: search for vtable references, follow typeinfo pointer
# typeinfo struct: {vtable_for_typeinfo, name_string, base_class_ptr}

Standard Library Patterns

std::string (libstdc++):
  SSO (Small String Optimization): inline buffer for ≤15 chars
  Layout: {char* ptr, size_t size, union{size_t cap, char buf[16]}}

std::vector<T>:
  {T* begin, T* end, T* capacity_end}

std::map<K,V>:
  Red-black tree: each node has {left, right, parent, color, key, value}

std::unordered_map<K,V>:
  Hash table: {bucket_array, size, load_factor_max, ...}

---

Rust Lifetime Escape via Compiler Bug #25860 (Hack.lu 2018)

Pattern: Rust compiler bug rust-lang/rust#25860 — higher-ranked lifetime variance was checked incorrectly, so a closure could "reborrow" a reference and unsoundly extend its lifetime to 'static. In a Rust-only sandbox that runs safe code (no unsafe block) this bug yields a UAF primitive: alias a Vec<u8> heap buffer as a (usize, usize, usize) tuple and read/write past its end.

// Triggering pattern — safe Rust only
fn extend<'a, 'b, T>(_: &'a &'b (), v: &'b T) -> &'a T { v }

fn bad<T>(v: T) -> &'static T {
    // Closure infers 'a = 'static because of the variance bug
    let f: fn(&'_ &'_ (), &'_ T) -> &'_ T = extend;
    f(&&(), &v) // returned ref now outlives v
}

fn main() {
    let aliased: &'static Vec<u8> = bad(vec![1u8, 2, 3]);
    // Reinterpret the Vec as its raw header: (ptr, len, cap)
    let header: &(usize, usize, usize) =
        unsafe { std::mem::transmute(aliased) };
    println!("ptr={:#x} len={} cap={}", header.0, header.1, header.2);
}

Key insight: A single soundness bug in the borrow checker turns a sandboxed Rust playground into an arbitrary-read-write primitive — no unsafe required. When a Rust CTF bans unsafe and pins a specific compiler version, grep the rust-lang issue tracker for soundness bugs fixed after that version: each one is an exploitation candidate. Repro this family with std::mem::transmute only after you have a lifetime-extended reference; the alias is the hard part.

References: Hack.lu CTF 2018 — Rusty CodePad, writeup 11859

---

Rust #[no_mangle] libc Override for seccomp Bypass (Hack.lu 2018)

Pattern: A sandboxed Rust binary calls prctl(PR_SET_SECCOMP, ...) early in main, then drops to user code. Because the sandboxed crate is linked statically alongside libc, defining an extern "C" function named prctl with #[no_mangle] shadows libc's symbol at link time. Returning 0 from the override disables seccomp, leaving every syscall reachable from the attacker's code.

// User-supplied code — linked into the same binary as the sandbox harness
#[no_mangle]
pub extern "C" fn prctl(_a: i64, _b: i64) -> i64 {
    0 // pretend success, do not install any filter
}

// When main() calls prctl(PR_SET_SECCOMP, ...) it hits our override
fn main() {
    // The real program runs without seccomp filtering
}

Key insight: Rust's static-linking default means extern "C" + #[no_mangle] is effectively a dynamic hook at compile time — any libc symbol the sandbox harness calls (prctl, chroot, seteuid, read) can be redefined by attacker crate code that ships inside the same binary. Harden by routing syscalls through libc::syscall(SYS_prctl, ...) directly (which bypasses the symbol table) or by using -Wl,-Bsymbolic to prefer the intended definitions.

References: Hack.lu CTF 2018 — Rusty CodePad seccomp variant, writeup 11864

Rust xmmword Constant Extraction via IDAPython (Insomnihack 2019)

Pattern: Rust stores literal byte buffers (flag expected values, XOR tables) as 16-byte xmmword constants in .rodata. IDA parses these as xmmword_xxxx tokens. Walk the .rodata range in IDAPython, read each xmmword, reverse any simple obfuscation (e.g., (dword >> 2) ^ 0xA), and dump the plaintext.

import idc, idaapi
start, end = 0x4A1000, 0x4A1100
for ea in range(start, end, 4):
    d = idc.get_wide_dword(ea)
    print(chr((d >> 2) ^ 0xA), end='')

Key insight: Rust binaries are hard to decompile but very easy to scan for literal data. Any check of the form input == const_buf leaves the expected value in .rodata untouched. Grep for mov reg, xmmword [rip+offset] to find the slot, then dump.

References: Insomnihack teaser 2019 — beginner_reverse, writeup 12910

---

Nuitka-Compiled Python — Module Stub Injection (X-MAS CTF 2018)

Pattern: Nuitka turns Python sources into a monolithic native binary, but still uses the standard Python import machinery at runtime. Put a dummy base64.py / midi.py / whatever_module.py in the current working directory before executing the binary; the module system prefers CWD, so your stub is loaded instead of the embedded one. Log every attribute access and incrementally build a shim of the API the binary uses.

# base64.py next to the binary
class _Trace:
    def __getattr__(self, name):
        def f(*a, **k):
            print(f'base64.{name}({a!r}, {k!r})')
            return b''
        return f
import sys; sys.modules[__name__] = _Trace()

Run ./target_bin; the printed calls reveal the algorithm without decompiling the Nuitka output.

Key insight: Any runtime that still resolves module names through sys.path (Nuitka, PyInstaller with --onefile, Py2Exe with --bundle_files=1 off, frozen CPython) can be shimmed at import time with CWD stubs. Grep strings output for module names to pick good hook targets.

References: X-MAS CTF 2018 — A Christmas Carol, writeup 12667

CTF Reverse - Competition-Specific Patterns (Part 2)

Table of Contents

• Multi-Layer Self-Decrypting Binary (DiceCTF 2026)
• Embedded ZIP + XOR License Decryption (MetaCTF 2026)
• Stack String Deobfuscation from .rodata XOR Blob (Nullcon 2026)
• Prefix Hash Brute-Force (Nullcon 2026)
• CVP/LLL Lattice for Constrained Integer Validation (HTB ShadowLabyrinth)
• Decision Tree Function Obfuscation (HTB WonderSMS)
• GF(2^8) Gaussian Elimination for Flag Recovery (ApoorvCTF 2026)
• ROP Chain Obfuscation in Modified Binary (PlaidCTF 2016)

---

Multi-Layer Self-Decrypting Binary (DiceCTF 2026)

Pattern (another-onion): Binary with N layers (e.g., 256), each reading 2 key bytes, deriving keystream via SHA-256 NI instructions, XOR-decrypting the next layer, then jumping to it. Must solve within a time limit (e.g., 30 minutes).

Oracle for correct key: Wrong key bytes produce garbage code. Correct key bytes produce code with exactly 2 call read@plt instructions (next layer's reads). Brute-force all 65536 candidates per layer using this oracle.

JIT execution approach (fastest):

// Map binary's memory at original virtual addresses into solver process
// Compile solver at non-overlapping address: -Wl,-Ttext-segment=0x10000000
void *text = mmap((void*)0x400000, text_size, PROT_RWX, MAP_FIXED|MAP_PRIVATE, fd, 0);
void *bss = mmap((void*)bss_addr, bss_size, PROT_RW, MAP_FIXED|MAP_SHARED, shm_fd, 0);

// Patch read@plt to inject candidate bytes instead of reading stdin
// Patch tail jmp/call to next layer with ret/NOP to return from layer

// Fork-per-candidate: COW gives isolated memory without memcpy
for (int candidate = 0; candidate < 65536; candidate++) {
    pid_t pid = fork();
    if (pid == 0) {
        // Child: remap BSS as MAP_PRIVATE (COW from shared file)
        mmap(bss_addr, bss_size, PROT_RW, MAP_FIXED|MAP_PRIVATE, shm_fd, 0);
        inject_key(candidate >> 8, candidate & 0xff);
        ((void(*)())layer_addr)();  // Execute layer as function call
        // Check: does decrypted code contain exactly 2 call read@plt?
        if (count_read_calls(next_layer_addr) == 2) signal_found(candidate);
        _exit(0);
    }
}

Performance tiers:

Shared BSS optimization: BSS (16MB+) stored in /dev/shm as MAP_SHARED in parent. Children remap as MAP_PRIVATE for COW. Reduces fork overhead from 16MB page-table setup to ~4KB.

Key insight: Multi-layer decryption challenges are fundamentally about building fast brute-force engines. JIT execution (mapping binary memory into solver, running code directly as function calls) is orders of magnitude faster than ptrace. Fork-based COW provides free memory isolation per candidate.

Gotchas:

• Real binary may use call (0xe8) instead of jmp (0xe9) for layer transitions — adjust tail patching
• BSS may extend beyond ELF MemSiz via kernel brk mapping — map extra space
• SHA-NI instructions work even when not advertised in /proc/cpuinfo

---

Embedded ZIP + XOR License Decryption (MetaCTF 2026)

Pattern (License To Rev): Binary requires a license file as argument. Contains an embedded ZIP archive with the expected license, and an XOR-encrypted flag.

Recognition:

• strings reveals EMBEDDED_ZIP and ENCRYPTED_MESSAGE symbols
• Binary is not stripped — nm or readelf -s shows data symbols in .rodata
• file shows PIE executable, source file named licensed.c

Analysis workflow: 1. Find data symbols:

readelf -s binary | grep -E "EMBEDDED|ENCRYPTED|LICENSE"
# EMBEDDED_ZIP at offset 0x2220, 384 bytes
# ENCRYPTED_MESSAGE at offset 0x21e0, 35 bytes

2. Extract embedded ZIP:

import struct
with open('binary', 'rb') as f:
    data = f.read()
# Find PK\x03\x04 magic in .rodata
zip_start = data.find(b'PK\x03\x04')
# Extract ZIP (size from symbol table or until next symbol)
open('embedded.zip', 'wb').write(data[zip_start:zip_start+384])

3. Extract license from ZIP:

unzip embedded.zip  # Contains license.txt

4. XOR decrypt the flag:

license = open('license.txt', 'rb').read()
enc_msg = open('encrypted_msg.bin', 'rb').read()  # Extract from .rodata
flag = bytes(a ^ b for a, b in zip(enc_msg, license))
print(flag.decode())

Key insight: No need to run the binary or bypass the expiry date check. The embedded ZIP and encrypted message are both in .rodata — extract and XOR offline.

Disassembly confirms:

• memcmp(user_license, decompressed_embedded_zip, size) — license validation
• Date parsing with sscanf("%d-%d-%d") on EXPIRY_DATE= field
• XOR loop: ENCRYPTED_MESSAGE[i] ^ license[i] → putc() per byte

Lesson: When a binary has named symbols (EMBEDDED_*, ENCRYPTED_*), extract data directly from the binary without execution. XOR with known plaintext (the license) is trivially reversible.

---

Stack String Deobfuscation from .rodata XOR Blob (Nullcon 2026)

Pattern (stack_strings_1/2): Binary mmaps a blob from .rodata, XOR-deobfuscates it, then uses the blob to validate input. Flag is recovered by reimplementing the verification loop.

Recognition:

• mmap() call followed by XOR loop over .rodata data
• Verification loop with running state (eax, ebx, r9) updated with constants like 0x9E3779B9, 0x85EBCA6B, 0xA97288ED
• rol32() operations with position-dependent shifts
• Expected bytes stored in deobfuscated buffer

Approach: 1. Extract .rodata blob with pyelftools:

   from elftools.elf.elffile import ELFFile
   with open(binary, "rb") as f:
       elf = ELFFile(f)
       ro = elf.get_section_by_name(".rodata")
       blob = ro.data()[offset:offset+size]

2. Recover embedded constants (length, magic values) by XOR with known keys from disassembly 3. Reimplement the byte-by-byte verification loop:

• Each iteration: compute two hash-like values from running state
• XOR them together and with expected byte to recover input byte
• Update running state with constant additions

Variant (stack_strings_2): Adds position permutation + state dependency on previous character:

• Position permutation: byte i may go to position pos[i] in the output
• State dependency: need = (expected - rol8(prev_char, 1)) & 0xFF
• Must track state variable that updates to current character each iteration

Key constants to look for:

• 0x9E3779B9 (golden ratio fractional, common in hash functions)
• 0x85EBCA6B (MurmurHash3 finalizer constant)
• 0xA97288ED (related hash constant)
• rol32() with shift i & 7

---

Prefix Hash Brute-Force (Nullcon 2026)

Pattern (Hashinator): Binary hashes every prefix of the input independently and outputs one digest per prefix. Given N output digests, the flag has N-1 characters.

Attack: Recover input one character at a time:

for pos in range(1, len(target_hashes)):
    for ch in charset:
        candidate = known_prefix + ch + padding
        hashes = run_binary(candidate)
        if hashes[pos] == target_hashes[pos]:
            known_prefix += ch
            break

Key insight: If each prefix hash is independent (no chaining/HMAC), the problem decomposes into N x |charset| binary executions. This is the hash equivalent of byte-at-a-time block cipher attacks.

Detection: Binary outputs multiple hash lines. Changing last character only changes last hash. Different input lengths produce different numbers of output lines.

---

CVP/LLL Lattice for Constrained Integer Validation (HTB ShadowLabyrinth)

Pattern: Binary validates flag via matrix multiplication where grouped input characters are multiplied by coefficient matrices and checked against expected 64-bit results. Standard algebra fails because solutions must be printable ASCII (32-126). Lattice-based CVP (Closest Vector Problem) with LLL reduction solves this efficiently.

Identification: 1. Binary groups input characters (e.g., 4 at a time) 2. Each group is multiplied by a coefficient matrix 3. Results compared against hardcoded 64-bit values 4. Need integer solutions in a constrained range (printable ASCII)

SageMath CVP solver:

from sage.all import *

def solve_constrained_matrix(coefficients, targets, char_range=(32, 126)):
    """
    coefficients: list of coefficient rows (e.g., 4 values per group)
    targets: expected output values
    char_range: valid character range (printable ASCII)
    """
    n = len(coefficients[0])  # characters per group
    mid = (char_range[0] + char_range[1]) // 2

    # Build lattice: [coeff_matrix | I*scale]
    # The target vector includes adjusted targets
    M = matrix(ZZ, n + len(targets), n + len(targets))
    scale = 1000  # Weight to constrain character range

    for i, row in enumerate(coefficients):
        for j, c in enumerate(row):
            M[j, i] = c
        M[n + i, i] = 1  # padding

    for j in range(n):
        M[j, len(targets) + j] = scale

    target_vec = vector(ZZ, [t - sum(c * mid for c in row)
                              for row, t in zip(coefficients, targets)]
                        + [0] * n)

    # LLL + CVP
    L = M.LLL()
    closest = L * L.solve_left(target_vec)  # or use Babai
    solution = [closest[len(targets) + j] // scale + mid for j in range(n)]
    return bytes(solution)

Two-phase validation pattern: 1. Phase 1 (matrix math): Solve via CVP/LLL → recovers first N characters 2. First N characters become AES key → decrypt file.bin (XOR last 16 bytes + AES-256-CBC + zlib decompress) 3. Phase 2 (custom VM): Decrypted bytecode runs in custom VM, validates remaining characters via another linear system (mod 2^32)

Modular linear system solving (Phase 2 — VM validation):

import numpy as np
from sympy import Matrix

# M * x = v (mod 2^32)
M_mod = Matrix(coefficients) % (2**32)
v_mod = Matrix(targets) % (2**32)
# Gaussian elimination in Z/(2^32)
solution = M_mod.solve(v_mod)  # Returns flag characters

Key insight: When a binary validates input through linear combinations with large coefficients and the solution must be in a small range (printable ASCII), this is a lattice problem in disguise. LLL reduction + CVP finds the nearest lattice point, recovering the constrained solution. Cross-reference: invoke /ctf-crypto for LLL/CVP fundamentals (advanced-math.md in ctf-crypto).

Detection: Binary performs matrix-like operations on grouped input, compares against 64-bit constants, and a brute-force search space is too large (e.g., 256^4 per group × 12 groups).

---

Decision Tree Function Obfuscation (HTB WonderSMS)

Pattern: Binary routes input through ~200+ auto-generated functions, each computing a polynomial expression from input positions, comparing against a constant, and branching left/right. The tree makes static analysis impractical without scripted extraction.

Identification: 1. Large number of similar functions with random-looking names (e.g., f315732804) 2. Each function computes arithmetic on specific input positions 3. Functions call other tree functions or a final validation function 4. Decompiled code shows if (expr cmp constant) call_left() else call_right()

Ghidra headless scripting for mass extraction:

# Extract comparison constants from all tree functions
# Run via: analyzeHeadless project/ tmp -import binary -postScript extract_tree.py
from ghidra.program.model.listing import *
from ghidra.program.model.symbol import *

fm = currentProgram.getFunctionManager()
results = []
for func in fm.getFunctions(True):
    name = func.getName()
    if name.startswith('f') and name[1:].isdigit():
        # Find CMP instruction and extract immediate constant
        inst_iter = currentProgram.getListing().getInstructions(func.getBody(), True)
        for inst in inst_iter:
            if inst.getMnemonicString() == 'CMP':
                operand = inst.getOpObjects(1)
                if operand:
                    results.append((name, int(operand[0].getValue())))

Constraint propagation from known output format: 1. Start from known output bytes (e.g., http://HTB{...}) → fix several input positions 2. Fixed positions cascade through arithmetic constraints → determine dependent positions 3. Tree root equation pins down remaining free variables 4. Recognize English words in partial flag to disambiguate multiple solutions

Key insight: Auto-generated decision trees look overwhelming but are repetitive by construction. Script the extraction (Ghidra, Binary Ninja, radare2) rather than reversing each function manually. The tree is just a dispatcher — the real logic is in the leaf function and its constraints.

Detection: Binary with hundreds of similarly-structured functions, 3-5 input position references per function, branching to two other functions or a common leaf.

---

GF(2^8) Gaussian Elimination for Flag Recovery (ApoorvCTF 2026)

Pattern (Forge): Stripped binary performs Gaussian elimination over GF(2^8) (Galois Field with 256 elements, using the AES polynomial). A matrix and augmentation vector are embedded in .rodata. The solution vector is the flag.

GF(2^8) arithmetic with AES polynomial (x^8+x^4+x^3+x+1 = 0x11b):

def gf_mul(a, b):
    """Multiply in GF(2^8) with AES reduction polynomial."""
    p = 0
    for _ in range(8):
        if b & 1:
            p ^= a
        hi = a & 0x80
        a = (a << 1) & 0xff
        if hi:
            a ^= 0x1b  # Reduction: x^8 = x^4+x^3+x+1
        b >>= 1
    return p

def gf_inv(a):
    """Brute-force multiplicative inverse (fine for 256 elements)."""
    if a == 0: return 0
    for x in range(1, 256):
        if gf_mul(a, x) == 1:
            return x
    return 0

Solving the linear system:

# Extract N×N matrix + N-byte augmentation from binary .rodata
N = 56  # Flag length
# Build augmented matrix: N rows × (N+1) cols

for col in range(N):
    # Find non-zero pivot
    pivot = next((r for r in range(col, N) if aug[r][col] != 0), -1)
    if pivot != col:
        aug[col], aug[pivot] = aug[pivot], aug[col]
    # Scale pivot row by inverse
    inv = gf_inv(aug[col][col])
    aug[col] = [gf_mul(v, inv) for v in aug[col]]
    # Eliminate column in all other rows
    for row in range(N):
        if row == col: continue
        factor = aug[row][col]
        if factor == 0: continue
        aug[row] = [v ^ gf_mul(factor, aug[col][j]) for j, v in enumerate(aug[row])]

flag = bytes(aug[i][N] for i in range(N))

Key insight: GF(2^8) is NOT regular integer arithmetic — addition is XOR, multiplication uses polynomial reduction. The AES polynomial (0x11b) is the most common; look for the constant 0x1b in disassembly. The binary may encrypt the result with AES-GCM afterward, but the raw solution vector (pre-encryption) is the flag.

Detection: Binary with a large matrix in .rodata (N² bytes), XOR-based row operations, constants 0x1b or 0x11b, and flag length matching sqrt of matrix size.

---

ROP Chain Obfuscation in Modified Binary (PlaidCTF 2016)

Pattern (quite quixotic quest): Modified curl binary with a custom --pctfkey KEY option. Key validation replaces esp with a buffer address and returns into a ~250KB ROP chain stored in a magic_buf symbol. The ROP chain validates the key through XOR, MD5, and constant comparisons.

Analysis approach:

1. Detect the ROP dispatch: Look for mov esp, eax; ret or similar stack pivot — this redirects execution into the ROP chain 2. Dump the ROP chain: Script GDB to disassemble instructions after each return address in the chain:

# GDB script to trace ROP gadgets
import gdb

magic_buf = 0x080b0000  # symbol address
buf_size = 0x40000       # quarter megabyte
offset = 0

while offset < buf_size:
    addr = int.from_bytes(gdb.selected_inferior().read_memory(magic_buf + offset, 4), 'little')
    gdb.execute(f'x/3i {addr}')
    # Advance past the gadget (typically 4 bytes per return address)
    offset += 4

3. Identify patterns in the chain: Look for unrolled loops (repeated gadget sequences), pop instructions that skip data, and ret imm16 that skip large blocks 4. Reconstruct the algorithm: The chain typically performs:

• Key length check (compare with constant)
• Character-level operations (sum ASCII values, XOR with constants)
• Hash computation (MD5 of derived value)
• Hash prefix comparison
• XOR of input with hash as keystream
• Comparison with embedded constants

5. Extract and solve: Dump the embedded constants, brute-force any intermediate values (e.g., character sum → MD5 with matching prefix), then XOR to recover the key:

import hashlib

# Brute-force the sum that produces correct MD5 prefix
target_prefix = 0xc0050bdd  # extracted from ROP chain
for s in range(128 * 0x35):  # max sum of printable chars * key_length
    h = hashlib.md5(str(s ^ xor_constant).encode()).hexdigest()
    if int(h[:8], 16) == target_prefix:
        md5_key = bytes.fromhex(h)
        break

# XOR embedded values with MD5 keystream to get flag
flag = bytes(v ^ md5_key[i % 16] for i, v in enumerate(embedded_values))

Key insight: ROP chain obfuscation ("ROPfuscation") hides algorithms in chains of return-oriented gadgets. The chain looks incomprehensible as raw addresses but becomes analyzable when you: (a) dump each gadget's disassembly, (b) filter repetitions and skip regions, (c) annotate register effects. The chain is functionally equivalent to normal code — it just uses ret instead of sequential execution. Large chains (100K+ gadgets) often contain unrolled loops that compress to ~1000 lines of pseudocode.

See also: patterns-ctf.md for Part 1 (hidden emulator opcodes, SPN static extraction, image XOR smoothness, byte-at-a-time cipher, mathematical convergence bitmap, Windows PE XOR bitmap OCR, two-stage RC4+VM loaders, GBA ROM meet-in-the-middle, Sprague-Grundy game theory, kernel module maze solving, multi-threaded VM channels). patterns-ctf-3.md for Part 3 (Z3 single-line Python circuit, sliding window popcount, keyboard LED Morse code, C++ destructor-hidden validation, syscall side-effect memory corruption, MFC dialog event handlers, VM sequential key-chain brute-force, Burrows-Wheeler transform inversion, OpenType font ligature exploitation, GLSL shader VM with self-modifying code, instruction counter as cryptographic state).