Core Dumps

Core Dumps

Purpose

Guide agents through enabling, collecting, and analysing core dumps for post-mortem crash investigation without rerunning the buggy program.

Triggers

• "My program crashed in production — how do I analyse the core?"
• "How do I enable core dumps on Linux?"
• "I have a core file but no symbols / source"
• "How do I use debuginfod to get symbols for a core?"
• "coredumpctl show me the crash"

Workflow

1. Enable core dumps (Linux)

# Per-session (lost on logout)
ulimit -c unlimited

# Persistent (add to /etc/security/limits.conf)
*   soft   core   unlimited
*   hard   core   unlimited

# Check current limit
ulimit -c

# Set core pattern (where and how cores are named)
# Default: 'core' in CWD — often not useful
sudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t
# %e = executable, %p = PID, %t = timestamp

# Persistent (add to /etc/sysctl.d/99-core.conf)
kernel.core_pattern=/tmp/core-%e-%p-%t
kernel.core_uses_pid=1

2. systemd/coredumpctl (modern Linux)

If systemd manages core dumps (common on Ubuntu 20+, Fedora, Arch):

# List recent crashes
coredumpctl list

# Show details of the latest crash
coredumpctl info

# Load latest crash in GDB
coredumpctl gdb

# Load specific PID crash
coredumpctl gdb 12345

# Export core file
coredumpctl dump -o myapp.core PID

Core storage location: /var/lib/systemd/coredump/.

3. Enable core dumps (macOS)

# macOS uses /cores by default (must be root-writable)
ulimit -c unlimited

# Check
ls /cores/

# launchd-launched services: set in plist
# <key>HardResourceLimits</key>
# <dict><key>Core</key><integer>9223372036854775807</integer></dict>

4. Analyse a core with GDB

# Load binary and core
gdb ./prog core.12345

# If the binary was stripped, provide the unstripped copy
gdb ./prog-with-symbols core.12345

# Essential first commands
(gdb) bt                    # call stack
(gdb) bt full               # stack + locals
(gdb) info registers        # CPU state at crash
(gdb) frame 2               # jump to interesting frame
(gdb) info locals           # local variables in frame
(gdb) print ptr             # inspect a pointer

# All threads (multi-threaded crash)
(gdb) thread apply all bt full

5. Analyse a core with LLDB

lldb ./prog -c core.12345

# Or
lldb
(lldb) target create ./prog --core core.12345

# Commands
(lldb) bt
(lldb) thread backtrace all
(lldb) frame select 2
(lldb) frame variable

6. Missing symbols: debuginfod

debuginfod serves debug symbols from a central server, mapping build IDs to DWARF data.

# Install client (Debian/Ubuntu)
sudo apt install debuginfod

# Enable (add to ~/.bashrc or /etc/environment)
export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com https://debuginfod.elfutils.org"

# GDB auto-fetches symbols when DEBUGINFOD_URLS is set
gdb ./prog core

# Manually query
debuginfod-find debuginfo <build-id>
debuginfod-find source <build-id> /path/to/file.c

7. Missing symbols: manual approach

# Check if binary has a build ID
readelf -n ./prog | grep Build

# Find the correct debug package
# Debian: apt install prog-dbg or prog-dbgsym
# RPM: dnf install prog-debuginfo

# Point GDB to debug symbols directory
(gdb) set debug-file-directory /usr/lib/debug

# Or use eu-readelf to dump build ID, then find .debug file
eu-readelf -n ./prog
find /usr/lib/debug -name "*.debug" | xargs eu-readelf -n 2>/dev/null | grep <build-id>

8. Strip binaries and keep symbols

Best practice: build with symbols, strip for distribution, keep an unstripped copy.

# Build
gcc -g -O2 -o prog main.c

# Separate debug info
objcopy --only-keep-debug prog prog.debug
objcopy --strip-debug prog prog.stripped

# Add a debuglink so GDB finds the debug file automatically
objcopy --add-gnu-debuglink=prog.debug prog.stripped

# Deploy prog.stripped; keep prog.debug in a symbols store indexed by build-id

9. Quick triage from core without full debug session

# Print backtrace non-interactively
gdb -batch -ex 'bt full' -ex 'thread apply all bt full' ./prog core 2>&1 | tee crash.txt

# Print registers
gdb -batch -ex 'info registers' ./prog core

# Check signal that caused crash
gdb -batch -ex 'info signal' ./prog core

For a full cheatsheet covering core pattern tokens, coredumpctl, GDB/LLDB commands, debuginfod servers, and strip/symbol workflows, see references/cheatsheet.md.

Related skills

• Use skills/debuggers/gdb for full GDB session details
• Use skills/debuggers/lldb for LLDB-based analysis
• Use skills/runtimes/sanitizers to catch the bug before it reaches production
• Use skills/binaries/elf-inspection for readelf, build IDs, and binary inspection

Core Dump Cheatsheet

Source: <https://man7.org/linux/man-pages/man5/core.5.html> Source: <https://sourceware.org/gdb/current/onlinedocs/gdb.html/Core-File-Generation.html> Source: <https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html-single/developing_c_and_cpp_applications_in_rhel_8/index>

Table of Contents

1. Enable core dumps 2. Core pattern configuration 3. systemd / coredumpctl 4. Analyse with GDB 5. Analyse with LLDB 6. debuginfod for symbols 7. Non-interactive triage 8. macOS cores 9. Stripping and symbol management

---

Enable core dumps

# Per-session (temporary)
ulimit -c unlimited

# Per-process (in code or shell script)
#include <sys/resource.h>
struct rlimit rl = { RLIM_INFINITY, RLIM_INFINITY };
setrlimit(RLIMIT_CORE, &rl);

# Persistent (all users): add to /etc/security/limits.conf
*   soft   core   unlimited
*   hard   core   unlimited

# Check current limit
ulimit -c              # current shell
cat /proc/self/limits  # current process (Linux)

# Check if cores are being generated
ls -la /tmp/ | grep core
ls -la $(cat /proc/sys/kernel/core_pattern | sed 's/%[epstu]//g' | xargs dirname 2>/dev/null)

---

Core pattern configuration

# Show current pattern
cat /proc/sys/kernel/core_pattern

# Common useful pattern (temporary)
sudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t
# %e = executable name, %p = PID, %t = Unix timestamp
# Other tokens: %u=UID, %g=GID, %s=signal, %h=hostname

# Persistent: create /etc/sysctl.d/99-core.conf
[Unit]
kernel.core_pattern=/tmp/core-%e-%p-%t
kernel.core_uses_pid=1

sudo sysctl -p /etc/sysctl.d/99-core.conf

# Check if pipe handler is active (like systemd-coredump or apport)
cat /proc/sys/kernel/core_pattern
# If it starts with '|', a pipe handler is active:
# | /usr/lib/systemd/systemd-coredump ...   <- systemd
# | /usr/share/apport/apport %p ...         <- Ubuntu apport

Core pattern tokens:

---

systemd / coredumpctl

# List all recorded crashes
coredumpctl list

# Show crashes for a specific executable
coredumpctl list myapp

# Show detailed info about latest crash
coredumpctl info

# Show info for specific PID
coredumpctl info 12345

# Open latest crash in GDB
coredumpctl gdb

# Open specific crash in GDB
coredumpctl gdb myapp
coredumpctl gdb PID

# Export core file
coredumpctl dump -o /tmp/myapp.core
coredumpctl dump PID -o /tmp/myapp.core

# Show journal log around crash
coredumpctl info --no-pager | grep -A5 "MESSAGE="

# Core storage location
ls /var/lib/systemd/coredump/

# Disable coredumpctl (revert to kernel default)
# In /etc/systemd/coredump.conf:
[Coredump]
Storage=none

---

Analyse with GDB

# Load binary and core
gdb ./prog /tmp/core-prog-12345-1700000000

# If binary is stripped, use the debug build
gdb ./prog-with-debug-symbols /tmp/core

# Essential first session
(gdb) bt                         # call stack
(gdb) bt full                    # stack + locals in each frame
(gdb) info registers             # CPU registers at crash
(gdb) frame 2                    # jump to frame 2
(gdb) info locals                # local variables in that frame
(gdb) print ptr                  # inspect a pointer
(gdb) x/10wx $rsp                # examine memory near stack pointer

# All threads
(gdb) thread apply all bt full

# What signal caused the crash?
(gdb) info signals               # list signal handling
(gdb) print $_siginfo            # signal info struct (Linux)

# Check for SIGABRT (assertion failure)
(gdb) bt                         # look for __assert_fail or abort in stack

# Print a struct nicely
(gdb) set print pretty on
(gdb) print *my_struct_ptr

# Find where a pointer points
(gdb) info symbol 0x7fff12345678
(gdb) x/s 0x7fff12345678         # if it might be a string

---

Analyse with LLDB

# Load core
lldb ./prog -c core.12345
# Or:
lldb
(lldb) target create ./prog --core core.12345

# Essential commands
(lldb) bt                         # backtrace
(lldb) bt all                     # all threads
(lldb) thread backtrace all
(lldb) frame select 2             # select frame
(lldb) frame variable             # locals
(lldb) register read              # CPU registers
(lldb) memory read -s8 -fx -c10 0x7fff0000  # examine memory

# Inspect signal
(lldb) thread info

---

debuginfod for symbols

# Install client
sudo apt install debuginfod      # Debian/Ubuntu
sudo dnf install elfutils-debuginfod-client  # Fedora/RHEL

# Set URL (add to ~/.bashrc or /etc/environment)
export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com https://debuginfod.elfutils.org"

# GDB auto-uses debuginfod when DEBUGINFOD_URLS is set
gdb ./stripped-prog core

# Check build ID (needed for debuginfod lookup)
readelf -n ./prog | grep 'Build ID'
file ./prog | grep BuildID

# Manual lookup
debuginfod-find debuginfo <build-id-hex>
debuginfod-find source  <build-id-hex> /path/to/source.c

# Warm cache manually
DEBUGINFOD_PROGRESS=1 gdb ./prog core

Public debuginfod servers:

---

Non-interactive triage

# Quick backtrace from core (CI / automated)
gdb -batch \
    -ex 'set print thread-events off' \
    -ex 'thread apply all bt full' \
    -ex 'info registers' \
    -ex 'quit' \
    ./prog core 2>&1 | tee crash_report.txt

# One-liner: print backtrace and registers
gdb -batch -ex 'bt full' -ex 'info registers' ./prog core

# Check signal number from core metadata
eu-readelf -n core | grep -i signal
readelf -n core 2>/dev/null | head -30

# Print core metadata without GDB
file core                 # shows signal, PID, architecture

---

macOS cores

# Enable
ulimit -c unlimited
# Cores go to /cores/core.<PID>

# Check
ls /cores/

# Load in LLDB
lldb ./prog -c /cores/core.12345

# Enable for a specific process
taskgated policy ...     # complex; usually just run with ulimit set

# Disable Crash Reporter (prevent dialog)
# sudo defaults write com.apple.CrashReporter DialogType none

# Check crash logs (Crash Reporter writes .crash files)
ls ~/Library/Logs/DiagnosticReports/
ls /Library/Logs/DiagnosticReports/

---

Stripping and symbol management

# Best practice: keep an unstripped copy indexed by build ID
BUILD_ID=$(readelf -n prog | grep 'Build ID' | awk '{print $3}')
mkdir -p /srv/symbols/${BUILD_ID:0:2}
cp prog /srv/symbols/${BUILD_ID:0:2}/${BUILD_ID:2}.debug

# Strip for distribution
objcopy --only-keep-debug prog prog.debug
strip --strip-debug prog
objcopy --add-gnu-debuglink=prog.debug prog

# Install debug symbols (Debian)
sudo apt install myapp-dbgsym   # or myapp-dbg

# Install debug symbols (Fedora/RHEL)
sudo dnf install myapp-debuginfo

# Tell GDB where debug files live
(gdb) set debug-file-directory /usr/lib/debug:/srv/symbols