Strace Ltrace

strace / ltrace

Purpose

Guide agents through tracing system calls with strace and library calls with ltrace — the most effective tools for diagnosing incorrect binary behaviour without a crash or debugger.

Triggers

• "My program behaves incorrectly — how do I trace what it's doing?"
• "How do I find what files a binary is opening?"
• "strace shows ENOENT — how do I interpret it?"
• "How do I trace network calls with strace?"
• "What is ltrace and how does it differ from strace?"
• "How do I trace a running process?"

Workflow

1. Basic strace usage

# Trace all syscalls of a command
strace ./myapp arg1 arg2

# Attach to running process
strace -p 12345

# Trace child processes too (-f = follow fork)
strace -f ./myapp

# Save to file (raw output — not stdout)
strace ./myapp 2> trace.txt

# Most useful: timestamps + summary
strace -t -f ./myapp 2>&1 | head -100

2. Filter by syscall category

# Trace file operations only
strace -e trace=file ./myapp

# Trace network syscalls
strace -e trace=network ./myapp

# Trace specific syscalls
strace -e trace=open,openat,read,write ./myapp

# Trace process management
strace -e trace=process ./myapp

# Trace memory operations
strace -e trace=memory ./myapp

# Trace signals
strace -e trace=signal ./myapp

# Multiple categories
strace -e trace=file,network ./myapp

3. Interpreting common errors

# See return values and errors
strace -e trace=file ./myapp 2>&1 | grep -E "ENOENT|EPERM|EACCES|ENOTSUP"

# Find what file is not found
strace ./myapp 2>&1 | grep 'ENOENT'
# Example output:
# openat(AT_FDCWD, "/etc/myapp.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
# → Config file expected at /etc/myapp.conf

4. Useful strace flags

# Show strings fully (default truncates at 32 chars)
strace -s 256 ./myapp

# Timestamps
strace -t ./myapp     # wall clock time
strace -T ./myapp     # time spent in each syscall
strace -r ./myapp     # relative timestamps

# System call count summary
strace -c ./myapp
# Shows count, time, errors per syscall — great for profiling

# Trace with PIDs in output (for -f)
strace -f -p ./myapp
# Output: [pid 12346] open("/etc/passwd", O_RDONLY) = 3

# Decode numerical arguments
strace -e verbose=all ./myapp

# Print instruction pointer at each syscall
strace -i ./myapp

5. ltrace — library call tracing

# Trace all library calls
ltrace ./myapp

# Trace specific library function
ltrace -e malloc,free,fopen ./myapp

# Trace nested calls (lib → lib)
ltrace -n 2 ./myapp   # indent nested calls

# Trace with syscalls too
ltrace -S ./myapp

# Attach to running process
ltrace -p 12345

# Summary statistics
ltrace -c ./myapp

Typical ltrace output:

malloc(1024) = 0x55a1b2c3d000
fopen("/etc/myapp.conf", "r") = 0
free(0x55a1b2c3d000) = <void>

strace vs ltrace:

6. Practical diagnosis workflows

# Find missing config file
strace -e trace=openat,open ./myapp 2>&1 | grep ENOENT

# Find what network connections are made
strace -e trace=network -f ./myapp 2>&1 | grep connect

# Debug dynamic library loading failures
strace -e trace=openat ./myapp 2>&1 | grep "\.so"

# Find permission issues
strace -e trace=file ./myapp 2>&1 | grep -E "EACCES|EPERM"

# Debug slow startup (find where time is spent)
strace -c ./myapp 2>&1
# Look for high % time in unexpected syscalls

# Watch IPC/shared memory
strace -e trace=ipc,shm ./myapp

# Find what the binary exec's
strace -e trace=execve -f ./myapp

7. seccomp filter debugging

If a program is killed by a seccomp policy, strace reveals which syscall triggered it:

strace -e trace=all ./myapp 2>&1 | tail -5
# Often shows the last syscall before SIGSYS

For strace output patterns and ltrace filtering examples, see references/strace-patterns.md.

Related skills

• Use skills/debuggers/gdb when strace shows the failing location and you need to inspect internals
• Use skills/binaries/elf-inspection to understand what libraries and symbols a binary uses
• Use skills/binaries/dynamic-linking for diagnosing LD_* and library loading issues
• Use skills/profilers/linux-perf for performance profiling (strace overhead is too high for perf)

strace/ltrace Patterns Reference

strace Output Format

syscall_name(arg1, arg2, ...) = return_value [error]

Examples:

openat(AT_FDCWD, "/etc/passwd", O_RDONLY) = 3
read(3, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 1234
write(1, "hello\n", 6) = 6
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1234567000
clone(child_stack=0, flags=CLONE_VM|CLONE_FS|...) = 12346
openat(AT_FDCWD, "/missing", O_RDONLY) = -1 ENOENT (No such file or directory)

Diagnosing Common Issues

Binary won't start

# Check dynamic linker issues
strace -e trace=openat ./myapp 2>&1 | grep -E "\.so|ENOENT"
# Look for missing shared libraries

# Check execve itself
strace -e execve ./myapp
# execve("/path/to/myapp", ["./myapp"], ...) = -1 ENOEXEC → wrong binary format

# Check interpreter line (#!)
strace -e execve ./myscript 2>&1 | head -5

File not found issues

# Comprehensive file access trace
strace -e trace=openat,open,stat,access,faccessat ./myapp 2>&1 | \
    grep -v "= [0-9]$" | \
    grep -E "ENOENT|EACCES"

# Show all paths the app searches
strace -e trace=openat ./myapp 2>&1 | \
    awk '/openat/ { match($0, /"([^"]+)"/, arr); print arr[1] }'

Network issues

# See all connections attempted
strace -e trace=connect -s 256 ./myapp 2>&1 | grep connect

# DNS resolution (usually getaddrinfo → /etc/resolv.conf + UDP)
strace -f -e trace=openat,connect,sendto,recvfrom ./myapp 2>&1 | \
    grep -E "resolv|dns|53"

# Show full socket addresses
strace -e trace=network -s 256 ./myapp 2>&1

# TLS handshake debugging
strace -e trace=network -s 4096 ./myapp 2>&1 | grep -A5 connect

Permission issues

# Capability check
strace -e trace=prctl,capget,capset ./myapp 2>&1

# Setuid / setgid issues
strace -e trace=setuid,setgid,setresuid ./myapp

# SELinux/seccomp kills
strace -e trace=all ./myapp 2>&1 | tail -20
# SIGSYS = seccomp killed; SIGKILL could be OOM or policy

Memory issues (strace side)

# Large mmap calls (potential memory exhaustion)
strace -e trace=mmap,munmap,brk ./myapp 2>&1 | grep "ENOMEM\|failed"

# Stack issues
strace -e trace=mmap,getrlimit,setrlimit ./myapp 2>&1 | head -30

strace -c Output Analysis

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 62.34    0.012456         124       100        12 read
 18.91    0.003782          37       102         0 write
 11.23    0.002246        1123         2         0 futex
  3.45    0.000690         690         1         0 nanosleep
  2.01    0.000402           4       100         0 close
  1.12    0.000224           2       112        10 openat
  0.94    0.000188           2        84         0 stat

Key columns:

• % time → where the process spends time in kernel
• usecs/call → average time per call (high = blocking call)
• errors → failed calls count (non-zero = something wrong)

ltrace Filter Patterns

# Trace memory allocation functions
ltrace -e "malloc,calloc,realloc,free" ./myapp

# Trace string functions
ltrace -e "strcmp,strcpy,strncpy,strcat,strlen,strdup" ./myapp

# Trace file functions (C stdlib layer)
ltrace -e "fopen,fclose,fread,fwrite,fgets,fputs,fprintf,fscanf" ./myapp

# Trace format functions (find format string bugs)
ltrace -e "printf,fprintf,sprintf,snprintf,vprintf" ./myapp -s 256

# Trace pthread functions
ltrace -e "pthread_*" ./myapp

# Trace dynamic linking
ltrace -e "dlopen,dlsym,dlclose" ./myapp

Combining strace + gdb

# Find the exact code location causing an error:
# 1. Identify failing syscall with strace
strace -e trace=openat ./myapp 2>&1 | grep ENOENT

# 2. Set a syscall catchpoint in GDB
gdb ./myapp
(gdb) catch syscall openat
(gdb) condition 1 $rax == -2  # -ENOENT
(gdb) run
# Stops at the exact code location that triggered ENOENT
(gdb) bt
(gdb) info locals

strace on Docker / Containers

# Docker: requires --cap-add=SYS_PTRACE
docker run --cap-add=SYS_PTRACE myimage strace ./myapp

# Or in Kubernetes: add securityContext
# securityContext:
#   capabilities:
#     add: ["SYS_PTRACE"]

# Check if strace works in container
strace echo test 2>&1 | head -3