Root Cause Tracing

Root Cause Tracing

Overview

Bugs often manifest deep in the call stack (git init in wrong directory, file created in wrong location, database opened with wrong path). Your instinct is to fix where the error appears, but that's treating a symptom.

Core principle: Trace backward through the call chain until you find the original trigger, then fix at the source.

This skill is a specialized technique within the systematic-debugging workflow, typically applied during Phase 1 (Root Cause Investigation) when dealing with deep call stacks.

When to Use This Skill

Use root-cause-tracing when:

• Error happens deep in execution (not at entry point)
• Stack trace shows long call chain
• Unclear where invalid data originated
• Need to find which test/code triggers the problem
• Symptom appears far from actual cause

Relationship with systematic-debugging:

• systematic-debugging: The overall framework (Phases 1-4)
• root-cause-tracing: A specific technique for Phase 1 investigation
• Use root-cause-tracing WITHIN systematic-debugging Phase 1

The Iron Law

NEVER FIX JUST WHERE THE ERROR APPEARS
ALWAYS TRACE BACK TO FIND THE ORIGINAL TRIGGER

Fixing symptoms creates bandaid solutions that mask root problems.

Core Principles

1. Trace Backward: Follow call chain from symptom to source 2. Find Original Trigger: Identify where bad data/state originated 3. Fix at Source: Address root cause, not symptom 4. Defense-in-Depth: Add validation at each layer after fixing source

Quick Start

The 5-Step Trace Process

1. Observe the Symptom: What error message? What failed operation? 2. Find Immediate Cause: What code directly causes this error? 3. Ask What Called This: Trace one level up the call stack 4. Keep Tracing Up: Continue until you find the original trigger 5. Fix at Source + Defense: Fix root cause and add layer validation

Decision Tree

Error appears deep in stack?
  → Yes: Start tracing backward
    → Can identify caller? → Trace one level up → Repeat
    → Cannot identify caller? → Add instrumentation (see advanced-techniques.md)
  → No: May not need tracing (error at entry point)

The Tracing Process

Example: Git init in wrong directory

Error symptom → execFileAsync('git', ['init'], { cwd: '' })
  ← WorktreeManager.createSessionWorktree(projectDir='')
  ← Session.create() → Project.create() → Test code
  ← ROOT CAUSE: setupCoreTest() returns { tempDir: '' } before beforeEach

At each level ask: Where did this value come from? Is this the origin?

For detailed tracing methodology, see [Tracing Techniques](references/tracing-techniques.md) For complete real-world examples, see [Examples](references/examples.md)

After Finding Root Cause

Fix at source (throw if accessed before initialization) + Add defense-in-depth (validate at Project.create, WorkspaceManager, environment guards, instrumentation).

This prevents similar bugs and catches issues earlier.

Navigation

For detailed information:

• [Tracing Techniques](references/tracing-techniques.md): Complete tracing methodology, patterns, and decision trees
• [Examples](references/examples.md): Real-world debugging scenarios with full trace chains
• [Advanced Techniques](references/advanced-techniques.md): Stack traces, instrumentation, test pollution detection
• [Integration](references/integration.md): How to use with systematic-debugging and other skills

Key Reminders

• NEVER fix just where the error appears
• ALWAYS trace back to find the original trigger
• Use console.error() for debugging in tests (logger may be suppressed)
• Log BEFORE the dangerous operation, not after it fails
• Include context: directory, cwd, environment, timestamps
• Add defense-in-depth after fixing source
• Document your trace as you go (write down the call chain)

Red Flags - STOP

STOP when thinking:

• "I'll just add validation here" (without finding source)
• "This will prevent the error" (symptom fix)
• "Too hard to trace back" (add instrumentation instead)
• "Quick fix for now" (creates technical debt)

ALL of these mean: Continue tracing to find root cause.

Integration with Other Skills

• systematic-debugging: Use root-cause-tracing during Phase 1
• defense-in-depth: Add after finding root cause
• verification-before-completion: Verify fix worked at source
• test-driven-development: Write test for root cause, not symptom

See Integration for complete workflow examples.

Real-World Impact

From debugging session (2025-10-03):

• Found root cause through 5-level trace
• Fixed at source (getter validation)
• Added 4 layers of defense
• 1847 tests passed, zero pollution
• Time saved: 3+ hours vs symptom-fix approach

Bottom line: Tracing takes 15-30 minutes. Symptom fixes take hours of whack-a-mole.

Advanced Tracing Techniques

Advanced methods for tracing bugs when manual inspection isn't sufficient: instrumentation, stack traces, test pollution detection, and async tracing.

When Manual Tracing Isn't Enough

Manual code inspection works well when:

• Code path is clear
• Stack traces are available
• Single-threaded execution
• You can identify all callers

Use advanced techniques when:

• Can't identify which code path triggers the issue
• Multiple async operations interleave
• Race conditions or timing issues
• Need to find which test causes pollution
• External code (libraries) involved
• Production issues you can't reproduce locally

Stack Trace Instrumentation

Purpose

Capture complete call stack at strategic points to understand execution flow.

Basic Stack Trace Capture

function suspiciousOperation(param: string) {
  // Capture stack trace BEFORE the operation
  const stack = new Error().stack;
  console.error('DEBUG suspiciousOperation:', {
    param,
    cwd: process.cwd(),
    timestamp: Date.now(),
    stack
  });

  // Now do the operation
  performOperation(param);
}

Key points:

• Use console.error() in tests (regular logger may be suppressed)
• Log BEFORE the operation, not after it fails
• Include context: parameters, environment, state
• Capture stack with new Error().stack

Analyzing Stack Traces

Run and capture output:

npm test 2>&1 | grep 'DEBUG suspiciousOperation'

Look for:

• Test file names in stack traces
• Line numbers that trigger the call
• Patterns: same test? same parameters?
• Call frequency: how many times called?

Example output:

DEBUG suspiciousOperation: {
  param: '',
  cwd: '/Users/jesse/project/packages/core',
  stack: 'Error
    at suspiciousOperation (file.ts:10)
    at WorktreeManager.create (worktree.ts:45)
    at Session.initialize (session.ts:78)
    at Project.create (project.ts:23)
    at Test.<anonymous> (project.test.ts:12)
    at Test.run (node:internal/test)'
}

Analysis: The call originates from project.test.ts:12 with empty parameter.

Conditional Instrumentation

Only log when conditions are suspicious:

function gitInit(directory: string) {
  // Only log if directory is empty or equals cwd
  if (!directory || directory === process.cwd()) {
    console.error('SUSPICIOUS git init:', {
      directory,
      cwd: process.cwd(),
      nodeEnv: process.env.NODE_ENV,
      stack: new Error().stack
    });
  }

  execFileAsync('git', ['init'], { cwd: directory });
}

This reduces noise while capturing problematic cases.

Stack Trace in Production

Warning: Stack traces have performance cost. Use carefully in production.

class ErrorTracker {
  private static captureInterval = 100; // Only capture 1% of calls
  private static counter = 0;

  static maybeCapture(operation: string, data: any) {
    this.counter++;
    if (this.counter % this.captureInterval === 0) {
      // Sample 1% of operations
      logger.warn('Sampled operation', {
        operation,
        data,
        stack: new Error().stack
      });
    }
  }
}

Finding Test Pollution

What is Test Pollution?

Tests that create files, directories, or state that persists after the test completes.

Common polluters:

• Creating files/directories outside temp dir
• Not cleaning up in afterEach
• Modifying global state
• Creating git repositories
• Writing to current directory

Detection Strategy

Symptoms:

• Files appear in source code directory
• Tests fail when run together but pass individually
• Side effects from one test affect another
• Cleanup code not running

Manual Detection

Check for artifacts after test run:

# Before tests
ls -la src/

# Run tests
npm test

# After tests - did anything appear?
ls -la src/

Common artifacts:

• .git directories
• node_modules/ subdirectories
• Temp files not cleaned up
• Config files created
• Log files

Automated Detection with Bisection

Use the find-polluter.sh script to automatically find which test creates pollution:

#!/bin/bash
# find-polluter.sh
# Usage: ./find-polluter.sh <artifact> <test-pattern>
# Example: ./find-polluter.sh '.git' 'src/**/*.test.ts'

ARTIFACT=$1
TEST_PATTERN=$2

if [ -z "$ARTIFACT" ] || [ -z "$TEST_PATTERN" ]; then
  echo "Usage: $0 <artifact> <test-pattern>"
  echo "Example: $0 '.git' 'src/**/*.test.ts'"
  exit 1
fi

# Get list of test files
TEST_FILES=($(ls $TEST_PATTERN))

echo "Testing ${#TEST_FILES[@]} files for artifact: $ARTIFACT"

for test_file in "${TEST_FILES[@]}"; do
  echo "Testing: $test_file"

  # Remove artifact if exists
  rm -rf "$ARTIFACT" 2>/dev/null

  # Run single test file
  npm test -- "$test_file"

  # Check if artifact was created
  if [ -e "$ARTIFACT" ]; then
    echo "FOUND POLLUTER: $test_file"
    exit 0
  fi
done

echo "No polluter found"
exit 1

Usage:

# Find which test creates .git directory
./find-polluter.sh '.git' 'src/**/*.test.ts'

# Find which test creates node_modules
./find-polluter.sh 'node_modules' 'src/**/*.test.ts'

Advanced version with binary search:

#!/bin/bash
# find-polluter-fast.sh - Uses binary search for faster detection

ARTIFACT=$1
TEST_PATTERN=$2
TEST_FILES=($(ls $TEST_PATTERN))

function test_files() {
  local files=("$@")
  rm -rf "$ARTIFACT" 2>/dev/null
  npm test -- "${files[@]}"
  [ -e "$ARTIFACT" ]
}

function binary_search() {
  local files=("$@")
  local count=${#files[@]}

  if [ $count -eq 0 ]; then
    echo "No polluter found"
    return 1
  fi

  if [ $count -eq 1 ]; then
    if test_files "${files[@]}"; then
      echo "FOUND POLLUTER: ${files[0]}"
      return 0
    fi
    return 1
  fi

  # Split in half
  local mid=$((count / 2))
  local left=("${files[@]:0:mid}")
  local right=("${files[@]:mid}")

  # Test left half
  if test_files "${left[@]}"; then
    binary_search "${left[@]}"
  else
    binary_search "${right[@]}"
  fi
}

binary_search "${TEST_FILES[@]}"

Preventing Test Pollution

Best practices: 1. Always use temp directories:

   let tempDir: string;

   beforeEach(() => {
     tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-'));
   });

   afterEach(() => {
     fs.rmSync(tempDir, { recursive: true });
   });

2. Validate test isolation:

   // Guard against operations outside temp dir
   if (process.env.NODE_ENV === 'test') {
     if (!directory.includes(os.tmpdir())) {
       throw new Error(`Test safety: operation outside tmpdir: ${directory}`);
     }
   }

3. Use cleanup verification:

   afterEach(() => {
     // Clean up
     fs.rmSync(tempDir, { recursive: true });

     // Verify no artifacts in source
     const gitInSource = fs.existsSync(path.join(__dirname, '.git'));
     if (gitInSource) {
       throw new Error('Test pollution: .git created in source directory');
     }
   });

Async Operation Tracing

The Challenge

Async operations obscure call chains:

async function a() {
  await b();
}

async function b() {
  await c();
}

async function c() {
  throw new Error('Something failed');
}

Stack trace might only show:

Error: Something failed
  at c (file.ts:10)

Missing the full chain: a → b → c

Node.js Async Stack Traces

Enable async stack traces:

node --async-stack-traces test.js

Or in code:

// At application entry point
Error.stackTraceLimit = 50; // Capture deeper stacks

In package.json:

{
  "scripts": {
    "test": "node --async-stack-traces ./node_modules/.bin/jest"
  }
}

Trace IDs for Async Operations

When multiple async operations interleave, use trace IDs:

import { randomUUID } from 'crypto';
import { AsyncLocalStorage } from 'async_hooks';

const asyncLocalStorage = new AsyncLocalStorage();

function withTraceId<T>(fn: () => T): T {
  const traceId = randomUUID();
  return asyncLocalStorage.run({ traceId }, fn);
}

function getTraceId(): string {
  const store = asyncLocalStorage.getStore() as { traceId: string };
  return store?.traceId || 'no-trace-id';
}

// Usage
async function operationA() {
  console.log(`[${getTraceId()}] Starting operation A`);
  await operationB();
}

async function operationB() {
  console.log(`[${getTraceId()}] Starting operation B`);
  await operationC();
}

// Run with trace ID
await withTraceId(async () => {
  await operationA();
});

Output:

[abc-123] Starting operation A
[abc-123] Starting operation B
[abc-123] Starting operation C

All operations from same call chain have same trace ID.

Debugging Race Conditions

Problem: Operations complete in wrong order, causing bugs.

Solution: Add timing instrumentation:

class TimingTracer {
  private events: Array<{ time: number; event: string; data: any }> = [];

  record(event: string, data: any = {}) {
    this.events.push({
      time: Date.now(),
      event,
      data
    });
  }

  dump() {
    const sorted = this.events.sort((a, b) => a.time - b.time);
    console.error('=== Timing Trace ===');
    let start = sorted[0]?.time || 0;
    sorted.forEach(({ time, event, data }) => {
      console.error(`+${time - start}ms: ${event}`, data);
      start = time;
    });
  }
}

// Usage
const tracer = new TimingTracer();

async function operation() {
  tracer.record('start');

  const promise1 = async1().then(() => tracer.record('async1 done'));
  const promise2 = async2().then(() => tracer.record('async2 done'));

  await Promise.all([promise1, promise2]);

  tracer.record('both done');
  tracer.dump();
}

Output shows operation order:

=== Timing Trace ===
+0ms: start {}
+45ms: async2 done {}
+67ms: async1 done {}
+67ms: both done {}

Shows async2 completed before async1.

Debugging Third-Party Libraries

When Library Behavior is Unexpected

Strategy: 1. Verify you're using the API correctly 2. Check library version and changelog 3. Read library source code 4. Add instrumentation around library calls

Wrapping Library Calls

// Wrap library function to add tracing
import { originalFunction } from 'third-party-lib';

const tracedFunction = (...args: any[]) => {
  console.error('Calling library function:', {
    args,
    stack: new Error().stack
  });

  const result = originalFunction(...args);

  console.error('Library function result:', result);

  return result;
};

// Use traced version
export { tracedFunction as originalFunction };

Checking Library Source

When to read library source:

• Documentation is unclear
• Behavior differs from documentation
• Need to understand edge cases
• Debugging library bug

How to read library source:

# Find library location
npm ls third-party-lib

# View source
code node_modules/third-party-lib/src/

# Or on GitHub
open https://github.com/author/third-party-lib

Environment-Specific Issues

Capturing Environment Context

function captureEnvironment() {
  return {
    nodeVersion: process.version,
    platform: process.platform,
    arch: process.arch,
    cwd: process.cwd(),
    env: {
      NODE_ENV: process.env.NODE_ENV,
      CI: process.env.CI,
      // Add relevant env vars
    },
    memory: process.memoryUsage(),
    uptime: process.uptime()
  };
}

// Log with every error
try {
  riskyOperation();
} catch (error) {
  console.error('Operation failed:', {
    error,
    environment: captureEnvironment(),
    stack: error.stack
  });
  throw error;
}

Reproducing Production Issues Locally

Techniques: 1. Match environment:

   nvm use <production-node-version>
   export NODE_ENV=production

2. Use production data (sanitized):

   # Dump production DB to local
   pg_dump production_db | psql local_db

3. Enable production logging locally:

   if (process.env.DEBUG_PROD) {
     logger.level = 'debug';
   }

4. Replay production requests:

   // Log requests in production
   app.use((req, res, next) => {
     logger.info('Request', {
       method: req.method,
       url: req.url,
       headers: req.headers,
       body: req.body
     });
     next();
   });

   // Replay locally
   const productionRequest = loadFromLogs();
   await fetch('http://localhost:3000' + productionRequest.url, {
     method: productionRequest.method,
     headers: productionRequest.headers,
     body: productionRequest.body
   });

Performance Profiling for Root Cause

Sometimes "bug" is performance issue. Trace to find bottleneck.

Node.js Built-in Profiler

# Generate CPU profile
node --cpu-prof app.js

# Analyze with Chrome DevTools
open chrome://inspect

Custom Performance Tracing

class PerformanceTracer {
  private timers = new Map<string, number>();

  start(label: string) {
    this.timers.set(label, Date.now());
  }

  end(label: string): number {
    const start = this.timers.get(label);
    if (!start) throw new Error(`No timer: ${label}`);

    const duration = Date.now() - start;
    console.log(`${label}: ${duration}ms`);
    this.timers.delete(label);
    return duration;
  }

  async measure<T>(label: string, fn: () => T): Promise<T> {
    this.start(label);
    try {
      return await fn();
    } finally {
      this.end(label);
    }
  }
}

// Usage
const tracer = new PerformanceTracer();

await tracer.measure('database query', async () => {
  return await db.query('SELECT ...');
});

await tracer.measure('API call', async () => {
  return await fetch('https://api.example.com');
});

Output:

database query: 1243ms  ← BOTTLENECK FOUND
API call: 89ms

Summary

Stack traces: Capture call chains with new Error().stack Test pollution: Use bisection to find polluting tests Async tracing: Use trace IDs and async stack traces Library issues: Wrap calls, read source, verify API usage Environment issues: Match production environment, replay requests Performance: Profile to find bottlenecks

When to use:

• Manual tracing hits dead end
• Multiple async operations involved
• Test pollution occurring
• Race conditions or timing issues
• Need production-level debugging

Real-World Tracing Examples

Detailed examples showing complete root cause tracing processes with full trace chains and solutions.

Example 1: Git Init in Wrong Directory

The Symptom

Observed behavior:

$ npm test
...
Error: fatal: .git directory created in /Users/jesse/project/packages/core

Tests were creating a .git directory in the source code folder instead of a temporary directory.

Initial Investigation

First thought: "Just validate the directory parameter in git init"

WRONG: This would be a symptom fix. We need to find WHY the directory is wrong.

The Trace

Step 1: Find Immediate Cause

// worktree-manager.ts:45
async function createSessionWorktree(projectDir: string, sessionId: string) {
  // This line creates .git in wrong place
  await execFileAsync('git', ['init'], { cwd: projectDir });
}

Discovery: projectDir is an empty string '' Why is this wrong: Empty string as cwd resolves to process.cwd() (current directory)

Step 2: Where Does Empty String Come From?

// session.ts:34
static async create(name: string, projectDir: string) {
  const session = new Session(name);
  await session.initializeWorkspace(projectDir); // ← calls worktree manager
  return session;
}

Discovery: Session.create() receives projectDir = '' Question: Where does Session.create() get this value?

Step 3: Trace to Caller

// project.ts:67
static async create(name: string, directory: string) {
  await Session.create(name, directory); // ← passes directory through
  // ...
}

Discovery: Project.create() also receives directory = '' Question: What calls Project.create() with empty string?

Step 4: Check Test Code

// project.test.ts:12
const context = setupCoreTest();
const PROJECT_DIR = context.tempDir; // ← Accessed at module load time!

describe('Project', () => {
  it('should create project', async () => {
    await Project.create('test-project', PROJECT_DIR);
  });
});

Discovery: PROJECT_DIR is set to context.tempDir at module load time Question: What is context.tempDir at module load time?

Step 5: Root Cause Found

// test-setup.ts
export function setupCoreTest() {
  let _tempDir = ''; // ← Initial value is empty string

  beforeEach(() => {
    _tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-'));
  });

  afterEach(() => {
    if (_tempDir) fs.rmSync(_tempDir, { recursive: true });
  });

  return {
    tempDir: _tempDir // ← Returns empty string at module load time
  };
}

ROOT CAUSE: The test accesses context.tempDir at module load time (when defining PROJECT_DIR), but _tempDir is only set during beforeEach. At module load time, it's still ''.

The Solution

Fix at Source

export function setupCoreTest() {
  let _tempDir: string | null = null; // ← null instead of empty string

  beforeEach(() => {
    _tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-'));
  });

  afterEach(() => {
    if (_tempDir) fs.rmSync(_tempDir, { recursive: true });
    _tempDir = null;
  });

  return {
    get tempDir(): string {
      if (!_tempDir) {
        throw new Error('tempDir accessed before beforeEach ran');
      }
      return _tempDir;
    }
  };
}

Why this works:

• Accessing context.tempDir at module load time now throws immediately
• Forces tests to access it only within test cases
• Clear error message guides developers to fix

Add Defense-in-Depth

// Layer 1: Project.create() validates directory
static async create(name: string, directory: string) {
  if (!directory || directory.trim() === '') {
    throw new Error('Project directory cannot be empty');
  }
  // ...
}

// Layer 2: WorkspaceManager validates not empty
async function initializeWorkspace(projectDir: string) {
  if (!projectDir) {
    throw new Error('projectDir cannot be empty');
  }
  // ...
}

// Layer 3: NODE_ENV guard refuses git init outside tmpdir
async function createSessionWorktree(projectDir: string, sessionId: string) {
  if (process.env.NODE_ENV === 'test' && !projectDir.includes('tmp')) {
    throw new Error(`Test safety: refusing git init outside tmpdir: ${projectDir}`);
  }
  await execFileAsync('git', ['init'], { cwd: projectDir });
}

// Layer 4: Stack trace logging before git init
async function createSessionWorktree(projectDir: string, sessionId: string) {
  if (!projectDir || projectDir === process.cwd()) {
    console.error('DEBUG git init:', {
      projectDir,
      cwd: process.cwd(),
      stack: new Error().stack
    });
  }
  await execFileAsync('git', ['init'], { cwd: projectDir });
}

Results

• 1847 tests passed
• Zero .git pollution
• Clear error message for similar issues
• Multiple layers prevent similar bugs

Time Saved

With root cause tracing: 45 minutes Without (symptom fixes): 3+ hours of whack-a-mole

Example 2: Database Connection URL Wrong

The Symptom

Error: Connection failed: database "undefined" does not exist
  at Database.connect (database.ts:34)

Application crashes on startup because database name is undefined.

The Trace

Step 1: Find Immediate Cause

// database.ts:34
async function connect(url: string) {
  this.connection = await pg.connect(url); // ← Crashes here
}

Discovery: url = "postgresql://localhost/undefined" Why is this wrong: Database name is literally the string "undefined"

Step 2: Where Does URL Come From?

// database.ts:12
constructor(config: DatabaseConfig) {
  this.url = `postgresql://${config.host}/${config.database}`; // ← constructs URL
}

Discovery: config.database = undefined Question: Where does config come from?

Step 3: Trace to Configuration Loading

// app.ts:23
const dbConfig: DatabaseConfig = {
  host: process.env.DATABASE_HOST || 'localhost',
  database: process.env.DATABASE_NAME, // ← No default value!
  port: parseInt(process.env.DATABASE_PORT || '5432')
};

Discovery: DATABASE_NAME environment variable is not set Question: Why isn't it set?

Step 4: Check Environment Setup

// .env file
DATABASE_HOST=localhost
DATABASE_PORT=5432
# DATABASE_NAME missing!

ROOT CAUSE: Environment variable not defined, and code doesn't validate required variables.

The Solution

Fix at Source

// config.ts - Load and validate environment
export function loadDatabaseConfig(): DatabaseConfig {
  const requiredEnvVars = ['DATABASE_NAME', 'DATABASE_HOST'];
  const missing = requiredEnvVars.filter(v => !process.env[v]);

  if (missing.length > 0) {
    throw new Error(`Required environment variables missing: ${missing.join(', ')}`);
  }

  return {
    host: process.env.DATABASE_HOST!,
    database: process.env.DATABASE_NAME!,
    port: parseInt(process.env.DATABASE_PORT || '5432')
  };
}

Add Defense-in-Depth

// Layer 1: Type-safe config with validation
interface DatabaseConfig {
  host: string;
  database: string;
  port: number;
}

// Layer 2: Constructor validates config
constructor(config: DatabaseConfig) {
  if (!config.database || config.database === 'undefined') {
    throw new Error('Database name cannot be empty or undefined');
  }
  this.url = `postgresql://${config.host}/${config.database}`;
}

// Layer 3: Validate URL before connect
async function connect(url: string) {
  if (url.includes('/undefined')) {
    throw new Error(`Invalid database URL: ${url}`);
  }
  this.connection = await pg.connect(url);
}

Results

• Application fails fast at startup with clear error
• Developers immediately know which env var is missing
• Prevents confusing "database undefined does not exist" error

Example 3: User ID = 0 in Database Query

The Symptom

Error: Cannot query user: id cannot be 0
  at UserRepository.findById (user-repository.ts:45)

Database query fails because user ID is 0 (invalid).

The Trace

Step 1: Find Immediate Cause

// user-repository.ts:45
async findById(id: number): Promise<User> {
  if (id === 0) {
    throw new Error('Cannot query user: id cannot be 0');
  }
  return await this.db.query('SELECT * FROM users WHERE id = ?', [id]);
}

Discovery: id = 0 being passed to query Question: Where does this come from?

Step 2: Trace to Caller

// auth-middleware.ts:67
async authenticate(req: Request): Promise<User> {
  const userId = this.extractUserId(req);
  return await this.userRepo.findById(userId); // ← userId is 0
}

Discovery: extractUserId() returns 0 Question: Why does it return 0?

Step 3: Check ID Extraction

// auth-middleware.ts:34
private extractUserId(req: Request): number {
  const token = req.headers.authorization?.replace('Bearer ', '');
  if (!token) return 0; // ← Default to 0 if no token!

  const decoded = jwt.verify(token, SECRET);
  return decoded.userId;
}

Discovery: Returns 0 when no authorization header Question: Why is there no authorization header?

Step 4: Check Request Handling

// router.ts:23
app.get('/api/user/profile', async (req, res) => {
  const user = await authMiddleware.authenticate(req); // ← Called on public route!
  res.json(user);
});

ROOT CAUSE: Authentication middleware called on public route that doesn't require authentication. When no token present, it defaults to userId=0.

The Solution

Fix at Source

// router.ts - Separate public and protected routes
app.get('/api/public/profile', async (req, res) => {
  // Public route - no authentication
  res.json({ message: 'Public profile' });
});

app.get('/api/user/profile',
  requireAuth(), // ← Middleware throws if no auth
  async (req, res) => {
    const user = await authMiddleware.authenticate(req);
    res.json(user);
  }
);

Add Defense-in-Depth

// Layer 1: Don't default to 0, throw instead
private extractUserId(req: Request): number {
  const token = req.headers.authorization?.replace('Bearer ', '');
  if (!token) {
    throw new UnauthorizedError('No authorization token provided');
  }

  const decoded = jwt.verify(token, SECRET);
  return decoded.userId;
}

// Layer 2: Validate userId before query
async findById(id: number): Promise<User> {
  if (!id || id <= 0) {
    throw new Error(`Invalid user ID: ${id}`);
  }
  return await this.db.query('SELECT * FROM users WHERE id = ?', [id]);
}

// Layer 3: Type system (use branded types)
type UserId = number & { readonly __brand: 'UserId' };

function validateUserId(id: number): UserId {
  if (!id || id <= 0) throw new Error('Invalid user ID');
  return id as UserId;
}

Results

• Authentication errors are clear and immediate
• Protected routes always require authentication
• No magic "0" default that causes confusing errors

Example 4: React Component Renders Wrong Data

The Symptom

Component shows data from previous user after logout/login

User logs out, logs in as different user, but sees previous user's data briefly.

The Trace

Step 1: Observe Behavior

• User A logs in → sees correct data
• User A logs out → data clears
• User B logs in → sees User A's data for ~100ms
• After 100ms → sees correct User B data

Discovery: Stale data being shown before new data loads

Step 2: Check Component Data Source

// UserDashboard.tsx
function UserDashboard() {
  const user = useSelector(state => state.auth.user);
  const data = useSelector(state => state.userData.data);

  useEffect(() => {
    dispatch(fetchUserData(user.id)); // ← Fetches new data
  }, [user.id]);

  return <div>{data.name}</div>; // ← Shows stale data during fetch
}

Discovery: Redux store still has old user's data when new user logs in Question: Why isn't the old data cleared?

Step 3: Check Login/Logout Actions

// auth.actions.ts
export function logout() {
  return { type: 'LOGOUT' }; // ← Only clears auth state
}

// auth.reducer.ts
case 'LOGOUT':
  return { user: null }; // ← Only clears user

// userData.reducer.ts (SEPARATE REDUCER)
case 'FETCH_USER_DATA':
  return { ...state, data: action.payload }; // ← Never cleared!

ROOT CAUSE: Logout action only clears auth state, not user data. User data reducer never listens to LOGOUT action.

The Solution

Fix at Source

// userData.reducer.ts
import { LOGOUT } from './auth.actions';

case LOGOUT:
  return initialState; // ← Clear data on logout

case 'FETCH_USER_DATA':
  return { ...state, data: action.payload };

Add Defense-in-Depth

// Layer 1: Clear all data on logout
export function logout() {
  return (dispatch) => {
    dispatch({ type: 'LOGOUT' });
    dispatch({ type: 'CLEAR_USER_DATA' });
    dispatch({ type: 'CLEAR_PREFERENCES' });
    dispatch({ type: 'CLEAR_CACHE' });
  };
}

// Layer 2: Check user ID matches before showing data
function UserDashboard() {
  const user = useSelector(state => state.auth.user);
  const userData = useSelector(state => state.userData.data);

  // Don't show data if user IDs don't match
  const dataIsValid = userData && userData.userId === user?.id;

  return <div>{dataIsValid ? userData.name : 'Loading...'}</div>;
}

// Layer 3: Reset all reducers on logout
const appReducer = combineReducers({
  auth: authReducer,
  userData: userDataReducer,
  // ...
});

const rootReducer = (state, action) => {
  if (action.type === 'LOGOUT') {
    state = undefined; // ← Reset entire store
  }
  return appReducer(state, action);
};

Results

• No stale data shown after logout
• Clean state for each user session
• Prevents data leakage between users

Common Patterns Across Examples

Pattern: Unvalidated Input at Boundaries

Examples:

• Git init: Empty string not caught at entry point
• Database: Missing env var not validated at startup
• User ID: Missing token defaulted to 0

Solution: Validate at system boundaries (entry points, config loading)

Pattern: State Not Cleared on Transitions

Examples:

• Test setup: tempDir accessed before initialization
• React: User data not cleared on logout

Solution: Explicit state transitions with cleanup

Pattern: Magic Default Values

Examples:

• Empty string defaulting to process.cwd()
• 0 as default user ID
• undefined becoming string "undefined"

Solution: No magic defaults - fail fast with errors

Takeaways

1. Never stop at symptoms - Always trace to root cause 2. Fix at source - Don't add bandaids at error point 3. Add defense - Multiple layers catch similar issues 4. Document trace - Write down call chain as you go 5. Verify fix - Ensure fix addresses root cause, not just symptom

Integration with Other Skills

How root-cause-tracing integrates with systematic-debugging and other debugging skills to form a complete debugging toolkit.

Relationship with Systematic Debugging

The Big Picture

systematic-debugging is the overall framework:

• Phase 1: Root Cause Investigation
• Phase 2: Pattern Analysis
• Phase 3: Hypothesis and Testing
• Phase 4: Implementation

root-cause-tracing is a specialized technique used within Phase 1 when dealing with deep call stacks and unclear data origins.

When to Use Which Skill

User reports bug
  ↓
Activate: systematic-debugging
  ↓
Phase 1: Root Cause Investigation
  ↓
Error deep in call stack?
  → Yes: Use root-cause-tracing
    → Trace backward to find source
    → Return to systematic-debugging Phase 2
  → No: Continue with Phase 1
    → Read error messages
    → Check recent changes
    → Continue to Phase 2

Integration Flow

Example: Database Connection Error

1. Start with systematic-debugging Phase 1:

• Read error message: "Connection failed: database 'undefined' does not exist"
• Reproduce consistently: Yes, happens every time
• Check recent changes: No recent changes to database code

2. Recognize need for root-cause-tracing:

• Error happens deep in execution (database.connect())
• Unclear where "undefined" value originates
• Long call chain from app startup to connection

3. Apply root-cause-tracing:

• Step 1: Observe symptom → database name is "undefined"
• Step 2: Find immediate cause → Database constructor receives undefined
• Step 3: Trace to caller → Config object has undefined database field
• Step 4: Continue tracing → Environment variable not set
• Step 5: Root cause → Missing DATABASE_NAME env var

4. Return to systematic-debugging Phase 2:

• Find working examples: Other services have DATABASE_NAME set
• Compare differences: This service's .env missing the variable
• Understand dependencies: App requires DATABASE_NAME to start

5. Proceed to Phase 3 (Hypothesis):

• Hypothesis: Adding DATABASE_NAME env var will fix the issue
• Test minimally: Add DATABASE_NAME=testdb to .env
• Verify: App starts successfully

6. Continue to Phase 4 (Implementation):

• Write test: App startup should fail if DATABASE_NAME missing
• Implement fix: Add validation for required env vars
• Verify: All tests pass, app fails fast with clear error

Integration with Defense-in-Depth

After finding root cause with tracing, apply defense-in-depth pattern.

The Pattern

1. Fix at source (what root-cause-tracing finds) 2. Add validation at intermediate layers (defense-in-depth) 3. Fail fast with clear errors (both skills)

Example: Git Init in Wrong Directory

Root-cause-tracing finds:

• Source: setupCoreTest() returns empty tempDir before initialization

Defense-in-depth adds layers:

// Layer 0: Fix at source (root-cause-tracing result)
get tempDir(): string {
  if (!this._tempDir) {
    throw new Error('tempDir accessed before initialization');
  }
  return this._tempDir;
}

// Layer 1: Validate at entry point
static async create(name: string, directory: string) {
  if (!directory) throw new Error('Directory required');
  // ...
}

// Layer 2: Validate at workspace level
async initializeWorkspace(projectDir: string) {
  if (!projectDir) throw new Error('projectDir required');
  // ...
}

// Layer 3: Environment guard
async function gitInit(directory: string) {
  if (process.env.NODE_ENV === 'test' && !directory.includes('tmp')) {
    throw new Error('Test safety: refusing git init outside tmpdir');
  }
  // ...
}

// Layer 4: Instrumentation
async function gitInit(directory: string) {
  if (!directory || directory === process.cwd()) {
    console.error('SUSPICIOUS git init', { directory, stack: new Error().stack });
  }
  // ...
}

Result: Bug impossible at multiple levels

Integration with Verification-Before-Completion

After implementing fix, verify it worked.

The Pattern

1. Find root cause (root-cause-tracing) 2. Implement fix (systematic-debugging Phase 4) 3. Verify fix (verification-before-completion)

Verification Checklist

// After fixing root cause, verify:

// ✓ 1. Fix addresses root cause (not symptom)
const fix = 'Added validation for tempDir before access';
const rootCause = 'tempDir accessed before initialization';
// Matches? YES

// ✓ 2. All tests pass
npm test
// Result: 1847 tests passed

// ✓ 3. Specific test case for root cause
it('should throw if tempDir accessed before beforeEach', () => {
  const context = setupCoreTest();
  expect(() => context.tempDir).toThrow('before initialization');
});

// ✓ 4. No pollution/side effects
ls -la src/
// No .git directory in source

// ✓ 5. Defense layers working
// Each validation layer tested independently

Only after ALL checks pass: Mark as complete

Integration with Test-Driven-Development

Use TDD to create test for root cause (not symptom).

The Pattern

1. Find root cause (root-cause-tracing) 2. Write failing test for root cause (TDD) 3. Fix root cause (systematic-debugging) 4. Verify test passes (verification-before-completion)

Example: Writing Test for Root Cause

Symptom: Git init in wrong directory Root cause: tempDir accessed before initialization

WRONG: Test for symptom

// ❌ Tests symptom, not root cause
it('should not create .git in source directory', async () => {
  await Project.create('test', '');
  expect(fs.existsSync('.git')).toBe(false);
});

RIGHT: Test for root cause

// ✅ Tests root cause
it('should throw if tempDir accessed before initialization', () => {
  const context = setupCoreTest();
  expect(() => context.tempDir).toThrow('before initialization');
});

// ✅ Test defense layer
it('should validate directory parameter', async () => {
  await expect(Project.create('test', ''))
    .rejects.toThrow('Directory required');
});

Why this matters:

• Root cause test fails if bug reintroduced
• Symptom test might pass even if bug exists (different code path)
• Root cause test documents the actual issue

Integration with Condition-Based-Waiting

Sometimes root cause is a timing/race condition.

The Pattern

1. Trace reveals race condition (root-cause-tracing) 2. Replace timeouts with condition-based waiting (condition-based-waiting)

Example: Database Query Before Connection

Trace finds:

Error: Connection not established
  ← database.query() called
    ← from constructor initialization
      ← constructor doesn't wait for connection

Root cause: Query runs before connection completes

WRONG: Add timeout

constructor() {
  this.connect();
  await new Promise(resolve => setTimeout(resolve, 100)); // ❌ Race condition
}

RIGHT: Condition-based waiting

private connectionReady: Promise<void>;

constructor() {
  this.connectionReady = this.connect();
}

async query(sql: string) {
  await this.connectionReady; // ✅ Wait for actual condition
  return this.db.execute(sql);
}

Skill Activation Decision Tree

User reports bug
  ↓
Use: systematic-debugging (overall framework)
  ↓
Phase 1: Root Cause Investigation
  ↓
Is error deep in call stack?
  → Yes: Apply root-cause-tracing
    → Manual tracing sufficient?
      → Yes: Use tracing-techniques.md
      → No: Use advanced-techniques.md
    → Root cause found? → Continue to Phase 2
  → No: Continue Phase 1 investigation
  ↓
Phase 2: Pattern Analysis
  → Timing issue found?
    → Yes: Consider condition-based-waiting
  ↓
Phase 3: Hypothesis and Testing
  ↓
Phase 4: Implementation
  → Write test: Use test-driven-development
  → Add defense: Use defense-in-depth
  → Verify: Use verification-before-completion

Complete Debugging Toolkit

The Core Skills

1. systematic-debugging: Overall debugging framework (Phases 1-4) 2. root-cause-tracing: Technique for Phase 1 (deep call stacks) 3. defense-in-depth: Pattern for Phase 4 (multiple validation layers) 4. verification-before-completion: Ensures fix worked before claiming success 5. test-driven-development: Write tests for root cause, not symptoms

When to Use Each

Real-World Workflow Example

Bug: Application Crashes on Startup

Step 1: Activate systematic-debugging

• Read error: "TypeError: Cannot read property 'host' of undefined"
• Reproduce: Yes, happens every time
• Recent changes: None

Step 2: Recognize deep call stack → Use root-cause-tracing

Error at database.connect()
  ← from DatabaseService.initialize()
    ← from App.start()
      ← from main()

Step 3: Trace backward

• database.connect() receives undefined config
• DatabaseService.initialize() gets config from ConfigService
• ConfigService.load() returns undefined
• Root cause: Config file not loaded before DatabaseService initialized

Step 4: Return to systematic-debugging Phase 2

• Find working examples: Other apps load config in main()
• Compare: This app tries to load config in service initialization
• Pattern: Initialization order problem

Step 5: Phase 3 - Hypothesis

• Hypothesis: Loading config before services will fix issue
• Test: Move config loading to top of main()
• Result: Works!

Step 6: Phase 4 - Implementation with TDD

// Test for root cause
it('should load config before initializing services', () => {
  expect(ConfigService.isLoaded()).toBe(true);
});

// Test defense layer
it('should throw if DatabaseService initialized before config', () => {
  ConfigService.reset();
  expect(() => new DatabaseService()).toThrow('Config not loaded');
});

Step 7: Add defense-in-depth

• Layer 1: Load config first in main()
• Layer 2: DatabaseService validates config on init
• Layer 3: ConfigService throws if accessed before loaded

Step 8: verification-before-completion

• ✓ All tests pass
• ✓ App starts successfully
• ✓ Config loaded before services
• ✓ Clear error if config missing

Result: Bug fixed at root cause with multiple defensive layers

Common Anti-Patterns

Anti-Pattern 1: Tracing Without Framework

Wrong:

Use root-cause-tracing alone
  → Find root cause
  → Quick fix
  → No verification
  → Bug reappears later

Right:

Use systematic-debugging framework
  → Use root-cause-tracing in Phase 1
  → Continue through all phases
  → Verify fix
  → Bug stays fixed

Anti-Pattern 2: Symptom Tests

Wrong:

Root cause: Config not loaded
Test: "Should not crash on startup"

Right:

Root cause: Config not loaded
Test: "Should throw if config accessed before load"
Test: "Should load config before services"

Anti-Pattern 3: Fix Without Defense

Wrong:

Find root cause
  → Fix at source only
  → No validation at layers
  → Similar bug appears elsewhere

Right:

Find root cause
  → Fix at source
  → Add defense-in-depth
  → Test each layer
  → Similar bugs prevented

Summary

Root-cause-tracing is a technique, not a complete framework:

• Use within systematic-debugging Phase 1
• Combine with defense-in-depth after finding root cause
• Write tests for root cause with test-driven-development
• Verify fix with verification-before-completion
• Use condition-based-waiting for timing issues

The complete workflow: 1. systematic-debugging (framework) 2. root-cause-tracing (investigation technique) 3. test-driven-development (test root cause) 4. defense-in-depth (multiple layers) 5. verification-before-completion (ensure success)

Result: Bugs fixed permanently with comprehensive prevention

Tracing Techniques

Complete methodology for tracing bugs backward through call chains to find original triggers.

The Tracing Methodology

Overview

Root cause tracing follows a systematic approach to walk backward through code execution until you find where bad data or invalid state originated.

Key insight: The place where an error manifests is rarely where the bug actually lives.

Manual Tracing Process

Step 1: Observe the Symptom

What to capture:

• Exact error message
• Stack trace (if available)
• Failed operation
• Wrong value/state
• Location where error occurred

Example:

Error: git init failed in /Users/jesse/project/packages/core
  at WorktreeManager.createSessionWorktree (worktree-manager.ts:45)

Questions to ask:

• What operation failed?
• What was the expected behavior?
• What value/state is wrong?
• Where did the error manifest?

Step 2: Find Immediate Cause

Look at the code where the error occurs.

Example:

async function createSessionWorktree(projectDir: string, sessionId: string) {
  // This is where it fails
  await execFileAsync('git', ['init'], { cwd: projectDir });
}

Questions to ask:

• What code directly causes this error?
• What parameters does it receive?
• What assumptions does it make?
• Are the parameters valid?

Step 3: Identify the Caller

Trace one level up the call stack.

From stack trace:

  at WorktreeManager.createSessionWorktree (worktree-manager.ts:45)
  at Session.initializeWorkspace (session.ts:78)
  at Session.create (session.ts:34)
  at Test.<anonymous> (project.test.ts:12)

Manual code inspection:

// In session.ts
static async create(name: string, projectDir: string) {
  const session = new Session(name);
  await session.initializeWorkspace(projectDir); // ← Caller
  return session;
}

Questions to ask:

• What function called this?
• What value did it pass?
• Where did that value come from?

Step 4: Continue Tracing Up

Repeat step 3 for each caller until you find the source.

Trace chain example:

Test code (project.test.ts:12)
  → Project.create(name, context.tempDir)
    → Session.create(name, projectDir)
      → Session.initializeWorkspace(projectDir)
        → WorktreeManager.createSessionWorktree(projectDir, sessionId)
          → execFileAsync('git', ['init'], { cwd: projectDir })
            → ERROR: projectDir is empty string

Working backward:

• Where did projectDir = '' come from? → Session.create()
• Where did Session.create() get it? → Project.create()
• Where did Project.create() get it? → Test code
• Where did test code get it? → context.tempDir
• Where did context.tempDir = '' come from? → ROOT CAUSE FOUND

Step 5: Identify Root Cause

The root cause is where bad data/state originates.

Common root causes:

• Uninitialized variables accessed too early
• Configuration not loaded
• Null/undefined not handled
• Wrong default value
• Timing issue (race condition)
• Environment-specific behavior

In our example:

// Root cause: Getter returns empty string before initialization
function setupCoreTest() {
  let _tempDir = ''; // ← BAD: Empty string default

  beforeEach(() => {
    _tempDir = createTempDir(); // ← Set later
  });

  return { tempDir: _tempDir }; // ← Returns '' initially!
}

// Test code runs at module load time
const context = setupCoreTest();
const PROJECT_DIR = context.tempDir; // ← '' before beforeEach runs!

Tracing Patterns

Pattern 1: Data Flow Tracing

When to use: Invalid data appears somewhere in execution

Process: 1. Identify the invalid data at error point 2. Trace backward to find where it was set 3. Continue to where it originated 4. Fix at origin

Example:

userId = 0 (invalid) at database query
  ← from request.userId
    ← from parseRequest(req)
      ← from req.headers['user-id']
        ← ROOT: Header not validated/defaulted

Pattern 2: Call Chain Tracing

When to use: Error happens deep in call stack

Process: 1. Start at error location 2. Examine immediate caller 3. Move up one level 4. Repeat until finding where invalid call originates

Example:

Error in database.execute(query)
  ← from UserService.findUser(id)
    ← from AuthMiddleware.authenticate()
      ← from Router.handleRequest()
        ← ROOT: No authentication check before calling

Pattern 3: State Mutation Tracing

When to use: Object/variable has wrong value at some point

Process: 1. Identify when state is wrong 2. Find all places that mutate state 3. Trace backward to find which mutation caused it 4. Find why that mutation happened

Example:

config.database = undefined at startup
  ← config.database set to undefined in validateConfig()
    ← because env.DATABASE_URL is undefined
      ← ROOT: Environment variable not set

Pattern 4: Timing/Ordering Tracing

When to use: Issue involves race conditions or execution order

Process: 1. Identify operations that ran in wrong order 2. Trace why the order is wrong 3. Find what controls the ordering 4. Fix ordering logic

Example:

Database query before connection established
  ← query() called in constructor
    ← constructor runs immediately
      ← connection.connect() called async in initialize()
        ← ROOT: Constructor doesn't wait for async initialization

Decision Trees

When Manual Tracing is Sufficient

Can you see the code path?
  → Yes: Use manual tracing
    → Stack trace available?
      → Yes: Follow stack trace
      → No: Inspect caller manually
  → No: Add instrumentation (see advanced-techniques.md)

Is error reproducible?
  → Yes: Can trace reliably
  → No: Must make reproducible first
    → Add logging to capture when it happens
    → Use instrumentation to understand timing

How Deep to Trace

Found where bad value originates?
  → Yes: Root cause found
  → No: Continue tracing
    → At system boundary (entry point)?
      → Yes: Root cause is at boundary
      → No: Keep tracing backward

Multiple callers with same issue?
  → Yes: Root cause is in shared caller
  → No: Root cause is in specific caller path

Fixing vs Adding Defense

Found root cause?
  → Yes:
    → Fix at source: YES
    → Add validation at intermediate layers: YES
    → Fix only at error point: NO
  → No:
    → Add defense at error point: TEMPORARY
    → Continue tracing: YES

Common Tracing Challenges

Challenge 1: Long Call Chains

Problem: 10+ levels in call stack Solution:

• Use binary search: check middle of chain first
• Skip obviously correct intermediate calls
• Focus on where data changes

Challenge 2: Async/Callback Hell

Problem: Callbacks and promises obscure call chain Solution:

• Use async stack traces (Node.js: --async-stack-traces)
• Add trace IDs to log all operations
• Use debugger to step through promises

Challenge 3: Multiple Code Paths

Problem: Error could come from multiple callers Solution:

• Add conditional logging at error point
• Use instrumentation to capture caller info
• Reproduce with minimal test case

Challenge 4: External Dependencies

Problem: Issue might be in third-party library Solution:

• Verify inputs to library are correct
• Check library version/compatibility
• Read library source if needed
• Consider if misusing library API

Challenge 5: No Stack Trace

Problem: Error doesn't produce stack trace Solution:

• Add stack capture: new Error().stack
• Use debugger to pause at error point
• Add logging at suspected callers
• Use process of elimination

Tips for Effective Tracing

Use Your Tools

IDE navigation:

• "Find References" to see all callers
• "Go to Definition" to jump to implementation
• Call hierarchy view
• Type hierarchy for inheritance

Debugger:

• Set breakpoint at error point
• Step out to see caller
• Examine call stack panel
• Watch variables as they change

Version control:

• git blame to see when code changed
• git log to see recent changes
• git bisect to find when bug was introduced

Document Your Trace

As you trace, write down the call chain:

ERROR: git init in wrong directory
  ← execFileAsync('git', ['init'], { cwd: '' })
  ← WorktreeManager.createSessionWorktree(projectDir='')
  ← Session.initializeWorkspace(projectDir='')
  ← Session.create(name, projectDir='')
  ← Project.create(name, context.tempDir='')
  ← Test: const PROJECT_DIR = context.tempDir
  ← ROOT: setupCoreTest() returns { tempDir: '' } before beforeEach

This helps you:

• Remember where you are in the trace
• Communicate findings to others
• Verify your understanding
• Identify patterns

Know When to Stop Tracing

Stop when you find:

• The origin of bad data
• The first place where invariant is violated
• The entry point where validation should happen
• The configuration/initialization that sets wrong value

Don't stop at:

• Where error manifests
• Where symptom is visible
• Intermediate validation that fails
• Defensive checks that catch the issue

Verification After Tracing

Once you think you've found the root cause:

1. Verify understanding:

• Can you explain how the bug happens?
• Does your explanation match all symptoms?
• Can you predict what will happen if you fix it?

2. Test your hypothesis:

• Add temporary fix at suspected root cause
• Does it resolve the issue?
• Does it resolve ALL instances of the issue?

3. Implement proper fix:

• Fix at root cause
• Add tests for root cause
• Add defense-in-depth at intermediate layers
• Verify no regressions

Summary

Manual tracing process: 1. Observe symptom → 2. Find immediate cause → 3. Identify caller → 4. Trace up → 5. Find root cause

Key principles:

• Trace backward from symptom to source
• Don't stop at first "cause" - find original trigger
• Fix at source, add defense at layers
• Document your trace for clarity

When to use advanced techniques:

• Can't manually trace (see advanced-techniques.md)
• Need to identify which test pollutes (see advanced-techniques.md)
• Multiple async operations involved (see advanced-techniques.md)