Debugging

Debugging

Comprehensive debugging framework combining systematic investigation, root cause tracing, defense-in-depth validation, and verification protocols.

Core Principle

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

Random fixes waste time and create new bugs. Find the root cause, fix at source, validate at every layer, verify before claiming success.

When to Use

Always use for: Test failures, bugs, unexpected behavior, performance issues, build failures, integration problems, before claiming work complete

Especially when: Under time pressure, "quick fix" seems obvious, tried multiple fixes, don't fully understand issue, about to claim success

The Four Techniques

1. Systematic Debugging (references/systematic-debugging.md)

Four-phase framework ensuring proper investigation:

• Phase 1: Root Cause Investigation (read errors, reproduce, check changes, gather evidence)
• Phase 2: Pattern Analysis (find working examples, compare, identify differences)
• Phase 3: Hypothesis and Testing (form theory, test minimally, verify)
• Phase 4: Implementation (create test, fix once, verify)

Key rule: Complete each phase before proceeding. No fixes without Phase 1.

Load when: Any bug/issue requiring investigation and fix

2. Root Cause Tracing (references/root-cause-tracing.md)

Trace bugs backward through call stack to find original trigger.

Technique: When error appears deep in execution, trace backward level-by-level until finding source where invalid data originated. Fix at source, not at symptom.

Includes: scripts/find-polluter.sh for bisecting test pollution

Load when: Error deep in call stack, unclear where invalid data originated

3. Defense-in-Depth (references/defense-in-depth.md)

Validate at every layer data passes through. Make bugs impossible.

Four layers: Entry validation → Business logic → Environment guards → Debug instrumentation

Load when: After finding root cause, need to add comprehensive validation

4. Verification (references/verification.md)

Run verification commands and confirm output before claiming success.

Iron law: NO COMPLETION CLAIMS WITHOUT FRESH VERIFICATION EVIDENCE

Run the command. Read the output. Then claim the result.

Load when: About to claim work complete, fixed, or passing

Quick Reference

Bug → systematic-debugging.md (Phase 1-4)
  Error deep in stack? → root-cause-tracing.md (trace backward)
  Found root cause? → defense-in-depth.md (add layers)
  About to claim success? → verification.md (verify first)

Red Flags

Stop and follow process if thinking:

• "Quick fix for now, investigate later"
• "Just try changing X and see if it works"
• "It's probably X, let me fix that"
• "Should work now" / "Seems fixed"
• "Tests pass, we're done"

All mean: Return to systematic process.

Defense-in-Depth Validation

Validate at every layer data passes through to make bugs impossible.

Core Principle

Validate at EVERY layer data passes through. Make bug structurally impossible.

When fix bug caused by invalid data, adding validation at one place feels sufficient. But single check can be bypassed by different code paths, refactoring, or mocks.

Why Multiple Layers

Single validation: "We fixed bug" Multiple layers: "We made bug impossible"

Different layers catch different cases:

• Entry validation catches most bugs
• Business logic catches edge cases
• Environment guards prevent context-specific dangers
• Debug logging helps when other layers fail

The Four Layers

Layer 1: Entry Point Validation

Purpose: Reject obviously invalid input at API boundary

function createProject(name: string, workingDirectory: string) {
  if (!workingDirectory || workingDirectory.trim() === '') {
    throw new Error('workingDirectory cannot be empty');
  }
  if (!existsSync(workingDirectory)) {
    throw new Error(`workingDirectory does not exist: ${workingDirectory}`);
  }
  if (!statSync(workingDirectory).isDirectory()) {
    throw new Error(`workingDirectory is not a directory: ${workingDirectory}`);
  }
  // proceed
}

Layer 2: Business Logic Validation

Purpose: Ensure data makes sense for this operation

function initializeWorkspace(projectDir: string, sessionId: string) {
  if (!projectDir) {
    throw new Error('projectDir required for workspace initialization');
  }
  // proceed
}

Layer 3: Environment Guards

Purpose: Prevent dangerous operations in specific contexts

async function gitInit(directory: string) {
  // In tests, refuse git init outside temp directories
  if (process.env.NODE_ENV === 'test') {
    const normalized = normalize(resolve(directory));
    const tmpDir = normalize(resolve(tmpdir()));

    if (!normalized.startsWith(tmpDir)) {
      throw new Error(
        `Refusing git init outside temp dir during tests: ${directory}`
      );
    }
  }
  // proceed
}

Layer 4: Debug Instrumentation

Purpose: Capture context for forensics

async function gitInit(directory: string) {
  const stack = new Error().stack;
  logger.debug('About to git init', {
    directory,
    cwd: process.cwd(),
    stack,
  });
  // proceed
}

Applying the Pattern

When find bug:

1. Trace data flow - Where does bad value originate? Where used? 2. Map all checkpoints - List every point data passes through 3. Add validation at each layer - Entry, business, environment, debug 4. Test each layer - Try to bypass layer 1, verify layer 2 catches it

Example from Real Session

Bug: Empty projectDir caused git init in source code

Data flow: 1. Test setup → empty string 2. Project.create(name, '') 3. WorkspaceManager.createWorkspace('') 4. git init runs in process.cwd()

Four layers added:

• Layer 1: Project.create() validates not empty/exists/writable
• Layer 2: WorkspaceManager validates projectDir not empty
• Layer 3: WorktreeManager refuses git init outside tmpdir in tests
• Layer 4: Stack trace logging before git init

Result: All 1847 tests passed, bug impossible to reproduce

Key Insight

All four layers were necessary. During testing, each layer caught bugs others missed:

• Different code paths bypassed entry validation
• Mocks bypassed business logic checks
• Edge cases on different platforms needed environment guards
• Debug logging identified structural misuse

Don't stop at one validation point. Add checks at every layer.

Root Cause Tracing

Systematically trace bugs backward through call stack to find original trigger.

Core Principle

Trace backward through call chain until finding original trigger, then fix at source.

Bugs often manifest deep in call stack (git init in wrong directory, file created in wrong location). Instinct is to fix where error appears, but that's treating symptom.

When to Use

Use when:

• Error happens deep in execution (not at entry point)
• Stack trace shows long call chain
• Unclear where invalid data originated
• Need to find which test/code triggers problem

The Tracing Process

1. Observe the Symptom

Error: git init failed in /Users/jesse/project/packages/core

2. Find Immediate Cause

What code directly causes this?

await execFileAsync('git', ['init'], { cwd: projectDir });

3. Ask: What Called This?

WorktreeManager.createSessionWorktree(projectDir, sessionId)
  → called by Session.initializeWorkspace()
  → called by Session.create()
  → called by test at Project.create()

4. Keep Tracing Up

What value was passed?

• projectDir = '' (empty string!)
• Empty string as cwd resolves to process.cwd()
• That's the source code directory!

5. Find Original Trigger

Where did empty string come from?

const context = setupCoreTest(); // Returns { tempDir: '' }
Project.create('name', context.tempDir); // Accessed before beforeEach!

Adding Stack Traces

When can't trace manually, add instrumentation:

async function gitInit(directory: string) {
  const stack = new Error().stack;
  console.error('DEBUG git init:', {
    directory,
    cwd: process.cwd(),
    stack,
  });

  await execFileAsync('git', ['init'], { cwd: directory });
}

Critical: Use console.error() in tests (not logger - may not show)

Run and capture:

npm test 2>&1 | grep 'DEBUG git init'

Analyze stack traces:

• Look for test file names
• Find line number triggering call
• Identify pattern (same test? same parameter?)

Finding Which Test Causes Pollution

If something appears during tests but don't know which test:

Use bisection script: scripts/find-polluter.sh

./scripts/find-polluter.sh '.git' 'src/**/*.test.ts'

Runs tests one-by-one, stops at first polluter.

Key Principle

NEVER fix just where error appears. Trace back to find original trigger.

When found immediate cause:

• Can trace one level up? → Trace backwards
• Is this the source? → Fix at source
• Then add validation at each layer (see defense-in-depth.md)

Real Example

Symptom: .git created in packages/core/ (source code)

Trace chain: 1. git init runs in process.cwd() ← empty cwd parameter 2. WorktreeManager called with empty projectDir 3. Session.create() passed empty string 4. Test accessed context.tempDir before beforeEach 5. setupCoreTest() returns { tempDir: '' } initially

Root cause: Top-level variable initialization accessing empty value

Fix: Made tempDir a getter that throws if accessed before beforeEach

Also added defense-in-depth:

• Layer 1: Project.create() validates directory
• Layer 2: WorkspaceManager validates not empty
• Layer 3: NODE_ENV guard refuses git init outside tmpdir
• Layer 4: Stack trace logging before git init

Systematic Debugging

Four-phase debugging framework that ensures root cause investigation before attempting fixes.

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If haven't completed Phase 1, cannot propose fixes.

The Four Phases

Must complete each phase before proceeding to next.

Phase 1: Root Cause Investigation

BEFORE attempting ANY fix:

1. Read Error Messages Carefully - Don't skip past errors/warnings, read stack traces completely 2. Reproduce Consistently - Can trigger reliably? Exact steps? If not reproducible → gather more data 3. Check Recent Changes - What changed? Git diff, recent commits, new dependencies, config changes 4. Gather Evidence in Multi-Component Systems

• For EACH component boundary: log data entering/exiting, verify environment propagation
• Run once to gather evidence showing WHERE it breaks
• THEN analyze to identify failing component

5. Trace Data Flow - Where does bad value originate? Trace up call stack until finding source (see root-cause-tracing.md)

Phase 2: Pattern Analysis

Find pattern before fixing:

1. Find Working Examples - Locate similar working code in same codebase 2. Compare Against References - Read reference implementation COMPLETELY, understand fully before applying 3. Identify Differences - List every difference however small, don't assume "that can't matter" 4. Understand Dependencies - What other components, settings, config, environment needed?

Phase 3: Hypothesis and Testing

Scientific method:

1. Form Single Hypothesis - "I think X is root cause because Y", be specific not vague 2. Test Minimally - SMALLEST possible change to test hypothesis, one variable at a time 3. Verify Before Continuing - Worked? → Phase 4. Didn't work? → NEW hypothesis. DON'T add more fixes 4. When Don't Know - Say "I don't understand X", don't pretend, ask for help

Phase 4: Implementation

Fix root cause, not symptom:

1. Create Failing Test Case - Simplest reproduction, automated if possible, MUST have before fixing 2. Implement Single Fix - Address root cause identified, ONE change, no "while I'm here" improvements 3. Verify Fix - Test passes? No other tests broken? Issue actually resolved? 4. If Fix Doesn't Work

• STOP. Count: How many fixes tried?
• If < 3: Return to Phase 1, re-analyze with new information
• If ≥ 3: STOP and question architecture

5. If 3+ Fixes Failed: Question Architecture

• Pattern: Each fix reveals new shared state/coupling problem elsewhere
• STOP and question fundamentals: Is pattern sound? Wrong architecture?
• Discuss with human partner before more fixes

Red Flags - STOP and Follow Process

If catch yourself thinking:

• "Quick fix for now, investigate later"
• "Just try changing X and see if it works"
• "Add multiple changes, run tests"
• "Skip the test, I'll manually verify"
• "It's probably X, let me fix that"
• "I don't fully understand but this might work"
• "One more fix attempt" (when already tried 2+)

ALL mean: STOP. Return to Phase 1.

Human Partner Signals You're Doing It Wrong

• "Is that not happening?" - Assumed without verifying
• "Will it show us...?" - Should have added evidence gathering
• "Stop guessing" - Proposing fixes without understanding
• "Ultrathink this" - Question fundamentals, not just symptoms
• "We're stuck?" (frustrated) - Approach isn't working

When see these: STOP. Return to Phase 1.

Common Rationalizations

Real-World Impact

From debugging sessions:

• Systematic approach: 15-30 minutes to fix
• Random fixes approach: 2-3 hours of thrashing
• First-time fix rate: 95% vs 40%
• New bugs introduced: Near zero vs common

Verification Before Completion

Run verification commands and confirm output before claiming success.

Core Principle

Evidence before claims, always.

Claiming work complete without verification is dishonesty, not efficiency.

The Iron Law

NO COMPLETION CLAIMS WITHOUT FRESH VERIFICATION EVIDENCE

If haven't run verification command in this message, cannot claim it passes.

The Gate Function

BEFORE claiming any status or expressing satisfaction:

1. IDENTIFY: What command proves this claim?
2. RUN: Execute FULL command (fresh, complete)
3. READ: Full output, check exit code, count failures
4. VERIFY: Does output confirm claim?
   - If NO: State actual status with evidence
   - If YES: State claim WITH evidence
5. ONLY THEN: Make claim

Skip any step = lying, not verifying

Common Failures

Red Flags - STOP

• Using "should", "probably", "seems to"
• Expressing satisfaction before verification ("Great!", "Perfect!", "Done!")
• About to commit/push/PR without verification
• Trusting agent success reports
• Relying on partial verification
• Thinking "just this once"
• Tired and wanting work over
• ANY wording implying success without having run verification

Rationalization Prevention

Key Patterns

Tests:

✅ [Run test command] [See: 34/34 pass] "All tests pass"
❌ "Should pass now" / "Looks correct"

Regression tests (TDD Red-Green):

✅ Write → Run (pass) → Revert fix → Run (MUST FAIL) → Restore → Run (pass)
❌ "I've written regression test" (without red-green verification)

Build:

✅ [Run build] [See: exit 0] "Build passes"
❌ "Linter passed" (linter doesn't check compilation)

Requirements:

✅ Re-read plan → Create checklist → Verify each → Report gaps or completion
❌ "Tests pass, phase complete"

Agent delegation:

✅ Agent reports success → Check VCS diff → Verify changes → Report actual state
❌ Trust agent report

When To Apply

ALWAYS before:

• ANY variation of success/completion claims
• ANY expression of satisfaction
• ANY positive statement about work state
• Committing, PR creation, task completion
• Moving to next task
• Delegating to agents

Rule applies to:

• Exact phrases
• Paraphrases and synonyms
• Implications of success
• ANY communication suggesting completion/correctness

The Bottom Line

No shortcuts for verification.

Run command. Read output. THEN claim result.

Non-negotiable.

find-polluter.sh Test Documentation

Purpose

Bisection script to find which test creates unwanted files or state pollution.

Manual Test Procedure

Setup Test Scenario

# Create test directory
mkdir -p /tmp/polluter-test && cd /tmp/polluter-test

# Create clean test
cat > test1.test.js << 'EOF'
console.log('Test 1: clean');
EOF

# Create polluter test
cat > test2.test.js << 'EOF'
const fs = require('fs');
fs.mkdirSync('.git', { recursive: true });
console.log('Test 2: creates pollution');
EOF

# Create another clean test
cat > test3.test.js << 'EOF'
console.log('Test 3: clean');
EOF

Run Script

# For projects with npm test
/path/to/find-polluter.sh '.git' 'src/**/*.test.ts'

# For node-only tests (modify script to use 'node' instead of 'npm test')
./find-polluter.sh '.git' '*.test.js'

Expected Output

🔍 Searching for test that creates: .git
Test pattern: *.test.js

Found 3 test files

[1/3] Testing: ./test1.test.js
[2/3] Testing: ./test2.test.js

🎯 FOUND POLLUTER!
   Test: ./test2.test.js
   Created: .git

Cleanup

rm -rf /tmp/polluter-test

Test Results

✅ Script logic verified (2025-11-11)

• Correctly iterates through test files
• Detects pollution creation
• Reports the polluting test file
• Exits early when polluter found

Usage Notes

Prerequisites:

• Test runner (npm test) must be configured in project
• Test pattern must match actual test files
• Pollution path must be accurate

Customization: If your project doesn't use npm test, modify line 42:

# Replace
npm test "$TEST_FILE" > /dev/null 2>&1 || true

# With your test command
node "$TEST_FILE" > /dev/null 2>&1 || true
# Or
jest "$TEST_FILE" > /dev/null 2>&1 || true

Common Use Cases

1. Find test creating .git directory:

   ./find-polluter.sh '.git' 'src/**/*.test.ts'

2. Find test creating node_modules:

   ./find-polluter.sh 'node_modules' 'test/**/*.spec.js'

3. Find test creating specific file:

   ./find-polluter.sh 'unwanted-file.txt' '**/*.test.js'