Systematic Debugging

Systematic Debugging

When to Use

• A bug, error, exception, or crash needs investigation
• Something is "not working" and the cause is unclear
• A test is failing and the reason isn't obvious
• Unexpected behavior needs troubleshooting in any language or framework

Core Workflow

Follow these five phases sequentially. Do not skip ahead to fixing before completing isolation and tracing.

Phase 1: Reproduce

Establish a reliable way to trigger the bug before doing anything else.

1. Read the full error message, stack trace, and logs — note exact text, line numbers, and error codes 2. Create a minimal reproduction case that triggers the issue consistently 3. Record the exact steps, inputs, and environment that cause the failure

Checkpoint: Can you trigger the bug on demand? If intermittent, gather more data before proceeding.

Phase 2: Isolate

Narrow down where the failure originates.

1. Use binary search to find the failing component — disable or stub out halves of the system 2. Check recent changes with git log --oneline -20 and git diff against the last known good state 3. Add targeted logging or use a debugger to observe state at key boundaries

# Find which commit introduced the bug
git bisect start
git bisect bad HEAD
git bisect good <last-known-good-commit>
# Git will checkout midpoints — test each one and mark good/bad

Checkpoint: The bug is traced to a specific function, module, or data flow.

Phase 3: Trace to Root Cause

Understand why the failure happens — not just where.

1. Read the code path completely from entry point through the failure site 2. Check assumptions: what does each function expect vs. what it actually receives? 3. Trace data flow backward — where does the bad value originate? 4. Verify with evidence: add assertions or print statements to confirm your hypothesis

# Example: verify assumptions about incoming data
def process_order(order):
    assert order.status == "pending", f"Expected pending, got {order.status}"
    assert order.items, "Order has no items"
    # ... rest of processing

Checkpoint: The chain of causation from trigger to symptom is explained, with supporting evidence (logs, assertions, debugger output).

Phase 4: Fix at Root Cause

Apply a targeted fix that addresses the actual cause, not just the symptom.

1. Fix the root cause, not a downstream effect 2. Keep the fix minimal — change only what's necessary 3. Avoid "band-aid" fixes that mask the underlying problem (e.g., adding a try/except around a crash without fixing why it crashes)

Phase 5: Verify

Confirm the fix works and doesn't introduce regressions.

1. Run the reproduction case from Phase 1 — confirm the bug is gone 2. Run the full test suite to check for regressions 3. Test edge cases related to the fix 4. If the bug was missing a test, add one that would have caught it

Checkpoint: Reproduction case passes, test suite is green, and you have a new test covering this bug.

Key Anti-Patterns to Avoid

• Shotgun debugging: making random changes hoping something works
• Fix-and-pray: applying a fix without understanding the cause
• Skipping reproduction: jumping to code changes without confirming you can trigger the bug
• Fixing symptoms: wrapping errors in try/catch instead of fixing what produces them

See [anti-patterns.md](references/anti-patterns.md) for the full catalog.

Related Skills

• [root-cause-tracing](../root-cause-tracing/SKILL.md): Deep call-stack tracing techniques — use after Phase 2 when the bug is deep in execution chains
• [verification-before-completion](../verification-before-completion/SKILL.md): Mandatory verification gates — reinforces Phase 5 before claiming a fix is complete

Deep-Dive References

• [workflow.md](references/workflow.md): Detailed phase-by-phase instructions with decision trees
• [examples.md](references/examples.md): Worked debugging examples across languages
• [troubleshooting.md](references/troubleshooting.md): Common debugging scenarios and solutions
• [anti-patterns.md](references/anti-patterns.md): Patterns to avoid and how to recognize them

Debugging Anti-Patterns

Part of: Systematic Debugging

Category: debugging

Reading Level: Intermediate

Purpose

Common debugging mistakes, rationalizations, and red flags to avoid. Learn to recognize when you're violating systematic debugging principles.

Red Flags - Recognize and Stop

"Quick Fix for Now, Investigate Later"

What It Sounds Like:

• "Let me just try this quick fix first"
• "We'll investigate the root cause after shipping"
• "This workaround will hold us over"

Why It's Wrong:

• "Later" never comes
• Quick fixes become permanent
• Masks underlying issues
• Creates technical debt

Reality Check:

Quick fix:     15 min → ships with bug hidden
Investigation: 30 min → bug returns different form
Total:         45 min + future debugging time

Systematic:    25 min → bug actually fixed
Total:         25 min, done forever

What to Do Instead: Spend 25 minutes finding root cause now vs 2+ hours fixing symptoms repeatedly.

"Just Try Changing X and See If It Works"

What It Sounds Like:

• "Maybe if we change this..."
• "Let's try increasing the timeout"
• "What if we add try-catch here?"
• "Could we just restart the service?"

Why It's Wrong:

• Random changes waste time
• If it works, you don't know why
• If it fails, you learned nothing
• Creates more bugs

Reality Check:

// Random attempt 1
setTimeout(() => process(), 5000);  // Didn't work

// Random attempt 2
setTimeout(() => process(), 10000); // Didn't work

// Random attempt 3
setTimeout(() => process(), 20000); // Works sometimes?

// You still don't know the actual problem

What to Do Instead: Gather evidence about why it's timing out, then fix root cause.

"Add Multiple Changes, Run Tests"

What It Sounds Like:

• "Let me fix all these issues at once"
• "While I'm here, I'll also..."
• "These changes are all related"

Why It's Wrong:

• Can't isolate what actually fixed it
• If tests fail, which change broke it?
• If it works, which change was necessary?
• Makes debugging harder

Reality Check:

Changed: Auth logic, error handling, timeout value
Test passes
Which change fixed it? Unknown
Are all changes necessary? Unknown
Can we safely refactor? Unknown

What to Do Instead: One change at a time, verify each change independently.

"I'll Write Test After Confirming Fix Works"

What It Sounds Like:

• "Let me verify the fix manually first"
• "I'll add tests once I know it works"
• "Testing can come after we fix it"

Why It's Wrong:

• Manual testing misses edge cases
• No regression protection
• Can't verify fix in CI
• Bug likely returns

Reality Check:

Manual test:
1. Try fix
2. Seems to work
3. Ship it
4. Bug returns in different scenario
5. Debug again

Test-first:
1. Write failing test
2. Fix until test passes
3. Test catches regressions forever

What to Do Instead: Write failing test reproducing bug, then fix until test passes.

"It's Probably X, Let Me Fix That"

What It Sounds Like:

• "I'm pretty sure it's..."
• "This looks like that bug we had before"
• "Usually this means..."
• "Based on my experience..."

Why It's Wrong:

• "Probably" is not evidence
• Each bug is unique
• Assumptions lead to wrong fixes
• Wastes time on wrong solution

Reality Check:

Assumption: "Probably a caching issue"
Fix:         Clear cache
Result:      Still broken
Cause:       Actually a validation bug
Time wasted: 45 minutes

What to Do Instead: Gather evidence, form hypothesis based on data, test hypothesis.

"One More Fix Attempt" (After 2+)

What It Sounds Like:

• "Let me try one more thing"
• "This fix should definitely work"
• "Third time's the charm"

Why It's Wrong:

• 3+ failures indicate architectural problem
• More symptom fixing won't help
• Digging deeper into wrong solution
• Need to question fundamentals

Reality Check:

Fix 1: Add retry logic         → Failed
Fix 2: Increase retry count    → Failed
Fix 3: Add exponential backoff → Failed

Pattern: Each fix assumes retrying will work
Reality: Problem isn't retry-able error

What to Do Instead: STOP. Question whether approach is fundamentally sound.

Common Rationalizations

"Issue Is Simple, Don't Need Process"

The Rationalization: "This is obviously a simple typo/config issue, systematic debugging is overkill"

Reality:

• Simple issues have root causes too
• "Simple" bugs often hide deeper problems
• Process is fast for simple bugs (5-10 minutes)
• "Simple" assumption often wrong

Counter-Example:

Seems simple: Variable name typo
Following process reveals: Copy-pasted code in 5 places
Systematic fix: Abstract to shared function
Random fix: Fix one instance, bug persists elsewhere

"Emergency, No Time for Process"

The Rationalization: "Production is down, we need to fix NOW, can't follow slow process"

Reality:

• Systematic debugging is FASTER than guess-and-check
• Random fixes in emergencies often make it worse
• Process ensures you actually fix it
• Rushing guarantees longer downtime

Time Comparison:

"Quick Fix" Approach:
Attempt 1:  10 min → makes it worse
Attempt 2:  15 min → partially works
Attempt 3:  20 min → still broken
Call expert: 30 min
Total:       75 min downtime

Systematic Approach:
Phase 1:    10 min → identify root cause
Phase 4:    10 min → correct fix
Total:      20 min downtime

"Reference Too Long, I'll Adapt the Pattern"

The Rationalization: "I understand the general idea, don't need to read all the details"

Reality:

• Partial understanding guarantees bugs
• Details matter - that's why they're documented
• Adaptation without full understanding fails
• Reading takes 10 minutes, fixing bugs takes hours

Counter-Example:

Reference: "Initialize connection pool before server starts"
Skimmed:   "Initialize connection pool" ✓
Missed:    "Before server starts"
Result:    Race condition, intermittent failures

"I See the Problem, Let Me Fix It"

The Rationalization: "I can see what's wrong, no need to investigate further"

Reality:

• Seeing symptoms ≠ understanding root cause
• "Obvious" fixes often miss deeper issues
• What you see is rarely the actual problem
• Quick "fixes" become permanent workarounds

Counter-Example:

Symptom:    NullPointerException at line 45
"Obvious":  Add null check at line 45
Root cause: Function at line 23 returns null unexpectedly
Correct fix: Fix function at line 23
Your "fix":  Hides real problem, bug persists elsewhere

Pattern Recognition: Bad Debugging

Pattern: Timeout Spiral

Progression: 1. Operation times out 2. Increase timeout 3. Still times out 4. Increase timeout more 5. Eventually works (sometimes) 6. Actual problem never fixed

Why It Fails: Timeouts are symptoms, not root causes

Correct Approach: 1. Why is it timing out? 2. What operation is slow? 3. Fix slow operation 4. Remove arbitrary timeout

Pattern: Try-Catch Cascade

Progression: 1. Error occurs 2. Add try-catch to hide error 3. Different error occurs 4. Add another try-catch 5. More errors appear elsewhere 6. Code full of error handling, problem not solved

Why It Fails: Catching errors doesn't fix causes

Correct Approach: 1. What's causing the error? 2. Fix root cause 3. Add error handling only for truly exceptional cases

Pattern: Configuration Whack-a-Mole

Progression: 1. Doesn't work 2. Change config setting 3. Different failure 4. Change another setting 5. Previous failure returns 6. Config becomes mystery state

Why It Fails: Random config changes without understanding

Correct Approach: 1. What does this config control? 2. What's the correct value for our use case? 3. Why was previous value wrong? 4. Set correct value once

Pattern: Copy-Paste from Stack Overflow

Progression: 1. Search error message 2. Find Stack Overflow answer 3. Copy-paste code 4. Seems to work 5. Causes subtle bugs later 6. Don't understand what it does

Why It Fails: Code without understanding

Correct Approach: 1. Find Stack Overflow answer 2. Read and understand it 3. Verify it applies to your case 4. Adapt it appropriately 5. Add comments explaining what and why

Human Partner Signals

Signal Categories

Questions = You Assumed Without Verifying

• "Is that not happening?"
• "Did you check...?"
• "Will it show us...?"
• "Are you sure...?"

Commands = You're Off Track

• "Stop guessing"
• "Gather evidence first"
• "One change at a time"
• "Write a test"

Frustration = Your Approach Isn't Working

• "We're stuck?"
• "This isn't working"
• "Let's try different approach"
• "Ultrathink this"

How to Respond

Don't:

• Argue or defend
• Explain why you did it that way
• Continue with current approach
• Make excuses

Do:

• STOP immediately
• Acknowledge the redirect
• Return to Phase 1
• Gather evidence
• Ask for guidance if unclear

Example Responses:

Signal: "Is that not happening?"
Bad:    "I assumed it would..."
Good:   "Let me verify that with evidence"

Signal: "Stop guessing"
Bad:    "I'm not guessing, I think..."
Good:   "You're right, let me investigate root cause first"

Signal: "We're stuck?"
Bad:    "Let me try one more thing..."
Good:   "Let me reconsider the approach from Phase 1"

Self-Assessment Questions

Ask yourself these questions to catch anti-patterns:

Before Making Changes

• [ ] Have I gathered evidence about root cause?
• [ ] Do I have a specific hypothesis?
• [ ] Can I explain why this change should work?
• [ ] Am I changing only one thing?
• [ ] Have I written a test case?

If Answer Is "No"

STOP. Return to Phase 1.

After Multiple Attempts

• [ ] How many fixes have I tried? (If ≥3, STOP)
• [ ] Am I fixing symptoms or root cause?
• [ ] Is each fix revealing new problems?
• [ ] Should I question the architecture?

If In Doubt

• [ ] Would systematic debugging be faster?
• [ ] Am I rationalizing shortcuts?
• [ ] What would I tell someone else to do?

Recovery from Anti-Patterns

If You Realize You're Doing It Wrong

Immediate Actions: 1. STOP - Don't make more random changes 2. Revert - Back to known good state 3. Document - What you tried (data for Phase 1) 4. Restart - Phase 1 with fresh perspective

Don't:

• Keep changes "that might help"
• Try "one more quick thing"
• Feel bad about time "wasted"
• Rush to "make up for lost time"

Do:

• Clean slate
• Systematic approach from start
• Use failed attempts as evidence
• Take time to do it right

Sunk Cost Fallacy

The Trap: "I've already spent 2 hours on random fixes, can't give up now"

The Reality:

• Those 2 hours are gone regardless
• Continuing wastes more time
• Starting over with systematic approach is faster
• 25 minutes systematic > 4 hours random

The Decision:

Option A: Continue random approach
Cost:     2 hours spent + 2 more hours likely
Result:   Maybe works, likely more bugs

Option B: Restart systematically
Cost:     2 hours spent + 25 minutes systematic
Result:   Definitely works, no new bugs

Option B is clearly better despite sunk cost

Summary

Common Anti-Patterns:

• Quick fixes without investigation
• Random changes hoping they work
• Multiple simultaneous changes
• Tests after instead of before
• "Probably" instead of evidence
• Continuing after 3+ failures

Common Rationalizations:

• "Too simple for process"
• "Emergency, no time"
• "I'll adapt the pattern"
• "I see the problem"

Recovery:

• Recognize the pattern
• STOP immediately
• Revert to known state
• Restart Phase 1 systematically

Remember: Systematic debugging is faster, even when it feels slow. Random fixes always take longer.

Related References

• Workflow: Correct four-phase process
• Examples: Real-world systematic debugging
• Troubleshooting: When debugging gets hard

Real-World Debugging Examples

Part of: Systematic Debugging

Category: debugging

Reading Level: Intermediate

Purpose

Real-world scenarios demonstrating systematic debugging in action, with step-by-step walkthroughs showing how to apply the four-phase process.

Example 1: API Integration Failure

Symptom

Error: API request failed with status 401
Tests pass locally but fail in CI

Phase 1: Root Cause Investigation

Read Error:

HTTP 401 Unauthorized
Response: {"error": "Invalid API key"}

Reproduce:

• ✓ Runs successfully locally
• ✗ Fails in CI every time
• Difference: Local has .env file, CI uses environment variables

Check Changes:

git log --oneline --since="3 days ago"
# Found: "Add API key authentication" 2 days ago

Gather Evidence:

# In CI config
echo "API_KEY present: ${API_KEY:+YES}${API_KEY:-NO}"
# Output: API_KEY present: NO

# In local env
echo $API_KEY
# Output: sk-abc123...

Root cause identified: API_KEY environment variable not set in CI

Phase 2: Pattern Analysis

Find working examples:

# Other workflows using secrets
env:
  DATABASE_URL: ${{ secrets.DATABASE_URL }}  # Works
  API_KEY: ${{ secrets.API_KEY }}            # Missing!

Identify difference: DATABASE_URL configured in GitHub secrets, API_KEY is not

Phase 3: Hypothesis and Testing

Hypothesis: "API_KEY secret not configured in GitHub repository settings"

Test: Check repository settings → Secrets → API_KEY not found

Confirmed: Secret needs to be added

Phase 4: Implementation

Test case (manual verification):

# After adding secret, check in CI
echo "API_KEY set: ${API_KEY:+YES}"
# Expected: YES

Fix: Add API_KEY to GitHub repository secrets

Verify: CI build passes, API calls succeed

Time: 15 minutes with systematic approach vs 2+ hours guessing

Example 2: Intermittent Test Failure

Symptom

Test 'processes user data' fails randomly
Passes 80% of the time, fails 20%
No clear pattern to failures

Phase 1: Root Cause Investigation

Read Error:

Expected: { id: 1, name: 'Alice', role: 'admin' }
Received: { id: 1, name: 'Alice', role: 'user' }

Reproduce:

• Run test 10 times → fails 2-3 times
• Failure seems random
• Not time-dependent

Check Recent Changes:

git diff HEAD~1 -- tests/user.test.ts
# No changes to test
# But role assignment logic changed

Gather Evidence:

// Add logging
test('processes user data', async () => {
  console.log('Test start:', new Date().toISOString());
  const user = await createUser({ name: 'Alice', role: 'admin' });
  console.log('User created:', user);
  console.log('Role at creation:', user.role);
  // ...
});

Pattern found: Role correct at creation, changes later

Trace Data Flow:

// Step through code
createUser() → saveToDatabase() → applyDefaults() → role changes!

// applyDefaults has bug:
function applyDefaults(user) {
  return {
    ...user,
    role: user.role || 'user'  // BUG: 'user' overwrites existing role
  };
}

Phase 2: Pattern Analysis

Find working examples:

// Correct default handling
function applyDefaults(user) {
  return {
    role: 'user',  // Default first
    ...user,       // Then overrides
  };
}

Identify difference: Order of spread operator matters

Phase 3: Hypothesis and Testing

Hypothesis: "Spread operator order causes role to be overwritten with default"

Test:

// Minimal test
const result = { ...{ name: 'Alice', role: 'admin' }, role: 'user' };
console.log(result.role);  // 'user' - overwrites!

const correct = { role: 'user', ...{ name: 'Alice', role: 'admin' } };
console.log(correct.role);  // 'admin' - correct!

Confirmed: Spread order is the issue

Phase 4: Implementation

Test case:

test('preserves existing role when applying defaults', () => {
  const user = { name: 'Alice', role: 'admin' };
  const result = applyDefaults(user);
  expect(result.role).toBe('admin');
});

Fix:

function applyDefaults(user) {
  return {
    role: 'user',  // Default first
    status: 'active',
    ...user,       // User values override
  };
}

Verify: Test passes 100 consecutive times, no more intermittent failures

Example 3: Performance Degradation

Symptom

Dashboard loads in 5+ seconds (was <1 second)
Users complaining about slowness
No recent deploys

Phase 1: Root Cause Investigation

Read Error: No explicit error, just slow performance

Reproduce:

# Measure consistently
time curl http://localhost:3000/dashboard
# ~5.2 seconds consistently

Check Recent Changes:

git log --oneline --since="1 week ago" -- src/dashboard/
# No code changes to dashboard

git log --oneline --since="1 week ago" -- database/
# Found: "Add index to improve user queries" 3 days ago

Gather Evidence:

-- Enable query logging
SET log_min_duration_statement = 100;

-- Analyze slow queries
EXPLAIN ANALYZE SELECT * FROM user_activity
WHERE user_id = 123
ORDER BY created_at DESC;

-- Output shows: Seq Scan on user_activity (cost=0.00..45678.23 rows=500000)

Pattern found: Query doing sequential scan despite new index

Phase 2: Pattern Analysis

Check index:

\d user_activity
-- Indexes:
--   "user_activity_user_created_idx" btree (user_id, created_at)

-- But query doesn't use it!

Find working examples:

-- This query uses index
SELECT * FROM user_sessions WHERE user_id = 123;

-- Our query doesn't - why?
SELECT * FROM user_activity WHERE user_id = 123 ORDER BY created_at DESC;

Identify difference: ORDER BY direction (DESC) doesn't match index (ASC)

Phase 3: Hypothesis and Testing

Hypothesis: "Index not used because ORDER BY DESC doesn't match index order ASC"

Test:

-- Drop and recreate with DESC
DROP INDEX user_activity_user_created_idx;
CREATE INDEX user_activity_user_created_idx
  ON user_activity (user_id, created_at DESC);

-- Test query performance
EXPLAIN ANALYZE SELECT * FROM user_activity
WHERE user_id = 123
ORDER BY created_at DESC;

-- Output: Index Scan using user_activity_user_created_idx (cost=0.43..123.45)

Confirmed: Index now used, performance improved

Phase 4: Implementation

Test case:

test('dashboard loads in under 1 second', async () => {
  const start = Date.now();
  await fetch('/dashboard');
  const duration = Date.now() - start;
  expect(duration).toBeLessThan(1000);
});

Fix: Update index definition to match query pattern

Verify:

• Dashboard loads in 0.4 seconds
• All other queries still work
• No performance regressions

Example 4: Multi-Component System Failure

Symptom

iOS app code signing fails in CI
Error: "No identity found"
Works on developer machines

Phase 1: Root Cause Investigation

Read Error:

error: No signing certificate "iOS Distribution" found
codesign failed with exit code 1

Reproduce:

• ✓ Works locally with Xcode
• ✗ Fails in CI every time
• Multi-layer system: CI → build → keychain → signing

Gather Evidence at Each Layer:

# Layer 1: CI environment
echo "=== Secrets available ==="
echo "CERT_P12: ${CERT_P12:+SET}${CERT_P12:-UNSET}"
echo "CERT_PASSWORD: ${CERT_PASSWORD:+SET}${CERT_PASSWORD:-UNSET}"
# Output: Both SET

# Layer 2: Keychain setup
echo "=== Keychain state ==="
security list-keychains
security find-identity -v -p codesigning
# Output: No identities found

# Layer 3: Certificate import
security import cert.p12 -k ~/Library/Keychains/build.keychain -P "$CERT_PASSWORD" -T /usr/bin/codesign
echo "Import exit code: $?"
# Output: Exit code 0 (success)

# But still no identity!
security find-identity -v -p codesigning
# Output: Still no identities

Pattern found: Import succeeds but identity not available

Phase 2: Pattern Analysis

Find working examples:

# Local machine
security find-identity -v
# Shows many identities in login keychain

# CI (after import)
security find-identity -v
# Shows nothing

Compare keychain paths:

# Local
security list-keychains
# "~/Library/Keychains/login.keychain-db"

# CI
security list-keychains
# Custom build.keychain not in search list!

Identify difference: Keychain created but not added to search list

Phase 3: Hypothesis and Testing

Hypothesis: "Identity imported but keychain not in search list, so codesign can't find it"

Test:

# Add keychain to search list
security list-keychains -s ~/Library/Keychains/build.keychain-db

# Check if identity now visible
security find-identity -v -p codesigning
# Output: Shows imported identity!

Confirmed: Keychain search list was the issue

Phase 4: Implementation

Test case (integration test in CI):

# After fix, verify identity available
security find-identity -v -p codesigning | grep "iOS Distribution" || exit 1

Fix:

# Complete keychain setup
security create-keychain -p "$KEYCHAIN_PASSWORD" build.keychain
security list-keychains -s ~/Library/Keychains/build.keychain-db  # Add to search
security unlock-keychain -p "$KEYCHAIN_PASSWORD" build.keychain
security import cert.p12 -k build.keychain -P "$CERT_PASSWORD" -T /usr/bin/codesign
security set-key-partition-list -S apple-tool:,apple: -s -k "$KEYCHAIN_PASSWORD" build.keychain

Verify: Code signing succeeds in CI

Time: 45 minutes systematic vs 4+ hours of random attempts

Key Patterns Across Examples

Common Root Causes

1. Configuration differences (Example 1): Local vs CI environment 2. Timing/order issues (Example 2): Race conditions, initialization order 3. Hidden assumptions (Example 3): Index order matching query 4. Multi-layer problems (Example 4): Issue at component boundaries

Systematic Approach Benefits

• Faster resolution: 15-45 minutes vs 2-4 hours
• First-time fix: 95%+ success rate
• No new bugs: Targeted fixes don't break other things
• Knowledge gained: Understanding root cause prevents recurrence

Red Flags That Would Have Failed

• "Just try adding the secret" (skips investigation)
• "Add try-catch around failing code" (masks problem)
• "Increase timeout" (hides real issue)
• "Maybe clear cache?" (random guess)

Summary

Systematic debugging: 1. Saves time (15-45 min vs 2-4 hours) 2. Fixes correctly first time (95% vs 40%) 3. Prevents new bugs (targeted vs random changes) 4. Builds understanding (root cause vs symptom)

Related References

• Workflow: Complete four-phase process
• Troubleshooting: When debugging gets stuck
• Anti-patterns: Common mistakes to avoid

Debugging Troubleshooting Guide

Part of: Systematic Debugging

Category: debugging

Reading Level: Advanced

Purpose

Solutions for common challenges that arise during systematic debugging, including what to do when you get stuck, when the process seems slow, or when human partners redirect you.

Common Debugging Challenges

Challenge 1: Cannot Reproduce Issue

Problem: Bug reported but can't trigger it reliably

Solution Approach:

1. Gather More Context

   Ask reporter:
   - Exact steps they took
   - Browser/OS/environment details
   - Screenshots or video
   - Error messages they saw
   - When it started happening

2. Check for Environmental Factors

• Time of day (server load patterns)
• User account state (permissions, data)
• Network conditions (latency, failures)
• Cache state (fresh vs cached)
• Concurrent operations

3. Add Comprehensive Logging

   // Log everything around suspected area
   logger.info('Function entry', { input, state });
   logger.info('Step 1 complete', { intermediate });
   logger.info('Decision point', { condition, willTake: path });
   logger.info('Function exit', { result });

4. Monitor for Patterns

• Does it happen at specific times?
• Only for certain users?
• Only on certain data?
• Percentage occurrence?

When to Stop: If truly irreproducible after thorough investigation, document findings and add defensive error handling.

Challenge 2: Too Many Possible Causes

Problem: Multiple things could cause this issue

Solution Approach:

1. Binary Search

   Disable half the functionality
   → Still broken? Issue in remaining half
   → Fixed? Issue in disabled half
   Repeat until narrowed to single component

2. Isolate Variables

   // Test each variable independently
   const test1 = processWithA(data);
   const test2 = processWithB(data);
   const test3 = processWithC(data);

   // Which fails?

3. Priority by Likelihood

   Most likely causes (based on evidence):
   1. Recent code changes → Check first
   2. Known fragile areas → Check second
   3. External dependencies → Check third
   4. Theoretical possibilities → Check last

4. Eliminate Systematically

• Start with most likely cause
• Test one hypothesis at a time
• Document which causes ruled out
• Don't test multiple theories simultaneously

Challenge 3: Error Messages Are Cryptic

Problem: Error message doesn't clearly indicate root cause

Solution Approach:

1. Search for Exact Error

   # Google with quotes
   "exact error message text"

   # Search codebase
   grep -r "exact error message" src/

   # Check documentation
   # Stack Overflow, GitHub issues

2. Understand Error Source

   // Where is error thrown?
   throw new Error('Cryptic message');

   // Check call stack - WHO called this?
   // Work backward from error

3. Increase Verbosity

   # Debug mode
   DEBUG=* npm start

   # Verbose flags
   command --verbose --debug

   # Enable all logging
   LOG_LEVEL=debug

4. Read Library Source

• If error from library, read library code
• Check library issue tracker
• Look for related error messages

Challenge 4: Fix Doesn't Work

Problem: Implemented fix but issue persists

Solution Approach:

1. Verify Fix Actually Applied

   // Add logging to confirm
   if (condition) {
     console.log('FIX APPLIED: New code path');
     // new code
   }

   // Did you see the log? If not, fix not reached.

2. Check Fix Location

• Fixed right file?
• Fixed right function?
• All instances fixed?
• Code recompiled/reloaded?

3. Verify Understanding

• Was hypothesis correct?
• Did you misunderstand the problem?
• Is there a deeper issue?

4. Count Attempts

• Attempt 1 failed → Re-analyze
• Attempt 2 failed → Deeper investigation
• Attempt 3 failed → STOP, question architecture

Challenge 5: Multiple Bugs Overlap

Problem: Fixing one bug reveals another

Solution Approach:

1. Separate Issues

   Bug A: Authentication fails
   Bug B: Authorization missing
   Bug C: Error handling wrong

   These are THREE separate bugs
   Fix ONE at a time

2. Priority Order

• Fix deepest issue first (authentication)
• Then dependent issues (authorization)
• Then surface issues (error handling)

3. Track Each Separately

   Create separate:
   - Test cases
   - Fixes
   - Verifications

   Don't bundle fixes

4. If They Keep Cascading

• This indicates architectural problem
• STOP fixing symptoms
• Discuss fundamental design with human partner

Challenge 6: Debugging Takes Too Long

Problem: Hours spent, no progress

Solution Approach:

1. Assess Current Phase

   Still in Phase 1? → Not enough evidence gathered
   Stuck in Phase 3? → Hypotheses too vague
   Repeated Phase 4? → Wrong architecture

2. Check for Red Flags

• Making random changes? → Return to Phase 1
• Guessing without evidence? → Gather more data
• Fixing symptoms? → Find root cause
• 3+ fix attempts? → Question architecture

3. Start Over

   If stuck after 2+ hours:
   1. Write down everything you know
   2. List assumptions you've made
   3. Question each assumption
   4. Start Phase 1 fresh

4. Ask for Help

• Explain to human partner
• Describe evidence gathered
• Share hypotheses tested
• Ask for guidance

Challenge 7: Human Partner Signals You're Wrong

Problem: Human partner redirects you with questions

Common Signals:

Solution Approach:

1. Recognize the Signal

• Human partner is redirecting for a reason
• They see something you're missing
• Don't argue or defend

2. STOP Current Approach

• Whatever you're doing isn't working
• Return to Phase 1
• Gather more evidence

3. Respond Appropriately

   Signal: "Is that not happening?"
   Response: "Let me verify that assumption with evidence"

   Signal: "Stop guessing"
   Response: "You're right, let me investigate the root cause first"

   Signal: "We're stuck?"
   Response: "Let me reconsider the approach from Phase 1"

4. Learn the Pattern

• Note what triggered the redirect
• Don't repeat that mistake
• Adjust debugging approach

When Systematic Approach Seems Slow

Perception vs Reality

Feeling: "This process is taking too long, just try a quick fix"

Reality:

• Systematic: 15-45 minutes → correct fix
• Random: 2-4 hours thrashing → maybe works

Remember: Time spent in Phase 1 is time saved avoiding wrong fixes

Time Breakdown

Phase 1 (Root Cause):    10-20 minutes
Phase 2 (Pattern):        5-10 minutes
Phase 3 (Hypothesis):     5-10 minutes
Phase 4 (Implementation): 5-15 minutes
TOTAL:                   25-55 minutes

Random Fix Approach:
Attempt 1:               15 min → fails
Attempt 2:               20 min → fails
Attempt 3:               30 min → partially works
Debug new issues:        60 min
TOTAL:                   125+ minutes

When It Actually IS Slow

If systematic approach taking 2+ hours:

1. Check Evidence Quality

• Logs detailed enough?
• Reproduction reliable?
• All layers instrumented?

2. Check Hypothesis Quality

• Too vague?
• Not testable?
• Based on assumptions?

3. Check Fix Scope

• Trying to fix too much at once?
• Bundling multiple issues?
• Over-engineering solution?

Process Shortcuts (When Appropriate)

Tiny Obvious Bugs

When: Typo in variable name, missing comma, etc.

Shortcut: Can fix immediately if:

• [ ] You can see exact problem in error message
• [ ] Fix is one-line change
• [ ] Can verify fix in <30 seconds
• [ ] Zero risk of side effects

Still Required: Quick verification that fix works

Repeated Known Issues

When: Same bug pattern seen before

Shortcut: Can use known solution if:

• [ ] 100% certain it's identical issue
• [ ] Previous solution documented
• [ ] Can verify match with evidence
• [ ] Test case exists

Still Required: Verify it actually matches

Development vs Production

Development: Can be slightly less rigorous

• Faster iteration acceptable
• Can test fixes quickly
• Easy rollback

Production: ALWAYS full systematic process

• High cost of being wrong
• Limited testing ability
• Difficult rollback

Recovery Strategies

If You Made Random Changes

Situation: Violated systematic process, made changes without investigation

Recovery: 1. Revert ALL changes - back to known state 2. Document what you tried - data for Phase 1 3. Start Phase 1 fresh - use failed attempts as evidence 4. Don't keep "parts that might work" - clean slate

If You're in Guess Loop

Situation: Tried multiple fixes, none worked, now guessing

Recovery: 1. Stop immediately - more guesses won't help 2. Count attempts - 3+? Architecture problem 3. Gather evidence - instrument everything 4. Question fundamentals - is approach wrong?

If Time Pressure Mounting

Situation: Manager wants it fixed NOW, pressure to skip process

Recovery: 1. Communicate reality - "Systematic is faster than guessing" 2. Show progress - "Phase 1 complete, identified root cause" 3. Set expectation - "15 more minutes for correct fix" 4. Don't skip steps - rushing guarantees rework

Tools for When Stuck

Rubber Duck Debugging

Explain problem out loud (or in writing): 1. What you're trying to debug 2. What you've discovered 3. What hypotheses you've tested 4. What you're confused about

Often reveals the issue.

Five Whys

Keep asking "Why?" until you hit root cause:

Bug: Dashboard slow
Why? → API calls taking long
Why? → Database queries slow
Why? → Missing indexes
Why? → Recent migration didn't add them
Why? → Migration script had bug

Minimal Reproduction

Strip everything until only broken part remains:

• Remove unrelated code
• Simplify inputs
• Isolate single operation
• Proves exact point of failure

Summary

When debugging gets hard:

• Cannot reproduce → Gather more context, add logging
• Too many causes → Binary search, eliminate systematically
• Cryptic errors → Search, increase verbosity, read source
• Fix doesn't work → Verify applied, count attempts
• Multiple bugs → Separate and prioritize
• Taking too long → Assess phase, check for red flags
• Human redirects → Recognize signal, return to Phase 1

Remember: Systematic approach is FASTER than random fixes, especially when it seems slow.

Related References

• Workflow: Complete four-phase process
• Examples: Real-world scenarios
• Anti-patterns: What NOT to do

Complete Debugging Workflow

Part of: Systematic Debugging

Category: debugging

Reading Level: Intermediate

Purpose

Complete step-by-step workflow for all four phases of systematic debugging, including detailed instructions, decision trees, and verification criteria.

Phase 1: Root Cause Investigation

Goal: Understand WHAT and WHY before attempting fixes.

Step 1: Read Error Messages Carefully

Don't skip past errors or warnings:

• Read stack traces completely
• Note line numbers, file paths, error codes
• Error messages often contain the exact solution
• Write down exact error text

Step 2: Reproduce Consistently

Can you trigger it reliably?

Can you reproduce the issue?
├─ Yes → Proceed to Step 3
├─ Intermittent → Gather more data
│  ├─ Check for race conditions
│  ├─ Look for environmental factors
│  ├─ Add logging around suspected area
│  └─ Document patterns (time of day, load, etc.)
└─ No → Issue may be environmental
   ├─ Check configuration differences
   ├─ Verify dependencies
   └─ Compare runtime environments

Document:

• Exact steps to reproduce
• Required preconditions
• Expected vs actual behavior
• Success rate (every time? 50%? 10%?)

Step 3: Check Recent Changes

What changed that could cause this?

# Git history
git log --oneline --since="1 week ago"
git diff HEAD~5 -- path/to/relevant/file

# Recent commits
git show <commit-hash>

# Blame for specific line
git blame path/to/file.ts | grep -A5 -B5 "problem line"

Look for:

• Code changes in affected area
• New dependencies added
• Configuration changes
• Environmental differences (dev vs prod)
• Database schema changes
• API version changes

Step 4: Gather Evidence in Multi-Component Systems

WHEN system has multiple components (CI → build → signing, API → service → database):

BEFORE proposing fixes, add diagnostic instrumentation:

For EACH component boundary:
  - Log what data enters component
  - Log what data exits component
  - Verify environment/config propagation
  - Check state at each layer

Run once to gather evidence showing WHERE it breaks
THEN analyze evidence to identify failing component
THEN investigate that specific component

Example: Multi-layer System

# Layer 1: Workflow
echo "=== Secrets available in workflow: ==="
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"
echo "API_KEY: ${API_KEY:+SET}${API_KEY:-UNSET}"

# Layer 2: Build script
echo "=== Environment vars in build script: ==="
env | grep IDENTITY || echo "IDENTITY not in environment"
env | grep API_KEY || echo "API_KEY not in environment"

# Layer 3: Service layer
echo "=== Service initialization: ==="
echo "Config loaded: $CONFIG_PATH"
echo "Database connection: $DB_HOST:$DB_PORT"

# Layer 4: Actual operation
echo "=== Operation execution: ==="
set -x  # Enable command tracing
./actual-operation --verbose
set +x

This reveals: Which layer fails (secrets → workflow ✓, workflow → build ✗)

Step 5: Trace Data Flow

WHEN error is deep in call stack:

Backward tracing technique: 1. Where does bad value originate? 2. What called this function with bad value? 3. Keep tracing up until you find the source 4. Fix at source, not at symptom

Example:

Error: Cannot read property 'id' of undefined
  at processUser (user.service.ts:45)
  at handleRequest (request.handler.ts:23)
  at middleware (auth.middleware.ts:12)

Investigation:
45: const userId = user.id;  // user is undefined - WHERE did it come from?
23: processUser(req.user);   // req.user is undefined - WHERE set?
12: req.user = await getUser(token);  // getUser returned undefined - WHY?

Root cause: getUser returns undefined when token expired
Fix: Handle undefined in getUser, not at usage site

Phase 2: Pattern Analysis

Goal: Find working examples and identify differences.

Step 1: Find Working Examples

Locate similar working code in same codebase:

# Find similar patterns
grep -r "similar_function" src/
grep -r "similar_pattern" src/

# Find similar test cases
find tests/ -name "*similar*test*"

Questions:

• What works that's similar to what's broken?
• How is working code different?
• What dependencies does working code use?

Step 2: Compare Against References

If implementing pattern, read reference implementation COMPLETELY:

DON'T:

• Skim the documentation
• Copy-paste without understanding
• "Adapt" the pattern without reading fully

DO:

• Read every line of reference
• Understand WHY each part exists
• Note all dependencies and setup
• Check for hidden requirements

Step 3: Identify Differences

List every difference, however small:

Working Code          | Broken Code
---------------------|--------------------
Uses async/await     | Uses callbacks
Has error handling   | No error handling
Validates input      | Assumes valid input
Imports from '@lib'  | Imports from '../lib'

Don't assume "that can't matter" - small differences often cause bugs.

Step 4: Understand Dependencies

What other components does this need?

• Required packages and versions
• Configuration settings
• Environment variables
• Database schema
• External services
• Initialization order

Phase 3: Hypothesis and Testing

Goal: Form and test specific hypotheses scientifically.

Step 1: Form Single Hypothesis

Write it down explicitly:

❌ Bad: "Something is wrong with the database" ✅ Good: "Database connection times out because connection pool is exhausted"

❌ Bad: "The calculation is incorrect" ✅ Good: "Division by zero when denominator is empty list"

Good hypothesis characteristics:

• Specific: Names exact variable/function/line
• Testable: Can be verified with single change
• Falsifiable: Could be proven wrong
• Evidence-Based: Supported by logs/observations

Step 2: Test Minimally

Make the SMALLEST possible change to test hypothesis:

// Hypothesis: Function fails when input array is empty

// Minimal test
if (array.length === 0) {
  console.log('HYPOTHESIS TEST: Empty array detected');
}

// DON'T bundle multiple changes
if (array.length === 0) {
  throw new Error('Empty array');  // Multiple changes
}

One variable at a time:

• Change only what tests hypothesis
• Keep changes minimal
• Comment your reasoning
• Revert if hypothesis wrong

Step 3: Verify Before Continuing

Execute reproduction case and observe outcome:

# Run specific test
npm test path/to/failing.test.ts

# Or run reproduction script
node reproduce-bug.js

Outcomes:

• Hypothesis Confirmed → Proceed to Phase 4
• Hypothesis Rejected → Return to Phase 3 Step 1 with new data
• Inconclusive → Refine test or gather more data

Step 4: When You Don't Know

Be honest about knowledge gaps:

• Say "I don't understand X"
• Don't pretend to know
• Ask for help
• Research more before forming hypothesis

Better to say "I need to investigate further" than propose wrong fix.

Phase 4: Implementation

Goal: Fix the root cause, not the symptom.

Step 1: Create Failing Test Case

Simplest possible reproduction:

// Good: Minimal failing test
test('handles empty array', () => {
  const result = processArray([]);
  expect(result).toEqual([]);
});

// Bad: Too complex
test('complex scenario', () => {
  const db = setupDatabase();
  const user = createUser(db);
  const items = fetchItems(user);
  const result = processArray(items);
  // What exactly are we testing?
});

Requirements:

• Automated test if framework available
• One-off test script if no framework
• MUST have before fixing
• Should be fast to run

Step 2: Implement Single Fix

Address the root cause identified:

// Good: Fixes root cause
function processArray(items: Item[]): Result[] {
  if (items.length === 0) {
    return [];  // Handle edge case
  }
  return items.map(transform);
}

// Bad: Fixes symptom
function processArray(items: Item[]): Result[] {
  try {
    return items.map(transform);
  } catch (e) {
    return [];  // Hides real problem
  }
}

Rules:

• ONE change at a time
• No "while I'm here" improvements
• No bundled refactoring
• Focus only on the bug

Step 3: Verify Fix

Run full verification:

# Test passes now?
npm test path/to/test.test.ts

# No other tests broken?
npm test

# Issue actually resolved?
node reproduce-bug.js  # Should work now

Checklist:

• [ ] Failing test now passes
• [ ] All other tests still pass
• [ ] Original bug no longer occurs
• [ ] No new warnings or errors
• [ ] Performance not degraded

Step 4: If Fix Doesn't Work

STOP and reassess:

Track attempts:

• Fix #1 failed → Return to Phase 1, re-analyze
• Fix #2 failed → Return to Phase 1, gather more evidence
• Fix #3 failed → STOP: Architecture problem

DON'T attempt Fix #4 without architectural discussion.

Step 5: If 3+ Fixes Failed - Question Architecture

Pattern indicating architectural problem:

• Each fix reveals new shared state/coupling in different place
• Fixes require "massive refactoring" to implement
• Each fix creates new symptoms elsewhere
• You're "fighting the framework"

STOP and question fundamentals:

• Is this pattern fundamentally sound?
• Are we "sticking with it through sheer inertia"?
• Should we refactor architecture vs continue fixing symptoms?

Discuss with human partner before attempting more fixes.

This is NOT a failed hypothesis - this is wrong architecture.

Verification Checklist

Before marking debugging complete:

• [ ] Root cause identified (not just symptoms)
• [ ] Hypothesis formed and tested
• [ ] Fix addresses root cause
• [ ] Test case created
• [ ] Test passes
• [ ] No regressions introduced
• [ ] Solution documented
• [ ] Learned from the issue

Summary

• Phase 1 (Root Cause): Read, reproduce, gather evidence
• Phase 2 (Pattern): Find working examples, compare
• Phase 3 (Hypothesis): Form theory, test minimally
• Phase 4 (Implementation): Create test, fix, verify
• If 3+ fixes fail: Question architecture

Related References

• Examples: Real-world debugging scenarios
• Troubleshooting: Common debugging challenges
• Anti-patterns: Common mistakes to avoid