Debug

dot-skills Debugging Best Practices

Comprehensive debugging methodology guide for software engineers, containing 54 rules across 10 categories prioritized by impact. Based on research from Andreas Zeller's "Why Programs Fail" and academic debugging curricula.

Operational Loop

Use this loop before reaching for the detailed rules:

1. Build a fast, deterministic feedback loop that can fail on the reported bug. 2. Reproduce the user's symptom with that loop. 3. Write 3-5 ranked, falsifiable hypotheses before testing fixes. 4. Instrument the narrowest point that distinguishes those hypotheses. 5. Fix the cause, then add or preserve a regression test at the highest useful test boundary. 6. Re-run the original feedback loop and remove temporary debug instrumentation.

If no reliable loop can be built, stop and name exactly what evidence is missing: logs, trace payloads, a failing fixture, a screen recording, environment access, or a reproduction script. Do not guess without a loop.

Feedback Loop Options

Try these in order, choosing the cheapest loop that reproduces the real symptom:

1. Failing unit, integration, component, route, or end-to-end test. 2. CLI command with fixture input and an expected stdout/stderr snapshot. 3. HTTP script or curl request against a local or staging server. 4. Browser automation that asserts DOM, console, network, or visual state. 5. Captured trace replay: network request, webhook payload, event log, or job payload. 6. Throwaway harness around the smallest runnable subsystem. 7. Property, fuzz, stress, or repeated-run loop for nondeterministic failures. 8. Bisection or differential loop across commits, versions, configs, or datasets.

Improve the loop itself when it is slow, flaky, or vague. A sharp 2-second loop is more valuable than a broad 2-minute suite when debugging.

Instrumentation Rules

• Map every probe to a specific hypothesis.
• Change one variable at a time.
• Prefer debugger/REPL inspection when available.
• Use targeted logs at decision boundaries, not broad log spam.
• Tag temporary logs with a unique prefix such as [DEBUG-20260607-auth].
• Grep and remove every temporary tag before finishing.

For performance regressions, measure first. Establish a baseline, capture timing or profiler evidence, and bisect before changing code.

When to Apply

Reference these guidelines when:

• Investigating a bug or unexpected behavior
• Debugging code during development
• Code produces wrong results or crashes
• Performance issues need root cause analysis
• Triaging incoming bug reports and prioritizing fixes
• Conducting root cause analysis for incidents
• Reviewing debugging approaches or code for common bug patterns

Rule Categories by Priority

Quick Reference

1. Problem Definition (CRITICAL)

• prob-reproduce-before-debug - Reproduce the bug before investigating
• prob-minimal-reproduction - Create minimal reproduction cases
• prob-document-symptoms - Document symptoms precisely
• prob-separate-symptoms-causes - Separate symptoms from causes
• prob-state-expected-actual - State expected vs actual behavior
• prob-recent-changes - Check recent changes first

2. Hypothesis-Driven Search (CRITICAL)

• hypo-scientific-method - Apply the scientific method
• hypo-binary-search - Use binary search to localize bugs
• hypo-one-change-at-time - Test one hypothesis at a time
• hypo-where-not-what - Find WHERE before asking WHAT
• hypo-rule-out-obvious - Rule out obvious causes first
• hypo-rubber-duck - Explain the problem aloud

3. Observation Techniques (HIGH)

• obs-strategic-logging - Use strategic logging
• obs-log-inputs-outputs - Log function inputs and outputs
• obs-breakpoint-strategy - Use breakpoints strategically
• obs-stack-trace-reading - Read stack traces bottom to top
• obs-watch-expressions - Use watch expressions for state
• obs-trace-data-flow - Trace data flow through system

4. Root Cause Analysis (HIGH)

• rca-five-whys - Use the 5 Whys technique
• rca-fault-propagation - Trace fault propagation chains
• rca-last-known-good - Find the last known good state
• rca-question-assumptions - Question your assumptions
• rca-examine-boundaries - Examine system boundaries

5. Tool Mastery (MEDIUM-HIGH)

• tool-conditional-breakpoints - Use conditional breakpoints
• tool-logpoints - Use logpoints instead of modifying code
• tool-step-commands - Master step over/into/out
• tool-call-stack-navigation - Navigate the call stack
• tool-memory-inspection - Inspect memory and object state
• tool-exception-breakpoints - Use exception breakpoints

6. Bug Triage and Classification (MEDIUM)

• triage-severity-vs-priority - Separate severity from priority
• triage-user-impact-assessment - Assess user impact before prioritizing
• triage-reproducibility-matters - Factor reproducibility into triage
• triage-quick-wins-first - Identify and ship quick wins first
• triage-duplicate-detection - Detect and link duplicate bug reports

7. Common Bug Patterns (MEDIUM)

• pattern-null-pointer - Recognize null pointer patterns
• pattern-off-by-one - Spot off-by-one errors
• pattern-race-condition - Identify race condition symptoms
• pattern-memory-leak - Detect memory leak patterns
• pattern-type-coercion - Watch for type coercion bugs
• pattern-async-await-errors - Catch async/await error handling mistakes
• pattern-timezone-issues - Recognize timezone and date bugs

8. Fix Verification (MEDIUM)

• verify-reproduce-fix - Verify with original reproduction
• verify-regression-check - Check for regressions
• verify-understand-why-fix-works - Understand why fix works
• verify-add-test - Add test to prevent recurrence

9. Anti-Patterns (MEDIUM)

• anti-shotgun-debugging - Avoid shotgun debugging
• anti-quick-patch - Avoid quick patches without understanding
• anti-tunnel-vision - Avoid tunnel vision on initial hypothesis
• anti-debug-fatigue - Recognize debugging fatigue
• anti-blame-tool - Don't blame the tool too quickly

10. Prevention & Learning (LOW-MEDIUM)

• prev-document-solution - Document bug solutions
• prev-postmortem - Conduct blameless postmortems
• prev-defensive-coding - Add defensive code at boundaries
• prev-improve-error-messages - Improve error messages

How to Use

Read individual reference files for detailed explanations and code examples:

• Section definitions - Category structure and impact levels
• Rule template - Template for adding new rules
• Example rules: prob-reproduce-before-debug, hypo-binary-search

Full Compiled Document

For the complete guide with all rules expanded: AGENTS.md

Attribution

The operational loop incorporates debugging workflow ideas adapted from Matt Pocock's MIT-licensed diagnose skill.

Debugging

Version 1.0.0 dot-skills January 2025

Note:

This document is mainly for agents and LLMs to follow when maintaining,

generating, or refactoring codebases. Humans may also find it useful,

but guidance here is optimized for automation and consistency by AI-assisted workflows.

---

Abstract

Comprehensive debugging methodology guide for software engineers, designed for AI agents and LLMs. Contains 54 rules across 10 categories, prioritized by impact from critical (problem definition, hypothesis-driven search) to incremental (prevention and learning). Includes bug triage, common bug patterns, and root cause analysis. Each rule includes detailed explanations, real-world examples comparing incorrect vs. correct approaches, and specific impact metrics to guide systematic bug investigation.

---

Table of Contents

1. Problem Definition — CRITICAL

• 1.1 Check Recent Changes First — CRITICAL (80%+ of bugs are caused by recent changes; reduces search space dramatically)
• 1.2 Create Minimal Reproduction Cases — CRITICAL (Reduces debugging scope by 80-95%, making root cause obvious in many cases)
• 1.3 Document Symptoms Precisely — CRITICAL (Prevents misdiagnosis and enables pattern matching across similar issues)
• 1.4 Reproduce Before Debugging — CRITICAL (Prevents 50%+ of wasted debugging time on unreproducible or misunderstood issues)
• 1.5 Separate Symptoms from Causes — CRITICAL (Prevents fixing symptoms while root cause continues creating new bugs)
• 1.6 State Expected vs Actual Behavior — CRITICAL (Provides clear success criteria and prevents fixing the wrong thing)

2. Hypothesis-Driven Search — CRITICAL

• 2.1 Apply the Scientific Method — CRITICAL (Eliminates 80%+ of random debugging; provides systematic path to root cause)
• 2.2 Explain the Problem Aloud (Rubber Duck) — CRITICAL (Reveals gaps in understanding; 50%+ of bugs found during explanation)
• 2.3 Find WHERE Before Asking WHAT — CRITICAL (Location narrows problem space by 90%+; understanding comes faster with context)
• 2.4 Rule Out Obvious Causes First — CRITICAL (60%+ of bugs have simple causes; checking obvious things first saves hours)
• 2.5 Test One Hypothesis at a Time — CRITICAL (Prevents confounding variables; ensures you know which change fixed the bug)
• 2.6 Use Binary Search to Localize Bugs — CRITICAL (Reduces search space by 50% per iteration; finds bug in O(log n) steps)

3. Observation Techniques — HIGH

• 3.1 Log Function Inputs and Outputs — HIGH (Reveals data transformation issues; enables replay debugging)
• 3.2 Read Stack Traces Bottom to Top — HIGH (5-10× faster error localization; reveals full call chain context)
• 3.3 Trace Data Flow Through the System — HIGH (2-5× faster bug localization; pinpoints exact transformation that corrupts data)
• 3.4 Use Breakpoints Strategically — HIGH (10× faster inspection than print statements; enables state exploration)
• 3.5 Use Strategic Logging Over Random Print Statements — HIGH (5× faster bug localization; structured logs enable automated analysis)
• 3.6 Use Watch Expressions for Complex State — HIGH (3-5× faster state tracking; auto-updates computed values on each step)

4. Root Cause Analysis — HIGH

• 4.1 Examine System Boundaries — HIGH (70%+ of bugs occur at boundaries; interfaces are high-risk areas)
• 4.2 Find the Last Known Good State — HIGH (O(log n) regression detection via git bisect; establishes working baseline)
• 4.3 Question Your Assumptions — HIGH (Uncovers hidden bugs; 40%+ of debugging time is wasted on false assumptions)
• 4.4 Trace Fault Propagation Chains — HIGH (2-3× faster root cause discovery; traces infection chain from symptom to origin)
• 4.5 Use the 5 Whys Technique — HIGH (Reaches true root cause instead of surface symptoms; prevents recurrence)

5. Tool Mastery — MEDIUM-HIGH

• 5.1 Inspect Memory and Object State — MEDIUM-HIGH (Catches 90%+ of reference vs value bugs; reveals prototype chain and hidden properties)
• 5.2 Master Step Over, Step Into, Step Out — MEDIUM-HIGH (Efficient navigation through code; 5× faster than random stepping)
• 5.3 Navigate the Call Stack — MEDIUM-HIGH (3× faster context discovery; reveals parameter values at each call level)
• 5.4 Use Conditional Breakpoints — MEDIUM-HIGH (100× faster than hitting breakpoint manually in loops; targets exact conditions)
• 5.5 Use Exception Breakpoints — MEDIUM-HIGH (5× faster exception debugging; catches errors at throw point with full context)
• 5.6 Use Logpoints Instead of Modifying Code — MEDIUM-HIGH (100% clean commits; zero risk of shipping debug statements to production)

6. Bug Triage and Classification — MEDIUM

• 6.1 Assess User Impact Before Prioritizing — MEDIUM (10× improvement in value delivered per engineering hour)
• 6.2 Detect and Link Duplicate Bug Reports — MEDIUM (prevents duplicate investigation effort)
• 6.3 Factor Reproducibility into Triage — MEDIUM (prevents wasted investigation time)
• 6.4 Identify and Ship Quick Wins First — MEDIUM (3-5× more bugs fixed per sprint)
• 6.5 Separate Severity from Priority — MEDIUM (enables correct resource allocation)

7. Common Bug Patterns — MEDIUM

• 7.1 Catch Async/Await Error Handling Mistakes — MEDIUM (prevents unhandled promise rejections)
• 7.2 Detect Memory Leak Patterns — MEDIUM (prevents out-of-memory crashes)
• 7.3 Identify Race Condition Symptoms — MEDIUM (prevents intermittent production failures)
• 7.4 Recognize Null Pointer Patterns — MEDIUM (prevents 20-30% of runtime errors)
• 7.5 Recognize Timezone and Date Bugs — MEDIUM (prevents date calculation errors across timezones)
• 7.6 Spot Off-by-One Errors — MEDIUM (prevents 10-15% of logic errors)
• 7.7 Watch for Type Coercion Bugs — MEDIUM (prevents silent data corruption bugs)

8. Fix Verification — MEDIUM

• 8.1 Add a Test to Prevent Recurrence — MEDIUM (100% regression prevention for this specific bug; serves as executable documentation)
• 8.2 Check for Regressions After Fixing — MEDIUM (Prevents fix from breaking existing functionality; catches unintended side effects)
• 8.3 Understand Why the Fix Works — MEDIUM (Prevents cargo cult fixes; ensures fix is correct, not accidental)
• 8.4 Verify Fix With Original Reproduction — MEDIUM (Confirms fix actually works; prevents false confidence from unrelated changes)

9. Anti-Patterns — MEDIUM

• 9.1 Avoid Blaming the Tool Too Quickly — MEDIUM (95%+ of bugs are in your code, not libraries; premature blame wastes time)
• 9.2 Avoid Quick Patches Without Understanding — MEDIUM (Prevents technical debt and recurring bugs; quick fixes often mask real problems)
• 9.3 Avoid Shotgun Debugging — MEDIUM (Prevents hours of wasted effort; random changes make bugs harder to find)
• 9.4 Avoid Tunnel Vision on Initial Hypothesis — MEDIUM (Prevents wasted hours pursuing wrong theory; 30%+ of bugs aren't where we first look)
• 9.5 Recognize and Address Debugging Fatigue — MEDIUM (Prevents stupid mistakes from tiredness; fresh perspective finds bugs faster)

10. Prevention & Learning — LOW-MEDIUM

• 10.1 Add Defensive Code at System Boundaries — LOW-MEDIUM (Catches bugs earlier with better context; prevents cascade failures)
• 10.2 Conduct Blameless Postmortems — LOW-MEDIUM (Prevents recurrence through systemic fixes; builds team debugging culture)
• 10.3 Document Bug Solutions for Future Reference — LOW-MEDIUM (Reduces future debugging time by 40-60%; creates team knowledge base)
• 10.4 Improve Error Messages When You Debug — LOW-MEDIUM (Reduces future debugging time; helps next developer (including future you))

---

References

1. https://www.whyprogramsfail.com/ 2. https://web.mit.edu/6.031/www/sp17/classes/11-debugging/ 3. https://www.cs.cornell.edu/courses/cs312/2006fa/lectures/lec26.html 4. https://code.visualstudio.com/docs/debugtest/debugging 5. https://developer.chrome.com/docs/devtools/javascript/reference/ 6. https://rubberduckdebugging.com/ 7. https://git-scm.com/docs/git-bisect

---

Source Files

This document was compiled from individual reference files. For detailed editing or extension:

Rule Title Here

1-3 sentences explaining WHY this matters for debugging effectiveness.

Incorrect (what's wrong):

// Bad example with comments explaining the cost

Correct (what's right):

// Good example with comments explaining the benefit

When NOT to use this pattern:

• Exception 1
• Exception 2

Reference: Source Title

Debugging Best Practices

Comprehensive debugging methodology skill for AI coding agents. Based on research from Andreas Zeller's "Why Programs Fail" and academic debugging curricula.

Overview

This skill provides 54 rules across 10 categories to help developers debug systematically instead of randomly. Rules are prioritized by impact, from critical problem definition techniques to prevention practices.

Structure

debugging/
├── SKILL.md              # Entry point with quick reference
├── AGENTS.md             # Compiled comprehensive guide
├── metadata.json         # Version, references, metadata
├── README.md             # This file
├── references/
│   ├── _sections.md      # Category definitions
│   ├── prob-*.md         # Problem definition rules (6)
│   ├── hypo-*.md         # Hypothesis-driven search rules (6)
│   ├── obs-*.md          # Observation technique rules (6)
│   ├── rca-*.md          # Root cause analysis rules (5)
│   ├── tool-*.md         # Tool mastery rules (6)
│   ├── triage-*.md       # Bug triage rules (5)
│   ├── pattern-*.md      # Common bug pattern rules (7)
│   ├── verify-*.md       # Fix verification rules (4)
│   ├── anti-*.md         # Anti-pattern rules (5)
│   └── prev-*.md         # Prevention rules (4)
└── assets/
    └── templates/
        └── _template.md  # Rule template

Getting Started

Using in Claude Code

This skill automatically activates when you're working on:

• Debugging code issues or exceptions
• Investigating stack traces or errors
• Root cause analysis for bugs
• Troubleshooting unexpected behavior
• Bug triage and prioritization

Manual Commands

# Install dependencies (if contributing)
pnpm install

# Build AGENTS.md from rules
pnpm build

# Validate skill structure
pnpm validate

Creating a New Rule

1. Determine the category based on the rule's primary concern 2. Use the appropriate prefix from the table below 3. Copy assets/templates/_template.md as your starting point 4. Fill in frontmatter and content

Prefix Reference

Rule File Structure

Each rule follows this template:

---
title: Rule Title Here
impact: CRITICAL|HIGH|MEDIUM-HIGH|MEDIUM|LOW-MEDIUM|LOW
impactDescription: Quantified impact (e.g., "2-10× faster localization")
tags: prefix, technique, related-concepts
---

## Rule Title Here

1-3 sentences explaining WHY this matters for debugging effectiveness.

**Incorrect (what's wrong):**

// Bad example with comments explaining the cost

**Correct (what's right):**

// Good example with comments explaining the benefit

Reference: [Link](https://example.com)

File Naming Convention

Rule files follow the pattern: {prefix}-{description}.md

Examples:

• prob-reproduce-before-debug.md - Problem definition, about reproducing bugs first
• hypo-binary-search.md - Hypothesis-driven, about binary search localization
• tool-conditional-breakpoints.md - Tool mastery, about conditional breakpoints

Impact Levels

Scripts

Contributing

1. Check existing rules to avoid duplication 2. Use the rule template (assets/templates/_template.md) 3. Include both incorrect and correct examples 4. Quantify impact where possible 5. Reference authoritative sources 6. Run validation before submitting

Key Principles

1. Reproduce Before Debugging - Never debug until you can reliably trigger the bug 2. Apply Scientific Method - Form hypotheses, predict outcomes, test systematically 3. Binary Search Localization - Narrow down by 50% with each checkpoint 4. Find WHERE Before WHAT - Locate first, understand second 5. One Change at a Time - Isolate variables to avoid confounding 6. Question Assumptions - Many bugs hide behind unquestioned beliefs

Acknowledgments

This skill draws from:

• Why Programs Fail - Andreas Zeller
• MIT 6.031 - Debugging
• Cornell CS312 - Debugging Techniques
• VS Code Debugging

License

MIT

Sections

This file defines all sections, their ordering, impact levels, and descriptions. The section ID (in parentheses) is the filename prefix used to group rules.

---

1. Problem Definition (prob)

Impact: CRITICAL Description: Unclear problem statements waste 50%+ of debugging time; precise reproduction and symptom documentation prevent chasing wrong issues entirely.

2. Hypothesis-Driven Search (hypo)

Impact: CRITICAL Description: Scientific method approach with systematic hypothesis testing eliminates 80%+ of guesswork; binary search techniques halve the search space per iteration.

3. Observation Techniques (obs)

Impact: HIGH Description: Proper logging, strategic breakpoints, and execution tracing reveal actual vs expected behavior without relying on assumptions or mental simulation.

4. Root Cause Analysis (rca)

Impact: HIGH Description: Finding true causes vs symptoms prevents recurring bugs; structured techniques like 5 Whys and cause-effect chains reach core issues systematically.

5. Tool Mastery (tool)

Impact: MEDIUM-HIGH Description: Advanced debugger features like conditional breakpoints, watchpoints, and memory inspection provide 10× faster insight than print statement debugging.

6. Bug Triage and Classification (triage)

Impact: MEDIUM Description: Proper severity and priority classification ensures development resources focus on highest-impact issues. Distinguish technical severity from business priority to make informed decisions.

7. Common Bug Patterns (pattern)

Impact: MEDIUM Description: Recognizing classic bug patterns—null pointers, race conditions, off-by-one errors, memory leaks—enables faster diagnosis by matching symptoms to known causes.

8. Fix Verification (verify)

Impact: MEDIUM Description: Confirming fixes actually resolve the root cause and don't introduce regressions prevents bug ping-pong and ensures permanent resolution.

9. Anti-Patterns (anti)

Impact: MEDIUM Description: Recognizing and avoiding shotgun debugging, quick patches, assumption traps, and other counterproductive habits saves hours of wasted effort.

10. Prevention & Learning (prev)

Impact: LOW-MEDIUM Description: Post-mortems, defensive coding, and knowledge sharing reduce future debugging time by 40-60% through systematic learning from past issues.

Avoid Blaming the Tool Too Quickly

When debugging, assume the bug is in your code, not in the framework, library, or language. While tools do have bugs, they're far less common than bugs in application code. Blaming the tool prematurely stops productive investigation.

Incorrect (blaming the tool):

// Bug: Data not saving correctly
// Developer's conclusion: "React useState must be broken"

const [user, setUser] = useState(null);

const updateUser = (newData) => {
  setUser(newData);
  console.log(user);  // Still shows old value!
  // "useState is broken, it's not updating!"
  // Files bug report against React
  // Spends hours searching for React bugs
};

// Actual issue: React state updates are asynchronous
// Developer didn't understand the tool, not a tool bug

Correct (assume it's your code):

// Bug: Data not saving correctly
// Hypothesis: My usage of useState is incorrect

const [user, setUser] = useState(null);

const updateUser = (newData) => {
  setUser(newData);
  console.log(user);  // Shows old value

  // Question: Is this expected behavior?
  // Check React documentation on useState...
  // Found: "setState doesn't immediately mutate state"

  // Understanding: useState is working correctly
  // My expectation was wrong

  // Fix: Use callback for immediate value, or useEffect for side effects
  setUser(newData);
  console.log(newData);  // Use newData directly

  // Or:
  useEffect(() => {
    if (user) {
      console.log('User updated:', user);
    }
  }, [user]);
};

Before blaming the tool, verify:

1. Did you read the documentation? 2. Does a minimal example reproduce the issue? 3. Can you find others with the same "bug"? 4. Is your version up to date? 5. Are you using the API correctly? 6. Have you isolated the issue from your application code?

When tool bugs are actually likely:

• Minimal reproduction in isolation still fails
• Issue documented in tool's bug tracker
• Using edge case or new/deprecated feature
• Multiple independent developers report same issue
• Worked before tool version update

When NOT to use this pattern:

• You've verified correct usage through documentation
• Minimal reproduction clearly shows tool issue
• Tool's bug tracker confirms the issue

Reference: MIT 6.031 - Debugging

Recognize and Address Debugging Fatigue

Long debugging sessions lead to diminishing returns. Recognize when you're tired, frustrated, or stuck in a loop. Taking a break often leads to faster resolution than pushing through.

Incorrect (pushing through fatigue):

# Hour 1: Start investigating login bug
# Hour 2: Still looking, getting frustrated
# Hour 3: Making random changes, reverting them
# Hour 4: Re-reading same code, finding nothing
# Hour 5: "It makes no sense, the code is correct"
# Hour 6: Finally spot the typo: "usernaem" instead of "username"

# 6 hours wasted on a bug that fresh eyes would catch in minutes
# Fatigue signs ignored: frustration, circular thinking, missing obvious things

Correct (recognize and address fatigue):

# Hour 1: Start investigating login bug, make progress
# Hour 2: Progress slowing, re-reading same sections

# Fatigue check:
# - Am I making progress? No, last 30 minutes unproductive
# - Am I frustrated? Yes
# - Have I tried the same approach twice? Yes
# - When did I last take a break? 2 hours ago

# Decision: Take a break

# After 15-minute walk:
# Return to code, immediately see: "usernaem" typo
# Total time: 2 hours 15 minutes (not 6 hours)

Fatigue warning signs:

• Re-reading the same code repeatedly
• Making changes without clear reasoning
• Frustration or anger at the code
• Thinking "this is impossible"
• Missing obvious things (typos, wrong file, wrong branch)
• Forgetting what you've already tried

Fatigue recovery strategies:

The overnight effect: Many developers report solving bugs "immediately" the next morning. Sleep consolidates learning and provides fresh perspective that fatigued brains can't achieve.

When NOT to use this pattern:

• Critical production issue requiring immediate fix
• You're making steady progress (don't interrupt flow)

Reference: Psychology of Debugging

Avoid Quick Patches Without Understanding

Don't apply quick fixes that mask symptoms without understanding the root cause. These create technical debt, hide real problems, and often lead to worse failures later.

Incorrect (quick patch):

# Bug: Occasional NullPointerException in process_order

def process_order(order):
    # Quick patch: Just check for None everywhere
    if order is None:
        return None
    if order.user is None:
        return None
    if order.items is None:
        return None

    total = sum(item.price for item in order.items if item is not None)
    if total is None:  # Can't even be None but added anyway
        return None

    return charge_user(order.user, total)

# Symptoms hidden, but:
# - WHY was order None? Still happening somewhere
# - WHY was order.user None? Data corruption? Race condition?
# - These None returns silently fail, no alerts
# - Real bug continues causing data inconsistencies

Correct (understand then fix):

# Bug: Occasional NullPointerException in process_order

# Step 1: Investigate why order was None
# Added logging: "order is None when called from webhook handler"
# Found: Webhook sends order_id, not order object

# Step 2: Fix at the source
def handle_webhook(data):
    order_id = data.get('order_id')
    if not order_id:
        logger.error("Webhook missing order_id", extra={"data": data})
        raise ValueError("order_id required")

    order = Order.query.get(order_id)
    if not order:
        logger.error(f"Order not found: {order_id}")
        raise ValueError(f"Order {order_id} not found")

    return process_order(order)  # Now guaranteed to have valid order

def process_order(order):
    # No defensive checks needed - contract enforced by caller
    total = sum(item.price for item in order.items)
    return charge_user(order.user, total)

Quick patch red flags:

• Adding try/except that catches and ignores all exceptions
• Adding null checks everywhere without investigating why nulls occur
• Adding retries without understanding what fails
• Suppressing warning/error logs instead of fixing cause

When quick patches are acceptable:

• Critical production issue (patch now, investigate later)
• Third-party bug you can't fix (document and work around)
• Truly defensive code at system boundaries

Always follow up:

• Create ticket to investigate root cause
• Schedule time to properly fix after stabilization
• Add monitoring to catch if symptom returns

Reference: Root Cause Analysis

Avoid Shotgun Debugging

Don't make random changes hoping something will fix the bug. Shotgun debugging wastes time, introduces new bugs, and makes the codebase harder to understand. Each change should test a specific hypothesis.

Incorrect (shotgun debugging):

# Bug: User registration fails

def register_user(data):
    # Try adding a sleep
    time.sleep(0.5)

    # Try encoding fix
    data = data.encode('utf-8').decode('utf-8')

    # Try null check
    if data is None:
        data = {}

    # Try lowercase
    data['email'] = data.get('email', '').lower()

    # Try adding retry
    for attempt in range(3):
        try:
            return create_user(data)
        except:
            pass

    # None of this is based on understanding the actual bug
    # Code is now a mess, might work by accident

Correct (hypothesis-driven debugging):

# Bug: User registration fails

# Step 1: What exactly is failing?
# Error: "IntegrityError: duplicate key email"

# Step 2: Hypothesis: Email uniqueness check is case-sensitive
# Test: Check if different-case duplicate exists
existing = User.query.filter_by(email=data['email'].lower()).first()
print(f"Existing user with email: {existing}")  # Found "ALICE@example.com"

# Step 3: Confirmed hypothesis - make targeted fix
def register_user(data):
    email = data.get('email', '').lower()  # Normalize case
    if User.query.filter_by(email=email).first():
        raise ValueError("Email already registered")
    return create_user({**data, 'email': email})

Signs of shotgun debugging:

• Adding code without knowing why
• Trying multiple "fixes" at once
• Changes are reverted frequently
• Comments like "not sure why this works"
• Copy-pasting solutions from Stack Overflow without understanding

Alternative approach:

1. Stop and reproduce the bug cleanly 2. Form a hypothesis about the cause 3. Design a test for that hypothesis 4. Make ONE change based on results 5. Repeat until bug is found

When shotgun debugging seems tempting:

• Take a break - you're likely frustrated
• Explain the problem to someone (rubber duck)
• Write down what you've tried and results
• Start fresh with systematic approach

Reference: Why Programs Fail

Avoid Tunnel Vision on Initial Hypothesis

Don't get stuck on your first guess about where the bug is. If evidence doesn't support your hypothesis, let it go and consider alternatives. Confirmation bias makes us see evidence that supports our theory and ignore evidence against it.

Incorrect (tunnel vision):

# Bug: Slow page load
# Developer's hypothesis: "Database is slow"

# Spends 3 hours:
# - Adding database indexes
# - Optimizing queries
# - Enabling query caching
# - Profiling database
# Database performance improved 50%, but page still slow

# Evidence ignored:
# - API response time was <100ms in browser network tab
# - Slow load happened even on cache-hit pages
# - JavaScript bundle was 5MB (not checked)

# Actual cause: Unoptimized JavaScript, not database

Correct (open to alternatives):

# Bug: Slow page load
# Initial hypothesis: "Database is slow"

# Test hypothesis 1: Database
# - Add timing logs: database query = 50ms ✓ Fast
# - Conclusion: Not the database

# Hypothesis falsified! Generate new hypotheses:
# - API processing time?
# - Network latency?
# - Frontend rendering?
# - JavaScript bundle size?

# Test hypothesis 2: API processing
# - Total API time = 80ms ✓ Fast
# - Conclusion: Not the backend

# Test hypothesis 3: Frontend
# - Network tab shows: 5MB JavaScript bundle
# - Parse/execute time: 3 seconds
# - Conclusion: Found it! Unoptimized frontend bundle

# Fixed by code splitting and lazy loading

Signs of tunnel vision:

• Spending hours on one area without progress
• Ignoring contradictory evidence
• Thinking "it HAS to be here"
• Not considering other possibilities
• Defensive when others suggest alternatives

Breaking out of tunnel vision:

1. Set a time limit for each hypothesis (e.g., 30 minutes) 2. Write down ALL possible causes before investigating 3. List evidence FOR and AGAINST your current theory 4. Ask someone else to suggest alternatives 5. Take a break and return with fresh eyes

When NOT to use this pattern:

• Strong evidence points to specific location
• You've methodically eliminated other possibilities

Reference: A Systematic Approach to Debugging

Use Binary Search to Localize Bugs

When you know a bug exists somewhere in a code path, use binary search to find it. Insert a checkpoint halfway through, determine which half contains the bug, and repeat. This finds bugs in O(log n) steps instead of O(n).

Incorrect (linear search):

// Bug: Data is corrupted somewhere in this pipeline
function transformOrder(input) {
  const step1 = validate(input);      // Check here... OK
  const step2 = normalize(step1);     // Check here... OK
  const step3 = transform(step2);     // Check here... OK
  const step4 = enrich(step3);        // Check here... still looking...
  const step5 = format(step4);        // Check here...
  const step6 = compress(step5);      // Check here...
  const step7 = encrypt(step6);       // Check here...
  const step8 = serialize(step7);     // Found it!
  return step8;
}
// 8 checkpoints examined linearly = 8 iterations

Correct (binary search):

// Bug: Data is corrupted somewhere in this 8-step pipeline
function transformOrder(input) {
  const step1 = validate(input);
  const step2 = normalize(step1);
  const step3 = transform(step2);
  const step4 = enrich(step3);

  console.log('Checkpoint (step 4):', isDataValid(step4));  // Iteration 1
  // Result: VALID - bug is in steps 5-8

  const step5 = format(step4);
  const step6 = compress(step5);

  console.log('Checkpoint (step 6):', isDataValid(step6));  // Iteration 2
  // Result: VALID - bug is in steps 7-8

  const step7 = encrypt(step6);

  console.log('Checkpoint (step 7):', isDataValid(step7));  // Iteration 3
  // Result: INVALID - bug is in step 7 (encrypt)

  const step8 = serialize(step7);
  return step8;
}
// 3 checkpoints examined with binary search = log2(8) = 3 iterations

Binary search debugging process:

1. Identify the range: first known-good point to first known-bad point 2. Test the midpoint 3. If midpoint is good, bug is in second half 4. If midpoint is bad, bug is in first half 5. Repeat until you've isolated the bug location

For git history, use git bisect:

git bisect start
git bisect bad HEAD          # Current is broken
git bisect good v1.0.0       # This version worked
# Git picks middle commit, test it, mark good/bad
# Finds culprit commit in log2(n) tests

When NOT to use this pattern:

• Non-deterministic bugs that don't reproduce reliably
• Bugs that depend on specific data only present at certain points

Reference: Code with Jason - Binary Search Debugging

Test One Hypothesis at a Time

Change only one variable per experiment. Making multiple changes simultaneously prevents you from knowing which change had which effect, leading to false conclusions and unfixed bugs.

Incorrect (multiple simultaneous changes):

# Bug: API returns 500 error intermittently

# Developer makes several "fixes" at once:
def get_user(user_id):
    try:
        # Change 1: Add timeout
        response = requests.get(url, timeout=30)
        # Change 2: Add retry logic
        if response.status_code != 200:
            response = requests.get(url, timeout=30)
        # Change 3: Add null check
        data = response.json()
        if data is None:
            return default_user
        # Change 4: Add caching
        cache.set(user_id, data)
        return data
    except Exception as e:
        # Change 5: Better error handling
        logger.error(f"Failed: {e}")
        return default_user

# Bug seems fixed... but WHICH change fixed it?
# What if changes 1, 2, 4, 5 are unnecessary overhead?
# What if change 3 is hiding a different bug?

Correct (test one change at a time):

# Bug: API returns 500 error intermittently

# Original code (reproduce the bug first):
def get_user(user_id):
    response = requests.get(url)
    return response.json()

# Hypothesis 1: Request timeout causes 500
# Test: Add ONLY timeout, test multiple times
def get_user_v1(user_id):
    response = requests.get(url, timeout=30)  # Only change
    return response.json()
# Result: Still fails. Timeout is not the cause.

# Hypothesis 2: Server overloaded, needs retry
# Test: Add ONLY retry (revert timeout)
def get_user_v2(user_id):
    response = requests.get(url)
    if response.status_code == 500:
        time.sleep(1)
        response = requests.get(url)  # Only change
    return response.json()
# Result: Works! Retry after brief delay fixes it.

# Now you KNOW: Server needs brief cooldown between requests
# Can investigate WHY and fix properly, or add targeted retry

Controlled experiment checklist:

• Revert to known-broken state before each test
• Change exactly one thing
• Test thoroughly (multiple runs if intermittent)
• Document result before moving to next hypothesis
• Keep working changes, revert ineffective ones

When NOT to use this pattern:

• Emergency production fixes (fix first, understand later)
• When changes are clearly interdependent

Reference: A Systematic Approach to Debugging

Explain the Problem Aloud (Rubber Duck)

Explain your code and the bug to someone (or something) else, line by line. The act of verbalizing forces you to examine assumptions and often reveals the bug before you finish explaining.

Incorrect (debugging silently in your head):

// Bug: Function returns wrong result
function calculateDiscount(price, quantity, memberLevel) {
  let discount = 0;
  if (quantity > 10) discount = 0.1;
  if (quantity > 50) discount = 0.2;
  if (memberLevel === 'gold') discount += 0.05;
  if (memberLevel === 'platinum') discount += 0.1;
  return price * quantity * discount;  // Stare at code...
}                                        // Looks right...

// Developer stares at code for 30 minutes
// "I don't see the bug..."
// Keeps re-reading the same lines

Correct (explain to a rubber duck):

// Bug: Function returns wrong result
function calculateDiscount(price, quantity, memberLevel) {
  let discount = 0;

  // "OK duck, this function calculates the discounted price."
  // "First, discount starts at 0..."
  if (quantity > 10) discount = 0.1;
  // "If quantity is over 10, discount is 10%..."
  if (quantity > 50) discount = 0.2;
  // "If over 50, discount is 20%..."
  if (memberLevel === 'gold') discount += 0.05;
  // "Gold members get 5% extra..."
  if (memberLevel === 'platinum') discount += 0.1;
  // "Platinum gets 10% extra..."

  return price * quantity * discount;
  // "Then return price times quantity times discount..."
  // "Wait. That gives the DISCOUNT AMOUNT, not the final price!"
  // "It should be: price * quantity * (1 - discount)"
  // BUG FOUND IN 2 MINUTES
}

Rubber duck debugging process:

1. Get a rubber duck (or any object, or a colleague) 2. Explain what the code is SUPPOSED to do 3. Explain what it ACTUALLY does, line by line 4. Explain your data at each step 5. The bug usually reveals itself during explanation

Why this works:

• Forces you to slow down and be precise
• Exposes assumptions you didn't realize you were making
• Shifts perspective from "writer" to "explainer"
• Engages verbal reasoning alongside visual code reading

When NOT to use this pattern:

• Race conditions and timing bugs (hard to verbalize)
• Bugs in code you don't understand at all yet

Reference: Rubber Duck Debugging

Rule Out Obvious Causes First

Before diving deep, check common causes that explain most bugs. Many developers waste hours on complex investigations when the cause is a simple typo, wrong config, or stale cache.

Incorrect (jumping to complex causes):

# Bug: API returns 404 for endpoint that exists

# Developer assumes complex cause:
# "Must be a routing conflict or middleware issue..."
# Spends 2 hours debugging router configuration
# Checks middleware ordering
# Reviews authentication logic
# Adds extensive logging

# Finally runs: curl -v http://localhost:3000/api/users
# Response: "connection refused"
# Server wasn't running.

Correct (check obvious causes first):

# Bug: API returns 404 for endpoint that exists

# OBVIOUS CAUSES CHECKLIST (5 minutes max):

# 1. Is the server running?
curl -v http://localhost:3000/health
# ✗ Connection refused - START THE SERVER

# 2. Is the URL correct?
# Check for typos: /api/users vs /api/user vs /users

# 3. Is the method correct?
# POST vs GET vs PUT?

# 4. Is the environment correct?
# Dev vs staging vs prod URL?

# 5. Is the code deployed?
git status  # Uncommitted changes?
git log -1  # Is this the version you think it is?

# 6. Is there a cache involved?
# Browser cache? CDN cache? API cache?

# 7. Did you save the file?
# IDE might not have auto-saved

# 8. Is the config correct?
# Environment variables set? Config file loaded?

The "stupid things" checklist (check first, always):

1. Is it running? (server, service, database) 2. Is it the right environment? (dev/staging/prod) 3. Is the code saved and deployed? 4. Is there a cache to clear? 5. Are credentials/config correct? 6. Is there a typo in the name/path/URL? 7. Have you tried restarting it? 8. Are you testing the right thing?

When NOT to use this pattern:

• You've already verified the obvious causes
• The bug only appeared after a specific code change

Reference: Debugging Best Practices

Apply the Scientific Method

Debug like a scientist: observe, hypothesize, predict, test, conclude. This systematic approach eliminates guesswork and ensures you converge on the root cause rather than wandering aimlessly through code.

Incorrect (random exploration):

# Bug: Users sometimes get wrong totals in shopping cart

def calculate_total(cart):
    total = 0
    for item in cart.items:
        total += item.price * item.quantity  # Check this
    return total + cart.shipping  # Or check this

# Developer starts adding random print statements
print(f"total: {total}")  # Looks right...
print(f"shipping: {cart.shipping}")  # Looks right...
# Adds more prints, removes some, changes code, reverts...
# 2 hours later: still confused, code is a mess

Correct (scientific method):

# Bug: Users sometimes get wrong totals in shopping cart

# STEP 1: OBSERVE
# - Total is sometimes higher than expected
# - Happens with certain items, not all orders
# - Difference is always exactly one item's price

# STEP 2: HYPOTHESIZE
# Hypothesis 1: Item counted twice somewhere
# Hypothesis 2: Old cart data mixing with new
# Hypothesis 3: Currency conversion error

# STEP 3: PREDICT (for Hypothesis 1)
# If item is counted twice, the excess should equal
# exactly one item's price in the cart

# STEP 4: TEST
def calculate_total_debug(cart):
    seen_items = []
    total = 0
    for item in cart.items:
        if item.id in seen_items:
            print(f"DUPLICATE FOUND: {item.id}")  # Test prediction
        seen_items.append(item.id)
        total += item.price * item.quantity
    return total + cart.shipping

# STEP 5: CONCLUDE
# Test revealed: item.id appears twice when user clicks
# "Add to Cart" rapidly. Root cause: no debounce on button.

Scientific debugging template:

1. Observe: What exactly do you see? (symptoms, data, errors) 2. Hypothesize: What could cause this? (at least 2-3 options) 3. Predict: If hypothesis X is true, what else should be true? 4. Test: Design experiment to verify/falsify prediction 5. Conclude: Was hypothesis correct? Update understanding

When NOT to use this pattern:

• Trivial bugs where cause is immediately obvious
• Build/syntax errors with clear error messages

Reference: MIT 6.031 - Scientific Debugging

Find WHERE Before Asking WHAT

Focus first on locating exactly WHERE the bug occurs, not understanding WHAT the code does. Once you know the precise location, understanding the bug becomes much easier with the surrounding context.

Incorrect (trying to understand everything):

// Bug: User subscription status is wrong
// Developer tries to understand the entire subscription system first

// "Let me read through all the subscription code..."
// subscription/types.ts (200 lines)
// subscription/service.ts (500 lines)
// subscription/webhook.ts (300 lines)
// subscription/sync.ts (400 lines)
// billing/integration.ts (600 lines)

// 3 hours later: "I understand how subscriptions work now"
// But still don't know where the bug is

Correct (locate first, understand second):

// Bug: User subscription status is wrong
// Step 1: Find WHERE status becomes wrong

// Add checkpoints at system boundaries:
console.log('After webhook received:', status);       // CORRECT
console.log('After webhook processed:', status);      // CORRECT
console.log('After sync to database:', status);       // CORRECT
console.log('After read from database:', status);     // WRONG! <-- HERE

// Step 2: NOW narrow focus to this specific area
// Only need to understand: database write + read logic
// Read 50 lines instead of 2000

// Step 3: Understand just this section
async function getSubscriptionStatus(userId: string) {
  const cached = await cache.get(`sub:${userId}`);
  if (cached) return cached;  // BUG: Cache not invalidated on update!
  return await db.subscriptions.findOne({ userId });
}

// Found bug in 20 minutes by locating first

WHERE-first debugging steps:

1. Add coarse checkpoints at major boundaries (API, service, database) 2. Identify which section contains the bug 3. Add finer checkpoints within that section 4. Repeat until you find the exact line/function 5. NOW read and understand just that code

The "traffic light" technique:

# Mark checkpoints with pass/fail status
print("🟢 Checkpoint A: data valid")     # Known good
print("🟡 Checkpoint B: checking...")    # Testing now
print("🔴 Checkpoint C: data corrupted")  # Known bad
# Bug is between 🟢 and 🔴

When NOT to use this pattern:

• Architectural/design bugs requiring broad understanding
• When you're new to the codebase and need context anyway

Reference: Code with Jason - Binary Search Debugging

Use Breakpoints Strategically

Place breakpoints at decision points and state transitions, not randomly. Strategic breakpoints let you inspect program state interactively, enabling deeper exploration than static print statements.

Incorrect (breakpoints everywhere):

def process_payment(order, payment_method):
    # Breakpoint here
    total = order.total
    # Breakpoint here
    tax = calculate_tax(total)
    # Breakpoint here
    final = total + tax
    # Breakpoint here
    result = charge_card(payment_method, final)
    # Breakpoint here
    if result.success:
        # Breakpoint here
        update_inventory(order)
        # Breakpoint here
        send_receipt(order)
    # Breakpoint here
    return result

# Developer hits F5/Continue 8 times, loses track of state

Correct (strategic breakpoints):

def process_payment(order, payment_method):
    total = order.total
    tax = calculate_tax(total)
    final = total + tax

    # BREAKPOINT 1: Before external API call
    # Inspect: final, payment_method, order state
    result = charge_card(payment_method, final)

    # BREAKPOINT 2: After external call, before branching
    # Inspect: result.success, result.error, result.transaction_id
    if result.success:
        update_inventory(order)
        send_receipt(order)

    # BREAKPOINT 3: Before return (conditional)
    # Only hit if debugging return value issues
    return result

Strategic breakpoint locations:

1. Before external calls (APIs, database) - verify inputs 2. After external calls - verify responses 3. At decision points (if/switch) - understand branching 4. Loop entry - verify initial state 5. After complex calculations - verify results

Power features to use:

# Conditional breakpoint (IDE dependent)
# Break only when: order.total > 1000

# Logpoint (print without stopping)
# Log: f"Processing order {order.id}, total={order.total}"

# Hit count breakpoint
# Break after 10 hits (useful for loops)

# Exception breakpoint
# Break when specific exception is raised

When NOT to use this pattern:

• Remote debugging where breakpoints cause timeouts
• Multi-threaded race conditions (breakpoints change timing)
• Production environments

Reference: VS Code Debugging

Log Function Inputs and Outputs

When debugging data issues, log what goes into and comes out of functions. This creates a data flow trace that reveals exactly where values become incorrect.

Incorrect (logging inside function only):

function transformData(items) {
  console.log("transforming items");  // No actual data
  const result = items.map(item => ({
    id: item.id,
    total: item.price * item.qty,
    name: item.name.toUpperCase()
  }));
  console.log("transform complete");  // Still no data
  return result;
}

// Bug: Totals are wrong
// Logs tell you nothing about actual values

Correct (log inputs and outputs):

function transformData(items) {
  // Log input with identifying info
  console.log("transformData INPUT:", JSON.stringify({
    count: items.length,
    sample: items[0],  // First item as example
    itemIds: items.map(i => i.id)
  }, null, 2));

  const result = items.map(item => ({
    id: item.id,
    total: item.price * item.qty,
    name: item.name.toUpperCase()
  }));

  // Log output with same structure
  console.log("transformData OUTPUT:", JSON.stringify({
    count: result.length,
    sample: result[0],
    totals: result.map(r => ({ id: r.id, total: r.total }))
  }, null, 2));

  return result;
}

// Output:
// transformData INPUT: {
//   "count": 3,
//   "sample": { "id": 1, "price": 10, "qty": "2", "name": "Widget" }
//                                           ^^^ String not number!
// transformData OUTPUT: {
//   "totals": [{ "id": 1, "total": NaN }]  // Reveals the bug

Input/output logging patterns:

# Decorator pattern for automatic logging
def log_io(func):
    def wrapper(*args, **kwargs):
        logger.debug(f"{func.__name__} called", extra={
            "args": str(args)[:200],  # Truncate large data
            "kwargs": str(kwargs)[:200]
        })
        result = func(*args, **kwargs)
        logger.debug(f"{func.__name__} returned", extra={
            "result_type": type(result).__name__,
            "result_preview": str(result)[:200]
        })
        return result
    return wrapper

@log_io
def calculate_total(items):
    return sum(item.price * item.qty for item in items)

When NOT to use this pattern:

• Functions called thousands of times (performance impact)
• Functions with huge inputs/outputs (log summaries instead)
• Sensitive data (passwords, PII)

Reference: Effective Debugging and Logging

Read Stack Traces Bottom to Top

Stack traces show the call chain with the most recent (and usually most relevant) frame at the bottom. Start reading from the bottom where the error occurred, then work up to understand how you got there.

Incorrect (reading top to bottom):

Traceback (most recent call last):
  File "main.py", line 5, in <module>      # Start here? No!
    run_app()
  File "app.py", line 23, in run_app
    process_request(request)
  File "handler.py", line 45, in process_request
    data = parse_input(request.body)
  File "parser.py", line 12, in parse_input
    return json.loads(text)
  File "/usr/lib/python3/json/__init__.py", line 346
    raise JSONDecodeError(...)
json.decoder.JSONDecodeError: Expecting value: line 1 column 1

# Developer starts debugging main.py - wrong place!

Correct (reading bottom to top):

Traceback (most recent call last):
  File "main.py", line 5, in <module>
    run_app()
  File "app.py", line 23, in run_app
    process_request(request)
  File "handler.py", line 45, in process_request
    data = parse_input(request.body)         # 3. Called parse_input
  File "parser.py", line 12, in parse_input  # 2. With what body?
    return json.loads(text)                   # 1. START HERE: json.loads failed
  File "/usr/lib/python3/json/__init__.py", line 346
    raise JSONDecodeError(...)
json.decoder.JSONDecodeError: Expecting value: line 1 column 1

# Reading bottom-to-top:
# 1. BOTTOM: JSONDecodeError on empty/invalid JSON
# 2. UP: In parse_input, calling json.loads(text)
# 3. UP: Called from process_request with request.body
# Question: What was request.body? Likely an empty string.

Stack trace reading strategy:

BOTTOM (Error Location):
└── What exception? What message?
└── What line threw it?
└── What were the arguments?

MIDDLE (Your Code):
└── Find the topmost frame in YOUR code
└── This is usually where the bug actually is
└── Look at the data being passed

TOP (Entry Point):
└── How did we get here?
└── What triggered this code path?
└── Useful for understanding context

Skip library frames, focus on your code:

# Look for frames in YOUR code, not libraries:
  File "parser.py", line 12, in parse_input  # <-- YOUR CODE
    return json.loads(text)
  File "/usr/lib/python3/json/__init__.py"   # <-- LIBRARY (skip)

When NOT to use this pattern:

• Errors in library code due to version bugs
• Stack traces truncated or missing

Reference: Python Documentation - Traceback

Use Strategic Logging Over Random Print Statements

Replace scattered print statements with structured, leveled logging at strategic points. Good logging provides context, can be filtered, persists for analysis, and doesn't require removal before commit.

Incorrect (random print statements):

def process_order(order):
    print("here1")  # Where? What?
    print(order)    # Raw dump, hard to parse
    user = get_user(order.user_id)
    print("got user")
    print(user)
    total = calculate_total(order)
    print(f"total is {total}")  # No context
    if total > 1000:
        print("big order!")
        discount = apply_discount(total)
        print(discount)  # Is this the discount or discounted total?
    result = save_order(order)
    print("done")
    return result

# Output:
# here1
# <Order object at 0x...>
# got user
# <User object at 0x...>
# total is 1250
# big order!
# 125
# done
# Which is which? Hard to read, no timestamp, can't filter

Correct (strategic structured logging):

import logging

logger = logging.getLogger(__name__)

def process_order(order):
    logger.info("Processing order", extra={
        "order_id": order.id,
        "user_id": order.user_id,
        "item_count": len(order.items)
    })

    user = get_user(order.user_id)
    logger.debug("User retrieved", extra={
        "user_id": user.id,
        "member_level": user.member_level
    })

    total = calculate_total(order)
    logger.debug("Total calculated", extra={
        "order_id": order.id,
        "total": total,
        "currency": "USD"
    })

    if total > 1000:
        logger.info("Large order discount applied", extra={
            "order_id": order.id,
            "original_total": total,
            "discount_percent": 10
        })
        total = apply_discount(total)

    result = save_order(order)
    logger.info("Order processed successfully", extra={
        "order_id": order.id,
        "final_total": total
    })
    return result

# Output (JSON format, filterable, parseable):
# {"level": "INFO", "msg": "Processing order", "order_id": 123, ...}
# {"level": "DEBUG", "msg": "User retrieved", "user_id": 456, ...}

Strategic logging placement:

• Function entry/exit (INFO)
• Major decision points (DEBUG)
• External service calls (INFO with timing)
• Error conditions (ERROR with full context)
• State changes (DEBUG)

When NOT to use this pattern:

• Quick one-off investigation (print is fine)
• Performance-critical hot paths (logging has overhead)

Reference: Advanced Debug Logging Techniques

Trace Data Flow Through the System

When data is incorrect, trace its journey through the system to find where it becomes corrupted. Add checkpoints at each transformation to see values before and after.

Incorrect (guessing where corruption happens):

# Bug: Final price is wrong
# Developer checks random places, can't find issue

def handle_purchase(product_id, quantity, user):
    product = get_product(product_id)  # Check here
    price = calculate_price(product, quantity)  # Or here
    discount = get_discount(user)  # Or here
    final = apply_discount(price, discount)  # Or here
    return charge(user, final)  # Or here

# After 2 hours: still don't know where price goes wrong

Correct (trace data through each stage):

# Bug: Final price is wrong
# Trace data through every transformation

def handle_purchase(product_id, quantity, user):
    product = get_product(product_id)
    print(f"[1] product.price = {product.price}")  # 29.99 ✓

    price = calculate_price(product, quantity)
    print(f"[2] price (qty={quantity}) = {price}")  # 59.98 ✓ (2 × 29.99)

    discount = get_discount(user)
    print(f"[3] discount = {discount}")  # 0.1 ✓ (10%)

    final = apply_discount(price, discount)
    print(f"[4] final = {final}")  # 5.998 ✗ WRONG!
                                   # Expected ~53.98, got 5.998
                                   # apply_discount is buggy!

    return charge(user, final)

# Found in 5 minutes: apply_discount does price*discount, not price*(1-discount)

Data flow tracing template:

def trace_data(label, data):
    """Consistent checkpoint format"""
    print(f"[{label}] type={type(data).__name__}, value={data}")
    return data  # Pass through for chaining

# Usage:
result = trace_data("step1", fetch_data())
result = trace_data("step2", transform(result))
result = trace_data("step3", validate(result))

For complex pipelines, create a flow diagram:

Input (correct)
    ↓
[Transform A] ✓
    ↓
[Transform B] ✓
    ↓
[Transform C] ✗ ← Bug is here
    ↓
Output (wrong)

When NOT to use this pattern:

• Bug is in control flow, not data
• Asynchronous flows where data trace is hard to follow

Reference: Why Programs Fail - Tracking Origins

Use Watch Expressions for Complex State

Add watch expressions for computed values and relationships you need to monitor. Watches update automatically as you step through code, revealing exactly when values change unexpectedly.

Incorrect (manually inspecting each time):

// Bug: Array index sometimes out of bounds
function processItems(items, startIndex) {
  for (let i = startIndex; i < items.length; i++) {
    // Manually type in debug console each step:
    // > items.length
    // > i
    // > i < items.length
    // > items[i]
    // Tedious and error-prone
    processItem(items[i]);
  }
}

Correct (watch expressions):

// Bug: Array index sometimes out of bounds
function processItems(items, startIndex) {
  // Set up watch expressions in debugger:
  // Watch 1: items.length
  // Watch 2: i
  // Watch 3: i < items.length  (the loop condition)
  // Watch 4: items[i]          (current item)
  // Watch 5: items[i + 1]      (lookahead)

  for (let i = startIndex; i < items.length; i++) {
    processItem(items[i]);
    // As you step, watches update automatically:
    // Iteration 0: length=3, i=0, i<length=true, [i]=item0
    // Iteration 1: length=3, i=1, i<length=true, [i]=item1
    // Iteration 2: length=2, i=2, i<length=false <-- length changed!
    // Someone modified items during iteration
  }
}

Useful watch expression patterns:

# Track object state
watch: user.__dict__
watch: len(items)
watch: type(response)

# Track computed values
watch: total - expected_total
watch: current_time - start_time

# Track relationships
watch: parent.children.includes(child)
watch: request.user.id == response.user_id

# Track conditions
watch: retry_count < max_retries
watch: buffer.length > threshold

Watch expressions vs variables panel:

When NOT to use this pattern:

• Expressions with side effects (will execute every step!)
• Very expensive computations (slows debugging)

Reference: VS Code Debugging - Watch

Catch Async/Await Error Handling Mistakes

Async/await makes asynchronous code look synchronous, but error handling behaves differently. Unhandled promise rejections, missing try/catch, and forgotten await keywords are common bugs.

Incorrect (async error handling mistakes):

// Bug 1: Missing try/catch
async function fetchUser(id) {
  const response = await fetch(`/api/users/${id}`)
  return response.json()  // Network errors crash the app
}

// Bug 2: Forgotten await
async function processOrder(orderId) {
  const order = await getOrder(orderId)
  validateOrder(order)  // If async, validation runs after return!
  return order
}

// Bug 3: Errors lost in Promise.all
async function loadDashboard() {
  const [users, orders, stats] = await Promise.all([
    fetchUsers(),    // If this fails...
    fetchOrders(),   // These still run but error is unclear
    fetchStats()
  ])
}

Correct (proper async error handling):

// Fixed 1: Try/catch for error handling
async function fetchUser(id) {
  try {
    const response = await fetch(`/api/users/${id}`)
    if (!response.ok) {
      throw new Error(`HTTP ${response.status}`)
    }
    return response.json()
  } catch (error) {
    logger.error('fetch_user_failed', { id, error: error.message })
    throw error  // Re-throw or return fallback
  }
}

// Fixed 2: Await all async operations
async function processOrder(orderId) {
  const order = await getOrder(orderId)
  await validateOrder(order)  // Properly awaited
  return order
}

// Fixed 3: Handle Promise.all failures gracefully
async function loadDashboard() {
  const results = await Promise.allSettled([
    fetchUsers(),
    fetchOrders(),
    fetchStats()
  ])
  // Check each result: { status: 'fulfilled', value } or { status: 'rejected', reason }
  const [usersResult, ordersResult, statsResult] = results
  if (usersResult.status === 'rejected') {
    logger.error('users_fetch_failed', { error: usersResult.reason })
  }
}

Async debugging tips:

• Add .catch() to all promises in development to surface errors
• Use unhandledRejection event handler to log missed errors
• ESLint rules: require-await, no-floating-promises

Reference: Coders.dev - The Art of Debugging

Detect Memory Leak Patterns

Memory leaks occur when allocated memory is never released. Symptoms: gradually increasing memory usage, eventual out-of-memory crashes, performance degradation over time. Look for event listeners not removed, caches without bounds, and circular references.

Incorrect (memory leak patterns):

// Leak 1: Event listeners never removed
class Dashboard {
  constructor() {
    window.addEventListener('resize', this.handleResize)  // Never removed
  }
  // Missing: componentWillUnmount to remove listener
}

// Leak 2: Unbounded cache
const cache = {}
function getCachedData(key) {
  if (!cache[key]) {
    cache[key] = fetchData(key)  // Cache grows forever
  }
  return cache[key]
}

// Leak 3: Closures holding references
function createHandlers(elements) {
  const handlers = []
  for (const el of elements) {
    handlers.push(() => {
      console.log(el)  // Each closure holds reference to element
    })
  }
  return handlers  // Elements can't be garbage collected
}

Correct (memory-safe patterns):

// Fixed 1: Remove event listeners
class Dashboard {
  constructor() {
    this.handleResize = this.handleResize.bind(this)
    window.addEventListener('resize', this.handleResize)
  }
  destroy() {
    window.removeEventListener('resize', this.handleResize)
  }
}

// Fixed 2: Bounded cache with LRU eviction
const cache = new LRUCache({ max: 1000 })
function getCachedData(key) {
  if (!cache.has(key)) {
    cache.set(key, fetchData(key))
  }
  return cache.get(key)
}

// Fixed 3: WeakRef for optional references
function createHandlers(elements) {
  return elements.map(el => {
    const weakRef = new WeakRef(el)
    return () => {
      const element = weakRef.deref()
      if (element) console.log(element)
    }
  })
}

Memory leak detection:

• Memory profilers: Chrome DevTools, Valgrind, dotMemory
• Monitor heap size over time in production
• Test with long-running automated scenarios

Reference: Netdata - How to Find Memory Leaks

Recognize Null Pointer Patterns

Null pointer dereferences occur when code assumes a value exists but it doesn't. Recognize the patterns: missing null checks, optional chaining neglected, uninitialized variables, and failed lookups assumed successful.

Incorrect (assuming value exists):

function getUserEmail(userId: string): string {
  const user = userRepository.findById(userId)
  return user.email  // Crashes if user not found
}

function getFirstItem(items: Item[]): string {
  return items[0].name  // Crashes if array empty
}

function processConfig(config: Config): void {
  const timeout = config.settings.network.timeout  // Crashes if any level missing
}

Correct (defensive null handling):

function getUserEmail(userId: string): string | null {
  const user = userRepository.findById(userId)
  if (!user) {
    logger.warn('user_not_found', { userId })
    return null
  }
  return user.email
}

function getFirstItem(items: Item[]): string | null {
  if (items.length === 0) {
    return null
  }
  return items[0].name
}

function processConfig(config: Config): void {
  const timeout = config?.settings?.network?.timeout ?? 30000
  // Uses optional chaining and default value
}

Common null pointer sources:

• Database/API lookups that return no results
• Array access with invalid index
• Object property access on undefined
• Map/dictionary lookups for missing keys
• Race conditions where value not yet initialized

Reference: Krishna Gupta - Understanding CWE-476 NULL Pointer Dereference

Spot Off-by-One Errors

Off-by-one errors occur at boundaries: loop iterations, array indices, string slicing. Check whether conditions should use < vs <=, whether indices start at 0 or 1, and whether ranges are inclusive or exclusive.

Incorrect (off-by-one in loop):

def process_items(items):
    # Bug: Skips last item
    for i in range(len(items) - 1):  # Should be range(len(items))
        process(items[i])

def get_substring(text, start, length):
    # Bug: Returns one character too many
    return text[start:start + length + 1]  # Should be start + length

def validate_page_number(page, total_pages):
    # Bug: Rejects valid last page
    if page > total_pages - 1:  # Should be page > total_pages or page >= total_pages
        raise InvalidPageError()

Correct (boundary-aware code):

def process_items(items):
    # Correct: Process all items
    for i in range(len(items)):
        process(items[i])
    # Or simply: for item in items: process(item)

def get_substring(text, start, length):
    # Correct: Python slicing is exclusive on end
    return text[start:start + length]

def validate_page_number(page, total_pages):
    # Correct: Pages 1 through total_pages are valid
    if page < 1 or page > total_pages:
        raise InvalidPageError()

Off-by-one checklist:

• [ ] Does the loop include or exclude the last element?
• [ ] Are indices 0-based or 1-based?
• [ ] Is the range/slice inclusive or exclusive on the end?
• [ ] Does <= vs < matter for the edge case?

Reference: FSU - Debugging Techniques

Identify Race Condition Symptoms

Race conditions occur when multiple threads or processes access shared state without proper synchronization. Symptoms: intermittent failures, results depend on timing, works in debugger but fails in production.

Incorrect (unsynchronized shared state):

public class Counter {
    private int count = 0;

    public void increment() {
        count++;  // Not atomic: read, add, write can interleave
    }

    public int getCount() {
        return count;
    }
}

// Two threads call increment() 1000 times each
// Expected: count = 2000
// Actual: count = 1847 (random, changes each run)

Correct (synchronized access):

public class Counter {
    private final AtomicInteger count = new AtomicInteger(0);

    public void increment() {
        count.incrementAndGet();  // Atomic operation
    }

    public int getCount() {
        return count.get();
    }
}

// Two threads call increment() 1000 times each
// Result: count = 2000 (always correct)

Race condition indicators:

• Bug "disappears" when adding logging or breakpoints
• Different results on each run
• Works on developer machine, fails in CI/production
• Failures correlate with load or concurrent users
• "Heisenbug" that changes when observed

Detection tools:

• Thread sanitizers (TSan, Helgrind)
• Static analysis for data races
• Stress testing with high concurrency

Reference: Valgrind Documentation - Helgrind Thread Analyzer

Recognize Timezone and Date Bugs

Date and timezone bugs are subtle and often only manifest for users in certain locations or at certain times. Symptoms: events on wrong day, off-by-one-day errors near midnight, DST transition bugs.

Incorrect (timezone-unaware date handling):

// Bug 1: Date comparison ignores timezone
function isToday(eventDate) {
  const today = new Date()
  return eventDate.getDate() === today.getDate()  // Fails across timezones
}

// Bug 2: Creating dates from strings
const deadline = new Date('2024-03-15')  // Parsed as UTC midnight
// In US Pacific (UTC-8): March 14th 4pm!

// Bug 3: Storing local time instead of UTC
const createdAt = new Date().toString()  // "Fri Mar 15 2024 10:30:00 GMT-0800"
// Comparing this string across timezones: chaos

Correct (timezone-aware date handling):

// Fixed 1: Compare using date strings
function isToday(eventDate) {
  const today = new Date()
  return eventDate.toISOString().slice(0, 10) === today.toISOString().slice(0, 10)
}

// Fixed 2: Be explicit about timezone
const deadline = new Date('2024-03-15T00:00:00-08:00')  // Pacific midnight
// Or use a date library:
import { parseISO } from 'date-fns'
import { zonedTimeToUtc } from 'date-fns-tz'

// Fixed 3: Store timestamps in UTC
const createdAt = new Date().toISOString()  // "2024-03-15T18:30:00.000Z"
// Or store Unix timestamp
const createdAtUnix = Date.now()  // 1710526200000

// Display in user's local timezone:
const displayTime = new Date(createdAt).toLocaleString('en-US', {
  timeZone: userTimezone
})

Timezone bug prevention:

• Store all dates in UTC (ISO 8601 or Unix timestamp)
• Convert to local time only for display
• Use date libraries (date-fns, Luxon) for manipulation
• Test with users in multiple timezones
• Test around DST transitions

Reference: Wikipedia - Falsehoods Programmers Believe About Time

Watch for Type Coercion Bugs

Type coercion bugs occur when languages implicitly convert between types. JavaScript is notorious for this: string concatenation instead of addition, truthy/falsy surprises, and loose equality comparisons.

Incorrect (implicit type coercion):

// Bug 1: String concatenation instead of addition
function calculateTotal(price, tax) {
  return price + tax  // If tax is "10" (string): "100" + "10" = "10010"
}
calculateTotal(100, document.getElementById('tax').value)  // Input values are strings!

// Bug 2: Falsy zero treated as missing
function getDiscount(discount) {
  return discount || 10  // Returns 10 when discount is 0!
}
getDiscount(0)  // Expected: 0, Actual: 10

// Bug 3: Loose equality surprises
if (userId == null) {  // True for both null AND undefined
  // ...
}
'0' == false  // true (wat)
[] == false   // true (double wat)

Correct (explicit type handling):

// Fixed 1: Parse input explicitly
function calculateTotal(price, tax) {
  const numericTax = parseFloat(tax)
  if (isNaN(numericTax)) {
    throw new Error('Invalid tax value')
  }
  return price + numericTax
}

// Fixed 2: Explicit undefined check
function getDiscount(discount) {
  return discount !== undefined ? discount : 10
  // Or with nullish coalescing: discount ?? 10
}
getDiscount(0)  // Correctly returns 0

// Fixed 3: Strict equality
if (userId === null) {  // Only true for null, not undefined
  // ...
}
'0' === false  // false (correct)
[] === false   // false (correct)

Type coercion danger zones:

• Form input values (always strings)
• JSON parsed numbers (may be strings)
• Query parameters (always strings)
• Arithmetic with mixed types
• Boolean coercion of 0, "", null, undefined

Reference: TMS Outsource - What is Debugging

Add Defensive Code at System Boundaries

Add validation and assertions at system boundaries (APIs, module interfaces, data ingestion points). Bugs caught at boundaries have better context and don't propagate through the system.

Incorrect (no boundary defense):

# Bug manifests deep in the system
def render_dashboard(data):
    for widget in data['widgets']:
        for metric in widget['metrics']:
            # NullPointerException here
            # No idea where bad data came from
            # Could be API, database, cache, or calculation
            value = metric['current'] / metric['previous']

Correct (defend at boundaries):

# API boundary - validate incoming requests
@app.route('/api/dashboard')
def get_dashboard():
    data = request.json
    # Validate at entry point
    if not data or 'user_id' not in data:
        raise ValueError("user_id required")  # Clear error source

    dashboard = generate_dashboard(data['user_id'])
    return validate_dashboard_response(dashboard)  # Validate output too

# Module boundary - validate inputs and outputs
def generate_dashboard(user_id: str) -> dict:
    assert user_id, "user_id must not be empty"

    widgets = load_widgets(user_id)
    assert isinstance(widgets, list), f"Expected list, got {type(widgets)}"

    return {
        'user_id': user_id,
        'widgets': [validate_widget(w) for w in widgets]
    }

def validate_widget(widget: dict) -> dict:
    """Validate widget structure at data boundary"""
    required = ['id', 'type', 'metrics']
    missing = [k for k in required if k not in widget]
    if missing:
        raise ValueError(f"Widget missing required fields: {missing}")

    for metric in widget.get('metrics', []):
        if metric.get('previous', 0) == 0:
            logger.warning(f"Zero previous value in widget {widget['id']}")
            metric['previous'] = 1  # Prevent division by zero with logged warning

    return widget

Where to add defensive code:

1. API endpoints - Validate request parameters 2. Database results - Check expected structure 3. External service responses - Verify format/status 4. Module public interfaces - Assert preconditions 5. Configuration loading - Validate required settings 6. User input - Sanitize and validate

Defense patterns:

# Assert preconditions
def calculate(items):
    assert items, "items must not be empty"
    assert all(isinstance(i, Item) for i in items), "all items must be Item type"

# Guard clauses
def process(data):
    if not data:
        logger.warning("process called with empty data")
        return None

# Fail fast with context
def fetch_user(user_id):
    if not user_id:
        raise ValueError(f"user_id required, got: {user_id!r}")

When NOT to use this pattern:

• Internal functions with trusted callers
• Hot paths where validation overhead matters
• Redundant validation already done upstream

Reference: Defensive Programming

Document Bug Solutions for Future Reference

After solving a non-trivial bug, document the symptoms, investigation process, root cause, and solution. This creates a searchable knowledge base that helps you and teammates solve similar issues faster.

Incorrect (no documentation):

# Git commit message
Fix: resolved login issue

# 6 months later, same symptoms appear
# No one remembers the original investigation
# Team spends another 4 hours debugging
# Realizes it's the same issue they fixed before

Correct (documented for reference):

# Bug Documentation: Login Timeout on High Load

## Symptoms
- Users report "Login failed" after 30 seconds
- Occurs during peak hours (9-10 AM)
- Server logs show no errors during failure window

## Investigation
1. Checked auth service logs - normal response times
2. Checked database connections - pool at 95% capacity!
3. Traced connection leak to failed auth attempts
4. Found: Connections not released when password check fails

## Root Cause
In `auth.py:validate_password()`, database connection acquired but
not released in the error path. Under high load, pool exhausted.

## Solution
- Added `finally` block to release connection (commit abc123)
- Added connection pool monitoring dashboard
- Added alert for pool > 80% capacity

## Verification
- Load tested with 1000 concurrent logins
- Pool usage stable at 20-30%
- No timeout errors

## Related
- Similar issue in password reset: PR #456
- Connection pool documentation: /docs/database.md

What to document:

• Symptoms: Exact error messages, conditions, frequency
• Investigation: Steps taken, dead ends, key findings
• Root cause: The actual underlying issue
• Solution: What was changed and why
• Verification: How you confirmed the fix
• Related: Links to similar issues, relevant docs

Where to document:

• Bug tracker comments (attached to original issue)
• Team wiki/knowledge base
• Code comments for tricky edge cases
• ADR (Architecture Decision Record) for significant changes

When NOT to use this pattern:

• Trivial bugs (typos, obvious mistakes)
• One-off issues unlikely to recur

Reference: Root Cause Analysis - Documentation

Improve Error Messages When You Debug

When a bug takes a long time to find because error messages were unhelpful, improve those messages as part of your fix. Pay forward the debugging effort to help the next person.

Incorrect (leaving poor error messages):

# Original code with unhelpful error
def process_order(order_id):
    order = get_order(order_id)
    if not order:
        raise Exception("Error")  # What error? What order?

# Developer spends 2 hours finding bug
# Fixes the immediate issue
# Leaves the poor error message unchanged
# Next developer will also struggle

Correct (improve messages while fixing):

# After spending 2 hours debugging, improve the error for next time
def process_order(order_id):
    if not order_id:
        raise ValueError(
            f"order_id is required but got: {order_id!r}. "
            f"Check if the API request includes 'order_id' field."
        )

    order = get_order(order_id)
    if not order:
        raise OrderNotFoundError(
            f"Order {order_id} not found. "
            f"Possible causes: "
            f"1) Order was deleted, 2) Wrong environment, "
            f"3) order_id from different tenant. "
            f"Query: SELECT * FROM orders WHERE id = '{order_id}'"
        )

    if order.status == 'cancelled':
        raise InvalidOrderStateError(
            f"Cannot process cancelled order {order_id}. "
            f"Order was cancelled at {order.cancelled_at} by {order.cancelled_by}. "
            f"Use /api/orders/{order_id}/reinstate to restore if needed."
        )

Good error message components:

• What happened: Clear description of the failure
• Context: Relevant data values (IDs, states)
• Why it might have happened: Common causes
• What to do: Suggested next steps or fixes

Error message improvements to make:

When NOT to use this pattern:

• Error message is already clear and helpful
• Security-sensitive errors (don't leak implementation details)
• High-frequency errors where message generation is expensive

Reference: Why Programs Fail - Observing Facts

Conduct Blameless Postmortems

After significant bugs or incidents, conduct a blameless postmortem focused on systemic improvements, not individual blame. This transforms bugs from failures into learning opportunities.

Incorrect (blame-focused):

## Incident Review: Production Database Deleted

### What happened
John accidentally ran DROP TABLE in production.

### Who's responsible
John. He should have been more careful.

### Action items
- Talk to John about being more careful
- John loses production access

# Result: Team fears reporting issues
# No systemic improvements made
# Similar incident happens 3 months later with different person

Correct (blameless postmortem):

## Postmortem: Production Database Table Drop Incident

### Summary
On 2024-01-15, the users table was dropped in production,
causing 2 hours of downtime. Data was restored from backup.

### Timeline
- 14:23 - Engineer connects to database to run migration
- 14:25 - DROP TABLE executed against wrong database
- 14:26 - Errors reported by monitoring
- 14:30 - Incident declared, rollback initiated
- 16:30 - Service restored from backup

### Root Cause Analysis
1. Production and staging connection strings are similar
2. No prompt/confirmation when connecting to production
3. No safeguard against destructive commands in production
4. Local environment defaults to production credentials

### What Went Well
- Monitoring detected issue within 1 minute
- Backup was recent (< 1 hour data loss)
- Team responded quickly

### What Could Be Improved
- No visual distinction between prod/staging terminals
- Destructive commands don't require confirmation
- Easy to accidentally use production credentials

### Action Items
| Item | Owner | Due Date |
|------|-------|----------|
| Add color-coded terminal prompts for production | Team | Jan 22 |
| Implement SQL safeguards requiring confirmation | DevOps | Jan 29 |
| Rotate production credentials, require VPN | Security | Feb 5 |
| Update onboarding docs with safety procedures | Docs | Jan 25 |

Blameless postmortem principles:

1. Assume good intent - People tried to do the right thing 2. Focus on systems - What allowed error to happen? 3. Share openly - Publish findings to whole team 4. Follow up - Track action items to completion 5. Celebrate learning - Finding issues is valuable

When to conduct postmortems:

• Production incidents affecting users
• Bugs that took >1 day to resolve
• Issues that revealed systemic weaknesses
• Near-misses that could have been serious

Reference: Google SRE - Postmortem Culture

Document Symptoms Precisely

Record exactly what you observe, not what you think is happening. Precise symptom documentation prevents misdiagnosis and creates a reference you can verify against when testing hypotheses.

Incorrect (vague description):

Bug Report:
- Title: "App is slow"
- Description: "The app feels sluggish sometimes"
- Steps: "Just use the app normally"
- Expected: "Should be fast"
- Actual: "It's slow"

// Developer has no idea where to start
// "Slow" could mean: startup, rendering, API calls, animations...

Correct (precise symptoms):

Bug Report:
- Title: "2-3 second freeze when opening Settings after using search"
- Description: "UI becomes unresponsive for 2-3 seconds"
- Environment: macOS 14.2, App v2.1.0, 16GB RAM
- Steps to reproduce:
  1. Launch app (fresh start, not from background)
  2. Use search feature to find any item
  3. Click Settings icon in top right
  4. OBSERVE: UI freezes, spinner does not appear
  5. After 2-3 seconds, Settings panel opens
- Expected: Settings opens in <200ms
- Actual: 2-3 second freeze with no visual feedback
- Frequency: 100% reproducible with steps above
- Does NOT occur: if Settings opened before search, or on second open

// Clear starting point: something search does affects Settings load

Symptom documentation checklist:

• What exactly did you observe? (not interpret)
• When did it start? What changed?
• How often does it occur?
• What are the exact error messages (copy/paste, don't paraphrase)
• What works correctly in similar situations?

When NOT to use this pattern:

• Obvious crashes with clear stack traces
• Well-understood issues you've seen before

Reference: MIT 6.031 - Debugging

Create Minimal Reproduction Cases

Reduce the failing case to the smallest possible example that still exhibits the bug. Minimal reproductions eliminate noise, reveal the essential trigger, and often make the root cause immediately obvious.

Incorrect (debugging in full application context):

// Bug: "User profile doesn't update after edit"
// Debugging in the full app with 200+ components...

// App.jsx (2000 lines)
// ProfilePage.jsx (500 lines)
// ProfileForm.jsx (300 lines)
// useProfile.js (150 lines)
// api/profile.js (100 lines)
// store/userSlice.js (200 lines)

// Developer spends 3 hours stepping through all layers
// Still unclear if issue is in form, API, or state management

Correct (create minimal reproduction):

// Isolate the suspected component chain
// minimal-repro.jsx - 30 lines total

import { useState } from 'react';

function MinimalRepro() {
  const [profile, setProfile] = useState({ name: 'Alice' });

  const updateProfile = async (newName) => {
    // Simulate API call
    const response = await fetch('/api/profile', {
      method: 'PUT',
      body: JSON.stringify({ name: newName })
    });
    const data = await response.json();
    console.log('API returned:', data);  // Debug point 1
    setProfile(data);
    console.log('State after set:', profile);  // Debug point 2
    // BUG FOUND: Logging stale state due to closure!
  };

  return <button onClick={() => updateProfile('Bob')}>Update</button>;
}

// 10 minutes to find bug vs 3 hours in full app

Simplification techniques:

• Remove unrelated features one at a time
• Replace real APIs with hardcoded data
• Use a fresh project/file if possible
• Delete code until bug disappears, then add last deletion back

When NOT to use this pattern:

• Bug only occurs with specific data interactions across systems
• Performance issues requiring full load

Reference: Why Programs Fail - Simplifying Problems

Check Recent Changes First

Most bugs are introduced by recent changes. Before deep investigation, check what changed since the code last worked correctly. This dramatically reduces your search space.

Incorrect (ignoring change history):

# Bug report: "Login stopped working yesterday"

# Developer starts reading the entire auth codebase
# 50 files, 5000 lines of code to review
# 4 hours later, still searching...

Correct (check recent changes):

# Bug report: "Login stopped working yesterday"

# Step 1: When did it last work?
git log --oneline --since="3 days ago" -- src/auth/

# Output:
# a1b2c3d Add rate limiting to login endpoint
# e4f5g6h Update password validation regex
# i7j8k9l Refactor session handling

# Step 2: Check the suspicious commits
git show e4f5g6h  # Password validation change

# Found it! Regex now rejects valid passwords with special chars
# 10 minutes instead of 4 hours

Using git bisect for systematic search:

# When you know a good commit and bad commit
git bisect start
git bisect bad HEAD                    # Current version is broken
git bisect good v2.0.0                 # This version worked
# Git checks out middle commit
# Test and mark as good or bad
git bisect good  # or: git bisect bad
# Repeat until Git identifies the first bad commit

# Automate with a test script:
git bisect run npm test -- --grep "login"

Change investigation checklist:

• What was the last known working version/date?
• What commits/deploys happened since then?
• Who made changes to related code?
• Were there any config/environment changes?
• Did dependencies update?

When NOT to use this pattern:

• Bug has existed unnoticed for a long time
• Legacy code with unclear change history
• Issues caused by external factors (data, load, third-party services)

Reference: Git Bisect Documentation

Reproduce Before Debugging

Never start debugging until you can reliably reproduce the bug. Without reproduction, you cannot verify your fix works and may spend hours chasing phantom issues or misunderstanding the actual problem.

Incorrect (jumping straight to code):

# Bug report: "App crashes sometimes when saving"
# Developer immediately starts reading save code...

def save_data(data):
    # Check if this null check helps
    if data is None:  # Add defensive check
        return False
    # Check the file permission
    with open(self.path, 'w') as f:  # Add try/except?
        json.dump(data, f)
    # Hours later: still no idea what causes the crash

Correct (reproduce first):

# Bug report: "App crashes sometimes when saving"
# Step 1: Gather reproduction details

"""
Reproduction attempt log:
1. Fresh install, create new file, save -> OK
2. Open existing file, edit, save -> OK
3. Open large file (>10MB), edit, save -> CRASH!
4. Open large file, small edit, save -> CRASH!
5. Open large file, don't edit, save -> OK

Hypothesis: Crash occurs when saving edits to large files
Minimal reproduction: Open 15MB JSON file, change one character, save
"""

# NOW start debugging with a clear, reproducible case

Reproduction checklist:

• Document exact steps to trigger the bug
• Note environment details (OS, version, config)
• Identify the minimal reproduction case
• Confirm you can trigger it consistently (or note frequency if intermittent)

When NOT to use this pattern:

• Obvious typos or syntax errors visible in stack trace
• Build failures with clear error messages

Reference: Why Programs Fail - Reproducing Problems

Separate Symptoms from Causes

Clearly distinguish between what you observe (symptoms) and why it happens (causes). Fixing symptoms without addressing causes leads to whack-a-mole debugging where bugs keep reappearing in different forms.

Incorrect (treating symptom as cause):

# Symptom observed: NullPointerException on line 42

def process_user(user_id):
    user = get_user(user_id)
    # Fix: Add null check (treating symptom)
    if user is None:
        return None  # "Fixed" the crash
    return user.calculate_score()

# Problem: WHY was user None?
# - Invalid user_id passed?
# - Database connection failed?
# - Race condition in user creation?
# - Cache returned stale/deleted user?
# The null check hides the real problem

Correct (investigate cause before fixing):

# Symptom observed: NullPointerException on line 42

def process_user(user_id):
    user = get_user(user_id)
    # Investigation: WHY is user None?
    if user is None:
        # Diagnostic logging to find root cause
        logger.error(f"User not found: {user_id}")
        logger.error(f"Called from: {traceback.format_stack()}")
        logger.error(f"DB connection status: {db.is_connected()}")
        logger.error(f"Cache status: {cache.get_stats()}")
        raise ValueError(f"User {user_id} not found - see logs for context")
    return user.calculate_score()

# Investigation revealed: user_id came from stale session data
# Real fix: Invalidate session when user is deleted
# Symptom (null user) and cause (stale session) are different

Questions to separate symptoms from causes:

• What did I observe? (Symptom)
• What could cause this observation? (Hypothesis)
• Is fixing this observation enough, or will the problem manifest elsewhere?
• If I prevent this symptom, does the underlying issue still exist?

When NOT to use this pattern:

• Simple bugs where symptom location IS the cause location
• Time-critical production fixes (but schedule root cause investigation)

Reference: Root Cause Analysis Guide

State Expected vs Actual Behavior

Explicitly define what should happen before investigating why it doesn't. Without clear expected behavior, you may "fix" code to match your assumptions rather than the actual requirements.

Incorrect (vague problem statement):

// "The sort doesn't work right"

function sortUsers(users) {
  return users.sort((a, b) => a.name - b.name);  // Bug somewhere here?
}

// Developer "fixes" to:
return users.sort((a, b) => a.name.localeCompare(b.name));

// But wait - what was the EXPECTED behavior?
// - Sort by name alphabetically?
// - Sort by name case-insensitive?
// - Sort by last name, then first name?
// - Sort with numbers handled naturally (item2 before item10)?
// The fix might still be wrong!

Correct (explicit expected vs actual):

// Expected behavior (from requirements):
// - Sort users by display name
// - Case-insensitive (Alice = alice)
// - Handle accented characters (Émile after Emily)
// - Null names sort last

// Actual behavior:
// - Sorts incorrectly: ["bob", "Alice", "émile"] → ["Alice", "bob", "émile"]
// - Should be: ["Alice", "bob", "émile"] (case-insensitive)
// - Also: null names cause crash instead of sorting last

// Clear specification for the fix:
function sortUsers(users) {
  return users.sort((a, b) => {
    // Handle nulls: sort last
    if (!a.name) return 1;
    if (!b.name) return -1;
    // Case-insensitive, locale-aware comparison
    return a.name.localeCompare(b.name, undefined, { sensitivity: 'base' });
  });
}

// Can verify fix matches ALL expected behaviors

Template for stating expected vs actual:

**Expected:** [What should happen according to spec/requirements]
**Actual:** [What currently happens - be specific]
**Difference:** [The gap that needs to be fixed]
**Verification:** [How you'll confirm the fix works]

When NOT to use this pattern:

• Crashes and exceptions (expected = no crash)
• Performance issues (expected = meets performance target)

Reference: Cornell CS312 - Debugging

Examine System Boundaries

Most bugs occur at boundaries: between modules, services, systems, or data formats. When debugging, pay special attention to these interfaces where assumptions from one side may not match the other.

Incorrect (ignoring boundaries):

# Bug: User data sometimes corrupted after save

# Developer examines UserService internal logic exhaustively
class UserService:
    def save_user(self, user):
        validated = self.validate(user)  # Checks this...
        normalized = self.normalize(validated)  # And this...
        return self.repository.save(normalized)  # Glances at this

# 4 hours later: "Internal logic is perfect, don't understand"
# Never examined the boundary with repository

Correct (examine boundaries first):

# Bug: User data sometimes corrupted after save

class UserService:
    def save_user(self, user):
        validated = self.validate(user)
        normalized = self.normalize(validated)

        # BOUNDARY EXAMINATION: Service → Repository
        print("Sending to repository:", {
            "type": type(normalized).__name__,
            "data": normalized.__dict__
        })

        result = self.repository.save(normalized)

        # BOUNDARY EXAMINATION: Repository → Service
        print("Received from repository:", {
            "type": type(result).__name__,
            "data": result.__dict__ if result else None
        })

        return result

# Output reveals:
# Sending: {"name": "José García", "email": "jose@..."}
# Received: {"name": "Jos\u00e9 Garc\u00eda", "email": "jose@..."}
# BUG: Repository driver encoding issue at boundary!

Common boundary types to examine:

┌─────────────────────────────────────────────────────────────┐
│  Frontend ←──────────────────────────────────→ API         │
│            • JSON serialization                             │
│            • Date format conversion                         │
│            • Type coercion (string→number)                  │
├─────────────────────────────────────────────────────────────┤
│  Service ←───────────────────────────────────→ Database    │
│            • ORM mapping                                    │
│            • Encoding/character sets                        │
│            • Null handling                                  │
├─────────────────────────────────────────────────────────────┤
│  Your Code ←─────────────────────────────────→ Library     │
│            • Version differences                            │
│            • Optional vs required params                    │
│            • Error handling conventions                     │
├─────────────────────────────────────────────────────────────┤
│  System ←────────────────────────────────────→ File/Network│
│            • Line endings (CRLF vs LF)                      │
│            • Timeouts                                       │
│            • Path formats                                   │
└─────────────────────────────────────────────────────────────┘

When NOT to use this pattern:

• Bug is clearly in pure business logic
• Single-module applications with few boundaries

Reference: Cornell CS312 - Debugging

Trace Fault Propagation Chains

Bugs often manifest far from their origin. Trace the chain backward from where you see the symptom to where the fault was introduced. The infection (bad state) propagates until it causes a visible failure.

Incorrect (fixing where symptom appears):

# Symptom: NullPointerException in render_profile()

def render_profile(user):
    # Symptom appears here
    return f"Welcome, {user.name}"  # NPE: user is None

# Developer adds null check here:
def render_profile(user):
    if user is None:
        return "Welcome, Guest"
    return f"Welcome, {user.name}"

# But WHY was user None? Bug still exists upstream!

Correct (trace propagation backward):

# Symptom: NullPointerException in render_profile()

# Step 1: Where does user come from?
def handle_request(request):
    user = authenticate(request)  # Returns user or None
    return render_profile(user)

# Step 2: Why does authenticate return None?
def authenticate(request):
    token = request.headers.get('Authorization')
    if not token:
        return None  # No token → None user
    return verify_token(token)

# Step 3: Why is there no token?
# Traced to: Frontend forgot to include auth header after refresh

# Propagation chain:
# Missing header → authenticate returns None → render_profile crashes
#    (origin)            (propagation)              (symptom)

# REAL FIX: Frontend must include auth header
# DEFENSE: authenticate should raise AuthError, not return None

Propagation chain diagram:

DEFECT (Origin)           INFECTION (Propagation)        FAILURE (Symptom)
┌─────────────────┐      ┌─────────────────────┐      ┌────────────────┐
│ Missing header  │ ───► │ user = None         │ ───► │ NPE in render  │
│ in frontend     │      │ passed around       │      │ (visible crash)│
└─────────────────┘      └─────────────────────┘      └────────────────┘
     Fix HERE            Don't just mask here         Not the root cause

Tracing questions:

1. Where did this bad value come from? 2. What function/module passed it here? 3. Where was it supposed to be set correctly? 4. What condition caused it to be wrong/missing?

When NOT to use this pattern:

• Single-location bugs (cause and symptom are same place)
• Bugs from external systems (trace ends at boundary)

Reference: Why Programs Fail - Cause-Effect Chains

Use the 5 Whys Technique

Ask "why" repeatedly (typically 5 times) to drill past symptoms to root causes. Each answer becomes the subject of the next question until you reach a cause you can actually fix permanently.

Incorrect (stopping at first answer):

Problem: Users can't log in

Why? → The authentication service returns 500 errors
Fix: Restart the auth service

# Service restarted, works for a day, then fails again
# Never found out WHY it was returning 500s

Correct (5 Whys to root cause):

Problem: Users can't log in

Why 1: The authentication service returns 500 errors
Why 2: The auth service runs out of database connections
Why 3: Connections are not being released after use
Why 4: A try/finally block is missing in the auth code
Why 5: The developer copied code from a tutorial that didn't include cleanup

ROOT CAUSE: Missing connection cleanup in auth module
ACTUAL FIX: Add proper connection release in finally block
PREVENTION: Add code review checklist item for resource cleanup

5 Whys template:

## 5 Whys Analysis

**Problem Statement:** [Clear description of the bug]

**Why 1:** [First-level cause]
**Why 2:** [Why does Why 1 happen?]
**Why 3:** [Why does Why 2 happen?]
**Why 4:** [Why does Why 3 happen?]
**Why 5:** [Why does Why 4 happen?] ← Usually the root cause

**Root Cause:** [The fundamental issue to fix]
**Corrective Action:** [How to fix the root cause]
**Preventive Action:** [How to prevent similar issues]

Tips for effective 5 Whys:

• Don't accept "human error" as a cause - ask why the error was possible
• Multiple valid paths may exist - explore each branch
• Stop when you reach something actionable and preventable
• Include technical AND process causes

When NOT to use this pattern:

• Trivial bugs with obvious immediate causes
• Time-critical fixes (do 5 Whys afterward)
• When multiple independent factors combined

Reference: Root Cause Analysis Guide

Find the Last Known Good State

Identify when the code last worked correctly. Comparing working vs broken states reveals what changed and caused the bug. This is especially effective for regressions.

Incorrect (debugging without baseline):

# Bug: "Search feature doesn't work"
# Developer looks at current code trying to find bug
# No reference point for what "working" looks like
# Hours of reading code without knowing what changed

Correct (find last known good state):

# Bug: "Search feature doesn't work"

# Step 1: When did it last work?
# "It worked last Tuesday, stopped working after Wednesday deploy"

# Step 2: Find the last working commit
git log --oneline --since="last Tuesday" --until="Wednesday"
# abc123 (Wednesday) Add search filters
# def456 (Tuesday) Update search index
# ghi789 (Tuesday) Fix search pagination  ← Last known working

# Step 3: Compare working vs broken
git diff ghi789 abc123 -- src/search/

# Step 4: The diff shows exactly what changed
# Found: Search filters broke when no filters selected

Using git bisect for automated search:

# Automated binary search through commits
git bisect start
git bisect bad HEAD                  # Current is broken
git bisect good ghi789              # Tuesday commit worked

# Git checks out middle commit, test it:
npm test -- --grep "search"
# If tests pass:
git bisect good
# If tests fail:
git bisect bad

# Repeat until:
# abc123 is the first bad commit
# This commit introduced the bug

Finding last known good without git:

# Check different data states
working_data = load_from_backup("tuesday_backup.json")
broken_data = load_from_current()

# Compare
def compare_data(good, bad):
    for key in good:
        if good[key] != bad[key]:
            print(f"Difference in {key}:")
            print(f"  Good: {good[key]}")
            print(f"  Bad: {bad[key]}")

compare_data(working_data, broken_data)

When NOT to use this pattern:

• Bug existed since feature was created (no "good" state)
• Bug depends on external state that can't be reproduced

Reference: Git Bisect

Question Your Assumptions

Explicitly list and verify your assumptions about how the code works. Many bugs hide behind incorrect assumptions that we never think to question.

Incorrect (assuming code works as expected):

// Bug: Totals don't match between pages

function getOrderTotal(orderId) {
  const order = orders.get(orderId);
  return order.total;  // ASSUMPTION: order.total is always current

  // Developer assumes:
  // - orders.get() returns live data (but is it cached?)
  // - order.total is calculated (but is it stale?)
  // - total means the same thing everywhere (but is it pre-tax here?)
  // Spends hours looking at calculation logic when data is just stale
}

Correct (verify assumptions):

// Bug: Totals don't match between pages

function getOrderTotal(orderId) {
  const order = orders.get(orderId);

  // Verify assumptions explicitly:
  console.log("ASSUMPTION CHECK:", {
    // Is orders.get() returning fresh data?
    "order from cache?": orders.isFromCache(orderId),

    // Is order.total calculated or stored?
    "total property type": typeof order.total,
    "has calculated total?": typeof order.calculateTotal === 'function',

    // What exactly is in total?
    "total value": order.total,
    "includes tax?": order.totalIncludesTax,

    // When was this order last updated?
    "last updated": order.updatedAt,
    "current time": new Date()
  });

  return order.total;
}

// Output reveals: order is cached, total is 2 hours stale
// BUG: Cache not invalidated on order updates

Common assumptions to question:

When NOT to use this pattern:

• Well-tested code with strong guarantees
• When assumptions are verified by type system

Reference: A Systematic Approach to Debugging

Navigate the Call Stack

Use the call stack panel to move up and down the execution chain. Each frame shows local variables at that point, helping you understand how you reached the current state.

Incorrect (only looking at current frame):

# Debugger stopped in deep function
def calculate_tax(amount, rate):
    # Breakpoint here
    return amount * rate  # rate is 0.25, but should be 0.08?

# Developer wonders: "Where did 0.25 come from?"
# Only looks at this function, can't see caller context

Correct (navigate call stack):

# Call Stack shows:
#   calculate_tax        ← Current frame (bottom)
#   apply_pricing
#   process_line_item
#   process_order
#   handle_request       ← Entry point (top)

# Click on apply_pricing frame to see:
def apply_pricing(item, region):
    rate = get_tax_rate(region)  # rate = 0.25 for "EU" region
    return calculate_tax(item.price, rate)

# Click on process_line_item to see:
def process_line_item(item, order):
    region = order.shipping_region  # region = "EU" (wrong!)
    return apply_pricing(item, region)

# Found it! shipping_region should be billing_region for tax

Call stack navigation techniques:

┌─────────────────────────────────────────────┐
│ Call Stack                                  │
├─────────────────────────────────────────────┤
│ ► calculate_tax (current)     ← Click here  │
│   apply_pricing               ← Or here     │
│   process_line_item           ← Or here     │
│   process_order                             │
│   handle_request                            │
└─────────────────────────────────────────────┘

Clicking a frame shows:
- Local variables at that point
- Line where next function was called
- Parameter values passed to called function

Questions answered by call stack:

• How did execution reach this point?
• What parameters were passed at each level?
• What were local variables in parent functions?
• Where should I set breakpoints to catch this earlier?

When NOT to use this pattern:

• Recursion with 1000+ frames (hard to navigate)
• Async callbacks (stack may not show full context)

Reference: VS Code Call Stack