Systematic Debugging

Systematic Debugging

Overview

Debugging is investigation, not experimentation. This skill enforces a rigorous 4-phase process — root cause investigation, pattern analysis, hypothesis testing, and architecture questioning — that prevents shotgun debugging and ensures every fix is understood before it is applied.

Announce at start: "I'm using the systematic-debugging skill to investigate this issue."

---

Core Principle

┌─────────────────────────────────────────────────────────────────┐
│  HARD-GATE: NEVER GUESS. NEVER SHOTGUN DEBUG.                  │
│  NEVER CHANGE CODE WITHOUT UNDERSTANDING WHY IT IS BROKEN.     │
│                                                                 │
│  You are a detective gathering evidence, not a gambler trying   │
│  random fixes. If you are changing code without understanding   │
│  the root cause, STOP immediately.                             │
└─────────────────────────────────────────────────────────────────┘

---

Phase 1: Root Cause Investigation

Goal: Understand exactly WHAT is happening, not what you think is happening.

Actions

1. Read the error message carefully. The entire message. Every line. Including the stack trace. 2. Reproduce the bug. If you cannot reproduce it, you cannot fix it. Find the exact steps. 3. Gather evidence. Collect:

• Full error message and stack trace
• Input that triggers the bug
• Expected behavior vs actual behavior
• Environment details (versions, config, OS)

4. Check recent changes. What changed since this last worked?

• Recent commits (git log, git diff)
• Dependency updates
• Configuration changes
• Environment changes

Evidence Gathering Checklist

• [ ] Full error message captured (not truncated)
• [ ] Stack trace read from bottom to top
• [ ] Bug reproduced reliably with specific steps
• [ ] Expected vs actual behavior documented
• [ ] Recent changes reviewed (git log --oneline -20)
• [ ] Relevant logs examined

STOP — HARD-GATE: Do NOT proceed to Phase 2 until:

• [ ] You can reproduce the bug consistently
• [ ] You have the full error message and stack trace
• [ ] You know what changed recently
• [ ] You can describe the bug precisely (not vaguely)

---

Phase 2: Pattern Analysis

Goal: Narrow down WHERE the problem lives and WHEN it occurs.

Actions

1. Find working examples. Does this feature work in other contexts? With other inputs? In other environments? 2. Compare working vs broken. What is different between the case that works and the case that does not? 3. Check dependencies. Are all required services/libraries/configs present and correct? 4. Isolate the scope. Can you reproduce with a minimal example? Strip away everything non-essential.

Comparison Matrix

Fill this out to identify the pattern:

STOP — HARD-GATE: Do NOT proceed to Phase 3 until:

• [ ] You have identified at least one working case for comparison
• [ ] You have compared working vs broken and identified differences
• [ ] You have isolated the scope to the smallest reproducible case
• [ ] Dependencies have been verified (versions, availability, config)

---

Phase 3: Hypothesis and Testing

Goal: Form ONE specific, testable hypothesis and verify it with the smallest possible change.

Actions

1. Form ONE hypothesis. Based on evidence from Phases 1-2, what is the single most likely cause?

• State it explicitly: "The bug occurs because [specific cause]"
• If you cannot state it specifically, go back to Phase 1 or 2

2. Design a minimal test. What is the smallest change to confirm or deny this hypothesis?

• Prefer adding a test case over modifying production code
• Prefer logging/assertions over code changes
• Prefer reverting a change over writing new code

3. Apply the change and test.

• Make ONLY the change needed to test the hypothesis
• Run the test suite
• Observe the result

4. Evaluate.

• If CONFIRMED: proceed with the fix, write a regression test
• If DENIED: record what you learned, form a new hypothesis, return to step 1

Hypothesis Log Template

Hypothesis #1: [description]
Test: [what you did]
Result: CONFIRMED / DENIED
Learning: [what this taught you]

Hypothesis #2: ...

Decision Table: Hypothesis Testing Approach

STOP — HARD-GATE: Do NOT proceed to Phase 4 unless:

• [ ] You have tested at least 3 hypotheses and ALL were denied
• [ ] Each hypothesis was specific and testable
• [ ] Each test was minimal (one change at a time)
• [ ] You recorded learnings from each failed hypothesis

---

Phase 4: Architecture Questioning

Goal: If 3+ hypotheses have failed, the problem may be structural. Step back and question assumptions.

This phase is triggered ONLY after Phase 3 has been attempted at least 3 times without success.

Actions

1. Question your assumptions. What have you been assuming is true that might not be?

• Is the data shaped the way you think it is?
• Is the control flow what you expect?
• Are the types what you think they are?
• Is the API contract what you assumed?

2. Question the design. Is the current approach fundamentally flawed?

• Is there a race condition in the design?
• Is there a state management problem?
• Is there an incorrect abstraction?
• Are responsibilities misplaced?

3. Consider redesign. Sometimes the fix is not a patch but a restructuring.

• Can you simplify the design to eliminate the bug class entirely?
• Is there a pattern that handles this case better?
• Should you replace rather than fix?

4. Seek external input. If you are stuck:

• Explain the problem to someone else (rubber duck debugging)
• Search for known issues in dependencies
• Check if others have encountered similar problems

STOP — HARD-GATE: Do NOT continue without:

• [ ] Written list of assumptions that were questioned
• [ ] Explicit decision: patch the current design OR redesign
• [ ] If redesigning: a plan before implementing
• [ ] If patching: a new hypothesis informed by the assumption review

---

Debugging Decision Flowchart

Error encountered
    |
    v
Can you reproduce it?
    |
    +-- NO --> Gather more information (logs, user reports, monitoring)
    |          Try different inputs, environments, timing
    |          Do NOT proceed until reproducible
    |
    +-- YES -> Read the FULL error message and stack trace
               |
               v
         Is the cause obvious from the error?
               |
               +-- YES -> Form hypothesis, test it (Phase 3)
               |          Still write a regression test
               |
               +-- NO --> Complete Phase 1 evidence gathering
                          |
                          v
                    Find working case for comparison (Phase 2)
                          |
                          v
                    Identify differences
                          |
                          v
                    Form and test hypotheses (Phase 3)
                          |
                          +-- Fixed --> Write regression test, verify
                          |
                          +-- 3+ failed hypotheses --> Phase 4

---

Red Flags Table

---

Anti-Patterns / Common Mistakes

---

Integration Points

---

Quick Reference: What NOT To Do

1. Do NOT change random things and see if the bug goes away 2. Do NOT add try/catch to suppress the error 3. Do NOT rewrite the feature from scratch as a first resort 4. Do NOT blame the framework/library without evidence 5. Do NOT skip writing a regression test after fixing 6. Do NOT fix symptoms instead of root causes 7. Do NOT debug for more than 45 minutes without stepping back 8. Do NOT ignore error messages or stack traces

---

Skill Type

RIGID — The 4-phase process is mandatory and must be followed in order. Each phase has a HARD-GATE that must be satisfied before proceeding. Never change code without understanding why it is broken.

Defense in Depth

Reference document for the systematic-debugging skill. Describes layered validation patterns that prevent bugs from reaching production and limit blast radius when they do.

---

The 4-Layer Validation Pattern

Every data flow through your system should pass through four layers of validation. Each layer is independent and assumes the previous layers might have failed. No layer trusts input from any source, including other layers.

┌─────────────────────────────────────────┐
│  Layer 1: INPUT VALIDATION (Boundary)   │
│  Reject malformed data at the gate      │
├─────────────────────────────────────────┤
│  Layer 2: BUSINESS LOGIC (Domain)       │
│  Enforce domain rules and invariants    │
├─────────────────────────────────────────┤
│  Layer 3: DATA ACCESS (Persistence)     │
│  Protect data integrity at storage      │
├─────────────────────────────────────────┤
│  Layer 4: OUTPUT VALIDATION (Response)  │
│  Verify outgoing data is correct/safe   │
└─────────────────────────────────────────┘

---

Layer 1: Input Validation (Boundary)

Purpose: Reject obviously invalid data before it enters the system. This is the first line of defense.

Where: API endpoints, form handlers, CLI argument parsers, message consumers, file parsers.

What to validate:

• Type: Is the data the right type? (string, number, array, object)
• Presence: Are required fields present?
• Format: Does the data match the expected format? (email, UUID, date, URL)
• Range: Are numbers within acceptable bounds?
• Length: Are strings and arrays within size limits?
• Character set: Does text contain only allowed characters?
• Structure: Does the object have the expected shape?

Principles:

• Validate BEFORE any processing occurs
• Return clear, actionable error messages
• Log invalid input for monitoring (but redact sensitive data)
• Use allowlists over denylists (accept known-good, not reject known-bad)
• Parse, don't validate: convert raw input into typed domain objects

Example pattern:

def handle_request(raw_input):
    # Layer 1: Validate and parse
    validated = validate_and_parse(raw_input)  # Throws on invalid

    # Now 'validated' is a typed domain object, not raw data
    result = process(validated)
    return result

---

Layer 2: Business Logic Validation (Domain)

Purpose: Enforce domain rules, business constraints, and invariants that go beyond format validation.

Where: Domain models, service layer, use case handlers.

What to validate:

• Business rules: Can this user perform this action? Is this transition allowed?
• Invariants: Are domain object invariants maintained? (e.g., order total matches line items)
• State transitions: Is this state change valid? (e.g., can't ship an unpaid order)
• Cross-field validation: Do fields have consistent values? (e.g., end date after start date)
• Authorization: Does the user have permission for this specific resource?
• Idempotency: Has this operation already been performed?

Principles:

• Domain validation belongs in domain objects, not controllers or utilities
• Enforce invariants in constructors and mutation methods
• Make illegal states unrepresentable through types where possible
• Use the type system to prevent invalid combinations

Example pattern:

class Order:
    def ship(self):
        if self.status != "paid":
            raise DomainError("Cannot ship unpaid order")
        if not self.items:
            raise DomainError("Cannot ship empty order")
        if not self.shipping_address:
            raise DomainError("Cannot ship without address")

        self.status = "shipped"
        self.shipped_at = now()

---

Layer 3: Data Access Validation (Persistence)

Purpose: Protect data integrity at the storage layer. This is the last line of defense before data is persisted.

Where: Database schemas, ORM models, repository implementations.

What to validate:

• Constraints: NOT NULL, UNIQUE, FOREIGN KEY, CHECK constraints
• Types: Column types match expected data types
• Referential integrity: Foreign keys point to existing records
• Uniqueness: Business-unique fields are enforced at DB level
• Concurrency: Optimistic locking, version columns, serializable transactions
• Data size: Column length limits, row size limits

Principles:

• Database constraints are the safety net, not the primary validation
• Always use foreign keys - they catch bugs that application code misses
• Use database transactions for operations that must be atomic
• Implement optimistic locking for concurrent updates
• Schema migrations should be backward compatible

Example pattern:

CREATE TABLE orders (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id),
    status VARCHAR(20) NOT NULL CHECK (status IN ('draft', 'paid', 'shipped', 'delivered')),
    total_cents INTEGER NOT NULL CHECK (total_cents >= 0),
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    version INTEGER NOT NULL DEFAULT 1,
    CONSTRAINT order_must_have_items CHECK (
        status = 'draft' OR EXISTS (
            SELECT 1 FROM order_items WHERE order_id = orders.id
        )
    )
);

---

Layer 4: Output Validation (Response)

Purpose: Verify that outgoing data is correct, complete, and safe before sending it to the client or downstream system.

Where: Response serializers, API response builders, template renderers, event publishers.

What to validate:

• Completeness: Are all required response fields present?
• Sanitization: Is user-generated content escaped/sanitized?
• Sensitive data removal: Are internal fields, passwords, tokens stripped?
• Contract compliance: Does the response match the API schema?
• Encoding: Is the response properly encoded (UTF-8, JSON)?
• Size: Is the response within acceptable size limits?

Principles:

• Never expose internal error details to external clients
• Strip sensitive fields from responses (use allowlist serialization)
• Validate against API schema before sending
• Log the response (redacted) for debugging

Example pattern:

class UserSerializer:
    ALLOWED_FIELDS = ["id", "name", "email", "created_at"]

    def serialize(self, user):
        result = {}
        for field in self.ALLOWED_FIELDS:
            value = getattr(user, field)
            result[field] = self.sanitize(field, value)

        # Verify completeness
        assert all(f in result for f in self.ALLOWED_FIELDS)
        return result

---

Fail-Safe Defaults

When in doubt, the system should fail into a secure, safe state. Never fail into an open or permissive state.

Principles

Implementation Pattern

# BAD: Fail-open
def is_authorized(user, resource):
    try:
        return check_permissions(user, resource)
    except PermissionServiceError:
        return True  # "Let them through, service is down"

# GOOD: Fail-closed
def is_authorized(user, resource):
    try:
        return check_permissions(user, resource)
    except PermissionServiceError:
        log.error("Permission service unavailable, denying access")
        return False

---

Principle of Least Privilege

Every component should have exactly the permissions it needs and no more. This limits the blast radius when something goes wrong.

Application

Implementation Checklist

• [ ] Database connections use least-privilege credentials
• [ ] API keys are scoped to minimum required permissions
• [ ] File system access is limited to required directories
• [ ] Network access is restricted to known endpoints
• [ ] Secrets are accessible only to services that need them
• [ ] Admin interfaces require separate authentication
• [ ] Temporary elevated permissions expire automatically

---

Defense Against Cascading Failures

When one component fails, prevent the failure from propagating through the system.

Patterns

1. Circuit Breaker

Monitor calls to an external dependency. When failures exceed a threshold, stop calling and fail fast instead of waiting for timeouts.

States:
  CLOSED  → Normal operation, calls pass through
  OPEN    → Dependency is down, fail immediately (don't even try)
  HALF-OPEN → After cooldown, allow one test call to check recovery

Transitions:
  CLOSED → OPEN:     When failure count exceeds threshold (e.g., 5 failures in 60 seconds)
  OPEN → HALF-OPEN:  After cooldown period (e.g., 30 seconds)
  HALF-OPEN → CLOSED: If test call succeeds
  HALF-OPEN → OPEN:   If test call fails

2. Bulkhead

Isolate components so that a failure in one doesn't exhaust resources for others.

• Separate thread pools for different dependencies
• Separate connection pools for different databases
• Separate rate limits for different API consumers
• Separate deployment units for different services

3. Timeout + Retry with Backoff

Never wait indefinitely. Always set timeouts. Retry with exponential backoff.

Attempt 1: immediate
Attempt 2: wait 1 second
Attempt 3: wait 2 seconds
Attempt 4: wait 4 seconds
Attempt 5: give up, return error

Add jitter: multiply wait by random(0.5, 1.5) to prevent thundering herd

4. Graceful Degradation

When a non-critical dependency fails, continue with reduced functionality rather than total failure.

5. Health Checks and Readiness Probes

Components should report their health status so that the system can route around failures.

• Liveness: "Am I running?" (restart if no)
• Readiness: "Can I serve traffic?" (remove from load balancer if no)
• Dependency health: "Are my dependencies available?" (degrade if no)

---

Putting It All Together

Defense in depth is not about any single technique. It's about layering multiple defenses so that when one fails (and it will), the next layer catches the problem.

Request arrives
    │
    ▼
[Layer 1: Input Validation] → Reject malformed data
    │
    ▼
[Layer 2: Business Rules] → Enforce domain constraints
    │
    ▼
[Layer 3: Data Integrity] → Database constraints catch remaining issues
    │
    ▼
[Layer 4: Output Validation] → Strip sensitive data, verify response
    │
    ▼
[Circuit Breaker] → Protect against dependency failures
    │
    ▼
[Fail-Safe Defaults] → When uncertain, fail safely
    │
    ▼
Response sent

Each layer assumes the previous layers might have failed. Each layer validates independently. No layer trusts its input, regardless of source.

Root Cause Tracing

Reference document for the systematic-debugging skill. These are structured techniques for tracing errors back to their true origin.

---

1. Backward Tracing (Error → Call Stack → Root)

The most fundamental debugging technique. Start from the symptom and trace backward through the execution path to find the origin.

The Process

SYMPTOM (what you see)
    ↑
PROXIMATE CAUSE (what directly triggered it)
    ↑
INTERMEDIATE CAUSE (what led to that)
    ↑
ROOT CAUSE (the actual defect)

Steps

1. Start at the error. Read the error message and note the exact line and file. 2. Read the stack trace bottom-to-top. The bottom is the origin, the top is where it manifested. Identify each frame:

• Which function was called?
• What arguments were passed?
• What was the expected state at that point?

3. Trace data flow backward. For each variable in the error:

• Where was it assigned its current value?
• Where did that value come from?
• At which point did the value diverge from expectations?

4. Find the divergence point. The root cause is the FIRST place where reality diverged from expectations.

Example

Error: TypeError: Cannot read property 'name' of undefined
  at formatUser (format.js:15)
  at processUsers (process.js:42)
  at handleRequest (handler.js:8)

Backward trace:
  formatUser(user) → user is undefined
  ↑ processUsers passes users[i] → users[i] is undefined
  ↑ users array has fewer items than expected
  ↑ handleRequest fetches users → API returned partial data
  ↑ ROOT CAUSE: API pagination not handled, only first page fetched

Key Principle

The error message tells you WHERE the problem manifested. The root cause is almost never at that location. Trace backward until you find the first place where something went wrong.

---

2. Binary Search Debugging (Bisect the Problem Space)

When you have a large codebase or a long history and don't know where the bug was introduced, use binary search to find it efficiently.

Git Bisect

When you know the bug was introduced between two commits:

git bisect start
git bisect bad                    # Current commit has the bug
git bisect good <known-good-sha>  # This commit was working

# Git checks out the middle commit
# Test it:
# - If bug exists: git bisect bad
# - If bug absent: git bisect good
# Repeat until git identifies the exact commit

git bisect reset  # When done

Automated bisect: If you have a test that exposes the bug:

git bisect start HEAD <known-good-sha>
git bisect run ./test-script.sh

Code Bisect (Without Git)

When the bug is in current code but you don't know which part:

1. Identify the full code path from entry point to error 2. Add a checkpoint at the midpoint (log statement or assertion) 3. Run the test:

• If midpoint state is correct → bug is in the second half
• If midpoint state is wrong → bug is in the first half

4. Repeat with the guilty half until you find the exact line

Data Bisect

When the bug is triggered by specific data:

1. Take the failing input dataset 2. Split it in half 3. Test each half independently 4. The half that fails contains the trigger 5. Repeat until you find the exact data element causing the issue

Efficiency

Binary search debugging finds the problem in O(log n) steps, where n is the number of commits, lines, or data elements. For 1000 commits, that's about 10 steps.

---

3. Diff-Based Debugging (What Changed?)

Most bugs are caused by recent changes. Systematically examine what changed to find the culprit.

What to Diff

The Diff Investigation Process

1. When did it last work? Find the last known-good state. 2. What changed between then and now?

   git log --oneline <last-good-sha>..HEAD
   git diff <last-good-sha>..HEAD --stat

3. Categorize changes by risk:

• High risk: Logic changes, dependency updates, config changes
• Medium risk: Refactoring, new features in adjacent code
• Low risk: Documentation, test additions, formatting

4. Investigate high-risk changes first. For each:

• Could this change cause the observed symptom?
• Does reverting this change fix the bug?

Revert Test

The fastest way to confirm a change caused a bug:

# Create a branch to test the revert
git checkout -b test-revert
git revert <suspect-commit> --no-commit
# Run tests
# If bug is gone → that commit is the cause
# If bug persists → that commit is innocent
git checkout - && git branch -D test-revert

---

4. Rubber Duck Methodology

When you're stuck, explain the problem out loud (or in writing) to an imaginary listener. The act of articulating forces you to organize your thoughts and often reveals gaps in your understanding.

The Process

1. State the problem clearly. "The system should do X, but instead it does Y." 2. Explain what you've already tried. Walk through each investigation step. 3. Explain your current understanding. Describe the code flow as you understand it. 4. Identify gaps. "The part I don't understand is..." 5. Question assumptions. "I'm assuming that... but what if...?"

Why It Works

• Forces you to be precise instead of vague
• Exposes assumptions you didn't realize you were making
• Converts fuzzy intuition into concrete statements that can be verified
• Breaks tunnel vision by requiring a linear explanation

Structured Rubber Duck Template

PROBLEM: [What should happen vs what actually happens]

EVIDENCE GATHERED:
- Error message: [exact text]
- Reproduction steps: [exact steps]
- Working case: [what works for comparison]

CODE FLOW (as I understand it):
1. [Entry point]
2. [Step 2]
3. [Step 3 — this is where I think it breaks]
4. [Expected step 4 vs what actually happens]

HYPOTHESES TESTED:
1. [Hypothesis] → [Result]
2. [Hypothesis] → [Result]

WHAT I DON'T UNDERSTAND:
- [Gap 1]
- [Gap 2]

ASSUMPTIONS I'M MAKING:
- [Assumption 1 — verified? yes/no]
- [Assumption 2 — verified? yes/no]

---

5. Log-Based Investigation Patterns

When the bug is in production, intermittent, or involves complex interactions, logs are your primary evidence source.

Strategic Log Placement

Add logs at these locations to trace execution flow:

Entry points      → Log inputs and request context
Decision points   → Log which branch was taken and why
External calls    → Log request and response (or error)
State mutations   → Log before and after values
Exit points       → Log outputs and return values
Error handlers    → Log full error with context

Log Levels for Debugging

Correlation Patterns

When debugging distributed systems or async operations:

1. Request ID: Assign a unique ID at the entry point, include it in every log line for that request 2. Timestamp precision: Use millisecond or microsecond timestamps to establish ordering 3. Context propagation: Pass context (request ID, user ID, operation name) through the call chain

Log Analysis Techniques

Timeline reconstruction:

# Extract all logs for a specific request
grep "request-id-abc123" application.log | sort -k1

# Find the last successful operation before failure
grep "request-id-abc123" application.log | grep -B5 "ERROR"

Pattern detection:

# Find common patterns in errors
grep "ERROR" application.log | sort | uniq -c | sort -rn | head -20

# Check error frequency over time
grep "ERROR" application.log | cut -d' ' -f1-2 | uniq -c

State reconstruction:

• Follow the sequence of state changes for the affected entity
• Compare with a successful entity's log sequence
• The first divergence point is likely the root cause

Anti-Patterns in Logging