Ce Debug

Debug and Fix

Find root causes, then fix them. This skill investigates bugs systematically — tracing the full causal chain before proposing a fix — and optionally implements the fix with test-first discipline.

<bug_description> #$ARGUMENTS </bug_description>

Core Principles

1. Investigate before fixing. Do not propose a fix until you can explain the full causal chain from trigger to symptom with no gaps. "Somehow X leads to Y" is a gap. 2. Predictions for uncertain links. When the causal chain has uncertain or non-obvious links, form a prediction — something in a different code path or scenario that must also be true. If the prediction is wrong but a fix "works," you found a symptom, not the cause. When the chain is obvious (missing import, clear null reference), the chain explanation itself is sufficient. 3. One change at a time. Test one hypothesis, change one thing. If you're changing multiple things to "see if it helps," stop — that is shotgun debugging. 4. When stuck, diagnose why — don't just try harder.

Execution Flow

Beyond the trivial-bug fast-path in Phase 0, no further phase skipping — complex bugs simply spend more time in each phase naturally. No further complexity tiers.

---

Phase 0: Triage

Parse the input and reach a clear problem statement.

If the input references an issue tracker, fetch it:

• GitHub (#123, org/repo#123, github.com URL): Parse the issue reference from <bug_description> and fetch with gh issue view <number> --json title,body,comments,labels. For URLs, pass the URL directly to gh.
• Other trackers (Linear URL/ID, Jira URL/key, any tracker URL): Attempt to fetch using available MCP tools or by fetching the URL content. If the fetch fails — auth, missing tool, non-public page — ask the user to paste the relevant issue content. Ensure the fetch includes the full comment thread, not just the opening description.

Read the full conversation — the original description AND every comment, with particular attention to the latest ones. Comments frequently contain updated reproduction steps, narrowed scope, prior failed attempts, additional stack traces, or a pivot to a different suspected root cause; treating the opening post as the whole picture often sends the investigation in the wrong direction. Extract reported symptoms, expected behavior, reproduction steps, and environment details from the combined thread. Then proceed to Phase 1.

Everything else (stack traces, test paths, error messages, descriptions of broken behavior): the problem statement is the input itself.

Trivial-bug fast-path: Once the problem is clear, decide whether the framework is needed at all. If the cause is immediately readable from the input (single-file typo, missing import, obvious null deref or off-by-one with a one-line fix) and verification doesn't require deep tracing, present the cause and the proposed one-line fix and run Phase 2's Fix it now / Diagnosis only user-choice gate before editing — the fast-path saves investigation ceremony, not the user's choice over whether to apply a fix. If the user picks fix, run Phase 3's Workspace and branch check (uncommitted-work confirmation and default-branch branch-creation prompt), apply the fix, leave a one-line note explaining the cause, and skip to Phase 4's structured summary. If diagnosis only, write the summary and stop. When in doubt, run the full framework; getting the wrong root cause costs more than the few minutes of ceremony.

Otherwise, proceed to Phase 1.

Questions:

• Do not ask questions by default — investigate first (read code, run tests, trace errors)
• Only ask when a genuine ambiguity blocks investigation and cannot be resolved by reading code or running tests
• When asking, ask one specific question

Prior-attempt awareness: If the user indicates prior failed attempts ("I've been trying", "keeps failing", "stuck"), ask what they have already tried before investigating. This avoids repeating failed approaches and is one of the few cases where asking first is the right call.

---

Phase 1: Investigate

1.1 Reproduce the bug

Confirm the bug exists and understand its behavior. Run the test, trigger the error, follow reported reproduction steps — whatever matches the input.

• Browser bugs: Prefer agent-browser if installed. Otherwise use whatever works — MCP browser tools, direct URL testing, screenshot capture, etc.
• Manual setup required: If reproduction needs specific conditions the agent cannot create alone (data states, user roles, external services, environment config), document the exact setup steps and guide the user through them. Clear step-by-step instructions save significant time even when the process is fully manual.
• Does not reproduce after 2-3 attempts: Read references/investigation-techniques.md for intermittent-bug techniques.
• Cannot reproduce at all in this environment: Document what was tried and what conditions appear to be missing.
• Writing the reproduction test: If the project has testing-conventions guidance — a dedicated testing skill, an AGENTS.md/CLAUDE.md testing section, or a clear style across existing tests — apply it when authoring the failing test. Otherwise write a minimal isolated test that fails on the current bug and passes once the corrected behavior lands; name it descriptively so the failure message itself explains the bug.

1.2 Verify environment sanity

Before deep code tracing, confirm the environment is what you think it is:

• Correct branch checked out; no unintended uncommitted changes
• Dependencies installed and up to date (bun install, npm install, bundle install, etc.) — stale node_modules/vendor is a frequent false lead
• Expected interpreter or runtime version (check .tool-versions, .nvmrc, Gemfile, etc. against what's actually active)
• Required env vars present and non-empty
• No stale build artifacts (dist/, .next/, compiled binaries from an earlier branch)
• Dependent local services (database, cache, queue) running at expected versions when the bug plausibly involves them

1.3 Trace the code path

Trace data flow backward from the symptom to where valid state first became invalid. Read code-shape to form a hypothesis, then verify with observed values — do not theorize from code alone.

Concrete recipe:

1. Read the stack trace bottom-to-top, opening each frame's source. The bottom frame is the symptom; the root cause is somewhere upstream. 2. Identify the first frame where the input data is already invalid — that's the upper bound on where to look. 3. Instrument the boundaries around that frame: targeted log/print statements, debugger breakpoints, or test assertions that capture actual values at function entry/exit. Assumed values lie; observed values don't. 4. Walk the boundaries until valid input becomes invalid output. That transition is the root cause site.

Do not stop at the first function that looks wrong — the root cause is where bad state originates, not where it is first observed.

As you trace:

• Check recent changes in files you are reading: git log --oneline -10 -- [file]
• If the bug looks like a regression ("it worked before"), use git bisect (see references/investigation-techniques.md)
• Check the project's observability tools for additional evidence:
• Error trackers (Sentry, AppSignal, Datadog, BetterStack, Bugsnag)
• Application logs
• Browser console output
• Database state
• Each project has different systems available; use whatever gives a more complete picture

---

Phase 2: Root Cause

Reminder: investigate before fixing. Do not propose a fix until you can explain the full causal chain from trigger to symptom with no gaps.

Read references/anti-patterns.md before forming hypotheses. As a load-time preview of the rationalizations it covers, stop and re-examine if the internal monologue contains any of these:

• "Quick fix for now, investigate later"
• "This should work" (without a tested prediction)
• "Let me just try..." (without a hypothesis)

These phrases mark mode-drift toward symptom patches, not progress on the root cause. ("One more attempt" after a failed fix and "works on my machine" are covered at the points they fire — Phase 3's invalidation step and the Smart Escalation table below.)

Assumption audit (before hypothesis formation): List the concrete "this must be true" beliefs your understanding depends on — the framework behaves as expected here, this function returns what its name implies, the config loads before this runs, the caller passes a non-null value, the database is in the state the test implies. For each, mark verified (you read the code, checked state, or ran it) or assumed. Assumptions are the most common source of stuck debugging. Many "wrong hypotheses" are actually correct hypotheses tested against a wrong assumption.

Form hypotheses ranked by likelihood. For each, state:

• What is wrong and where (file:line)
• At least one concrete observation that supports it — a runtime variable value, a log line, an instrumented boundary capture, a behavior delta against a working comparison case, or a specific code reference. "X seems off" is not evidence; "X equals null at line 42 because Y was never initialized in the constructor path that runs under condition Z" is. Hypotheses without grounding observations are theorizing — go back to Phase 1 and instrument.
• The causal chain: how the trigger leads to the observed symptom, step by step
• For uncertain links in the chain: a prediction — something in a different code path or scenario that must also be true if this link is correct

When the causal chain is obvious and has no uncertain links (missing import, clear type error, explicit null dereference), the chain explanation itself is the gate — no prediction required. Predictions are a tool for testing uncertain links, not a ritual for every hypothesis.

Before forming a new hypothesis, review what has already been ruled out and why.

Causal chain gate: Do not proceed to Phase 3 until you can explain the full causal chain — from the original trigger through every step to the observed symptom — with no gaps. The user can explicitly authorize proceeding with the best-available hypothesis if investigation is stuck.

Reminder: if a prediction was wrong but the fix appears to work, you found a symptom. The real cause is still active.

Present findings

Once the root cause is confirmed, present:

• The root cause (causal chain summary with file:line references)
• The proposed fix and which files would change
• Which tests to add or modify to prevent recurrence (specific test file, test case description, what the assertion should verify)
• Whether existing tests should have caught this and why they did not

Then offer next steps.

Use the platform's blocking question tool (AskUserQuestion in Claude Code, request_user_input in Codex, ask_question in Antigravity CLI (agy), ask_user in Pi (requires the pi-ask-user extension)). In Claude Code, call ToolSearch with select:AskUserQuestion first if its schema isn't loaded — a pending schema load is not a reason to fall back. Fall back to numbered options in chat only when no blocking tool exists in the harness or the call errors (e.g., Codex edit modes). Never silently skip the question.

Options to offer:

1. Fix it now — proceed to Phase 3 2. Diagnosis only — I'll take it from here — skip the fix, proceed to Phase 4's summary, and end the skill 3. Rethink the design (/ce-brainstorm) — only when the root cause reveals a design problem (see below)

Do not assume the user wants action right now. The test recommendations are part of the diagnosis regardless of which path is chosen.

When to suggest brainstorm: Only when investigation reveals the bug cannot be properly fixed within the current design — the design itself needs to change. Concrete signals observable during debugging:

• The root cause is a wrong responsibility or interface, not wrong logic. The module should not be doing this at all, or the boundary between components is in the wrong place. (Observable: the fix requires moving responsibility between modules, not correcting code within one.)
• The requirements are wrong or incomplete. The system behaves as designed, but the design does not match what users actually need. The "bug" is really a product gap. (Observable: the code is doing exactly what it was written to do — the spec is the problem.)
• Every fix is a workaround. You can patch the symptom, but cannot articulate a clean fix because the surrounding code was built on an assumption that no longer holds. (Observable: you keep wanting to add special cases or flags rather than a direct correction.)

Do not suggest brainstorm for bugs that are large but have a clear fix — size alone does not make something a design problem.

Smart escalation

If 2-3 hypotheses are exhausted without confirmation, diagnose why:

Parallel investigation option: When hypotheses are evidence-bottlenecked across clearly independent subsystems, dispatch read-only sub-agents in parallel, each with an explicit hypothesis and structured evidence-return format. No code edits by sub-agents, and skip this when hypotheses depend on each other's outcomes. If the platform does not support parallel sub-agent dispatch, run the same hypothesis probes sequentially in ranked-likelihood order instead — the parallelism is a latency optimization, not a correctness requirement.

Present the diagnosis to the user before proceeding.

---

Phase 3: Fix

Reminder: one change at a time. If you are changing multiple things, stop.

If the user chose "Diagnosis only" at the end of Phase 2, skip this phase and go straight to Phase 4 for the summary — the skill's job was the diagnosis. If they chose "Rethink the design", control has transferred to /ce-brainstorm and this skill ends.

Workspace and branch check: Before editing files:

• Check for uncommitted changes (git status). If the user has unstaged work in files that need modification, confirm before editing — do not overwrite in-progress changes.
• If the current branch is the default branch, ask whether to create a feature branch first using the platform's blocking question tool (see Phase 2 for the per-platform names). To detect the default branch, compare against main, master, or the value of git rev-parse --abbrev-ref origin/HEAD with its origin/ prefix stripped (the raw output is origin/<name>, so an unstripped comparison will never match the local branch name). Default to creating one; derive a name from the bug and run git checkout -b <name>. On any other branch, proceed.

Test-first: 1. Write a failing test that captures the bug (or use the existing failing test) 2. Verify it fails for the right reason — the root cause, not unrelated setup 3. Implement the minimal fix — address the root cause and nothing else. Do not bundle drive-by refactors, formatting, or unrelated cleanup into a bug-fix change; those belong in separate commits. 4. Verify the test passes 5. Run the broader test suite for regressions 6. Self-review the diff before declaring the fix done: read every changed line and check for style violations, missed edge cases, regressions in adjacent behavior, and missing test coverage for the fix. For non-trivial fixes (multiple files, risky surface area), also run the harness's lightweight review tool (e.g., /review in Claude Code; the equivalent in other harnesses) — not the full ce-code-review multi-agent flow, which is PR-tier and over-sized for a single bug fix.

On a failed fix: return to Phase 2 and explicitly invalidate the current hypothesis before forming a new one. State out loud what evidence ruled out the prior hypothesis, then form a new one with its own grounding observation and prediction. Do not retry variants of the same theory ("maybe it was the other branch", "let me also catch this case") — that is the rationalization spiral, not iteration.

3 failed fix attempts = smart escalation. Diagnose using the same table from Phase 2. If fixes keep failing, the root cause identification was likely wrong. Return to Phase 2.

Conditional defense-in-depth (trigger: grep for the root-cause pattern found it in 3+ other files, OR the bug would have been catastrophic if it reached production): Read references/defense-in-depth.md for the four-layer model (entry validation, invariant check, environment guard, diagnostic breadcrumb) and choose which layers apply. Skip when the root cause is a one-off error with no realistic recurrence path.

Conditional post-mortem (trigger: the bug was in production, OR the pattern appears in 3+ locations): Analyze how this was introduced and what allowed it to survive. Note any systemic gap or repeated pattern found — it informs Phase 4's decision on whether to offer learning capture.

---

Phase 4: Handoff

Structured summary — always write this first:

## Debug Summary
**Problem**: [What was broken]
**Root Cause**: [Full causal chain, with file:line references]
**Recommended Tests**: [Tests to add/modify to prevent recurrence, with specific file and assertion guidance]
**Fix**: [What was changed — or "diagnosis only" if Phase 3 was skipped]
**Prevention**: [Test coverage added; defense-in-depth if applicable]
**Confidence**: [High/Medium/Low]

If Phase 3 was skipped (user chose "Diagnosis only" in Phase 2), stop after the summary — the user already told you they were taking it from here. Do not prompt.

If Phase 3 ran, the next move depends on whether the skill created the branch in Phase 3.

Skill-owned branch (created in Phase 3): default to commit-and-PR without prompting

1. Check for contextual overrides first. Look at the user's original prompt, loaded memories, and the project's active instructions already in your context for preferences that conflict with auto commit-and-PR — for example, "always review before pushing", "open PRs as drafts", or "don't open PRs from skills". A signal must be an explicit instruction or a clearly applicable rule, not a vague tonal cue. If any apply, honor them — switch to the pre-existing-branch menu below, or skip the PR step entirely, whichever matches the user's stated preference. 2. Briefly preview what will happen — what will be committed, on what branch, and that a PR will be opened — then proceed without waiting for confirmation. The preview exists so the user can interrupt; it is not a blocking question. Format and length are your call; keep it scannable. 3. Run `/ce-commit-push-pr`. When the entry came from an issue tracker, include the appropriate auto-close syntax for that tracker in the location it requires — most trackers parse PR descriptions (e.g., Fixes #N for GitHub, Closes ABC-123 for Linear), but some only parse commit messages (e.g., Jira Smart Commits) — so the diagnosis and fix flow back to the issue and it closes on merge. Surface the resulting PR URL.

Pre-existing branch (skill did not create it): ask the user

Use the platform's blocking question tool (AskUserQuestion in Claude Code, request_user_input in Codex, ask_question in Antigravity CLI (agy), ask_user in Pi (requires the pi-ask-user extension)). In Claude Code, call ToolSearch with select:AskUserQuestion first if its schema isn't loaded — a pending schema load is not a reason to fall back. Fall back to numbered options in chat only when no blocking tool exists in the harness or the call errors. Never end the phase without collecting a response.

Options:

1. Commit and open a PR (`/ce-commit-push-pr`) — default for most cases 2. Commit the fix (`/ce-commit`) — local commit only 3. Stop here — user takes it from there

After a PR is open (either path): consider offering learning capture

Most bugs are localized mechanical fixes (typo, missed null check, missing import) where the only "lesson" is the bug itself. Compounding those clutters docs/solutions/ without adding value. Decide which path applies:

• Skip silently when the fix is mechanical and there's no generalizable insight. Default to this when in doubt.
• Offer neutrally when the lesson can be stated in one sentence — e.g., "X.foo() returns T | undefined when Y, not just T", or "the diagnostic path was non-obvious and worth recording." If you cannot articulate the lesson, skip rather than offer.
• Lean into the offer when the pattern appears in 3+ locations OR the root cause reveals a wrong assumption about a shared dependency, framework, or convention that other code is likely to repeat.

When offering, use the blocking question tool described above. If the user accepts, run /ce-compound, then commit the resulting learning doc to the same branch and push so the open PR picks up the new commit.

Debugging Anti-Patterns

Read this before forming hypotheses. These patterns describe the most common ways debugging goes wrong. They feel productive in the moment — that is what makes them dangerous.

---

Prediction Quality

The prediction requirement exists to prevent symptom-fixing. A prediction tests whether your understanding of the bug is correct, not just whether a fix makes the error go away.

Bad prediction (restates the hypothesis):

Hypothesis: The null pointer is because user is not initialized.

Prediction: user will be null when I log it.

This just re-describes the symptom. It cannot be wrong if the hypothesis is right — so it cannot catch a wrong hypothesis.

Good prediction (tests something non-obvious):

Hypothesis: The null pointer is because the auth middleware skips initialization on cached requests.

Prediction: Non-cached requests to the same endpoint will NOT produce the null pointer, and the X-Cache header will be present on failing requests.

This tests a different code path and a different observable. If the prediction is wrong — cached and non-cached requests both fail — the hypothesis is wrong even if "initializing user earlier" happens to fix the immediate error.

Rule of thumb: A good prediction names something you have not looked at yet. If confirming the prediction requires only looking at the same line of code you already identified, the prediction is not adding information.

---

Shotgun Debugging

Changing multiple things at once to "see if it helps."

How it feels: Productive. You're making changes, running tests, making progress.

What actually happens: If the bug goes away, you do not know which change fixed it. If it persists, you do not know which changes are relevant. You have introduced variables instead of eliminating them.

The fix: One hypothesis, one change, one test. If the first change does not fix it, revert it before trying the next. Changes should be additive to understanding, not cumulative to the codebase.

---

Confirmation Bias

Interpreting ambiguous evidence as supporting your current hypothesis.

How it looks:

• A log line that could support your theory — you treat it as proof
• A test passes after your change — you declare the bug fixed without checking if the test was actually exercising the failure path
• The error message changes slightly — you interpret the change as "getting closer" instead of recognizing a different failure mode

The defense: Before declaring a hypothesis confirmed, ask: "What evidence would DISPROVE this hypothesis?" If you cannot name something that would change your mind, you are not testing — you are justifying.

---

"It Works Now, Move On"

The bug stops appearing after a change. The temptation is to declare victory and move on.

When this is a trap: If you cannot explain WHY the change fixed the bug — the full causal chain from your change through the system to the symptom — you may have:

• Fixed a symptom while the root cause remains
• Introduced a change that masks the bug without resolving it
• Gotten lucky with timing (especially for intermittent bugs)

The test: Can you explain the fix to someone else without using the words "somehow" or "I think"? If not, the root cause is not confirmed.

---

Thoughts That Signal You Are About to Shortcut

These feel like reasonable next steps. They are warning signs that investigation is being skipped.

Proposing a fix before explaining the cause. If the words "I think we should change..." come before "the root cause is...", pause. The fix might be right, but without a confirmed causal chain there is no way to know. Explain the cause first.

Reaching for another attempt without new information. After 2-3 failed hypotheses, trying a 4th without learning something new from the failures is not debugging — it is guessing with increasing frustration. Stop and diagnose why previous hypotheses failed (see smart escalation).

Certainty without evidence. The feeling of "I know what this is" before reading the relevant code. Experienced developers have strong pattern-matching instincts, and they are right often enough to be dangerous when wrong. Read the code even when you are confident.

Minimizing the scope. "It is probably just..." — the word "just" signals an assumption that the problem is small. Small problems do not resist 2-3 fix attempts. If you are still debugging, it is not "just" anything.

Treating environmental differences as irrelevant. When something works in one environment and fails in another, the difference between environments IS the investigation. Do not dismiss it — compare them systematically.

---

Smart Escalation Patterns

When 2-3 hypotheses have been tested and none confirmed, the problem is not "I need hypothesis #4." The problem is usually one of these:

Different subsystems keep appearing. Hypothesis 1 pointed to auth, hypothesis 2 to the database, hypothesis 3 to caching. This scatter pattern means the bug is not in any one subsystem — it is in the interaction between them, or in an architectural assumption that cuts across all of them. This is a design problem, not a localized bug.

Evidence contradicts itself. The logs say X happened, but the code makes X impossible. The test fails with error A, but the code path that produces error A is unreachable from the test. When evidence contradicts, the mental model is wrong. Step back. Re-read the code from the entry point without any assumptions about what it does.

Works locally, fails elsewhere. The most common causes: environment variables, dependency versions, file system differences (case sensitivity, path separators), timing differences (faster/slower machines), and data differences (test fixtures vs production data). Systematically compare the two environments rather than debugging the code.

Fix works but prediction was wrong. This is the most dangerous pattern. The bug appears fixed, but the causal chain you identified was incorrect. The real cause is still present and will resurface. Keep investigating — you found a coincidental fix, not the root cause.

Defense-in-Depth

When a bug is caused by invalid state reaching a vulnerable code path, fixing just one layer leaves the door open for different code paths, refactors, or mocks to re-introduce the same bug. Defense-in-depth makes the bug structurally harder to re-create by validating at multiple layers.

Not every bug warrants this. Use when:

• The root-cause pattern exists in 3+ other files (grep the fix signature)
• The bug would have been catastrophic in production
• The vulnerable operation is dangerous regardless of caller (destructive side effects, security-sensitive, irreversible)

Skip when the root cause is a one-off logic error with no realistic recurrence path.

The four layers

Pick the layers that apply. Not every bug needs all four.

Applying the pattern

1. Trace the data flow from the bad value's origin through every function that passed it along. 2. Map the checkpoints: at which of those points could validation have rejected the bad value earlier? 3. Add guards at the appropriate layers. Each guard should be as narrow as possible — validating exactly what this layer is responsible for, not duplicating checks from other layers. 4. Test each guard independently: construct a case that bypasses layer 1 and verify layer 2 still catches it.

Common mistakes

• Duplicating the same check at every layer. Each layer should catch a distinct class of failure. If layer 2 just repeats layer 1, the second one is noise.
• Adding guards speculatively without a bug to justify them. Defense-in-depth is a response to an observed failure mode, not a generic code-hygiene practice.
• Leaving layer 4 (diagnostic breadcrumb) out. When layers 1-3 still get bypassed — they will, eventually — the breadcrumb is what makes the next bug debuggable.

Investigation Techniques

Techniques for deeper investigation when standard code tracing is not enough. Load this when a bug does not reproduce reliably, involves timing or concurrency, or requires framework-specific tracing.

---

Root-Cause Tracing

When a bug manifests deep in the call stack, the instinct is to fix where the error appears. That treats a symptom. Instead, trace backward through the call chain to find where the bad state originated.

Backward tracing:

• Start at the error
• At each level, ask: where did this value come from? Who called this function? What state was passed in?
• Keep going upstream until finding the point where valid state first became invalid — that is the root cause

Worked example:

Symptom: API returns 500 with "Cannot read property 'email' of undefined"
Where it crashes: sendWelcomeEmail(user.email) in NotificationService
Who called this? UserController.create() after saving the user record
What was passed? user = await UserRepo.create(params) — but create() returns undefined on duplicate key
Original cause: UserRepo.create() silently swallows duplicate key errors and returns undefined instead of throwing

The fix belongs at the origin (UserRepo.create should throw on duplicate key), not where the error appeared (NotificationService).

When manual tracing stalls, add instrumentation:

// Before the problematic operation
const stack = new Error().stack;
console.error('DEBUG [operation]:', { value, cwd: process.cwd(), stack });

Use console.error() in tests — logger output may be suppressed. Log before the dangerous operation, not after it fails.

---

Multi-Component Boundary Instrumentation

Root-cause tracing walks one call chain. When a bug crosses subsystems — CI → build → signing, API → service → database, frontend → API → background worker — the failure localizes poorly to a single chain. Instead, instrument every component boundary in one run, capture what enters and what exits each, and let the evidence point to the failing layer.

Shape:

1. List the component boundaries data crosses from trigger to observed symptom. 2. At each boundary, log what enters and what exits — include the values, relevant environment, and a short tag identifying the boundary. 3. Run the scenario once. 4. Read the log linearly, comparing each "exits" value to the next "enters" value. 5. The boundary where data first stops matching expectation is the failing layer.

Worked example (app signing on CI):

# Layer 1: workflow env
echo "=== workflow env ==="
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

# Layer 2: build script env
echo "=== build script env ==="
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

# Layer 3: signing stage keychain state
echo "=== keychain ==="
security list-keychains
security find-identity -v

# Layer 4: the actual signing call
codesign --sign "$IDENTITY" --verbose=4 "$APP"

One run, and the log shows precisely which layer drops the value — secrets → workflow ✓, workflow → build ✗ → focus investigation on the workflow-to-build-script inheritance, not on signing.

When this beats backward tracing: When the symptom is far from the trigger (many components apart), when components are owned by different systems (CI vs app code), when the "call stack" is conceptual rather than literal (message bus, HTTP, process boundaries). Backward tracing still applies within each layer once the failing layer is identified.

---

Git Bisect for Regressions

When a bug is a regression ("it worked before"), use binary search to find the breaking commit:

git bisect start
git bisect bad                    # current commit is broken
git bisect good <known-good-ref> # a commit where it worked
# git bisect will checkout a middle commit — test it
# mark as good or bad, repeat until the breaking commit is found
git bisect reset                  # return to original branch when done

For automated bisection with a test script:

git bisect start HEAD <known-good-ref>
git bisect run <test-command>

The test command should exit 0 for good, non-zero for bad.

---

Intermittent Bug Techniques

When a bug does not reproduce reliably after 2-3 attempts:

Logging traps. Add targeted logging at the suspected failure point and run the scenario repeatedly. Capture the state that differs between passing and failing runs.

Statistical reproduction. Run the failing scenario in a loop to establish a reproduction rate:

for i in $(seq 1 20); do echo "Run $i:"; <test-command> && echo "PASS" || echo "FAIL"; done

A 5% reproduction rate confirms the bug exists but suggests timing or data sensitivity.

Environment isolation. Systematically eliminate variables:

• Same test, different machine?
• Same test, different data seed?
• Same test, serial vs parallel execution?
• Same test, with vs without network access?

Data-dependent triggers. If the bug only appears with certain data, identify the trigger condition:

• What is unique about the failing input?
• Does the input size, encoding, or edge value matter?
• Is the data order significant (sorted vs random)?

Test-order pollution. If an individual test passes in isolation but fails when the suite runs, tests are leaking state between each other:

• Run the failing test alone — if it passes, pollution is confirmed
• Run the failing test's file alone — narrows pollution to same-file or cross-file
• Run the suite with randomized test order (most runners support a seed flag) — a different failing-test neighbor each run implies global state mutation
• Bisect the preceding tests: run the failing test with just the first half of the earlier tests, then the second half, then narrow

Common culprits once isolated: module-level state, mocks not torn down, temp files not cleaned up, database rows not rolled back, environment variables mutated and not restored.

---

Repro Minimization

Once a bug reproduces reliably, the reproduction is often large — a 500-line integration test, a huge payload, a lengthy form-filling sequence. A smaller reproduction makes every subsequent investigation step faster and localizes the actual trigger.

Delta debugging (manual):

1. Cut the reproduction in half. 2. Does it still fail? If yes, discard the other half; recurse on what remains. If no, the failing behavior depends on something in the half you cut — put it back and cut the other half instead. 3. Continue until no further reduction is possible without losing the failure.

For input payloads:

• Remove fields one at a time (or half at a time) while confirming the bug persists
• Shrink string values until the minimum length that still triggers the bug
• Replace complex nested structures with the smallest shape that reproduces

For test sequences:

• Remove setup steps that don't appear to affect the failing assertion
• Inline helpers into the test to see what actually runs
• Remove other assertions to isolate which one fails and on what state

The minimized repro often reveals the root cause directly — "the bug only triggers when the string contains a tab character" is a much louder signal than "the bug triggers in this 500-line integration test."

---

Framework-Specific Debugging

Rails

• Check callbacks: before_save, after_commit, around_action — these execute implicitly and can alter state
• Check middleware chain: rake middleware lists the full stack
• Check Active Record query generation: .to_sql on any relation
• Use Rails.logger.debug with tagged logging for request tracing

Node.js

• Async stack traces: run with --async-stack-traces flag for full async call chains
• Unhandled rejections: check for missing .catch() or await on promises
• Event loop delays: process.hrtime() before and after suspect operations
• Memory leaks: --inspect flag + Chrome DevTools heap snapshots

Python

• Traceback enrichment: traceback.print_exc() in except blocks
• pdb.set_trace() or breakpoint() for interactive debugging
• sys.settrace() for execution tracing
• logging.basicConfig(level=logging.DEBUG) for verbose output

---

Stepping Debugger vs Instrumentation

Print-debugging is the default reach — it is fast to add and scales across many cases. But there are cases where an interactive stepping debugger converges to the root cause far faster. The rule of thumb:

• Reach for a stepping debugger when: the failing code path is localized (a specific function or tight call chain), the bug is reliably reproducible, and you need precise state at a known point — values of many locals at once, the exact shape of a structure, or the progression of state across a loop. One break, inspect everything.
• Reach for instrumentation when: the bug is intermittent, spans many calls or distributed components, or happens in a context where breaking execution is disruptive (production, concurrent code whose timing matters, long-running processes). Instrumentation captures diffuse behavior across time and environments.

Mixed use is common: instrument first to localize, then attach a debugger at the localized point.

Entry points by language:

For test runs, most test runners integrate with the above — e.g., node --inspect-brk $(which jest), pytest --pdb, rspec with binding.pry, dlv test. Prefer the runner's integration over trying to attach post-hoc.

---

Race Condition Investigation

When timing or concurrency is suspected:

Timing isolation. Add deliberate delays at suspect points to widen the race window and make it reproducible:

// Simulate slow operation to expose race
await new Promise(r => setTimeout(r, 100));

Shared mutable state. Search for variables, caches, or database rows accessed by multiple threads or processes without synchronization. Common patterns:

• Global or module-level mutable state
• Cache reads without locks
• Database rows read then updated without optimistic locking

Async ordering. Check whether operations assume a specific execution order that is not guaranteed:

• Promise.all with dependent operations
• Event handlers that assume emission order
• Database writes that assume read consistency

Condition-based waits instead of arbitrary delays. Flaky tests are often built on setTimeout/sleep calls that guess at how long an operation takes. These pass on fast machines and fail under load or in CI. Replace the guess with polling the condition the test actually depends on, bounded by a timeout:

// before: races under load
await new Promise(r => setTimeout(r, 50));
expect(getResult()).toBeDefined();

// after: waits for the condition
await waitFor(() => getResult() !== undefined, 'result available', 5000);
expect(getResult()).toBeDefined();

Arbitrary delays remain correct only when testing actual timing behavior (debounce intervals, throttle windows) — in that case, comment why the specific duration is needed.

---

Heisenbugs and the Observer Effect

When adding console.log, attaching a debugger, or inserting instrumentation causes the bug to disappear, the observation is changing the system's behavior. That is itself diagnostic — do not conclude "fixed." The bug is still present; your instrumentation perturbed it out of sight.

What the disappearance tells you:

• Timing-sensitive: Instrumentation slowed the code enough that a race condition no longer wins. Investigate concurrency, async ordering, and shared mutable state rather than the nominal logic.
• Garbage-collection-sensitive: Logging allocated memory and triggered a GC that hid the symptom. Look at memory pressure, finalizers, object lifecycle.
• Optimization-dependent: Instrumentation prevented a compiler/JIT optimization that was producing wrong results. Rare but real (especially in C/C++/Rust release builds).
• Buffering-dependent: Log flushing changed I/O ordering. Often indicates unflushed writes elsewhere.
• Async-ordering-sensitive: Log I/O introduced a microtask boundary that reorders subsequent operations. Look for code that implicitly depends on synchronous ordering.

How to investigate without perturbing:

• Non-blocking instrumentation: write to a ring buffer in memory, dump it only after failure is observed
• Sampling profilers instead of tracing: external observation of what's running without injecting code into the path
• Platform-level instrumentation: strace, dtrace, eBPF, platform profilers that don't require code changes
• Post-mortem evidence: core dumps, heap snapshots, captured state from after the failure, without observing during

The defining rule: if the bug is sensitive to observation, the fix must survive re-introduction of the observation. A fix that only works while instrumentation is present is itself a heisenbug.

---

Browser Debugging

When investigating UI bugs with agent-browser or equivalent tools:

# Open the affected page
agent-browser open http://localhost:${PORT:-3000}/affected/route

# Capture current state
agent-browser snapshot -i

# Interact with the page
agent-browser click @ref          # click an element
agent-browser fill @ref "text"    # fill a form field
agent-browser snapshot -i         # capture state after interaction

# Save visual evidence
agent-browser screenshot bug-evidence.png

Port detection: If your in-context project instructions explicitly state the dev-server port, use it (don't grep instruction prose for a port — it's false-positive-prone); otherwise check package.json dev scripts, then .env files, falling back to 3000.

Console errors: Check browser console output for JavaScript errors, failed network requests, and CORS issues. These often reveal the root cause of UI bugs before any code tracing is needed.

Network tab: Check for failed API requests, unexpected response codes, or missing CORS headers. A 422 or 500 response from the backend narrows the investigation immediately.

---

Evidence Harvesting Across Systems

When a bug spans a real environment — production, staging, a multi-service setup — the richest evidence usually already exists in logs, traces, and error-tracker payloads. Use it rather than reproducing from scratch when possible.

Follow a single request end-to-end. Pick one concrete failing request (an exact timestamp, user ID, or event ID from an error tracker). Then:

• Search every relevant log source for that identifier — correlation ID, request ID, trace ID, user ID
• Assemble the timeline in order: edge → API → service → database → downstream calls → response
• Note where the timeline has gaps (missing logs) or contradictions (timestamps out of order, IDs that don't propagate)

One traced request usually reveals the root cause faster than a dozen attempts to reproduce.

Correlation IDs. Most web frameworks either attach a request ID automatically or accept one via header (X-Request-ID, traceparent). When the project has one, every log line and every downstream call should carry it. If it's missing or not propagated, that is itself a finding — propagation gaps mean the agent cannot assemble the timeline, and neither could the on-call human who investigates the next incident.

Timestamp triangulation. When the failing operation has no shared ID, timestamps are the fallback. Constrain every log query to a narrow window around the observed failure, then look for the first anomaly in order. Watch for clock skew between services — a 30-second drift between two hosts reorders evidence and misleads triangulation.

Error tracker payloads. Sentry, Bugsnag, Honeybadger, AppSignal and similar tools capture stack traces, breadcrumbs, user context, request state, and release metadata at the moment of failure. Read the full payload before tracing code — it often contains the exact file:line, the variable state, and the breadcrumbs leading to the error. Grouping rules sometimes hide frequency and variant information; expand to see every instance rather than just the representative one.

APM / distributed traces. When the project has Datadog APM, Honeycomb, New Relic, or an OpenTelemetry collector, the trace view shows the full call tree across services with timings. Look for: unexpectedly long spans (blocking or slow dependency), failed spans in the middle of the chain, spans that should exist but don't (missing instrumentation also masks bugs).

Preserve before investigating. Error trackers and log systems have retention windows. Before starting a long investigation, export or snapshot the key evidence (event ID, trace ID, full stack trace, breadcrumbs) so it doesn't age out mid-session.

---

System Boundary Checks

Many bugs live at the boundary between an application and the system it runs on — network, database, filesystem, OS. A fast pass through these boundaries often eliminates whole categories of suspicion before deep code tracing.

Network.

• DNS resolution: dig <host>, nslookup <host>, host <host> — does the name resolve to what you expect from this host?
• Reachability: curl -v https://host/path — full headers, redirects, TLS errors
• Status codes and headers: check response for 4xx/5xx, unexpected redirects, missing CORS headers, content-encoding surprises
• Connection state: ss -tan / netstat -an / lsof -i — open connections, listening ports, connections in TIME_WAIT or CLOSE_WAIT
• TLS: openssl s_client -connect host:443 — certificate chain, expiry, SNI mismatches

Database.

• Query plan: EXPLAIN / EXPLAIN ANALYZE on the suspect query — is it using the expected index, or scanning a large table?
• Slow query log / recent queries: most databases surface the N slowest recent queries — failing queries often show up there
• Locks and transactions: inspect the lock/transaction tables (pg_locks, information_schema.innodb_trx, sys.dm_tran_locks) — is the operation waiting on a long-held lock?
• Connection pool: is the app exhausting its pool? Are connections leaking?
• Replication lag (if read replicas are in the path): a read right after a write may hit a replica that hasn't caught up yet

Filesystem.

• Existence and permissions: ls -la <path> — does the file exist, is it readable/writable by the running user?
• Case sensitivity: bugs that only appear on Linux (not macOS) are often case mismatches
• Open handles: lsof <path> or lsof -p <pid> — is something still holding the file, preventing write/unlink?
• Disk space: df -h — out-of-space errors sometimes surface as cryptic write failures elsewhere
• File watching / inotify limits: EMFILE or "too many open files" often means an inotify/FD limit, not a leak in your code
• Path separators and encoding: Windows-style paths in Unix code, or UTF-8 paths in a non-UTF-8 locale

Processes and signals. Check whether the process is actually the version you think is running (ps aux | grep, cross-reference pid to build time). Zombies, orphaned workers, and crashed-then-restarted-with-old-code processes all masquerade as code bugs.

---

Bug-Class Pattern Checklist

Before deep tracing, run down this checklist. Many bugs match a recognizable class, and the class implies where to look first. Check whether the observed symptom fits any of these patterns:

• Time and timezone: off-by-hours errors near midnight, failures specifically during DST transitions, epoch/milliseconds confusion, naive vs timezone-aware datetimes mixed, UTC-vs-local assumed incorrectly
• Encoding and locale: mojibake in output, byte-vs-character length off-by-one, BOM at the start of a file breaking parsers, non-ASCII characters missing, locale-sensitive comparisons producing inconsistent results
• Floating-point precision: comparisons that "should" be equal but aren't, NaN propagating through a calculation and silently corrupting downstream results, very large or very small numbers losing precision
• Integer overflow / underflow: wraparound on bounded integer types, int32 overflows in languages without arbitrary-precision integers, negative values where non-negative was assumed
• Off-by-one and boundaries: empty-collection edge case, first or last element missing, inclusive vs exclusive range mismatch, fencepost errors
• Cache staleness: correct behavior immediately after a change, wrong behavior after some time, fixed by restart or cache flush; includes HTTP caches, CDN caches, app-level memoization, browser service workers
• Permissions / auth: works for one user and not another, works in dev without auth layer but fails in prod with it, works with superuser but not with the actual operating identity
• Dependency or version drift: works on one machine but not another, lockfile out of sync with manifest, transitive dependency updated and changed behavior, native module built against a different runtime version
• Path / case sensitivity: works on macOS and fails on Linux (case), works on Linux and fails on Windows (path separators, reserved names like CON/PRN)
• Concurrency / ordering: works in serial test mode, fails in parallel; works one way and fails another when randomized
• Stale build artifacts: dist/, .next/, compiled .pyc, generated code, Docker image layers — rebuild from clean and see if it reproduces
• Observer effect (heisenbug): bug vanishes when logging, debugger, or profiler is attached — see the Heisenbugs section above
• TOCTOU (time-of-check vs time-of-use): a check passed a moment ago but the underlying state changed before the dependent action ran

Pattern-matching here is cheap. Spending 30 seconds checking whether the symptom fits a known class can eliminate hours of speculative tracing.