Debugging Network Issues

Debugging Network Issues

Evidence-driven investigation methodology for incidents where the obvious cause is probably wrong. Built from a real 5-hour production case (see references/case-sse-rst-130s.md) where assumption-stacking wasted hours that a 10-minute layered experiment would have resolved.

Apply this skill when the user reports a network/streaming/protocol symptom and the investigator feels tempted to diagnose from one log line or one circumstantial data point. The skill's job is to slow that reflex down.

Triage first — is this a known domain?

Before applying the general methodology below, check whether the symptom points at a stack that already has a dedicated skill in this repo. Those carry the domain-specific symptom→cause→fix tables this skill deliberately stays general about — start there, and come back here for methodology if the root cause turns out to be elsewhere.

If none match — or you tried a domain skill and the evidence points elsewhere — continue below. The methodology generalizes to any multi-layer system.

Note for this skill specifically: If the symptom is a Cloudflare 524/522 on a large `POST` body (e.g., /<openrouter-path> with Content-Length > 1 MB), the failure is often upload time to origin exceeding Cloudflare's origin read timeout, not backend slowness. Use the upload-vs-processing checklist below before assuming a backend stall.

Core principles

1. Evidence over assumption

If you cannot point to a concrete artifact — log line, pcap frame, probe output, metric sample — you are guessing, not diagnosing. Before stating "X is the cause", require yourself to name the direct evidence. If it does not exist yet, add instrumentation (see references/instrumentation-patterns.md) or capture it (see references/packet-capture-recipes.md) before continuing.

2. Falsification over confirmation

N independent sources "confirming" a hypothesis does not make it true. One falsifying observation rules it out. Before acting on a hypothesis, answer:

"What observation would make me abandon this hypothesis?"

If the answer is "nothing" or "I cannot think of one", the hypothesis is unfalsifiable and must not drive the investigation. If the answer is concrete, go look for that observation before committing to action.

3. Layered isolation

Multi-hop systems (client → CDN → LB → reverse proxy → app → upstream) concentrate bugs at the seams between layers. When a symptom could plausibly come from several layers, do not reason about which layer; test. The canonical technique: run the same logical request through three or more paths that differ by exactly one hop, then compare where the symptom appears. This resolves in minutes what stacking hypotheses cannot resolve in hours. See references/layered-isolation-experiment.md.

4. Counter-review before committing

Before committing to a root cause or shipping a fix, have independent reviewers challenge the conclusion — not confirm it. Agents are good at surfacing risks a single investigator did not think of; they are bad at weighing them. Apply the four-question filter (see references/counter-review-pattern.md) to every finding before it shapes action.

Workflow

Copy this checklist into the investigation notes and check items off:

Investigation Progress:
- [ ] Step 0:   Scope the symptom (exact error, exact times, who, who-not, what changed)
- [ ] Step 0.5: Verify the premise — does direct evidence show the symptom is actually happening?
- [ ] Step 0.6: **For large POST bodies: distinguish upload-timeout from processing-timeout** (see recipe below)
- [ ] Step 1:   Gather direct evidence at every hop before hypothesizing
- [ ] Step 2:   Frame ≥3 hypotheses; for each, name (a) what falsifies it, (b) which layer boundary the intervention would target
- [ ] Step 3:   Design a decisive experiment (for network: layered isolation)
- [ ] Step 4:   Add instrumentation if evidence gaps block direct observation
- [ ] Step 5:   Execute, record actual vs predicted
- [ ] Step 6:   Counter-review before acting
- [ ] Step 7:   Fix + re-run the same experiment to verify
- [ ] Step 8:   Document wrong turns as teaching material

Step 0: Scope

A tight scope is the difference between a 20-minute investigation and a 5-hour one. Before looking at anything, extract:

• Exact error string (copy-paste, not paraphrase). socket closed is not the same as ECONNRESET is not the same as HTTP/2 RST_STREAM INTERNAL_ERROR (err 2).
• Exact timestamps (ISO-8601 with timezone, not "yesterday evening")
• Reproducibility (every time / intermittent / only specific users)
• Who is affected, who is not (differential observations narrow the search)
• What changed recently (deploys, config, upstream dependencies, client versions)

Distinguish symptom from diagnosis. "Slow" is not a symptom. "Request took 130.898s then returned HTTP/2 INTERNAL_ERROR" is.

Step 0.5: Verify the premise

Before investing in a full investigation, confirm the reported symptom is actually happening — not just inferred from downstream effects or user frustration. One cheap direct observation beats hours spent investigating a non-problem.

Ask: "What direct evidence shows this symptom is real?"

• If the user reports "timeout at 130s": is that from a timestamped log, a browser network panel, or a recollection?
• If the user reports "connection reset": did they see the packet or is it inferred from a retry spike?
• If the user reports "fails for some but not others": has it been reproduced in a controlled test, or is it anecdotal?

Acceptable premises:

• Log line with timestamp and error string
• Browser DevTools Network screenshot showing the failure
• Reproduction command that shows the symptom on demand
• Metrics chart showing the specific error count rising

Not sufficient as premise:

• "Users are saying it feels slow"
• "The alert fired but I did not check what actually failed"
• "Last week someone mentioned..."

If the premise fails verification, the fix is observation — not investigation. Add the missing telemetry, wait for the next occurrence with instrumentation in place, and return when you have real data. Resist the sunk-cost instinct to investigate anyway "since we are already here".

Step 0.6: Upload-timeout vs processing-timeout for large POST bodies

For CDN-fronted POST/PUT endpoints with large bodies, the most common misdiagnosis is blaming backend slowness when the real problem is time-to-upload-body exceeding the CDN/proxy origin timeout.

Apply this sub-checklist when the symptom is a 524/522/504 on a request with Content-Length > ~500 KB:

1. Locate the edge/reverse-proxy access log (Caddy, nginx, Envoy, Cloudflare Logpush). 2. Compare `bytes_read` (or equivalent) to `Content-Length`:

• bytes_read == Content-Length and status is an error → likely backend/processing problem.
• bytes_read < Content-Length and the connection closed around the timeout window → upload problem.

3. Check `duration` / `request_time` semantics:

• Caddy duration = wall time from first byte read to response end.
• nginx $request_time = same.
• <upstream-capture-service> / app request_time = time backend spent processing after body was fully received.
• If proxy duration ≈ timeout but upstream request_time is short or never logged, the body upload is the bottleneck.

4. Look for `status=0` (Caddy) or `-` (nginx):

• status=0 means the proxy never wrote an HTTP response, usually because the downstream/client side closed first.

5. Correlate with upstream logs:

• If the request ID / ray ID / trace ID does not appear in upstream (<new-api-container>, <upstream-capture-service>, app) logs, the request never finished uploading.

Example signature of an upload-timeout 524:

{
  "status": 0,
  "duration": 125.0,
  "bytes_read": 4111422,
  "request": {
    "headers": { "Content-Length": ["6042141"] }
  }
}

Interpretation: the proxy kept the connection for 125 s, read 4.1 MB of a 6 MB body, then Cloudflare closed it and returned 524.

Example signature of a processing-timeout:

{
  "status": 504,
  "duration": 120.1,
  "bytes_read": 6042141,
  "request": { "headers": { "Content-Length": ["6042141"] } }
}

Interpretation: full body uploaded, but backend did not respond before proxy timeout → backend/processing problem.

Step 1: Gather direct evidence at every hop

Before framing hypotheses, collect:

• Server-side logs at every hop in the request path
• Client-side logs (browser devtools HAR, CLI debug log, SDK traces)
• Metrics over the incident window (RPS, latency, error rate, connection count, CPU/mem)
• Distributed trace if available
• Packet capture if the symptom is at the wire level (see references/packet-capture-recipes.md)

If any of these is missing and relevant, fill the gap before guessing. Adding a TRACE_* env flag and restarting a container beats an hour of hypothesis-stacking. The instrumentation patterns in references/instrumentation-patterns.md are low-risk, env-gated, and safe to ship into production permanently.

Reading reverse-proxy access logs for upload/processing split

Caddy and nginx logs are the cheapest way to falsify "backend is slow". Focus on three fields:

Key patterns:

• bytes_read < Content-Length and duration ≈ timeout → upload-timeout.
• bytes_read == Content-Length and status is 5xx → processing-timeout.
• status == 0 and bytes_read < Content-Length → client/CDN closed before upload finished.

Tracing a single request across the stack

For the <project> stack (Cloudflare → Caddy → <provider-gateway-service> → <upstream-capture-service> → <new-api-container>), the canonical trace is:

1. Cloudflare: get Cf-Ray and timestamp from the client error or Cloudflare Logpush. 2. Caddy: docker logs <gateway-container> | grep <Cf-Ray> → extract X-Request-Id (Caddy uuid) and confirm bytes_read, duration, status. 3. <provider-gateway-service>: docker logs <provider-gateway-service> for Client request error: aborted or request/response logs. 4. <upstream-capture-service>: grep <X-Request-Id or timestamp> /data/<upstream-capture-service>/log/access.log → confirms whether the request reached <new-api-container> and how long upstream processing took. 5. <new-api-container>: docker logs <new-api-container> for billing/channel errors.

If the request ID never appears in steps 3–5, the failure happened at the edge or during body upload.

Aggregating by client IP to spot patterns

A single 524 can be a fluke; a pattern of 524s concentrated on one IP + one path is a smoking gun. Run an aggregation like:

# Caddy JSON example: count failures by IP and body size for an endpoint
python3 -c "
import sys, json
from collections import Counter, defaultdict
stats = defaultdict(lambda: {'total': 0, 'fail': 0, 'slow': 0, 'max_cl': 0})
for line in sys.stdin:
    d = json.loads(line)
    req = d.get('request', {})
    if req.get('uri', '').startswith('/<openrouter-path>'):
        ip = req.get('headers', {}).get('Cf-Connecting-Ip', [''])[0]
        cl = int(req.get('headers', {}).get('Content-Length', ['0'])[0] or 0)
        dur = d.get('duration', 0)
        status = d.get('status', 0)
        s = stats[ip]
        s['total'] += 1
        s['max_cl'] = max(s['max_cl'], cl)
        if status == 0:
            s['fail'] += 1
        elif status == 200 and dur > 60:
            s['slow'] += 1
for ip, s in sorted(stats.items(), key=lambda x: -x[1]['fail']):
    print(f\"{ip}: total={s['total']} fail={s['fail']} slow={s['slow']} max_cl={s['max_cl']}\")
" < caddy-access-log.jsonl

If one IP dominates failures and its max_cl is large, investigate upload bandwidth/path before backend.

Step 2: Hypotheses with falsifiers and threat-model boundaries

List three or more plausible causes. For each, write three sentences:

• What would confirm it? (easy and often misleading)
• What would refute it? (the falsifier — this is what matters)
• Which layer boundary would the intervention target? (the threat-model question — forces you to be precise about where the fix would apply)

The third question prevents a common anti-pattern: proposing a fix that operates on the wrong hop. For example, a "keepalive" fix that writes bytes downstream to the client is useless for an _upstream_ idle timeout — the intervention targets a different boundary than the problem. Naming the boundary up-front surfaces this mismatch before coding starts.

If you cannot state a concrete refuter, the hypothesis is unfalsifiable. Flag it, but do not act on it. If you cannot state which boundary a proposed fix targets, you do not yet understand what the fix actually does.

Step 3: Decisive experiment

For network-layer problems, the default is layered isolation: three paths differing by exactly one hop. Example for a CDN-fronted service:

If only A fails, the CDN is the cause. If A and B fail but C passes, the LB is. Compose more variants as needed. See references/layered-isolation-experiment.md for a runnable template using a mock idle upstream — the experiment does not need a cooperating production request to trigger, the idle interval can be controlled precisely.

For non-network domains:

• Performance: controlled benchmark with one variable changed
• Correctness bug: failing test case that reproduces
• Intermittent: sampled tracing + wait for recurrence

Step 4: Instrumentation when needed

If the decisive experiment requires an observation that cannot currently be made, add it — do not skip it. The canonical pattern is env-gated instrumentation that:

• Defaults off (zero runtime cost in steady state)
• Turns on via one environment variable, without code changes
• Writes greppable log tags ([SSE-CHUNK] ts=... req=... bytes=...)
• Ships into production permanently — future incidents reuse it

See references/instrumentation-patterns.md for the exact template used to diagnose the <upstream-provider> 125-second upstream silence in this incident.

Step 5: Execute and record

Run the experiment once, fully documented: command, environment, inputs, observed outputs, wall-clock timestamps. Compare against the prediction made in Step 2. If actual matches predicted, the hypothesis is calibrated. If not, the hypothesis is wrong — do not rescue it with ad-hoc auxiliary hypotheses ("oh, but maybe X also interferes..."). Return to Step 2 and write new hypotheses from scratch.

Step 6: Counter-review

Before committing to a root cause or shipping a fix, spawn independent reviewers to challenge the conclusion. Give them the same evidence, ask them to falsify, not confirm. Apply the four-question filter to each finding they raise:

1. Probability — will this actually happen? 2. Cost — what is the cost of fixing versus ignoring? 3. Realistic scenario — does this apply to the user's actual business case? 4. Verification — can I cheaply confirm or refute this?

Classify every finding: real issue / partly right / unlikely / actively harmful. Never paste raw agent output to the user; filter first. See references/counter-review-pattern.md.

Step 7: Fix and verify

Apply the fix. Rerun the same decisive experiment from Step 3. Confirm the symptom no longer reproduces with the same setup that was reliably producing it. If the pre-fix state can no longer be reproduced after the fix, the fix cannot be proven — figure out why the repro was lost before declaring victory.

Step 8: Document wrong turns

The wrong turns in the investigation are more valuable than the right answer. Write an incident report capturing:

• Symptom + direct evidence
• Each hypothesis tried + how it was falsified
• Decisive experiment design + result
• Fix + verification
• New monitoring or instrumentation added

Future investigators — including future self — will read this to avoid the same cognitive traps.

Common cognitive traps

1. Circumstantial evidence convergence. Five indirect clues all pointing the same direction feel like proof. They are not. If a direct probe is cheap, run it. 2. Field-semantic confusion. duration=5.95s can mean total wall time (one tool), handler execution phase (another tool), or TTFB (a third). Never cite a numeric field without verifying its semantics against documentation or code. 3. Single-cause bias. Multi-layer systems fail from multi-layer defect compositions. Fix the direct cause but document the amplifying factors so the next layer of defense can also be hardened. 4. Naming assumption. A resource labeled spot-instance may not actually be a spot instance. Verify attributes via API, not metadata names. 5. Probe self-verification. A diagnostic that runs through the broken connection to test the broken connection yields uninterpretable results. Always cross-verify with an independent probe. 6. Assumption-rescue cycle. When evidence contradicts a hypothesis, the temptation is to add a modifier ("yes, but only in case X"). Resist. If the first falsifier fires, scrap the hypothesis. 7. Unverified premise. Investigating a symptom that was never directly observed — inferred from user frustration, alert titles, or downstream effects. Verify first (Step 0.5). Do not investigate anecdotes. 8. Threat-model mismatch. Proposing a fix that targets the wrong layer — writing bytes downstream to solve an upstream problem, tuning a timeout on a hop that never fires it. Naming the boundary each hypothesis targets (Step 2) surfaces this. 9. Reverse-path / directional asymmetry. A→B healthy ≠ B→A healthy. An external probe to a node proves only that node's return/inbound direction; network paths and congestion are directional. Measure the same direction the user's traffic flows, from the user's side (TCP-mode mtr/nexttrace from the affected origin), before declaring a hop healthy. 10. Edge timeouts masquerading as upstream client aborts. A 524 from Cloudflare can cause the origin proxy (Caddy/nginx) to log the upstream connection as a "client abort" (status=0, Client request error: aborted). The abort is real at the origin, but the _cause_ is the CDN edge timing out first. Always correlate edge error codes, edge timestamps, and origin logs before attributing an abort to the client. See the upload-vs-processing recipe in Step 0.6.

See references/cognitive-traps.md for extended examples including this case study.

Anti-patterns — things to explicitly avoid

• Jumping to a fix before a falsifier is found. "Probably it is X, let me restart / tweak / upgrade." This converts learning opportunities into mystery fixes that do not prevent recurrence.
• Accepting agent counter-review findings wholesale. Agents over-produce risk findings. Filter before acting (see four-question filter above).
• Ad-hoc production edits that bypass IaC. If the investigation requires changing production, change the source-of-truth first, then apply — otherwise the "fix" evaporates on the next deploy and the drift hides the real state.
• Declaring root cause from a single observation. Demand a falsifier attempt first.
• Writing "should work now" without re-running the failing experiment. Re-verify.

Case studies

Two canonical cases illustrate the methodology in different failure modes:

1. references/case-sse-rst-130s.md — a 5-hour investigation where the assistant repeatedly jumped to the wrong conclusion. The right answer — Cloudflare edge HTTP/2 stream idle timeout at 126 seconds, amplified by <upstream-provider> not emitting SSE ping during <model-name> tool_use generation — surfaced in 10 minutes once a subagent designed a 3-path layered isolation experiment with a mock idle upstream.

2. references/case-cloudflare-524-upload.md — a Cloudflare 524 on <api-domain>/<openrouter-path> where a ~6 MB POST body took longer to upload from the US client to the <origin-region> origin than Cloudflare's default origin read timeout allowed. The key insight came from comparing bytes_read (4.1 MB) to Content-Length (6.0 MB) and confirming the request never reached <upstream-capture-service> or <new-api-container>. This case is the source of the upload-vs-processing recipe and the "edge timeouts masquerading as client aborts" trap above.

Read both before applying this skill to an unfamiliar problem domain; the wrong-turn anatomy is the teaching.

Reference files

• references/layered-isolation-experiment.md — 3-path technique, mock upstream template, result matrix
• references/instrumentation-patterns.md — env-gated TRACE\_\*, greppable log tags, deployment checklist
• references/packet-capture-recipes.md — tcpdump filters for RST isolation, interface selection on Docker, HTTP/2 decoding
• references/counter-review-pattern.md — 4-agent team composition, 4-question filter, integration workflow
• references/cognitive-traps.md — extended examples, rescue-cycle warnings
• references/case-sse-rst-130s.md — canonical case study with wrong-turn timeline

Scripts

• scripts/mock-idle-upstream.py — SSE server that emits one frame then idles N seconds. Use as the upstream in layered isolation experiments to precisely control the idle interval.
• scripts/layered-isolation-probe.sh — Runs the 3-path A/B/C comparison and prints a diagnostic matrix.

Case study: Cloudflare 524 on a 6 MB <openrouter-service> request body

This case study walks through a 2026-06-12 incident on <api-domain> where a Cloudflare 524 was initially easy to misattribute to backend slowness. The actual cause was the request body upload time exceeding Cloudflare's default origin read timeout.

Symptom

Cloudflare returned a 524 for POST https://<api-domain>/<openrouter-path>:

{
  "type": "https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-524/",
  "title": "Error 524: A timeout occurred",
  "status": 524,
  "detail": "The origin web server did not return a complete response within the 120-second Proxy Read Timeout window.",
  "instance": "<cf-ray-id>",
  "ray_id": "<cf-ray-id>",
  "timestamp": "2026-06-12T11:30:28Z"
}

The error explicitly says "origin web server did not return a complete response", which naturally points at the origin. The risk is to start tuning backend timeouts or restarting services.

Direct evidence that changed the diagnosis

1. Caddy access log for the Ray ID

ssh root@<origin-ip> \
  'docker logs --since "2026-06-12T11:28:00Z" --until "2026-06-12T11:33:00Z" \
   <gateway-container> 2>&1 | grep "<cf-ray-id>"'

Key fields from the JSON log:

{
  "ts": 1781263801.0,
  "duration": 125.0,
  "status": 0,
  "size": 0,
  "bytes_read": 4111422,
  "request": {
    "method": "POST",
    "host": "<api-domain>",
    "uri": "/<openrouter-path>",
    "headers": {
      "Content-Length": ["6042141"],
      "Cf-Connecting-Ip": ["<client-ip>"],
      "User-Agent": ["<claude-cli-user-agent>"],
      "Cf-Ray": ["<cf-ray-id>-<colo>"]
    }
  }
}

The proxy read 4,111,422 bytes of a declared 6,042,141-byte body and never returned a response. That is an incomplete upload, not a backend hang.

2. Provider-gateway logged a client abort

ssh root@<origin-ip> \
  'docker logs --since "2026-06-12T11:28:00Z" --until "2026-06-12T11:35:00Z" \
   <provider-gateway-service> 2>&1 | head -20'

Output:

Client request error: aborted
Client request error: aborted
...

The relevant code:

// <provider-gateway-service-source>
req.on("error", (err) => {
  console.error("Client request error:", err.message);
  // Client aborted mid-request-body (Node surfaces it as 'aborted'/
  // ECONNRESET) or the request stream failed — record the terminal
  // client_abort row.
  maybeLogClientAbort();
  if (!res.headersSent) {
    res.writeHead(400);
    res.end("Bad Request");
  }
});

This tells us the request stream was aborted before it finished. It does not tell us _who_ aborted it. The edge (Cloudflare) aborting first and the origin seeing a downstream abort is the expected pattern for a CDN timeout.

3. The request never reached <upstream-capture-service> or <new-api-container>

ssh root@<origin-ip> \
  'grep -E "\"ts\":\"2026-06-12T11:30:0[0-9]" /data/<upstream-capture-service>/log/access.log | \
   python3 -c "import sys,json; [print(json.dumps({k:d.get(k) for k in [\"ts\",\"channel_id\",\"status\",\"request_length\",\"request_time\"]})) for d in (json.loads(l) for l in sys.stdin)]"'

Only one unrelated request appeared at 11:30:03 (channel 3, 174 KB). The 6 MB request never made it past the body-reading stage.

4. Successful large requests from the same client took ~122 s total

A successful request at 11:02:14 with the same body size:

• Caddy duration: 122.2 s
• Caddy status: 200
• <upstream-capture-service> request_time: 31.2 s (channel 7 DeepSeek)

The ~91 s gap is the time spent uploading the body. When the upload was slower than ~100–120 s, Cloudflare cut the connection and returned 524.

5. Pattern by client IP

Over a 3-hour window:

All failures were from the same US client IP, all on large bodies.

Root cause

Cloudflare's default origin read timeout (~100–120 s) was shorter than the time needed for this US client to upload a ~6 MB request body to the <origin-region> origin. When the upload did not finish in time, Cloudflare returned 524. Caddy and <provider-gateway-service> then observed the abort as a closed client stream.

Why this is easy to get wrong

1. The error message blames the origin. "Origin web server did not return a complete response" reads like a backend problem. 2. Provider-gateway logs look like a client problem. "Client request error: aborted" suggests the user killed the request, but the actual closer was Cloudflare. 3. Backend services are all healthy. Without checking bytes_read vs Content-Length, the natural next step is to look for slow upstream LLM calls.

Decisive observation

The falsifier for "backend is slow" was: if the request never reached <upstream-capture-service>, backend slowness is impossible. Confirming that falsifier required only two greps (Caddy + <upstream-capture-service>) and took under a minute.

Lessons

• Always compare bytes_read to Content-Length in proxy logs when diagnosing 524/522/504 on large POST bodies.
• status=0 in Caddy (or - in nginx) means no HTTP response was written — a strong signal that the connection was closed by an upstream-of-origin layer.
• Correlate edge error codes (Cloudflare Ray ID) with origin proxy logs; do not let origin-side "client abort" logs mislead you about who closed the connection.
• Aggregate by client IP and body size. A pattern concentrated on one IP is usually a network/path problem, not a systemic backend issue.

Related files in the main incident report

• docs/reports/incident-2026-06-12-cloudflare-524-openrouter-upload.md — full production incident report with exact commands and timeline.

Case Study: SSE HTTP/2 RST at 130s (130s RST incident, April 2026)

A production incident where a user's <cli-client> kept failing with ECONNRESET after exactly 130 seconds on long tool_use tasks. The investigation took 5 hours and produced three wrong root-cause conclusions before a structured experiment resolved it in 10 minutes.

This case study is the canonical teaching material for this skill. Read it to understand the anatomy of how assumption-first investigations go wrong, and how to recognize when to switch to evidence-first.

Contents

• Symptom
• Environment
• Investigation timeline (with wrong turns)
• The decisive experiment
• Final root cause
• Post-mortem lessons

Symptom

The reporting user (handle: User A), using <cli-client> 2.1.116 on Windows (Node v24.3.0), submits long tasks via ANTHROPIC_BASE_URL=https://api.example.com/openrouter. Tasks involving Claude <model-name> + long tool_use (writing long files with the Write tool) consistently fail with:

API Error: The socket connection was closed unexpectedly.
For more information, pass `verbose: true` in the second argument to fetch()

In CLI debug logs:

2026-04-22T13:43:38.261Z [DEBUG] [API REQUEST] /<openrouter-path>
2026-04-22T13:43:47.728Z [DEBUG] Stream started - received first chunk
2026-04-22T13:45:57.344Z [ERROR] Error in API request: The socket connection was closed unexpectedly.
2026-04-22T13:45:57.347Z [ERROR] Connection error details: code=ECONNRESET

From first chunk (t=9s) to ECONNRESET (t=130s): exactly 130 seconds. Reproducible across sessions.

Environment

• Client: <cli-client> 2.1.116, Windows 11, Node v24.3.0
• Server: api.example.com (Cloudflare-proxied, origin is aliyun Japan, 203.0.113.10)
• Architecture: CLI → CF → Caddy → <provider-gateway-service> → <new-api-container> → <upstream-provider> (Anthropic proxy) → <model-name>
• Not affected: same user's Haiku requests, other users' <model-name> requests with shorter duration (all <45s)

Investigation timeline (with wrong turns)

Hour 1: VPN theory (wrong)

Observation: Cf-Connecting-Ip varied between China (198.51.100.42) and US (203.0.113.x) across requests, sometimes in the same session.

Hypothesis: User's VPN is unstable, rotating exit nodes, TCP dies when the route changes.

Evidence gathered: IP log showed the flipping. CF Ray always terminated at SJC (US).

Action taken: recommend user disable VPN and retry.

Falsification: User responds "disabled VPN, still fails at 130s." The requests with China-origin IP also fail.

Trap: Circumstantial evidence convergence (Trap 1). The IP flipping was real but the VPN was not the cause.

Hour 2: CLI version bug theory (wrong)

Observation: Failing requests' response bodies archived to OSS all terminate at suspiciously similar byte positions (3538 / 3902 / 3946 bytes). In each case, the response truncates mid-way through a tool_use input_json_delta containing a path string with Chinese characters (D:\项目\AI剪辑\培训...).

Hypothesis: <cli-client> 2.1.116 has a bug parsing input_json_delta containing Chinese + Windows backslashes. Client closes the socket when it fails to parse.

Evidence gathered: Three failures, all at ~3.5-3.9 KB, all at similar character positions. Strong pattern.

Action tentative: recommend CLI upgrade to 2.1.117.

Falsification: User's CLI debug log shows the close happens 130 seconds after first chunk, not at the instant the byte is received. If it were a parse bug, the close would be within milliseconds of the problematic byte. Additionally, CLI 2.1.2 (older) worked fine on similar content.

Trap: Field-semantic confusion (Trap 2). A misread of Caddy's duration=5.95s field led to believing the close was fast (5s), which made the parse-bug theory plausible. When CLI debug logs were examined, the actual timing was 130s, contradicting the hypothesis.

Hour 3: Caddy IdleConnTimeout theory (wrong)

After a subagent suggested examining Caddy reverse_proxy defaults, a hypothesis formed: Caddy's HTTP/1.1 transport IdleConnTimeout defaults to 120s; combined with TTFB that explains the 130s.

Evidence: Caddy source code shows IdleConnTimeout = 120s default. 120 + 10s TTFB = 130s close match.

Action tentative: add explicit keepalive_timeout 30m to Caddy config.

Falsification: A probe from the server itself, curl --resolve api.example.com:443:127.0.0.1 https://api.example.com/<openrouter-path> ..., runs for 200+ seconds without closing. This path goes through Caddy (just not through CF). If Caddy's IdleConnTimeout were the cause, this probe would also fail at 130s. It does not.

Trap: Assumption-rescue cycle (Trap 6) was tempting — "maybe Caddy only triggers IdleConnTimeout under specific conditions" — but the direct probe was decisive. Abandoned the hypothesis cleanly.

Hour 4: <upstream-provider> no-ping theory (partial)

Observation: Scanning 38 archived response bodies shows all of them contain zero event: ping frames and zero SSE comment-only frames (: prefix lines). Anthropic protocol specifies periodic ping events during long inactivity. <upstream-provider> appears to strip or not forward them.

Hypothesis: <upstream-provider> does not forward SSE ping → connection looks idle to the CDN → CDN closes at its idle threshold.

Evidence: 38 bodies, ping count = 0, consistent.

Action tentative: deploy server-side keepalive in <provider-gateway-service> to compensate.

Partial refutation (from counter-review agent): Anthropic's own official API has been reported (GitHub issues claude-code#18028, claude-agent-sdk-typescript#44) to stall 59-138+ seconds with no event during tool_use generation. So <upstream-provider>'s no-ping behavior, while real, is not sufficient as an independent root cause — the upstream itself is silent, <upstream-provider> has nothing to forward.

This hypothesis is partly correct — <upstream-provider> does not emit ping — but it is not the direct cause of the close. It is an amplifying factor (absence of keepalive that would have prevented the idle timeout somewhere).

Progress: we now know "something on the wire is quiet for ~125 seconds during tool_use generation" but we still do not know which layer is closing the connection.

Hour 5: The decisive experiment

Subagent designed and executed a 3-path layered isolation experiment:

1. Mock upstream: Python/Flask on port 19999 that emits one SSE frame then sleeps 200 seconds — precisely simulating the observed upstream silence pattern.

2. Temporary routing: Added CF DNS record for test-idle.example.com pointing at origin (proxied: true), plus a Caddy conf snippet forwarding that hostname to the mock.

3. Three paths, run in parallel:

• Path A (via CF): curl https://test-idle.example.com/...
• Path B (bypass CF): env -i curl --resolve test-idle.example.com:443:203.0.113.10 ...
• Path C (server loopback): ssh server 'curl http://127.0.0.1:19999/probe-c'

Results (multiple runs, consistent):

Interpretation: Only Path A closes early. B and C traverse the same Caddy/origin/upstream stack but bypass CF. Therefore the close originates at Cloudflare's edge, not at any layer below it.

Additionally, curl's error HTTP/2 stream N was not closed cleanly: INTERNAL_ERROR (err 2) indicates the reset was a peer-sent RST_STREAM HTTP/2 frame, not a TCP RST. Confirmation that CF sent an HTTP/2-layer close while the TCP connection itself remained healthy for other streams.

One subtle wrinkle from the experiment

The first run of Path B appeared to close at 126s too, which would have falsely implicated the origin. Investigation revealed the client machine had a system-level Shadowrocket proxy on http_proxy=http://127.0.0.1:1082 that was silently routing the --resolve-bypassed traffic back through CF. env -i curl (strip all environment) then correctly showed Path B clean at 220s.

This is Trap 5 (probe self-verification) caught in real time. Mitigation: env -i curl ... is now a mandatory reflex when running bypass-comparison experiments.

Final root cause

Direct cause: Cloudflare edge closes HTTP/2 streams that go idle for ~126 seconds (empirically observed constant, matches the 120-130s close-time range on Path A across runs).

Amplifying factors:

1. <upstream-provider> proxy does not emit SSE event: ping during upstream silence (38/38 observed bodies had zero ping) 2. Upstream Claude <model-name> during tool_use generation emits initial output then batch-generates the tool_use.input for 100+ seconds with no interim chunks 3. <cli-client> does not implement a client-side idle watchdog to detect the stall before the peer resets

Any one of these, in isolation, would likely not produce the observed failure. All three together + the CF idle timeout produce it reliably for <model-name> + long tool_use requests from this account.

Remediation

Not shipped as part of this case study, but the intended fix vector:

• Server-side SSE keepalive in <provider-gateway-service>: if the upstream has not emitted for N seconds, inject : keepalive\n\n comment frames (SSE-safe, ignored by clients, but keeps bytes flowing on the wire). This prevents CF from observing a full N-second idle.
• Does not require client or upstream changes. Single point of defense for all client/upstream combinations.

Code reviewer counter-review (see counter-review-pattern.md) caught two non-trivial bugs in the first keepalive draft before they shipped:

• Writing keepalive bytes before the response header has been flushed triggers Node's implicit-header emission, corrupting non-streaming Anthropic JSON responses
• Several error-path clearInterval omissions that would leak timers

Counter-review also verified (not assumed) one claim that could have been silently wrong: SSE comment frames with `:` prefix are safely ignored by all standard clients. Cross-checked against the WHATWG EventSource spec (lines beginning with : are interpreted as comments and discarded), the @anthropic-ai/sdk source (SSEDecoder skips comment lines), the openai SDK (openai/streaming follows the same contract), and <chat-client>'s EventSourceParserStream. This verification was cheap — 5 minutes with grep — and removed an otherwise plausible failure mode ("what if some client treats our keepalive bytes as malformed SSE and errors out?") from the risk list. The lesson: when a code review produces a compatibility claim, verify it from primary sources (spec + SDK source), do not leave it as "probably fine".

Post-mortem lessons

What went wrong

• 5 hours before the experiment was proposed. The experiment was cheap (~10 minutes to set up) but nobody proposed it until a counter-review agent did. The main investigator was stuck in hypothesis-stacking mode.
• Three wrong hypotheses acted on before falsification. Each was plausible. Each had circumstantial supporting evidence. None had a cheap falsifier run before acting.
• One field-semantic mistake (Caddy duration=5.95s) anchored an entire wrong direction for ~1 hour.
• The user had to push back explicitly ("I turned off VPN, still fails") to break the first wrong direction. The system should have surfaced that test earlier.

What went right

• When the experiment was finally run, it was rigorous: 3 paths, multiple runs, server-side + client-side observation, explicit cleanup.
• Counter-review caught two real code bugs in the remediation.
• Instrumentation added mid-incident (env-gated TRACESSE_CHUNKS) yielded decisive evidence for the 125-second upstream silence, \_and is now permanent observability for future incidents.
• Docker compose was refactored mid-incident to bind-mount server.cjs from the host, eliminating the "rebuild image to add a log line" cycle permanently.

Transferable methodology (codified in this skill)

1. When circumstantial evidence converges on a cause, demand one direct falsifier before acting. See cognitive-traps.md, Trap 1. 2. When multiple layers could be responsible, do not reason — test. See layered-isolation-experiment.md. 3. When observability is missing, add it as an env-gated permanent feature. See instrumentation-patterns.md. 4. Before committing to a root cause or shipping a fix, counter-review. See counter-review-pattern.md. 5. When a probe shares infrastructure with the subject of the probe, it is not a valid probe. Cross-verify. See cognitive-traps.md, Trap 5.

These five rules, applied at hour 1, would have resolved the incident in ~30 minutes instead of 5 hours.

Cognitive Traps

Curated list of wrong-turn patterns observed in real investigations. Each entry: the trap, why it is seductive, a concrete example, and the counter-move.

Contents

• Trap 1: Circumstantial evidence convergence
• Trap 2: Field-semantic confusion
• Trap 3: Single-cause bias
• Trap 4: Naming assumption
• Trap 5: Probe self-verification
• Trap 6: Assumption-rescue cycle
• Trap 7: Time-of-symptom equals time-of-cause
• Trap 8: Duration of investigation biases conclusion weight
• Trap 9: Agent output equals ground truth
• Trap 10: Unverified premise
• Trap 11: Threat-model mismatch
• Trap 12: Reverse-path / directional asymmetry

Trap 1: Circumstantial evidence convergence

Five indirect clues all pointing toward hypothesis H feel like proof. They are not, because they share a common cause (your mental model) that selected them.

Example (from this case study): Initial assumption was "VPN node rotation". Supporting circumstantial evidence:

• Client IP flipped between CN and US across requests (real)
• Request to CF hit SJC PoP (real, expected for US-routed)
• Each failed request had short duration in some log field (misread — see Trap 2)
• User was known to sometimes use VPN (real)

All four looked consistent, and the main investigator committed to "VPN instability is the root cause". The user pushed back: "I turned off VPN and it still fails." One falsifying test broke the chain.

Counter-move: When circumstantial evidence converges, require at least one direct test before acting. "The IP flips" is circumstantial. "The same user reproduces the failure with VPN verifiably off" is direct.

Trap 2: Field-semantic confusion

A number from a log field means whatever that field's code defines — not what the name suggests.

Example (from this case study): Caddy's access log has duration=0 and a separate warning log has duration=5.95s. The investigator read "duration 5.95s" and concluded "connection lasted 5.95 seconds before being reset". But that particular field in Caddy's aborting with incomplete response warning is the elapsed time between the abort signal and the handler winding down — not the total request lifetime. The actual request lifetime (from CLI debug log) was 130 seconds.

The investigator then built a whole theory around "CLI fails at 5-8 seconds due to a bug in chunk parsing", which was wrong at the root.

Counter-move: Never cite a numeric field value as evidence without checking its semantics in the source code or vendor documentation. If the field is suggestive but ambiguous, treat it as unverified until its meaning is confirmed.

Trap 3: Single-cause bias

Real production failures often emerge from multiple cooperating defects. Finding one cause and stopping leaves the amplifying factors in place, which will trigger the next incident.

Example (case study in full resolution):

• Direct cause: Cloudflare edge HTTP/2 stream idle timeout at 126s
• Amplifying factor 1: <upstream-provider> proxy does not emit SSE event: ping during upstream stalls
• Amplifying factor 2: Upstream Claude <model-name> batches tool_use output (125s silences observed)
• Amplifying factor 3: <cli-client> has no client-side idle watchdog (GitHub issue documented)

Fixing only the direct cause (e.g., moving off CF) would leave factors 1-3. Factor 2 means even with a different CDN with a larger idle window, a different idle threshold eventually fires. Factor 3 means the client is blind to the stall. The durable fix addresses factor 1 at minimum and factor 2 via server-side keepalive as defense in depth.

Counter-move: After finding the direct cause, ask explicitly: "What amplifying factors enabled this? If the direct cause were fixed, what would still be wrong?" Document all layers, fix the most cost-effective ones.

Trap 4: Naming assumption

Labels, tags, and names are metadata assigned by humans; they do not reflect runtime attributes. Verify via API, not by reading the name.

Example: A cloud instance tagged claude4dev-spot was assumed to be a Spot pricing instance during an incident. The instance was actually PostPaid; the tag was legacy from a pre-migration period. The investigator spent 10 minutes down the wrong path (Spot reclamation theory) before checking DescribeInstanceAttribute.

Counter-move: In incident response, the first step when a property matters is to query the authoritative API, not to read the name.

Trap 5: Probe self-verification

A probe that uses the thing it is probing to deliver its result cannot independently verify that thing.

Example: Using curl through a VPN to test whether the VPN is dropping connections. If the VPN drops, curl reports an error, which is what you expected — but the same error would occur if the remote host rejected the connection. The probe did not isolate.

Counter-move: Probes must be structurally independent of the subject. To test the VPN, use a second network path (mobile hotspot) to compare. To test a CDN, bypass it with --resolve. The layered isolation experiment is this principle systematized.

Trap 6: Assumption-rescue cycle

When evidence contradicts a hypothesis, the temptation is to add a modifier: "yes, but only under condition X". This rescues the hypothesis at the cost of unfalsifiability — eventually the modifiers stack to "it fails when it fails".

Example (case study): After "VPN instability" was falsified by "still fails with VPN off", a rescue was "well, maybe the VPN client has a residual system-level hook". Adding more conditions without evidence.

Counter-move: When a falsifier fires, the correct response is to scrap the hypothesis, not to narrow its scope. Return to Step 2 of the workflow and write new hypotheses.

Trap 7: Time-of-symptom equals time-of-cause

The time the user notices a symptom is often much later than the time the cause first engaged. Correcting this requires examining upstream time series.

Example: Disk fills at midnight, various retries and degradations through the morning, user-facing failure at 10:30 AM. The 10:30 timestamp is when to start looking at logs, but if you examine only the 10:30 ± 5 minute window you will miss the midnight root cause.

Counter-move: Always extend the investigation window backward by at least 10x the symptom-to-report time, or to the last known-good state. Look for monotonic metric trends crossing thresholds, not just error spikes.

Trap 8: Duration of investigation biases conclusion weight

After four hours of deep investigation, the investigator has a strong psychological bias toward "we must be close" and against "start over". This leads to over-weighting marginal evidence that fits the current theory.

Example (case study): After 3 hours of circumstantial evidence for "VPN theory", then "CLI bug theory", then "Caddy IdleConnTimeout theory", the investigator was resistant to "start a fresh experiment from scratch". The user pushed to switch approach. The experiment resolved in 10 minutes what 3 hours of deep reasoning had not.

Counter-move: Time-box. If a hypothesis has not been confirmed (not just "consistent with evidence", but actually confirmed by a direct test) within a set time, switch to a structurally different approach. Layered isolation or an experiment is a good default switch.

Trap 9: Agent output equals ground truth

Spawning an agent to investigate returns text that reads authoritatively. Accepting that text without verification treats the agent as a peer reviewer, but agents do not have skin in the game — they over-produce risks and claims.

Example: A counter-review agent cites "Cloudflare proxy_read_timeout is 100s" with high confidence. This appears to match the observed 130s. The investigator concludes CF is the cause — except the actual CF limit in this case is a different timeout (HTTP/2 stream idle, ~126s), and "100s" was the agent generalizing from community posts without matching the exact protocol.

Counter-move: Every agent claim that feeds into an action needs at least one cheap verification step. If the agent says "X is 100s", test whether X is actually 100s in your environment (or find the primary source). Filter agent findings through the four-question filter.

Trap 10: Unverified premise

Investigating a symptom that was never directly observed. The premise enters the conversation as "users say X is happening" or "the alert fired so X must be failing" and drives hours of hypothesis-building before anyone checks whether X is actually occurring.

Example: A user reports "our SSE connections keep dropping at 130 seconds". The team spends 3 hours building a keepalive patch. On the verification run before ship, they realize the original symptom was a single-digit frequency over the last week — well within normal disconnect noise for that service — and the "130-second pattern" was coincidence across two samples.

Another example (surfaced by counter-review in this case study): the proposed fix was server-side SSE keepalive. Counter-review asked: "does the user have direct evidence the RST is actually happening right now, or is this inferred from a past incident?" The fix was for a real incident that _had_ occurred, so the question was answered correctly — but the habit of asking is what prevents investigating a non-problem.

Counter-move: Before investigating, answer one question: "What direct artifact (log line with timestamp, captured packet, screenshot) shows this symptom is currently real?" If the answer is "nothing I can point to", the first action is not investigation — it is adding the telemetry and waiting for the next real occurrence. This is faster and more correct than investigating on vapor.

See SKILL.md Step 0.5 for the verification checklist.

Trap 11: Threat-model mismatch

Proposing a fix that operates on the wrong hop. The hypothesis correctly identifies that _some layer_ is at fault, but the implementation lands at a different layer, so the fix cannot actually remediate the real cause.

Example (from eval-3 baseline in this skill's iteration-1 tests): a proposed SSE keepalive patch writes : keepalive\n\n to res (downstream client-facing connection). But the stated concern was "upstream idle > 15s". Writing bytes to the downstream socket does nothing to maintain the upstream TCP connection — if the idle timeout fires on the proxy→upstream hop, the keepalive is directed at the wrong boundary. The patch would ship without reducing the incidence of the original symptom.

This trap is particularly insidious because the hypothesis about "why" was correct (idle timeout somewhere) and the fix category was correct (keepalive). Only the _layer_ was wrong.

Counter-move: For every proposed fix, explicitly name which layer boundary it operates on, and then check whether that boundary is the one where the problem originates. If they do not match, the fix is targeting the wrong thing regardless of how reasonable it looks in isolation.

Phrased as a question: "My fix makes bytes flow at boundary X. Is X the same as the boundary where the problem manifests?"

In the SKILL.md workflow, this is the Step-2 third-question prompt. Do it before writing code.

Trap 12: Reverse-path / directional asymmetry

A→B healthy does not imply B→A healthy. Network paths are routinely asymmetric — forward and return routes differ, and congestion or interference on one direction is invisible from the other. Probing from the wrong end (or from only one end) systematically misses the failing direction.

Why it is seductive: a probe from a clean external vantage point (a cloud server in another region) to the suspect hop returns perfect numbers, and that feels like proof the hop is healthy. But that probe traversed the _return_ leg (or an entirely different path) — not the direction the user's traffic actually fails on.

Example (anonymized from a cross-border proxy investigation): user traffic home → relay → exit → site degraded badly at peak hours. To "prove the relay and exit nodes were healthy", the investigator drove probes _from an overseas server_ to those nodes and got a perfect score (30/30). The conclusion "the nodes are fine, so the fault is purely the user's last mile" was wrong: overseas→node is the lightly-loaded _inbound/return_ direction; the failing direction was the user's _outbound_ leg into those nodes, which the overseas probe never touched. In many networks the congested direction is structurally the one an external probe cannot reach — only in-country vantage points measure it. The 30/30 "proof" had zero bearing on the failing direction.

Counter-move: measure the _same direction the user's traffic flows, from the user's side_, before declaring a hop healthy. A clean external probe proves only that hop's externally-facing/return path — label it as such, never generalize it to "the hop is healthy". For directional confirmation, run TCP-mode mtr/nexttrace from the affected origin toward the target (not ICMP — see Trap 5 and the ICMP caveat) and read where loss first appears; or, if you must use a remote vantage point, deliberately point it at the _return_ leg (traffic toward the affected origin), not the outbound leg.

This is the directional sibling of Trap 5 (probe self-verification): Trap 5 is about the probe being structurally independent of the subject; this one is about the probe traversing the _same direction_ as the failure. Both fail identically — the measurement does not cover the thing it claims to.

Summary: the meta-move

All of these traps share a common structure: the investigator is willing to act on indirect evidence when a cheap direct test is available but was skipped.

The universal counter-move, restated:

Before acting on a conclusion, identify the cheapest direct test that could falsify it. Run that test.

If the test is expensive, accept the conclusion is provisional and design instrumentation that will make the test cheap next time.

Counter-Review Pattern

Contents

• Why counter-review (not peer-review)
• The four-agent team composition
• The four-question filter
• Integration workflow
• When NOT to counter-review
• The case study: what counter-review surfaced

Why counter-review

A lone investigator converging on a conclusion is highly susceptible to confirmation bias, especially after investing hours in a line of reasoning. Standard peer review is better but shares most of the investigator's context and inherits the same blind spots.

Counter-review is adversarial by design: the reviewer's job is to _falsify_ the conclusion, not confirm it. They start from the same evidence but are explicitly instructed to find what the investigator missed, find weaker-evidence conclusions the investigator over-weighted, and propose experiments that could disprove the current hypothesis.

Counter-review works best when:

• Multiple reviewers run in parallel with distinct framings
• Each reviewer has their own search/research capability (not just re-reading the same investigator's notes)
• Their outputs are filtered before acting, not accepted wholesale

The four-agent team composition

This composition was used in this investigation and proved effective. Roles are distinct on purpose — they cover orthogonal angles.

1. Independent diagnostician

Prompt framing: "You have the complete evidence set. Reach your own conclusion without being anchored by mine. Especially: what hypotheses did I not consider?"

Typical value: surfaces the "I forgot to check X" class of gap. In this case study, this agent was the first to raise Caddy's IdleConnTimeout default as a suspect — a layer the main investigator had not yet examined.

Agent type: general-purpose with SSH/Bash access

2. Assumption challenger

Prompt framing: "The main conclusion is X. Challenge it using external research (WebSearch authoritative sources, vendor docs, bug trackers). Cite URLs. Do not rely on training data."

Typical value: finds vendor-documented behavior that contradicts or qualifies the investigator's assumption. In this case study, this agent found published evidence that Anthropic's own official API also experiences SSE stalls during tool_use generation — downgrading the "<upstream-provider> does not forward ping" hypothesis from "root cause" to "amplifying factor".

Agent type: general-purpose with WebSearch

3. Code reviewer (if a fix is proposed)

Prompt framing: "The proposed fix is this [diff]. Audit for: race conditions, cleanup paths, unintended interactions with existing code, boundary conditions. Read the surrounding code."

Typical value: catches "my fix would break the JSON response path" class of bug. In this case study, this agent caught that a proposed setInterval keepalive would corrupt non-streaming Anthropic JSON responses because res.write before writeHead triggers Node's implicit-header emission — a subtle bug the main investigator would likely have shipped.

Agent type: Plan agent or codex:rescue with code-read access

4. Decisive experiment designer

Prompt framing: "Design and execute an experiment that decisively confirms or refutes the current root cause. Layered isolation preferred. Clean up after yourself."

Typical value: converts hypothesis-stacking into definitive answer. In this case study, this agent set up a mock idle upstream, deployed temporary CF DNS and Caddy routes, ran 3 parallel paths, observed 126s RST only on Path A, and cleaned everything up. What 5 hours of reasoning could not resolve, 10 minutes of experiment did.

Agent type: general-purpose with Bash/SSH/file access

Launch pattern

Launch all four in parallel (one message, four Agent tool calls with run_in_background: true). Process notifications as they return; do not wait for all before starting to read. Some will finish in minutes, the experiment designer often takes longer.

The four-question filter

Agent reviewers over-produce findings. A code reviewer will generate 10 risk items even when 3 are worth acting on; a challenger will list 6 counter-hypotheses even when 1 is plausible. Paste-the-raw-agent-output is the anti-pattern. Filter every finding through four questions:

1. Probability — will this actually happen?

Distinguish real risks from theoretical risks. A race condition in a code path that runs once at startup on a single thread is theoretical. A race condition in a request handler under load is real.

Ask: in the actual deployment, under actual load patterns, with actual input distributions, does this failure mode fire with >1% probability? If not, defer.

2. Cost — what is the cost of fixing versus ignoring?

For each finding:

• Cost of fixing: engineering time + regression risk + complexity added
• Cost of ignoring: expected incidents × their impact

Some findings are real but cheap to accept (log a warning, move on). Others are real and cheap to fix (one-line guard). Prioritize the second.

3. Realistic scenario — does this apply to the user's actual business case?

Agents generalize. A counter-review of an internal-only tool often surfaces findings appropriate for a consumer product (rate limiting, CSRF, input fuzzing). Filter to the actual deployment context.

4. Verification — can I cheaply confirm or refute this?

For findings that survive 1-3, can you test the claim in under 5 minutes? If yes, test. If the test comes back negative, discard. If positive, elevate to actionable.

Never accept a finding as real without at least one cheap verification.

Classification after filtering

Classify each finding:

• Real issue — act on it
• Partly right — acknowledge and narrow scope before acting
• Unlikely — log but do not act
• Actively harmful — the suggested fix would introduce a new bug; explicitly reject

Report to the user with classification, not raw agent output.

Integration workflow

┌────────────────────────────────────────────────────────┐
│ 1. Main investigator reaches tentative conclusion      │
│    and draft remediation                               │
├────────────────────────────────────────────────────────┤
│ 2. Launch 4 counter-review agents in parallel,         │
│    one message, run_in_background=true                 │
├────────────────────────────────────────────────────────┤
│ 3. As each returns, read full output                   │
├────────────────────────────────────────────────────────┤
│ 4. Apply 4-question filter to every finding            │
│    - probability, cost, realism, verifiability         │
├────────────────────────────────────────────────────────┤
│ 5. For every finding that survives filter,             │
│    run cheap verification                              │
├────────────────────────────────────────────────────────┤
│ 6. Reclassify findings after verification              │
│    (real / partly / unlikely / harmful)                │
├────────────────────────────────────────────────────────┤
│ 7. Update root cause / fix based on classified         │
│    findings                                            │
├────────────────────────────────────────────────────────┤
│ 8. Report to user: classification table + justified    │
│    action list. No raw paste.                          │
└────────────────────────────────────────────────────────┘

When NOT to counter-review

Counter-review has overhead (5-30 minutes of parallel agent runs + filter time). Skip it when:

• The root cause is already directly verified (you have the smoking gun, not circumstantial evidence)
• The fix is mechanical (e.g., updating a hardcoded version number) with no design decisions
• The incident is ongoing and the priority is stabilization, not perfect root cause
• The cost of getting it 80% right and iterating is lower than the cost of getting it 100% right the first time

In other words: counter-review is for the conclusion phase, not the stabilization phase.

The case study: what counter-review surfaced

Main investigator's tentative conclusion before counter-review: "<upstream-provider> does not forward SSE ping, causing some middle layer to RST after ~130s. Fix: add server-side keepalive in <provider-gateway-service>."

Counter-review surfaced:

Raw count: 6 findings surfaced. Acted on: 3 (2 code fixes, 1 definitive experiment result). Discarded: 1 (wrong). Downgraded but kept as context: 2.

If the main investigator had pasted all six to the user as "here's what counter-review said", the user would have had to do the filtering work. The filter is the investigator's job.

Instrumentation Patterns

Contents

• When to instrument
• Env-gated TRACE pattern (the default)
• Log tag conventions
• Deployment checklist
• Worked example: TRACE_SSE_CHUNKS
• Analysis: extracting timing data from logs
• Persisting instrumentation versus removing it

When to instrument

Instrument when a hypothesis cannot be confirmed or refuted from currently-available observability. If the system already emits the signal you need (distributed trace, access log field, metric), use that. If not, add instrumentation rather than guess.

Symptoms that justify adding instrumentation:

• "We do not know what the upstream is doing between these timestamps" — add chunk-level or event-level logging at the boundary
• "We think the client side disconnects but have no proof" — log client-close events on the server
• "The fix might not actually be triggering" — log entry/exit at the new code path

Do not instrument for symptoms that already have direct evidence. If tcpdump shows the RST, you do not need a new log line to confirm it.

Env-gated TRACE pattern (the default)

Instrumentation added mid-incident tends to become tech debt. The right pattern makes it permanent but invisible:

1. Defaults off. Zero runtime cost in steady state. No risk to enable-by-default performance. 2. One environment variable toggles it. No code changes required to enable in production. 3. Greppable log tag. Single bracketed prefix (e.g., [SSE-CHUNK]) makes every emission easy to filter. 4. Structured, key=value output. ts=... req=... bytes=... total=... parses into a DataFrame in three lines of Python. 5. Ships into production permanently. Future incidents reuse the same knob without re-adding code.

Template (Node.js)

// Near config section
const TRACE_SSE_CHUNKS =
  (process.env.TRACE_SSE_CHUNKS || "false").toLowerCase() === "true";
if (TRACE_SSE_CHUNKS) console.log("[SSE-CHUNK] instrumentation ENABLED");

// At the observation point
proxyRes.on("data", (chunk) => {
  if (TRACE_SSE_CHUNKS && isAnthropicMessagesPath && isStreaming) {
    const reqId =
      (proxyRes.headers && proxyRes.headers["x-oneapi-request-id"]) || "n/a";
    const total = chunks.reduce((a, c) => a + c.length, 0);
    console.log(
      "[SSE-CHUNK] ts=" +
        Date.now() +
        " req=" +
        reqId +
        " bytes=" +
        chunk.length +
        " total=" +
        total,
    );
  }
  // ... existing logic untouched
});

Template (Python)

import os, time, logging
TRACE_SSE_CHUNKS = os.environ.get('TRACE_SSE_CHUNKS', '').lower() == 'true'
if TRACE_SSE_CHUNKS:
    logging.info('[SSE-CHUNK] instrumentation ENABLED')

# At the observation point
def on_chunk(chunk, req_id, running_total):
    if TRACE_SSE_CHUNKS:
        logging.info(
            f'[SSE-CHUNK] ts={int(time.time()*1000)} '
            f'req={req_id} bytes={len(chunk)} total={running_total}'
        )

Enabling in a containerized deployment

# Edit docker-compose env or apply shell env then recreate:
TRACE_SSE_CHUNKS=true docker compose up -d <service>

# Or in Kubernetes:
kubectl set env deployment/<name> TRACE_SSE_CHUNKS=true

Disabling is the inverse — unset the variable and restart. No code change, no git commit, no deploy pipeline.

Log tag conventions

Pick tags that are unlikely to false-match other logs. A good tag:

• Starts with a bracket to survive grep
• Uses SCREAMING-KEBAB-CASE for visual distinction from regular logs
• Is specific enough to identify the instrumentation site, not just the subsystem

Good: [SSE-CHUNK], [UPSTREAM-CONNECT-RTT], [CLIENT-DISCONNECT] Bad: [DEBUG], [TRACE], log.info("chunk arrived")

Pair an ENABLED log line with the toggle so the presence of instrumentation is visible at service start — no guessing whether it actually took effect.

Deployment checklist

When adding instrumentation to a running system:

• [ ] The gate defaults off (confirmed by reading the code — do not trust the comment)
• [ ] The gate reads from env at startup (warn the user that changes require restart, not a runtime-reload)
• [ ] The output is structured (key=value) — no prose log messages
• [ ] The tag is unique (grep the codebase for conflicts)
• [ ] Sampling or volume cap if high-frequency (e.g., per-chunk logs on a 1000-rps service need either sampling or a max-size file sink)
• [ ] An ENABLED banner line is emitted at startup when the gate is on
• [ ] The change is committed to source of truth (not only applied to the running server — see the IaC trap below)

Worked example: TRACE_SSE_CHUNKS

Real artifact from this investigation. Goal: observe the upstream chunk arrival pattern to confirm/refute the hypothesis "<upstream-provider> batches chunks and goes silent for >120s during tool_use generation".

Before instrumentation: the only available signal was aggregate duration_ms in the archive metadata. This told us the request took 315 seconds total but said nothing about _when_ within those 315s bytes flowed.

After instrumentation (10 lines added):

[SSE-CHUNK] ts=1776870300212 req=202604221504562... bytes=128 total=1993
[SSE-CHUNK] ts=1776870300213 req=202604221504562... bytes=131 total=2124
[SSE-CHUNK] ts=1776870300213 req=202604221504562... bytes=127 total=2251
...
[SSE-CHUNK] ts=1776870300627 req=202604221504562... bytes=128 total=5583
... (30 chunks over 1.2 seconds, then silence)
[SSE-CHUNK] ts=1776870425235 req=202604221504562... bytes=74 total=3865

Extracted: 30 chunks in the first 1.2 seconds (3791 bytes total), then a 125-second gap with zero bytes, then 74 more bytes. The hypothesis was confirmed: <upstream-provider> emits the beginning of the response in a burst, then stays silent for over 2 minutes while the model generates the tool_use arguments internally.

Without the instrumentation, this would have been invisible. With 10 lines of code gated on one env var, it became a permanent observability capability.

Analysis: extracting timing data from logs

Once instrumentation is emitting structured logs, a few lines of Python turns log output into inter-arrival time analysis:

import sys, re
from collections import defaultdict

chunks = []
for line in sys.stdin:
    m = re.search(r'ts=(\d+) req=(\S+) bytes=(\d+) total=(\d+)', line)
    if m:
        ts, req, b, tot = m.groups()
        chunks.append((int(ts), req, int(b), int(tot)))

by_req = defaultdict(list)
for ts, req, b, tot in chunks:
    by_req[req].append((ts, b, tot))

for req, seq in by_req.items():
    if len(seq) < 2: continue
    span = seq[-1][0] - seq[0][0]
    big_gaps = [seq[i][0] - seq[i-1][0] for i in range(1, len(seq)) if seq[i][0] - seq[i-1][0] > 1000]
    print(f'req={req[:20]} chunks={len(seq)} span={span}ms gaps>1s={len(big_gaps)} max_gap={max(big_gaps) if big_gaps else 0}ms')

Pipe docker logs <container> | python analyze.py and you have a per-request latency histogram. A request with max_gap=125023ms jumps out immediately.

Persisting instrumentation versus removing it

Traditional wisdom says "remove debug logging after fix". That wisdom predates this pattern. With the env-gate approach, the correct default is:

Keep the instrumentation code. Leave the env toggle off. Document the toggle in an ops runbook.

Rationale:

• Adding instrumentation mid-incident under pressure is error-prone. Far better to have the gate already in place.
• Zero runtime cost when off.
• The env variable name is self-documenting.
• The next incident is cheaper.

The only time to remove instrumentation is when it has been superseded by better observability (e.g., you instrumented chunk timing, then later added full distributed tracing that subsumes it).

The IaC trap

If the service is deployed via Infrastructure-as-Code (Terraform, Ansible, Kubernetes manifests), instrumentation applied directly to the running instance (docker exec, live file edit) will be overwritten on the next deploy. The drift hides the real state and frustrates the next investigator.

Always apply the code change to the source of truth first:

1. Edit the source repo (e.g., js/service-name/server.js) 2. Commit and push, or at minimum sync to the deploy pipeline's input 3. Run the normal deploy to propagate 4. Only after that, enable the env toggle

If time-critical: apply directly to the running server _and_ to the source, in the same session. Never only to the running server.

Layered Isolation Experiment

Contents

• Why layered isolation
• The 3-path pattern
• Mock upstream pattern
• Result matrix and interpretation
• Failure modes and probe self-verification
• Extended variants (4+ paths, client-side variation)
• Canonical reference case (case-study SSE RST)

Why layered isolation

Multi-hop network systems concentrate bugs at the seams. A request from a user to a backend service typically traverses: client → ISP → CGNAT → CDN/edge → load balancer → reverse proxy → application → upstream dependency. Each hop introduces a timeout policy, a connection pool, a rewrite rule, a header translation, or a flow-control window. When something fails, hypothesis-stacking ("maybe it's the CDN, no maybe the LB, actually probably the app…") tends to burn hours with circumstantial evidence on each candidate.

Layered isolation inverts the approach: instead of reasoning about which hop caused the symptom, run the same logical request through several paths that differ by exactly one hop, then observe where the symptom appears. The differential directly names the responsible layer.

The 3-path pattern

For a CDN-fronted service with the topology Client → CDN → LB → Origin:

Interpretation:

• If A fails, B passes, C passes: the CDN is the cause
• If A fails, B fails, C passes: the LB / origin external network path is the cause
• If A fails, B fails, C fails: the cause is in the application or upstream dependency (hypothesis-stacking was wrong from the start)
• If all three pass: the failure condition was not actually reproduced; re-examine the assumed trigger

Add a fourth path as needed — e.g., bypass only the LB by hitting the origin VM's private IP from within the VPC, or test from a different client geography if ISP/CGNAT is suspect.

Mock upstream pattern

The experiment needs a way to reliably and repeatably trigger the failure condition. For idle-timeout symptoms (the most common class addressed by this skill), the cleanest trigger is a mock upstream that emits one response header + one data frame, then goes silent for a controlled duration.

See scripts/mock-idle-upstream.py for a runnable Flask implementation. The essence:

def gen():
    yield b'event: message_start\ndata: {"type":"message_start"}\n\n'
    time.sleep(IDLE_SECONDS)  # configurable, e.g. 200
    yield b'event: message_stop\ndata: {"type":"message_stop"}\n\n'

Why this is better than using a real long-tail production request to trigger the failure:

• Controlled timing: 200-second idle is a knob; a real <model-name> request has variable thinking duration
• Cheap: no model inference cost
• Reproducible: deterministic bytes, deterministic timing
• Isolated from app bugs: rules out "maybe the app itself has a bug"
• Safe: does not consume user quota or affect real traffic

Deploy the mock on a port the reverse proxy can reach, add a temporary route in the proxy config pointing a test hostname/path to it, and run the 3 paths against that hostname.

Result matrix and interpretation

After running the experiment, tabulate:

              | Path A (via CDN) | Path B (bypass CDN) | Path C (loopback) |
--------------|------------------|---------------------|-------------------|
Result        | RST @ 126s       | Clean @ 220s        | Clean @ 220s      |
Observed by   | curl + server    | curl + server       | curl + server     |

Always record observations from both ends (curl-side time_total AND server-side peer-close timestamp). Discrepancies between the two are themselves diagnostic: if curl reports a close at 69s but the server saw the connection alive for 126s, the close happened in a middlebox between them.

The observed constant in the failing path (here: 126s) is usually close to a known layer's default idle policy. Cross-reference against:

Do not treat this table as authoritative for your environment. These are starting points to cross-check against your measured constant. Confirm via vendor documentation or direct testing before citing as cause.

Failure modes and probe self-verification

The experiment is only as valid as its isolation. Common ways to poison the result:

Local proxy contamination

If the client has a system-level HTTP proxy (Shadowrocket, corporate proxy, VPN client), curl will silently route through it even when --resolve is set. Path B (intended to bypass CDN) gets routed through the same proxy and the isolation fails.

Mitigation: use env -i curl to strip the environment before running the probe, and explicitly unset http_proxy / https_proxy / HTTP_PROXY / HTTPS_PROXY / ALL_PROXY / NO_PROXY. Verify the path with curl -v and check the * Trying IP:port line matches the expected target.

Real example from this case study: run 1 Path B appeared to fail at 126s, which would have falsely implicated the origin. It turned out the client's Shadowrocket was proxying localhost-targeted requests back through Cloudflare. env -i curl --resolve ... reproduced the clean 220s that correctly exonerated the origin.

Probe self-verification

If the probe depends on the infrastructure being tested, its output is not independent evidence. Example: running mtr through the CDN to test CDN behavior — if the CDN drops ICMP, mtr shows gaps that look like the symptom but are artifacts. Always compare against at least one structurally different probe (e.g., curl + server-side tcpdump alongside mtr).

Container network namespace differences

When the target runs in Docker, Path C (loopback) behavior differs depending on whether you run curl from the host or from inside the container. Host curl localhost:3002 may hit a port-mapped container, while container-internal curl localhost:3002 hits the service directly. They are different isolation paths — pick based on which hops you are trying to include/exclude and be explicit about which one you ran.

Extended variants

4-path with client-side variation

A: client via CDN via LB
B: different client (mobile hotspot) via CDN via LB   # ISP/CGNAT differential
C: client --resolve to origin IP (bypass CDN only)
D: server loopback

Use when Path A fails and you suspect client-side network (ISP/VPN/NAT) is the cause.

Time-of-day variant

For symptoms correlated with time (load, scheduled jobs), run the same matrix at a known-good window and a known-failing window. Compare.

Observability variant

For each path, record at minimum: HTTP status code, total elapsed time, bytes received, close reason (if available from curl or tcpdump). Paths that all return "success" but with vastly different byte counts hint at partial-response truncation rather than a clean failure/success dichotomy.

Canonical reference case

The 130s RST incident (documented in case-sse-rst-130s.md) ran exactly this 3-path matrix after 5 hours of hypothesis-stacking failed to converge. The result:

Interpretation was immediate: the RST comes from Cloudflare's edge, not the origin or any network in between. What 5 hours of circumstantial reasoning could not resolve (Caddy? <upstream-provider>? VPN? CGNAT? CLI bug?), the 3-path experiment resolved in the 10 minutes it took to set up.

The lesson: when multiple layers could be the cause, do not reason about which one — test.

Packet Capture Recipes

Contents

• When to capture packets
• Interface selection on Docker hosts
• Essential filters for RST isolation
• HTTP/2 specifics (stream RST is not TCP RST)
• Correlating pcap with application logs
• Common pitfalls

When to capture packets

Reach for tcpdump when the question is at Layer 3-4 and application logs cannot answer it:

• Who sent the RST? (application does not see the peer's RST as it is OS-delivered to the socket)
• Was this a TCP-level reset or an HTTP/2 stream-level reset?
• Did the server send a FIN or was the connection torn down mid-response?
• What was the exact byte on the wire at the time of close?

If the application already tells you "client disconnected at T", you do not need pcap for that.

Interface selection on Docker hosts

Container traffic traverses multiple interfaces on a Docker host. Picking the wrong interface shows only part of the story.

Typical layout:

Use tcpdump -i any to capture across all interfaces, at the cost of higher volume. For specific scoping:

• Capture between a container and an upstream (e.g., origin to Cloudflare): tcpdump -i eth0 host <upstream-ip>
• Capture inside a compose network (container-to-container): tcpdump -i br-<hash> (get the hash via docker network ls)
• Capture only one container's veth: docker exec <container> ip route to find its IP, then tcpdump -i any host <container-ip>

Essential filters for RST isolation

The goal is usually: "find RST packets relevant to my incident, ignore everything else".

All TCP RST packets

tcpdump -i any -nn 'tcp[tcpflags] & tcp-rst != 0'

Note the != 0 — not == tcp-rst — because RST can be combined with ACK (RST-ACK packets).

RST scoped to a target port

tcpdump -i any -nn '(tcp[tcpflags] & tcp-rst != 0) and (port 443 or port 80)'

Watch the operator precedence in compound filters — always parenthesize. A naive tcp-rst != 0 and port 443 or port 80 will match port 80 without the RST constraint.

RST to/from a specific IP

tcpdump -i any -nn '(tcp[tcpflags] & tcp-rst != 0) and host <ip>'

With write to file for later analysis

tcpdump -i any -s 0 -w /tmp/capture.pcap 'host <target> and port 443'
# ... reproduce the incident ...
# Then:
tcpdump -r /tmp/capture.pcap -nn | head -50

-s 0 captures full packets (default truncates to 68 bytes for performance).

Ring-buffer capture for long-running collections

tcpdump -i any -w /tmp/cap-%Y%m%d-%H%M%S.pcap -G 60 -W 10 'host <target>'

60-second files, 10-file rotation, self-cleaning. Safe to leave running overnight.

HTTP/2 specifics

A key trap: HTTP/2 has its own RST mechanism at the stream level (RST_STREAM frame) that is unrelated to TCP-level RST. The two failure modes look different:

The case study was the second kind: tcpdump showed no TCP RST on the client→origin path but curl reported HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2). The reset was a peer-initiated HTTP/2 RST_STREAM, sent as a data frame on a connection that otherwise stayed open for other streams.

Decoding HTTP/2 with tshark

tcpdump alone cannot show HTTP/2 frames because they are TLS-encrypted. If you control both endpoints, you can:

1. Export the TLS session keys via SSLKEYLOGFILE=/tmp/keylog.log curl ... 2. Open the pcap in Wireshark with the keylog to decrypt 3. Filter: http2.type == 3 (RST_STREAM frames)

Alternatively, for internal services where you can intercept before TLS:

tshark -i any -f 'host <target>' -Y 'http2.type == 3'  # requires plaintext or pre-TLS interception

In most production debugging, the HTTP/2 error code is observable from the client-side log (curl, browser devtools Network tab "Status", Node SDK error message) without needing to decrypt pcap.

Correlating pcap with application logs

Tie pcap to application activity via the request identifier:

1. Log the request ID server-side at request start (e.g., [REQ-START] req=abc123 src=1.2.3.4:54321 ts=...) 2. Capture pcap with source IP and port filter 3. Cross-reference: the pcap flow on (1.2.3.4:54321 ↔ server:443) maps to application log entries for req=abc123

This resolves ambiguities like "which of the 20 concurrent connections is the one that failed?"

Common pitfalls

Wrong filter syntax leading to silent over-capture

tcp[tcpflags] & tcp-rst != 0 and port 443 or port 3002 without parentheses evaluates as (tcp-rst != 0 and port 443) or port 3002, capturing all traffic on port 3002 regardless of RST. The resulting pcap contains thousands of packets the investigator did not intend to capture, and the actual RSTs are drowned out.

Fix: always parenthesize compound filters — (tcp[tcpflags] & tcp-rst != 0) and (port 443 or port 3002).

Capturing nothing because of interface mismatch

tcpdump -i eth0 'host 10.0.0.5' on a Docker host where the target is only reachable via br-abc123 captures nothing. Symptom: "I ran tcpdump but got zero packets even though traffic is flowing."

Fix: use -i any first to verify, then narrow once you see traffic.

Confusing timestamps across tools

tcpdump timestamps are in the local timezone of the capture host. Application logs may be in UTC, or in a different timezone. When correlating, convert to a single timezone before comparing timestamps — use epoch seconds if unsure.

Missing the RST because it happened before capture started

tcpdump only captures from the moment it starts. If the RST already happened, there is no going back. The fix is to start capture _before_ reproducing the incident (or to leave a ring-buffer capture running in advance for known-intermittent issues).

Forgetting snaplen

Default -s 68 (or -s 262144 on modern systems depending on version) may truncate large frames. Use -s 0 to capture full packets if you intend to inspect payload bytes. For RST-only analysis, the default is fine (RST is a small packet).