Ci Debug Workflow

CI Debug Workflow

Debug failing CI pipelines, containers, and reproduce bugs locally.

Trigger Patterns

• "fix failing CI"
• "debug this pipeline"
• "why is CI red"
• "container won't start"
• "reproduce this bug"

Workflow

1. Gather Context

Read CI logs first. Identify:

• Which step failed
• Error message/stack trace
• Environment differences from local

For bug reports, extract reproduction steps. See bug-thread-extraction.md.

2. Reproduce Locally

Never fix blind. Reproduce failure before changing code:

# Run same commands CI runs
npm ci && npm test

# Or match CI environment
docker build -t debug-image .
docker run --rm debug-image npm test

3. Identify Root Cause

Common CI failure patterns in ci-patterns.md:

• Dependency version mismatches
• Missing environment variables
• Timing/race conditions
• Platform differences (Linux vs macOS)

Container issues in docker-debug.md.

4. Apply Fix

Fix the actual issue, not symptoms:

• Pin dependency versions explicitly
• Add missing env vars to CI config
• Fix flaky tests with proper waits
• Use platform-agnostic paths

5. Verify

Push fix and confirm CI passes. Do not mark done until green.

git push && gh run watch

Anti-patterns

• Rerunning CI hoping it passes
• Fixing locally without reproducing CI environment
• Disabling failing tests
• Adding broad || true to mask failures

References

• ci-patterns.md - Common CI failure patterns
• docker-debug.md - Container troubleshooting
• bug-thread-extraction.md - Parse bug reports

Bug Thread Extraction

Extract actionable reproduction steps from bug reports and threads.

Extraction Process

1. Identify Key Elements

Scan for:

• Steps to reproduce - numbered lists, "when I...", "after..."
• Expected vs actual - "should", "but instead", "expected"
• Environment - versions, OS, browser, config
• Error output - stack traces, logs, screenshots

2. Filter Noise

Skip:

• Unrelated discussion
• Duplicate reports
• Workarounds (note separately)
• Speculation without evidence

Focus on:

• Original report
• Confirmed reproductions
• Maintainer responses with clarification

3. Normalize Steps

Convert prose to concrete steps:

Before:

So I was trying to do X and then clicked the button and it broke

After:

1. Navigate to /settings
2. Click "Save" button
3. Observe error in console

Common Bug Report Sources

GitHub Issues

gh issue view <number>
gh issue view <number> --comments

CI Failure Threads

Look for:

• Failed step name
• Full error output
• Commit that introduced failure

Slack/Discord Threads

Extract:

• Initial problem statement
• Any shared logs/screenshots
• Resolution if found

Structured Output Template

## Summary

[One-line description]

## Environment

- OS:
- Version:
- Config:

## Steps to Reproduce

1.
2.
3.

## Expected

[What should happen]

## Actual

[What happens instead]

## Error Output

[paste logs/trace]

## Notes
- [Related issues]
- [Known workarounds]

Verification Checklist

Before fixing:

• [ ] Can reproduce locally
• [ ] Same error as reported
• [ ] Minimal reproduction (no extra steps)
• [ ] Environment matches report

Red Flags

Watch for:

• "Works on my machine" - environment delta
• "Sometimes fails" - race condition or flaky test
• "Stopped working after update" - check git blame
• Multiple users, different symptoms - multiple bugs

CI Failure Patterns

Dependency Issues

Version Mismatch

Error: Cannot find module 'foo'

Cause: package-lock.json out of sync or missing. Fix: Commit lockfile. Run npm ci not npm install.

Peer Dependency Conflicts

npm WARN peer dep missing

Fix: Install peer deps explicitly or use --legacy-peer-deps.

Cache Poisoning

Symptom: Works after cache clear. Fix: Invalidate CI cache. Add cache key versioning.

Environment Variables

Missing Secrets

Error: API_KEY is not defined

Fix: Add secret to CI settings. Check secret name spelling.

Wrong Environment

Symptom: Tests hit production API. Fix: Set NODE_ENV=test. Mock external services.

Timing Issues

Race Conditions

Symptom: Intermittent failures. Passes on retry. Fix: Add explicit waits. Use waitFor() in tests. Avoid sleep().

Timeouts

Error: Timeout of 5000ms exceeded

Fix: Increase timeout or fix slow operation. Check for hanging promises.

Platform Differences

Path Separators

Error: ENOENT /home/runner/work/foo\bar

Fix: Use path.join() not string concatenation.

Line Endings

Symptom: Git shows all files changed. Fix: Configure .gitattributes. Set core.autocrlf.

Case Sensitivity

Symptom: Works on macOS, fails on Linux. Fix: Match exact filename case in imports.

Resource Limits

Out of Memory

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed

Fix: Increase NODE_OPTIONS=--max-old-space-size=4096.

Disk Space

ENOSPC: no space left on device

Fix: Clean artifacts between steps. Reduce build output.

Debugging Commands

# View CI environment
env | sort

# Check available resources
free -h
df -h

# Verify installed versions
node -v && npm -v

# Test network connectivity
curl -I https://registry.npmjs.org

Docker Debug Reference

Container Won't Start

Check Logs

docker logs <container>
docker logs --tail 100 -f <container>

Inspect Container

docker inspect <container>
docker inspect --format='{{.State.ExitCode}}' <container>

Common Exit Codes

• 0 - Success
• 1 - Application error
• 137 - OOM killed (128 + 9)
• 139 - Segfault (128 + 11)
• 143 - SIGTERM (128 + 15)

Build Failures

Cache Issues

# Rebuild without cache
docker build --no-cache -t myimage .

# Prune build cache
docker builder prune

Layer Debugging

# Build up to specific stage
docker build --target builder -t debug .

# Run shell in intermediate image
docker run -it --rm <image-id> /bin/sh

Runtime Issues

Shell Into Running Container

docker exec -it <container> /bin/sh
# or for bash
docker exec -it <container> /bin/bash

Shell Into Failed Container

# Commit failed container state
docker commit <container> debug-image
docker run -it --rm debug-image /bin/sh

Override Entrypoint

docker run -it --rm --entrypoint /bin/sh myimage

Resource Problems

Check Resource Usage

docker stats <container>
docker system df

Memory Limits

# Run with more memory
docker run -m 2g myimage

# Check if OOM killed
docker inspect --format='{{.State.OOMKilled}}' <container>

Network Issues

Check Network

docker network ls
docker network inspect <network>

Test Connectivity

docker exec <container> curl -I http://other-service:8080
docker exec <container> nslookup other-service

Port Mapping

docker port <container>
docker run -p 8080:80 myimage

Volume Problems

Check Mounts

docker inspect --format='{{.Mounts}}' <container>

Permission Issues

# Check user inside container
docker exec <container> id
docker exec <container> ls -la /app

Docker Compose

View Combined Logs

docker compose logs -f
docker compose logs <service>

Rebuild Single Service

docker compose up -d --build <service>

Shell Into Service

docker compose exec <service> /bin/sh