Resilience Review

Resilience Review

Evaluate how your application behaves when things go wrong — network failures, API errors, slow connections, missing data, and edge cases. Most apps are built for the happy path; this review systematically tests the unhappy paths that real users encounter.

When to use

Use /shiplight:resilience-review when:

• Before launching a user-facing feature
• After adding new API integrations or data sources
• When reliability is critical (healthcare, finance, e-commerce checkout)
• After production incidents caused by unhandled errors
• When moving from prototype to production quality

Standards Referenced

• Google SRE Principles — Error budgets, graceful degradation
• Netflix Chaos Engineering Principles — Verify steady state, inject real-world failures
• OWASP Error Handling — Secure and user-friendly error responses
• Nielsen Norman Group — Error message usability heuristics

Phase Overview

Phase 1: EDUCATE   → Why resilience matters and what we test
Phase 2: SCOPE     → Map failure points, dependencies, critical flows
Phase 3: ANALYZE   → Browser-based fault injection and edge case testing
Phase 4: REPORT    → Findings with evidence and user impact assessment
Phase 5: REMEDIATE → Fix guidance + YAML regression tests

---

Phase 1: Educate

Why this matters: Users don't experience your app in ideal conditions. 53% of mobile visits are abandoned if a page takes >3 seconds. Error pages with no guidance increase support tickets 5x. A blank screen is the worst possible failure mode — it tells the user nothing and offers no recovery path. Resilient apps maintain trust even when backend systems fail.

This review simulates real-world failure conditions in the browser and evaluates how your UI responds.

---

Phase 2: Scope

Gather context

1. Auto-detect from codebase:

• API calls and their endpoints
• Error boundary components (React ErrorBoundary, Vue errorHandler)
• Loading state implementations (spinners, skeletons, suspense)
• Empty state components
• Retry logic / error recovery patterns
• Offline support (service workers, cache strategies)
• Third-party service dependencies

2. Ask the user (one at a time):

• Target URL: Where is the app running?
• Critical user flows: Which flows must never show a blank screen? (auto-detect from routes)
• Key API dependencies: Which APIs does the frontend depend on? (auto-detected)
• Known fragile areas: Any pages/features that break frequently? (optional)

3. Map failure points:

• API endpoints the frontend calls (and what happens if each fails)
• Third-party dependencies (CDN, auth provider, analytics, maps, payment)
• Data-dependent UI (what shows when data is empty, missing, or malformed)
• User input edge cases (long text, special characters, empty submissions)

---

Phase 3: Analyze

Open a browser session with new_session using record_evidence: true. Run all applicable check categories.

Category A: Error Handling (ERR)

Browser validation: Use CODE blocks to intercept network requests via page.route() to simulate failures. Check UI state after each failure. Use get_browser_console_logs for JavaScript errors.

// Example: Mock API 500 error
await page.route('**/api/**', route => {
  route.fulfill({ status: 500, body: JSON.stringify({ error: 'Internal Server Error' }) });
});

Category B: Graceful Degradation (DEG)

Browser validation: Use page.route() to block specific resources. Use CDP to simulate network conditions. Disable JavaScript via browser settings. Verify each degradation scenario.

Category C: Empty & Edge States (EDGE)

Browser validation: Navigate to pages and test each edge case. Use act to interact with forms, submit empty/extreme data. Use JavaScript to check for UI overflow, frozen states.

Category D: API Contract & Data Handling (API)

Browser validation: Use page.route() to mock each response scenario. Verify UI state after each mock.

Category E: Recovery & User Communication (REC)

Browser validation: Use fault injection then verify recovery paths.

---

Phase 4: Report

Generate a structured report saved to shiplight/reports/resilience-review-{date}.md:

# Resilience Review Report
**Date:** {date}
**URL:** {url}
**Critical flows tested:** {list}
**API dependencies tested:** {count}
**Failure scenarios simulated:** {count}

## Overall Score: {X}/10 | Confidence: {X}%

## Score Breakdown
| Category | Score | Findings |
|----------|-------|----------|
| Error Handling (ERR) | 5/10 | 2 critical, 1 high |
| Graceful Degradation (DEG) | 6/10 | 1 high, 2 medium |
| Empty & Edge States (EDGE) | 4/10 | 1 critical, 3 high |
| API Contract (API) | 7/10 | 1 high, 1 medium |
| Recovery (REC) | 3/10 | 2 high, 1 medium |

## Failure Matrix
| Failure Scenario | Expected Behavior | Actual Behavior | Status |
|-----------------|-------------------|-----------------|--------|
| API returns 500 | Error message + retry | Blank screen | FAIL |
| Network timeout | Loading → timeout message | Infinite spinner | FAIL |
| Empty data set | "No results" message | Blank page | FAIL |
| ... | | | |

## Findings
(structured findings with evidence, screenshots of failure states)

Confidence Scoring

• 90-100%: Fault injected and failure behavior verified in browser
• 70-89%: Code analysis shows missing error handling, not validated at runtime
• 50-69%: Pattern-based assessment (e.g., no error boundary detected)
• Below 50%: Don't report

---

Phase 5: Remediate

1. Fix guidance (example)

#### ERR-01: API error shows blank screen instead of error message
**Impact:** Users see empty page, think app is broken, leave
**File:** src/pages/Dashboard.tsx:45
**Current:** `const data = await fetch('/api/data').then(r => r.json())`
**Problem:** No error handling — fetch throws on network error, .json() throws on non-JSON response
**Fix:**
- Wrap in try/catch
- Add error state: `const [error, setError] = useState(null)`
- Render error UI with retry button
- Add React Error Boundary as fallback

2. YAML regression test

- name: err-01-api-error-shows-message
  description: Verify API failure shows user-friendly error message instead of blank screen
  severity: critical
  standard: UX-Error-Handling
  steps:
    - CODE: |
        await page.route('**/api/data**', route => {
          route.fulfill({
            status: 500,
            contentType: 'application/json',
            body: JSON.stringify({ error: 'Internal Server Error' })
          });
        });
    - URL: /dashboard
    - WAIT_UNTIL: Page has finished attempting to load data
      timeout_seconds: 15
    - VERIFY: An error message is visible explaining that data could not be loaded
    - VERIFY: A retry button or recovery action is available to the user
    - VERIFY: The page is NOT blank — navigation and header are still visible

Save all YAML tests to shiplight/tests/resilience-review.test.yaml.

---

Tips

• Use page.route() in CODE blocks — it's the primary tool for fault injection
• Test the most critical user flows first (checkout, signup, core feature)
• A blank screen is always a CRITICAL finding — it's the worst failure mode
• Check get_browser_console_logs for uncaught promise rejections — they indicate missing error handling
• Edge case testing (EDGE category) often reveals the most bugs per minute spent
• Close session with close_session and use generate_html_report for evidence