Review

Review Orchestrator

The single entry point for application reviews. It triages what matters, then runs one or more domain reviews and merges them into a unified report. Each domain lives in references/<domain>.md and is loaded only when selected.

When to use

• User wants a review but isn't sure which kind
• Pre-launch readiness assessment
• Post-incident review planning
• A targeted request for one domain ("check my app's security", "review SEO")

Modes

• Triage (default, /review) — ask context questions, recommend a plan, run it.
• Full suite (/review --all) — run every applicable domain.
• Targeted (/review <domain>) — jump straight into one domain, skipping

triage. E.g. /review security, /review seo. Accepts an optional depth flag (--quick / --thorough).

Domains

Each row maps to a reference file. Load the file only when the domain is selected.

Shared conventions (phases, scoring, confidence, severity, output paths) live in references/report-format.md — every domain follows them.

Steps

1. Gather context

• Read the project: tech stack, framework, package.json, routes, components.
• Check git diff for recent changes.
• Look for existing reports in shiplight/reports/.
• Auto-detect compliance markers (HIPAA/PHI, PCI/payment fields, GDPR/cookie consent).

If invoked as /review <domain>, skip to step 4 for that domain.

2. Ask targeted questions (max 4)

One at a time, with auto-detected defaults:

1. What type of application? (SaaS, healthcare, fintech, e-commerce, internal tool, marketing site, API-only) 2. What triggered this review? (pre-launch, new feature, dependency update, security incident, audit prep, routine) 3. Any compliance requirements? (none, HIPAA, SOC2, PCI-DSS, GDPR, multiple) — auto-detect from codebase 4. Specific concerns? (open-ended, optional)

3. Generate review plan

Categorize each applicable domain as CRITICAL (must run), RECOMMENDED (meaningful value), or OPTIONAL (nice to have), with estimated depth (quick / standard / thorough).

SEO vs GEO prioritization by product type:

Present a decision matrix:

4. Execute

Ask: "Run all CRITICAL reviews now? [Y/n] Or pick specific ones."

For each selected domain, read `references/<domain>.md` and follow its five phases, applying references/report-format.md for scoring, severity, and output paths. Run domains sequentially; show a brief summary after each before moving on.

5. Unified report

After the selected domains complete, merge their per-domain reports into one, saved to shiplight/reports/review-{date}.md:

• Overall readiness score (0–10) and per-domain scores
• Top 5 findings across all domains, by severity
• Regression test summary (total YAML tests generated, in shiplight/tests/)

Tips

• Run /review before every major launch.
• /review <domain> is the fast path when you already know what you need.
• Reports accumulate in shiplight/reports/ — the orchestrator can show trends.
• YAML regression tests from reviews accumulate in shiplight/tests/.

Compliance Review

Evaluate your application against industry-specific regulatory requirements. This review translates dense compliance frameworks into concrete, testable technical checks — and validates them through browser-based testing. Non-compliance can result in severe fines, legal action, and loss of business.

When to use

Use this review when:

• Building applications for regulated industries (healthcare, finance, government)
• Preparing for a compliance audit (SOC 2, HIPAA, PCI-DSS)
• Adding payment processing or health data features
• Expanding to GDPR-regulated markets
• After infrastructure or architecture changes that affect data handling

Standards Referenced

• HIPAA — Health Insurance Portability and Accountability Act (Technical Safeguards §164.312)
• SOC 2 — Service Organization Control (Trust Service Criteria)
• PCI-DSS v4.0 — Payment Card Industry Data Security Standard
• GDPR — General Data Protection Regulation (Technical Measures)

Phase Overview

Phase 1: EDUCATE   → Compliance context and applicable frameworks
Phase 2: SCOPE     → Determine which frameworks apply, identify regulated data
Phase 3: ANALYZE   → Browser-based checks against framework requirements
Phase 4: REPORT    → Findings mapped to specific regulatory sections
Phase 5: REMEDIATE → Fix guidance + YAML regression tests for continuous compliance

---

Phase 1: Educate

Why this matters: HIPAA violations: up to $1.9M per violation category per year. PCI-DSS non-compliance: $5,000-$100,000/month in fines plus liability for breaches. SOC 2 failures: loss of enterprise customers who require it. GDPR: up to 4% of global annual revenue. These aren't theoretical — enforcement is active and increasing.

Compliance frameworks are large documents. This review extracts the technical requirements testable in a web application — not the organizational/procedural requirements (policies, training, vendor management) that require human process review.

---

Phase 2: Scope

Determine applicable frameworks

1. Auto-detect from codebase:

• Health data handling (HIPAA indicators: HL7, FHIR, patient records, PHI references)
• Payment processing (PCI-DSS indicators: Stripe, Braintree, credit card fields, payment forms)
• EU user data (GDPR indicators: consent banners, cookie notices, EU deployments)
• Audit logging (SOC 2 indicators: audit trail, event logging, access logs)

2. Ask the user:

• Which frameworks apply? (auto-detected, confirm)
• [ ] HIPAA — healthcare / protected health information
• [ ] SOC 2 — enterprise SaaS / customer data
• [ ] PCI-DSS — payment card data
• [ ] GDPR — EU personal data
• [ ] Other (specify)
• Target URL: Where is the app running?
• Regulated data types: What regulated data does the app handle? (auto-detected)
• Test credentials: Accounts with access to regulated data for testing

3. Map regulated data flows:

• Where regulated data enters the system (forms, APIs, imports)
• Where it's displayed (dashboards, reports, exports)
• Where it's stored client-side (if anywhere)
• Where it's transmitted (API endpoints, third-party services)

---

Phase 3: Analyze

Run only the sections applicable based on Phase 2 scoping. Open a browser session with new_session using record_evidence: true.

HIPAA Technical Safeguards (HIP)

Applicable when: application handles Protected Health Information (PHI).

Browser validation: Navigate to pages with PHI. Test access controls. Check for PHI in URLs, storage, console. Test session timeout by waiting. Test logout completeness.

SOC 2 Trust Service Criteria (SOC)

Applicable when: enterprise SaaS handling customer data.

Browser validation: Test authentication boundaries, password policies, MFA flows, role-based access, audit logging visibility.

PCI-DSS v4.0 (PCI)

Applicable when: application processes, stores, or transmits cardholder data.

Browser validation: Walk through the payment flow. Check card display masking. Inspect storage and console for cardholder data. Verify payment form is iframe/hosted (SAQ A compliance).

GDPR Technical Requirements (GDPR)

Applicable when: application handles EU personal data. (Note: privacy-specific checks are in the privacy review — this section covers GDPR's technical/compliance obligations.)

Browser validation: Test consent flows, data export, account deletion, privacy policy accessibility. Check third-party script origins for cross-border transfer concerns.

---

Phase 4: Report

Generate a structured report saved to shiplight/reports/compliance-review-{date}.md:

# Compliance Review Report
**Date:** {date}
**URL:** {url}
**Frameworks evaluated:** {HIPAA, SOC 2, PCI-DSS, GDPR}
**Regulated data types:** {PHI, cardholder data, EU personal data}

## Overall Compliance Score: {X}/10 | Confidence: {X}%

## Framework Scores
| Framework | Score | Pass | Fail | N/A | Critical Gaps |
|-----------|-------|------|------|-----|---------------|
| HIPAA | 6/10 | 8 | 3 | 1 | Session timeout, PHI in URL |
| SOC 2 | 7/10 | 9 | 2 | 1 | No MFA, weak audit trail |
| PCI-DSS | 8/10 | 10 | 1 | 1 | Card data in console |
| GDPR | 5/10 | 5 | 4 | 1 | Consent, data export, erasure |

## Compliance Status by Check
(Full table of all checks with PASS/FAIL/N-A status, evidence, and confidence)

## Critical Non-Compliance Items
(Findings that could result in regulatory action, ordered by risk)

## Audit Preparation Checklist
- [ ] Fix all CRITICAL findings
- [ ] Fix all HIGH findings
- [ ] Document accepted risks for MEDIUM findings
- [ ] Run YAML regression tests before audit date
- [ ] Prepare evidence documentation from this report

Confidence Scoring

• 90-100%: Browser-validated, compliance violation confirmed (e.g., PHI visible without auth, card number in console)
• 70-89%: Strong evidence from inspection (e.g., missing header, no timeout behavior)
• 50-69%: Architectural concern based on code patterns (e.g., audit logging might be incomplete)
• Below 50%: Don't report — compliance findings must be substantiated

---

Phase 5: Remediate

1. Fix guidance (example)

#### HIP-02: No session auto-timeout
**Regulation:** HIPAA §164.312(a)(2)(iii) — Automatic logoff
**Risk:** Unattended sessions with PHI visible
**Current:** Sessions persist indefinitely
**Fix:** Implement idle timeout (HIPAA recommends ≤15 minutes for PHI access)
- Add client-side idle detection (mouse, keyboard events)
- Server-side session expiry as backup
- Show warning dialog at 12 minutes
- Auto-logout and clear screen at 15 minutes

2. YAML regression test

- name: hip-02-session-auto-timeout
  description: Verify session auto-timeout for HIPAA compliance
  severity: critical
  standard: HIPAA-164.312(a)(2)(iii)
  steps:
    - URL: /login
    - intent: Log in with test credentials
      action: fill
      locator: "getByLabel('Email')"
      value: "test@example.com"
    - intent: Enter password
      action: fill
      locator: "getByLabel('Password')"
      value: "testpass123"
    - intent: Submit login form
      action: click
      locator: "getByRole('button', { name: 'Sign in' })"
    - WAIT_UNTIL: Dashboard with PHI is visible
      timeout_seconds: 15
    - VERIFY: Session timeout warning appears after inactivity period
      timeout_seconds: 900
    - VERIFY: User is automatically logged out after timeout expires
      timeout_seconds: 300

Save all YAML tests to shiplight/tests/compliance-review.test.yaml.

---

Depth Levels

• `--quick`: Critical checks only — authentication boundaries + data exposure. ~3 minutes.
• default: Full applicable framework. ~10-15 minutes.
• `--thorough`: All checks + multi-role testing + edge cases + documentation review. ~25-40 minutes.

Tips

• Run the compliance review specific to your framework: "run HIPAA checks only"
• Compliance requires evidence — use record_evidence: true and generate_html_report for audit documentation
• YAML regression tests from this review serve as continuous compliance monitoring
• This review covers technical requirements only — organizational requirements (policies, training) need human review
• For privacy-specific concerns, complement with the privacy review (/review privacy)
• For security-specific concerns, complement with the security review (/review security)
• Close session with close_session and use generate_html_report for evidence

Design Review

Evaluate your application's visual quality and usability against established design standards. This review catches issues that typically require a trained designer's eye — responsive breakpoints, accessibility compliance, visual hierarchy, spacing consistency, and internationalization readiness.

When to use

Use this review when:

• Shipping UI without a designer reviewing it
• Before a launch or demo
• After significant UI changes or redesigns
• Checking accessibility compliance (WCAG 2.1 AA)
• Validating responsive behavior across devices

Standards Referenced

• WCAG 2.1 Level AA — Web Content Accessibility Guidelines
• Material Design / Human Interface Guidelines — spacing, touch targets, typography scales
• ISO 9241-110 — Interaction principles (suitability, self-descriptiveness, conformity)
• APCA — Advanced Perceptual Contrast Algorithm (next-gen contrast)

Phase Overview

Phase 1: EDUCATE   → Brief context on what we check and why
Phase 2: SCOPE     → Identify pages, breakpoints, and focus areas
Phase 3: ANALYZE   → Browser-based checks with evidence capture
Phase 4: REPORT    → Findings with screenshots, scores, confidence
Phase 5: REMEDIATE → Fix guidance + YAML regression tests

---

Phase 1: Educate

Why this matters: 94% of first impressions are design-related. Poor visual quality erodes trust even when functionality is correct. Accessibility issues affect 15-20% of users and carry legal risk (ADA lawsuits increased 300% since 2018).

This review checks your app against objective, measurable design criteria — not subjective taste. Every finding references a specific standard.

---

Phase 2: Scope

Gather context

1. Auto-detect — scan the project for:

• Framework (React, Vue, Next.js, etc.)
• CSS approach (Tailwind, CSS modules, styled-components, etc.)
• Design system in use (if any)
• Route structure → list of pages
• Existing a11y tooling (eslint-plugin-jsx-a11y, axe-core, etc.)

2. Ask the user (one at a time, with auto-detected defaults):

• Target URL: Where is the app running? (auto-detect dev server)
• Key pages: Which pages matter most? (recommend top 3-5 from routes)
• Target devices: Desktop only? Mobile-first? Both? (default: both)
• Brand guidelines: Any specific colors, fonts, or design system? (default: evaluate against general best practices)
• Focus areas: Any known concerns? (optional)

3. Define breakpoints to test:

• Mobile: 375px (iPhone SE), 390px (iPhone 14)
• Tablet: 768px (iPad)
• Desktop: 1280px, 1920px
• (Adjust based on user's target audience)

---

Phase 3: Analyze

Open a browser session with new_session using record_evidence: true. For each page in scope, run the following check categories.

Category A: Responsive Design (RES)

Browser validation: For each breakpoint, use act to resize the viewport, then inspect_page to capture DOM and screenshot. Check for overflow elements, measure sizes via JavaScript.

Category B: Accessibility (A11Y)

Browser validation: Use inspect_page to extract the DOM. Run JavaScript via act to compute contrast ratios, check ARIA attributes, extract heading hierarchy. Use keyboard navigation (Tab, Enter, Escape) to test focus management.

Category C: Visual Consistency (VIS)

Browser validation: Use JavaScript to extract computed styles, compare across elements and pages. Screenshot comparison between pages for visual consistency.

Category D: Typography & Readability (TYP)

Category E: Internationalization Readiness (I18N)

Browser validation: Use JavaScript to modify dir attribute, inject longer text, change locale settings. Screenshot at each state.

---

Phase 4: Report

Generate a structured report saved to shiplight/reports/design-review-{date}.md:

# Design Review Report
**Date:** {date}
**URL:** {url}
**Pages reviewed:** {list}
**Breakpoints tested:** {list}

## Overall Score: {X}/10 | Confidence: {X}%

## Score Breakdown
| Category | Score | Findings |
|----------|-------|----------|
| Responsive (RES) | 7/10 | 2 high, 1 medium |
| Accessibility (A11Y) | 5/10 | 1 critical, 3 high |
| Visual Consistency (VIS) | 8/10 | 1 medium |
| Typography (TYP) | 9/10 | 1 low |
| i18n Readiness (I18N) | 6/10 | 2 medium |

## Findings

### CRITICAL

#### A11Y-01: Insufficient color contrast on primary buttons
- **Standard:** WCAG 1.4.3 AA (minimum 4.5:1)
- **Finding:** Primary button (#4A90D2 on #FFFFFF) has contrast ratio 3.1:1
- **Evidence:** [screenshot with annotation]
- **Pages affected:** All pages with primary CTA
- **Confidence:** 97%

### HIGH
...

### MEDIUM
...

### LOW / INFO
...

Confidence Scoring

• 90-100%: Browser-validated, measured programmatically (contrast ratio calculated, element size measured)
• 70-89%: Strong evidence from DOM inspection, screenshot supports finding
• 50-69%: Heuristic-based, may vary by context (e.g., "spacing looks inconsistent")
• Below 50%: Don't report

---

Phase 5: Remediate

For each finding, provide:

1. Fix guidance

#### A11Y-01: Insufficient color contrast
**File:** src/components/Button.tsx:23
**Current:** `background: #4A90D2` (contrast 3.1:1 against white)
**Fix:** `background: #2563EB` (contrast 4.8:1 against white) — maintains blue hue, meets AA
**Alternative:** `background: #1D4ED8` (contrast 7.1:1) — meets AAA

2. YAML regression test

- name: a11y-01-button-contrast
  description: Verify primary button meets WCAG AA contrast ratio
  severity: critical
  standard: WCAG-1.4.3-AA
  steps:
    - URL: /
    - VERIFY: Primary action buttons have sufficient color contrast (minimum 4.5:1 ratio for normal text)
      timeout_seconds: 15

Save all YAML tests to shiplight/tests/design-review.test.yaml.

---

Tips

• Use inspect_page to read the DOM first — it's cheaper than screenshots and provides element indices for act.
• For contrast checking, use JavaScript via act with window.getComputedStyle() to extract actual rendered colors.
• Test keyboard navigation by using act with keyboard actions (Tab, Enter, Escape, Arrow keys).
• Run this review at multiple breakpoints — many issues only appear at specific viewport sizes.
• For i18n text expansion testing, use act with JavaScript to modify textContent to longer strings.
• Close the session with close_session and use generate_html_report for a shareable evidence report.

GEO Review

Evaluate how well your application and content are optimized for AI-powered search and answer engines — ChatGPT, Perplexity, Google AI Overviews, Claude, and other generative AI systems that cite web sources. Traditional SEO gets you ranked in a link list; GEO gets you cited in AI-generated answers.

When to use

Use this review when:

• Your product is discovered through AI assistants (developer tools, SaaS, APIs)
• You want to appear in Google AI Overviews
• Users find your product by asking AI "what's the best X for Y?"
• You publish documentation, guides, or educational content
• Your competitors are showing up in AI answers and you're not
• Building thought leadership content that AI should reference
• Launching a new product where AI-driven discovery matters

Why GEO Matters Now

• 40% of Gen Z uses TikTok and AI chatbots instead of Google for search (Adobe 2024)
• Google AI Overviews now appear for ~30% of search queries, pushing traditional results below the fold
• Perplexity processes 100M+ queries/month, citing web sources in every answer
• ChatGPT with browsing and search is becoming a primary research tool
• AI systems don't rank links — they select and cite sources based on different signals than traditional SEO
• Being the source an AI quotes is the new "position #1"

Standards & Frameworks Referenced

• GEO research (Georgia Tech / Princeton / IIT Delhi, 2024) — "GEO: Generative Engine Optimization"
• Google E-E-A-T — Experience, Expertise, Authoritativeness, Trustworthiness
• Schema.org — Structured data for entity understanding
• llms.txt — Emerging standard for AI crawler instructions (similar to robots.txt for LLMs)
• Retrieval-Augmented Generation (RAG) — How AI systems fetch and cite content

Phase Overview

Phase 1: EDUCATE   → How AI search works differently from traditional search
Phase 2: SCOPE     → Identify content types, target queries, AI visibility goals
Phase 3: ANALYZE   → Content analysis + browser-based AI search validation
Phase 4: REPORT    → Findings with citation gap analysis and confidence scores
Phase 5: REMEDIATE → Fix guidance + YAML regression tests

---

Phase 1: Educate

How AI search is different: Traditional search engines crawl, index, and rank pages by relevance signals (backlinks, keywords, authority). AI answer engines do something fundamentally different — they retrieve content, understand it semantically, and synthesize answers by selecting the most citation-worthy sources. Your content needs to be clear, specific, authoritative, and directly answerable to be selected.

Key insight: AI systems prefer content that makes specific, verifiable claims with supporting evidence. Vague marketing copy is ignored. Concrete statements with data, comparisons, and clear structure get cited.

---

Phase 2: Scope

Gather context

1. Auto-detect from codebase/content:

• Content pages (docs, blog, landing pages, about, pricing, FAQ)
• Existing structured data (JSON-LD, Schema.org)
• Content management approach (static, CMS, MDX, etc.)
• llms.txt presence
• Sitemap and content organization
• Author/expertise signals
• Publication dates and freshness signals

2. Ask the user (one at a time):

• Product type: What does your product/site do? (needed to understand AI query context)
• Target URL: Where is the content published?
• Target AI queries: What questions should AI answer with your content? (e.g., "best CI/CD tool for startups", "how to implement OAuth in Node.js")
• Competitors: Who else shows up when AI answers these queries? (optional but valuable)
• Content goals: Documentation? Thought leadership? Product discovery? All of the above?

3. Map content landscape:

• Key content pages and their purpose
• Target queries each page should satisfy
• Current AI citation status (test a few queries in ChatGPT/Perplexity)
• Content gaps vs competitors

---

Phase 3: Analyze

Open a browser session with new_session using record_evidence: true. Run all applicable check categories.

Category A: Content Citation-Worthiness (CITE)

Browser validation: Navigate to content pages. Extract text content. Analyze for claim density, statistics, quotable statements. Compare against target queries for direct answer matching.

Category B: Content Structure for AI Retrieval (STRUCT)

Browser validation: Extract heading structure, count FAQ patterns, measure section lengths, check for tables and lists via DOM inspection.

Category C: Authority & Trust Signals (AUTH)

Browser validation: Navigate to content pages, about page, author pages. Extract dates, author info, citation links.

Category D: Technical AI Discoverability (TECH)

Browser validation: Fetch llms.txt, check robots.txt for AI bot rules, verify SSR content, inspect structured data.

Category E: Entity & Brand Clarity (ENTITY)

Browser validation: Navigate key pages and extract product positioning, feature lists, use cases, pricing structure. Check for entity-clear statements.

Category F: AI Citation Testing (TEST)

This category is unique to GEO — it tests actual AI visibility.

Browser validation: Use new_session to navigate to Perplexity and Google. Search target queries. Screenshot results. Check for citations to the user's domain. This provides real-world evidence of current AI visibility.

Important: TEST category results are the ground truth — they show whether your content is actually being cited, regardless of what the other categories suggest.

---

Phase 4: Report

Generate a structured report saved to shiplight/reports/geo-review-{date}.md:

# GEO Review Report
**Date:** {date}
**URL:** {url}
**Product type:** {description}
**Target AI queries tested:** {list}

## Overall GEO Score: {X}/10 | Confidence: {X}%

## Score Breakdown
| Category | Score | Findings |
|----------|-------|----------|
| Citation-Worthiness (CITE) | 5/10 | 2 high, 2 medium |
| Content Structure (STRUCT) | 6/10 | 1 high, 2 medium |
| Authority Signals (AUTH) | 7/10 | 1 medium |
| Technical Discoverability (TECH) | 4/10 | 1 critical, 2 high |
| Entity Clarity (ENTITY) | 5/10 | 2 high |
| AI Citation Testing (TEST) | 3/10 | Not cited in 4/5 target queries |

## AI Citation Status
| Target Query | Perplexity | Google AI Overview | Cited? | Competitor Cited? |
|-------------|------------|-------------------|--------|------------------|
| "best X for Y" | Not cited | Not in overview | ❌ | CompetitorA: ✅ |
| "how to do Z" | Cited (#3 source) | Cited | ✅ | CompetitorB: ✅ |
| ... | | | | |

## Citation Gap Analysis
What competitors' cited content has that yours doesn't:
- Specific performance benchmarks (CompetitorA cites "40% faster than...")
- Comparison tables (CompetitorB has detailed feature matrices)
- Direct answer paragraphs (CompetitorA leads sections with the conclusion)

## Findings
(structured findings with evidence and priority)

Confidence Scoring

• 90-100%: Verified via live AI search — content is/isn't cited (TEST category)
• 70-89%: Strong structural evidence — content has/lacks citation-worthy patterns
• 50-69%: Heuristic assessment of content quality signals
• Below 50%: Don't report

---

Phase 5: Remediate

1. Fix guidance (example)

#### CITE-01: Landing page lacks specific, verifiable claims
**Impact:** AI systems skip vague marketing copy — your landing page is invisible to AI answers
**Current:** "We're the fastest platform for modern teams"
**Fix:** Add specific, citable claims:
- "Deploys complete in 47 seconds on average (based on 10,000 deployments in Q4 2025)"
- "Used by 2,300 companies including [notable names]"
- "Reduces CI/CD pipeline time by 62% compared to Jenkins (internal benchmark, Jan 2026)"
**Principle:** AI cites facts, not adjectives. Every claim should be verifiable.

#### TECH-01: No llms.txt present
**Impact:** AI crawlers have no guidance on how to understand your site
**Fix:** Create /llms.txt at site root:

# [Your Product Name]

> One-sentence description of what your product does.

## Docs
- [Getting Started](/docs/getting-started): How to set up and configure [Product]
- [API Reference](/docs/api): Complete API documentation
- [Guides](/docs/guides): Step-by-step tutorials

## Key Pages
- [Pricing](/pricing): Plans and pricing
- [Changelog](/changelog): Recent updates and releases
- [About](/about): Company and team information

Also create /llms-full.txt with expanded content for deeper AI understanding.

2. YAML regression tests

- name: tech-01-llms-txt-present
  description: Verify llms.txt exists and is properly formatted
  severity: high
  standard: llms-txt-standard
  steps:
    - URL: /llms.txt
    - VERIFY: The page loads successfully and contains structured information about the site
    - description: Assert llms.txt is present and uses markdown headings
      js: |
        const content = await page.textContent('body');
        if (!content || content.trim().length < 50) {
          throw new Error('llms.txt is missing or too short');
        }
        if (!content.includes('#')) {
          throw new Error('llms.txt should use markdown heading structure');
        }
        console.log(`llms.txt found (${content.length} chars)`);

- name: tech-06-ai-crawlers-allowed
  description: Verify robots.txt allows AI search crawlers
  severity: high
  standard: AI-Discoverability
  steps:
    - URL: /robots.txt
    - description: Assert robots.txt does not block AI crawlers
      js: |
        const content = await page.textContent('body');
        const blockedBots = ['GPTBot', 'ClaudeBot', 'PerplexityBot', 'Google-Extended'];
        const blocked = blockedBots.filter(bot => {
          const pattern = new RegExp(`User-agent:\\s*${bot}[\\s\\S]*?Disallow:\\s*/`, 'i');
          return pattern.test(content);
        });
        if (blocked.length > 0) {
          throw new Error(`AI crawlers blocked in robots.txt: ${blocked.join(', ')}`);
        }
        console.log('All major AI crawlers are allowed');
    - VERIFY: robots.txt does not block major AI search engine crawlers

- name: cite-01-specific-claims-present
  description: Verify key pages contain specific, citable claims with data
  severity: high
  standard: GEO-Citation-Worthiness
  steps:
    - URL: /
    - description: Assert the landing page has citable statistics
      js: |
        const text = await page.textContent('main') || await page.textContent('body');
        // Check for specific numbers/statistics
        const hasNumbers = /\d+[%xX]|\$[\d,.]+|\d{1,3}(,\d{3})+|\d+\s*(users|customers|companies|teams|downloads)/i.test(text);
        if (!hasNumbers) {
          throw new Error('Landing page lacks specific statistics or data points that AI can cite');
        }
        console.log('Found specific, citable claims with data');
    - VERIFY: Landing page contains specific statistics, benchmarks, or verifiable data points

Save all YAML tests to shiplight/tests/geo-review.test.yaml.

---

Depth Levels

• `--quick`: llms.txt check + robots.txt AI crawler check + landing page claim analysis. ~2 minutes.
• default: All content categories + 3 target query tests in Perplexity. ~10-15 minutes.
• `--thorough`: All categories + full AI citation testing across multiple engines + competitor citation analysis + content gap recommendations. ~25-40 minutes.

Tips

• The TEST category (live AI search testing) is the most valuable — it shows ground truth, not theory
• Perplexity is the best testing ground because it always shows citations
• llms.txt is emerging but increasingly adopted — it's low effort, high signal
• AI systems update their knowledge at different speeds — changes may take weeks to reflect in citations
• Focus on content that answers specific questions, not brand awareness content
• The #1 GEO principle: AI cites facts, not adjectives — replace every vague claim with a specific one
• Close session with close_session and use generate_html_report for evidence

Performance Review

Measure and evaluate your application's performance against Google's Core Web Vitals thresholds and industry benchmarks. This review catches performance issues that are invisible during development but impact real users — bundle bloat, layout shifts, slow interactions, unoptimized images, and render-blocking resources.

When to use

Use this review when:

• Before launching or after major feature additions
• Page load feels slow but you're not sure why
• Preparing for high-traffic events
• After adding new dependencies or third-party scripts
• SEO rankings depend on performance scores
• Users report slowness or abandonment

Standards Referenced

• Google Core Web Vitals — LCP, INP, CLS (2024 thresholds)
• Google Lighthouse — Performance scoring methodology
• HTTP Archive — Web performance benchmarks (median, p75, p90)
• Web.dev Performance Guidelines — Best practices
• RAIL Model — Response, Animation, Idle, Load budgets

Phase Overview

Phase 1: EDUCATE   → Performance impact on business and what we measure
Phase 2: SCOPE     → Identify key pages, performance budget, baseline
Phase 3: ANALYZE   → Browser-based performance measurement
Phase 4: REPORT    → Findings with metrics, scores, and comparisons
Phase 5: REMEDIATE → Fix guidance + YAML regression tests

---

Phase 1: Educate

Why this matters: A 1-second delay in page load reduces conversions by 7% (Akamai). Google uses Core Web Vitals as ranking signals since 2021. 53% of mobile visitors leave a page that takes >3 seconds to load (Google). Amazon found every 100ms of latency costs 1% of sales. Performance is a feature — and its absence is a bug.

This review measures real performance in a browser, not just static analysis. We capture actual load times, rendering behavior, and interaction responsiveness.

---

Phase 2: Scope

Gather context

1. Auto-detect from codebase:

• Build system (Webpack, Vite, Next.js, etc.)
• Bundle analysis setup (if any)
• Image optimization pipeline (sharp, next/image, etc.)
• Font loading strategy
• Code splitting configuration
• Service worker / caching strategy
• CDN configuration

2. Ask the user (one at a time):

• Target URL: Where is the app running? (production preferred for realistic measurements)
• Key pages: Which pages matter most for performance? (recommend: landing page, main feature page, data-heavy page)
• Performance budget: Any existing targets? (default: Core Web Vitals "Good" thresholds)
• Known concerns: Any pages that feel slow? (optional)

3. Define measurement plan:

• Pages to test (3-5 key pages)
• Conditions: desktop and mobile simulated (Moto G4 / Slow 4G)
• Metrics: Core Web Vitals + supplementary metrics
• Baseline: first run establishes baseline for comparison

---

Phase 3: Analyze

Open a browser session with new_session using record_evidence: true. For each page in scope, run all measurement categories.

Category A: Core Web Vitals (CWV)

Browser validation: Navigate to each page and capture metrics via JavaScript:

// LCP
new PerformanceObserver((list) => {
  const entries = list.getEntries();
  const lcp = entries[entries.length - 1];
  console.log('LCP:', lcp.startTime);
}).observe({ type: 'largest-contentful-paint', buffered: true });

// CLS
let clsValue = 0;
new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    if (!entry.hadRecentInput) clsValue += entry.value;
  }
  console.log('CLS:', clsValue);
}).observe({ type: 'layout-shift', buffered: true });

Category B: Page Load Performance (LOAD)

Browser validation: Use Performance API and performance.getEntries() to gather all metrics.

Category C: Resource Optimization (RES)

Browser validation: Use JavaScript to inspect all loaded resources, their types, sizes, and loading attributes. Use performance.getEntriesByType('resource') for detailed resource metrics.

Category D: Bundle Analysis (BUN)

Browser validation: Use Performance API to measure transfer sizes. Check for source map URLs. Analyze script domain origins.

Category E: Runtime Performance (RUN)

Browser validation: Navigate and interact with the app while measuring performance metrics via JavaScript.

---

Phase 4: Report

Generate a structured report saved to shiplight/reports/performance-review-{date}.md:

# Performance Review Report
**Date:** {date}
**URL:** {url}
**Pages tested:** {list}
**Conditions:** Desktop + Mobile (simulated Moto G4 / Slow 4G)

## Overall Score: {X}/10 | Confidence: {X}%

## Core Web Vitals Summary
| Metric | Desktop | Mobile | Status |
|--------|---------|--------|--------|
| LCP | 1.8s | 3.2s | ⚠️ Mobile needs work |
| INP | 95ms | 180ms | ✅ Good |
| CLS | 0.05 | 0.15 | ⚠️ Mobile needs work |

## Score Breakdown
| Category | Score | Findings |
|----------|-------|----------|
| Core Web Vitals (CWV) | 6/10 | 1 high, 1 medium |
| Page Load (LOAD) | 7/10 | 1 high |
| Resources (RES) | 5/10 | 2 high, 2 medium |
| Bundle (BUN) | 6/10 | 1 high, 1 medium |
| Runtime (RUN) | 8/10 | 1 medium |

## Resource Waterfall
(Top 10 slowest resources with load times)

## Bundle Breakdown
| Category | Size (gzipped) | Budget | Status |
|----------|---------------|--------|--------|
| First-party JS | 180KB | 250KB | ✅ |
| Third-party JS | 220KB | 150KB | ❌ Over budget |
| CSS | 45KB | 100KB | ✅ |
| Images | 1.2MB | 1.5MB | ✅ |
| Fonts | 85KB | 100KB | ✅ |

## Findings
(structured findings with metrics and evidence)

Confidence Scoring

• 90-100%: Measured in browser with specific values (e.g., LCP: 3.2s)
• 70-89%: Derived from resource analysis (e.g., unoptimized images detected)
• 50-69%: Code-level pattern (e.g., no lazy loading attributes found)
• Below 50%: Don't report

---

Phase 5: Remediate

1. Fix guidance (example)

#### RES-01: Images not using modern formats
**Impact:** ~40% larger images than necessary, adds ~500KB to page weight
**Current:** 8 PNG images totaling 1.2MB
**Fix:** Convert to WebP with fallback:
- Use `<picture>` with WebP source and PNG fallback
- Or use Next.js `<Image>` / `sharp` for automatic format negotiation
- Expected savings: ~480KB (40% reduction)
**Priority files:**
- /images/hero.png (320KB → ~190KB as WebP)
- /images/features.png (280KB → ~165KB as WebP)

2. YAML regression test

- name: cwv-01-lcp-under-threshold
  description: Verify Largest Contentful Paint is under 2.5 seconds
  severity: high
  standard: Core-Web-Vitals-LCP
  steps:
    - description: Set up an LCP observer before navigation
      js: |
        // Set up LCP observer before navigation
        await page.evaluateOnNewDocument(() => {
          window.__lcp = 0;
          new PerformanceObserver((list) => {
            const entries = list.getEntries();
            window.__lcp = entries[entries.length - 1].startTime;
          }).observe({ type: 'largest-contentful-paint', buffered: true });
        });
    - URL: /
    - WAIT_UNTIL: Page has fully loaded including all images and content
      timeout_seconds: 30
    - description: Assert Largest Contentful Paint is under 2.5s
      js: |
        const lcp = await page.evaluate(() => window.__lcp);
        if (lcp > 2500) {
          throw new Error(`LCP is ${lcp}ms, exceeds 2500ms threshold`);
        }
        console.log(`LCP: ${lcp}ms (threshold: 2500ms)`);
    - VERIFY: Page loaded with Largest Contentful Paint under 2.5 seconds

Save all YAML tests to shiplight/tests/performance-review.test.yaml.

---

Depth Levels

• `--quick`: Core Web Vitals only on the main page. ~2 minutes.
• default: All categories on key pages, desktop + mobile. ~8-12 minutes.
• `--thorough`: All categories + extended pages + multiple runs for statistical confidence + runtime profiling. ~20-30 minutes.

Tips

• Measure on production (or production-like build) — dev mode performance is misleading
• Run multiple times — performance measurements vary; look for patterns, not single data points
• Mobile simulation reveals issues that desktop hides — always test both
• Use performance.getEntries() — it's the richest source of performance data in the browser
• Focus on Core Web Vitals first — they're the metrics Google uses for ranking
• Close session with close_session and use generate_html_report for evidence

Privacy Review

Evaluate how your application handles personal data — where it's collected, processed, stored, transmitted, and potentially leaked. This review catches privacy issues that code review alone misses: runtime data flows, third-party tracking, console/network leaks, and consent implementation gaps.

When to use

Use this review when:

• Your app collects any personal information (names, emails, addresses, etc.)
• Before launching in GDPR/CCPA jurisdictions
• Adding third-party analytics, tracking, or marketing tools
• After a data incident or privacy complaint
• Building features that handle sensitive data (health, financial, biometric)
• Integrating with third-party APIs that receive user data

Standards Referenced

• GDPR — EU General Data Protection Regulation (Articles 5, 6, 7, 12-22, 25, 32)
• CCPA/CPRA — California Consumer Privacy Act
• OWASP Privacy Risks Top 10
• NIST Privacy Framework
• ISO 27701 — Privacy Information Management
• ePrivacy Directive — Cookie consent requirements

Phase Overview

Phase 1: EDUCATE   → Privacy principles and what we check
Phase 2: SCOPE     → Map data flows, PII types, third parties
Phase 3: ANALYZE   → Browser-based validation of privacy practices
Phase 4: REPORT    → Findings with evidence and confidence scores
Phase 5: REMEDIATE → Fix guidance + YAML regression tests

---

Phase 1: Educate

Why this matters: GDPR fines reached €2.1B in 2023. CCPA gives consumers the right to sue for data breaches ($100-$750 per consumer per incident). Beyond compliance, privacy violations erode user trust — 79% of consumers say they'd stop engaging with a brand after a privacy breach. Many privacy issues are invisible in code review but obvious in runtime behavior.

This review focuses on observable runtime privacy behavior — what actually happens in the browser when users interact with your app.

---

Phase 2: Scope

Gather context

1. Auto-detect from codebase:

• Forms that collect user input (registration, profile, payment, contact)
• Analytics/tracking scripts (Google Analytics, Mixpanel, Segment, Hotjar, etc.)
• Cookie-setting code and cookie consent mechanisms
• Logging statements that might include PII
• API calls that transmit user data
• Third-party SDKs and their data sharing behavior
• Privacy policy and terms of service pages

2. Ask the user (one at a time):

• Target URL: Where is the app running?
• Data types: What personal data does your app collect? (auto-detected, confirm)
• Jurisdictions: Where are your users? (determines GDPR/CCPA/other applicability)
• Third parties: What analytics/tracking/marketing tools do you use? (auto-detected, confirm)
• Known concerns: Any specific privacy areas you're worried about? (optional)

3. Build data flow map:

• PII entry points (forms, URL params, imports)
• PII processing (client-side or server-side)
• PII storage (cookies, localStorage, server DB)
• PII transmission (API calls, third-party scripts)
• PII display (profile pages, admin panels, logs)

---

Phase 3: Analyze

Open a browser session with new_session using record_evidence: true. Run all applicable check categories.

Category A: Data Collection & Consent (CON)

Browser validation: Load page in fresh session (no cookies). Use get_browser_console_logs and monitor network via JavaScript. Check cookies before and after consent interaction. Use act to interact with consent banner.

Category B: PII Leakage Detection (LEAK)

Browser validation: Navigate through user flows. After each action, check URLs, console logs, storage, and network requests for PII patterns (email regex, phone patterns, SSN patterns, etc.). Use JavaScript to inspect performance.getEntries() for request URLs.

Category C: Third-Party Tracking Inventory (TRACK)

Browser validation: Load page with fresh session. Use JavaScript to enumerate all <script> sources, all cookie domains, all network requests to external domains. Check for fingerprinting API usage (Canvas, WebGL, AudioContext).

Category D: Data Storage & Retention (STOR)

Browser validation: Inspect all cookies (name, value, domain, expiry, flags). Check localStorage/sessionStorage. Test logout + back button behavior.

Category E: User Rights Implementation (RIGHTS)

Browser validation: Navigate to account settings, test data export, profile editing, and account deletion flows. Verify each right is accessible and functional.

---

Phase 4: Report

Generate a structured report saved to shiplight/reports/privacy-review-{date}.md:

# Privacy Review Report
**Date:** {date}
**URL:** {url}
**PII types handled:** {list}
**Jurisdictions:** {GDPR, CCPA, etc.}
**Third parties detected:** {count and list}

## Overall Score: {X}/10 | Confidence: {X}%

## Score Breakdown
| Category | Score | Findings |
|----------|-------|----------|
| Consent (CON) | 5/10 | 1 critical, 2 high |
| PII Leakage (LEAK) | 7/10 | 1 high, 1 medium |
| Tracking Inventory (TRACK) | 4/10 | 2 high, 1 medium |
| Data Storage (STOR) | 8/10 | 1 medium |
| User Rights (RIGHTS) | 6/10 | 1 high, 1 medium |

## Data Flow Map
(visual representation of PII flows through the application)

## Third-Party Tracking Inventory
| Domain | Type | Cookies Set | Data Sent | Consent Required |
|--------|------|-------------|-----------|-----------------|
| google-analytics.com | Analytics | _ga, _gid | Page URL, user agent | Yes |
| ... | | | | |

## Findings
(structured findings with evidence, severity, confidence)

Confidence Scoring

• 90-100%: Browser-validated — observed PII in console, URL, or network request
• 70-89%: Strong evidence from storage/header inspection
• 50-69%: Code-level pattern match, may not manifest at runtime
• Below 50%: Don't report

---

Phase 5: Remediate

1. Fix guidance (example)

#### LEAK-01: Email address in URL parameter after form submit
**Risk:** PII in URL is logged by servers, proxies, browser history, and analytics
**File:** src/pages/search.tsx:34
**Current:** `router.push(`/results?email=${email}`)`
**Fix:** Use POST request or session state
- `router.push('/results')` with email in request body or session
- Add `Referrer-Policy: no-referrer` header as defense-in-depth

2. YAML regression test

- name: leak-01-no-pii-in-urls
  description: Verify email addresses are not exposed in URL parameters
  severity: high
  standard: OWASP-Privacy-1
  steps:
    - URL: /search
    - intent: Enter email in search form
      action: fill
      locator: "getByLabel('Email')"
      value: "test@example.com"
    - intent: Submit the search form
      action: click
      locator: "getByRole('button', { name: 'Search' })"
    - WAIT_UNTIL: Search results are displayed
      timeout_seconds: 15
    - description: Assert no email address appears in the URL
      js: |
        const url = page.url();
        if (/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/.test(url)) {
          throw new Error(`PII found in URL: ${url}`);
        }
    - VERIFY: No email addresses appear in the browser URL

Save all YAML tests to shiplight/tests/privacy-review.test.yaml.

---

Tips

• Use a fresh browser session (no stored cookies) to test consent behavior accurately
• PII patterns to search for: email (@), phone (\d{3}[-.]?\d{3}[-.]?\d{4}), SSN, credit card numbers, names from test accounts
• Third-party scripts often load more scripts — check for cascade loading
• get_browser_console_logs often reveals PII that developers left in debug logging
• Test with consent rejected AND accepted — both paths matter
• Close session with close_session and use generate_html_report for evidence

Shared Review Conventions

Conventions every domain review follows. Each references/<domain>.md defines the domain-specific checks; this file defines the cross-cutting rules they share, so the router can merge their output into one consistent report.

The five-phase model

Every domain review runs the same shape:

Phase 1: EDUCATE   → why this domain matters + what we check
Phase 2: SCOPE     → auto-detect from the codebase, confirm with the user, map surface
Phase 3: ANALYZE   → browser-based checks against the domain's categories
Phase 4: REPORT    → findings with evidence, scores, and confidence
Phase 5: REMEDIATE → fix guidance + YAML regression tests

Browser evidence

Domain analysis runs against a live target through the Shiplight MCP. Open the session with new_session using record_evidence: true, and close it with close_session. Use generate_html_report for an auditable artifact. Findings that claim runtime behavior need browser evidence, not code-reading alone.

Scoring and confidence

• Score: each review reports an overall {X}/10 plus a per-category

breakdown. The router's unified report carries each domain's score.

• Confidence (per finding):
• 90–100% — browser-validated; the issue was observed at runtime.
• 70–89% — strong evidence from inspection (headers, storage, DOM).
• 50–69% — code-level pattern; may not manifest at runtime.
• Below 50% — do not report; too speculative.

Each domain tailors the examples for these bands; the bands themselves are uniform.

Severity

Order findings by severity: CRITICAL → HIGH → MEDIUM → LOW / INFO. The router surfaces the top findings across all domains by this ordering.

Output paths

• Per-domain report: shiplight/reports/<domain>-review-{date}.md
• YAML regression tests: shiplight/tests/<domain>-review.test.yaml
• Unified report (router): shiplight/reports/review-{date}.md

Regression tests accumulate in shiplight/tests/ so a review's findings become durable checks, not one-off observations.

Resilience Review

Evaluate how your application behaves when things go wrong — network failures, API errors, slow connections, missing data, and edge cases. Most apps are built for the happy path; this review systematically tests the unhappy paths that real users encounter.

When to use

Use this review when:

• Before launching a user-facing feature
• After adding new API integrations or data sources
• When reliability is critical (healthcare, finance, e-commerce checkout)
• After production incidents caused by unhandled errors
• When moving from prototype to production quality

Standards Referenced

• Google SRE Principles — Error budgets, graceful degradation
• Netflix Chaos Engineering Principles — Verify steady state, inject real-world failures
• OWASP Error Handling — Secure and user-friendly error responses
• Nielsen Norman Group — Error message usability heuristics

Phase Overview

Phase 1: EDUCATE   → Why resilience matters and what we test
Phase 2: SCOPE     → Map failure points, dependencies, critical flows
Phase 3: ANALYZE   → Browser-based fault injection and edge case testing
Phase 4: REPORT    → Findings with evidence and user impact assessment
Phase 5: REMEDIATE → Fix guidance + YAML regression tests

---

Phase 1: Educate

Why this matters: Users don't experience your app in ideal conditions. 53% of mobile visits are abandoned if a page takes >3 seconds. Error pages with no guidance increase support tickets 5x. A blank screen is the worst possible failure mode — it tells the user nothing and offers no recovery path. Resilient apps maintain trust even when backend systems fail.

This review simulates real-world failure conditions in the browser and evaluates how your UI responds.

---

Phase 2: Scope

Gather context

1. Auto-detect from codebase:

• API calls and their endpoints
• Error boundary components (React ErrorBoundary, Vue errorHandler)
• Loading state implementations (spinners, skeletons, suspense)
• Empty state components
• Retry logic / error recovery patterns
• Offline support (service workers, cache strategies)
• Third-party service dependencies

2. Ask the user (one at a time):

• Target URL: Where is the app running?
• Critical user flows: Which flows must never show a blank screen? (auto-detect from routes)
• Key API dependencies: Which APIs does the frontend depend on? (auto-detected)
• Known fragile areas: Any pages/features that break frequently? (optional)

3. Map failure points:

• API endpoints the frontend calls (and what happens if each fails)
• Third-party dependencies (CDN, auth provider, analytics, maps, payment)
• Data-dependent UI (what shows when data is empty, missing, or malformed)
• User input edge cases (long text, special characters, empty submissions)

---

Phase 3: Analyze

Open a browser session with new_session using record_evidence: true. Run all applicable check categories.

Category A: Error Handling (ERR)

Browser validation: Use description: + js: statements to intercept network requests via page.route() to simulate failures. Check UI state after each failure. Use get_browser_console_logs for JavaScript errors.

// Example: Mock API 500 error
await page.route('**/api/**', route => {
  route.fulfill({ status: 500, body: JSON.stringify({ error: 'Internal Server Error' }) });
});

Category B: Graceful Degradation (DEG)

Browser validation: Use page.route() to block specific resources. Use CDP to simulate network conditions. Disable JavaScript via browser settings. Verify each degradation scenario.

Category C: Empty & Edge States (EDGE)

Browser validation: Navigate to pages and test each edge case. Use act to interact with forms, submit empty/extreme data. Use JavaScript to check for UI overflow, frozen states.

Category D: API Contract & Data Handling (API)

Browser validation: Use page.route() to mock each response scenario. Verify UI state after each mock.

Category E: Recovery & User Communication (REC)

Browser validation: Use fault injection then verify recovery paths.

---

Phase 4: Report

Generate a structured report saved to shiplight/reports/resilience-review-{date}.md:

# Resilience Review Report
**Date:** {date}
**URL:** {url}
**Critical flows tested:** {list}
**API dependencies tested:** {count}
**Failure scenarios simulated:** {count}

## Overall Score: {X}/10 | Confidence: {X}%

## Score Breakdown
| Category | Score | Findings |
|----------|-------|----------|
| Error Handling (ERR) | 5/10 | 2 critical, 1 high |
| Graceful Degradation (DEG) | 6/10 | 1 high, 2 medium |
| Empty & Edge States (EDGE) | 4/10 | 1 critical, 3 high |
| API Contract (API) | 7/10 | 1 high, 1 medium |
| Recovery (REC) | 3/10 | 2 high, 1 medium |

## Failure Matrix
| Failure Scenario | Expected Behavior | Actual Behavior | Status |
|-----------------|-------------------|-----------------|--------|
| API returns 500 | Error message + retry | Blank screen | FAIL |
| Network timeout | Loading → timeout message | Infinite spinner | FAIL |
| Empty data set | "No results" message | Blank page | FAIL |
| ... | | | |

## Findings
(structured findings with evidence, screenshots of failure states)

Confidence Scoring

• 90-100%: Fault injected and failure behavior verified in browser
• 70-89%: Code analysis shows missing error handling, not validated at runtime
• 50-69%: Pattern-based assessment (e.g., no error boundary detected)
• Below 50%: Don't report

---

Phase 5: Remediate

1. Fix guidance (example)

#### ERR-01: API error shows blank screen instead of error message
**Impact:** Users see empty page, think app is broken, leave
**File:** src/pages/Dashboard.tsx:45
**Current:** `const data = await fetch('/api/data').then(r => r.json())`
**Problem:** No error handling — fetch throws on network error, .json() throws on non-JSON response
**Fix:**
- Wrap in try/catch
- Add error state: `const [error, setError] = useState(null)`
- Render error UI with retry button
- Add React Error Boundary as fallback

2. YAML regression test

- name: err-01-api-error-shows-message
  description: Verify API failure shows user-friendly error message instead of blank screen
  severity: critical
  standard: UX-Error-Handling
  steps:
    - description: Mock the data API to return a 500 error
      js: |
        await page.route('**/api/data**', route => {
          route.fulfill({
            status: 500,
            contentType: 'application/json',
            body: JSON.stringify({ error: 'Internal Server Error' })
          });
        });
    - URL: /dashboard
    - WAIT_UNTIL: Page has finished attempting to load data
      timeout_seconds: 15
    - VERIFY: An error message is visible explaining that data could not be loaded
    - VERIFY: A retry button or recovery action is available to the user
    - VERIFY: The page is NOT blank — navigation and header are still visible

Save all YAML tests to shiplight/tests/resilience-review.test.yaml.

---

Tips

• Use page.route() in description: + js: statements — it's the primary tool for fault injection
• Test the most critical user flows first (checkout, signup, core feature)
• A blank screen is always a CRITICAL finding — it's the worst failure mode
• Check get_browser_console_logs for uncaught promise rejections — they indicate missing error handling
• Edge case testing (EDGE category) often reveals the most bugs per minute spent
• Close session with close_session and use generate_html_report for evidence

Security Review

Evaluate your application's security posture against industry standards and validate findings through browser-based penetration testing. This review covers the attack surface that static analysis tools miss — runtime behavior, header configuration, authentication flows, and client-side vulnerabilities.

When to use

Use this review when:

• Before launching a new application or feature
• After adding authentication or authorization changes
• When handling sensitive data (user credentials, payment info, PII)
• Preparing for a security audit
• After a security incident to check for similar issues
• Reviewing third-party integrations

Standards Referenced

• OWASP Top 10 (2021) — Top web application security risks
• OWASP ASVS v4.0 — Application Security Verification Standard
• OWASP Session Management Cheat Sheet
• NIST 800-63B — Digital Identity Guidelines (authentication)
• CWE/SANS Top 25 — Most Dangerous Software Weaknesses
• Mozilla Observatory — HTTP security header best practices

Phase Overview

Phase 1: EDUCATE   → Security context and what we check
Phase 2: SCOPE     → Identify attack surface, auth mechanisms, data flows
Phase 3: ANALYZE   → Automated checks + browser-based penetration testing
Phase 4: REPORT    → Findings with evidence, CVE references, confidence scores
Phase 5: REMEDIATE → Fix guidance + YAML regression tests

---

Phase 1: Educate

Why this matters: The average cost of a data breach is $4.45M (IBM 2023). 83% of web applications have at least one critical vulnerability. Many security issues are only detectable at runtime — misconfigured headers, insecure token storage, broken access controls — which is exactly what browser-based testing catches.

This review checks your app against objective security criteria with browser-based validation. Every finding references a specific standard (OWASP, CWE, NIST).

---

Phase 2: Scope

Gather context

1. Auto-detect from codebase:

• Authentication mechanism (JWT, sessions, OAuth, API keys)
• Framework security features in use (CSRF tokens, CORS config, CSP)
• Dependencies with known vulnerabilities (npm audit / pip audit)
• API routes and endpoints
• Environment variable handling
• File upload capabilities
• Third-party scripts and CDN usage

2. Ask the user (one at a time):

• Target URL: Where is the app running?
• Auth mechanism: How do users log in? (auto-detected, confirm)
• Test credentials: Do you have test accounts I can use? (needed for authenticated testing)
• Sensitive data: What sensitive data does the app handle? (PII, payments, health records)
• Known concerns: Any specific areas you're worried about? (optional)

3. Map the attack surface:

• List all user input points (forms, URL params, file uploads, WebSocket messages)
• List all API endpoints with their auth requirements
• List all third-party integrations
• Identify data flow: where does sensitive data enter, process, store, and exit?

---

Phase 3: Analyze

Open a browser session with new_session using record_evidence: true. Run all applicable check categories.

Category A: HTTP Security Headers (HDR)

Browser validation: Use JavaScript via act to inspect document.querySelector('meta[http-equiv]') and fetch response headers via a same-origin request. Use get_browser_console_logs to check for mixed content warnings.

Category B: Authentication & Session Management (AUTH)

Browser validation: Log in via act, inspect cookies with JavaScript (document.cookie — HttpOnly cookies won't appear, which is correct). Check localStorage. Perform logout, replay requests. Attempt brute force (5 wrong passwords). Modify JWT tokens and test.

Category C: Input Validation & Injection (INJ)

Browser validation: Use act to fill form fields with test payloads. Capture page state after submission. Check for script execution, error messages, unexpected behavior. Use get_browser_console_logs for JavaScript errors that indicate injection vectors.

Important: These are non-destructive test payloads for detection only. Do not attempt actual exploitation. Alert-based XSS tests use alert(1) which is harmless.

Category D: Access Control (AC)

Browser validation: Navigate to protected pages without auth. Try accessing resources belonging to other users. Check JavaScript source for hardcoded secrets using act with JavaScript to scan script contents.

Category E: Client-Side Security (CLI)

Browser validation: Use JavaScript via act to enumerate localStorage keys, check script tags for SRI, list all network requests to external domains. Use get_browser_console_logs to check for leaked data.

Category F: Dependency & Supply Chain (DEP)

Validation: Run dependency audit commands. Cross-reference with codebase scan from Phase 2.

---

Phase 4: Report

Generate a structured report saved to shiplight/reports/security-review-{date}.md:

# Security Review Report
**Date:** {date}
**URL:** {url}
**Auth mechanism:** {type}
**Attack surface:** {summary}

## Overall Score: {X}/10 | Confidence: {X}%

## Score Breakdown
| Category | Score | Findings |
|----------|-------|----------|
| HTTP Headers (HDR) | 6/10 | 1 critical, 2 high |
| Auth & Sessions (AUTH) | 4/10 | 2 critical, 1 high |
| Input Validation (INJ) | 7/10 | 1 high, 2 medium |
| Access Control (AC) | 8/10 | 1 medium |
| Client-Side (CLI) | 5/10 | 1 critical, 1 high |
| Dependencies (DEP) | 9/10 | 1 low |

## Findings

### CRITICAL

#### AUTH-01: JWT stored in localStorage — XSS leads to full account takeover
- **Standard:** OWASP ASVS 3.3.2 / CWE-922
- **Finding:** Access token stored in `localStorage` under key `auth_token`, accessible to any XSS payload
- **Evidence:** [screenshot of Application > Storage showing JWT]
- **Attack scenario:** Any XSS vulnerability (even via third-party script) can exfiltrate all user tokens
- **CVSS estimate:** 8.1 (High)
- **Confidence:** 95%

...

Confidence Scoring

• 90-100%: Exploited and verified in browser (e.g., XSS payload executed, unauthorized access confirmed)
• 70-89%: Strong evidence from inspection (e.g., missing header confirmed, insecure cookie flags observed)
• 50-69%: Code-level evidence, not fully validated at runtime
• Below 50%: Don't report — too speculative

---

Phase 5: Remediate

For each finding, provide:

1. Fix guidance

#### AUTH-01: JWT stored in localStorage
**Risk:** Any XSS → full account takeover
**File:** src/lib/auth.ts:47
**Current:** `localStorage.setItem('auth_token', jwt)`
**Fix:** Move to HttpOnly cookie set by the server
- Server: `Set-Cookie: token=<jwt>; HttpOnly; Secure; SameSite=Strict; Path=/`
- Client: Remove all localStorage token operations
- API calls: Cookies sent automatically (remove Authorization header)
**Migration steps:**
1. Add cookie-setting endpoint on server
2. Update API middleware to read from cookie
3. Remove client-side token storage
4. Update CORS to allow credentials

2. YAML regression test

- name: auth-01-no-tokens-in-localstorage
  description: Verify authentication tokens are not stored in localStorage
  severity: critical
  standard: OWASP-ASVS-3.3.2
  steps:
    - URL: /login
    - intent: Enter test username
      action: fill
      locator: "getByLabel('Email')"
      value: "test@example.com"
    - intent: Enter test password
      action: fill
      locator: "getByLabel('Password')"
      value: "testpass123"
    - intent: Click login button
      action: click
      locator: "getByRole('button', { name: 'Sign in' })"
    - WAIT_UNTIL: User is logged in and dashboard is visible
      timeout_seconds: 15
    - description: Assert no auth tokens are stored in localStorage
      js: |
        const keys = Object.keys(localStorage);
        const tokenKeys = keys.filter(k =>
          /token|jwt|auth|session|access/i.test(k)
        );
        if (tokenKeys.length > 0) {
          throw new Error(
            `Auth tokens found in localStorage: ${tokenKeys.join(', ')}`
          );
        }
    - VERIFY: No authentication tokens are stored in browser localStorage

Save all YAML tests to shiplight/tests/security-review.test.yaml.

---

Penetration Test Depth Levels

• `--quick`: Headers (HDR) + Cookie flags (AUTH-02/03/04) + localStorage check (AUTH-01) + dependency audit (DEP-01). ~2 minutes.
• default: All categories, standard payloads. ~10 minutes.
• `--thorough`: All categories + extended injection payloads + IDOR enumeration + brute force testing + full third-party script analysis. ~20-30 minutes.

Tips

• Always use test credentials, never production credentials
• XSS test payloads are non-destructive (alert(1)) — safe for staging environments
• For authenticated testing, save the session with save_storage_state after login
• Run npm audit before the browser-based review to catch known CVEs early
• Use get_browser_console_logs — many security issues produce console warnings
• Close the session with close_session and use generate_html_report for evidence

SEO Review

Evaluate your application's search engine optimization and discoverability. This review catches SEO issues that are invisible during normal development — missing meta tags, broken structured data, poor crawlability, missing sitemaps, and social sharing problems. Many SPAs ship with zero SEO consideration.

When to use

Use this review when:

• Before launching a public-facing website or application
• After redesigning or migrating a website
• When organic traffic is a growth channel
• After switching to an SPA framework (React, Vue, Angular)
• When social sharing (Open Graph) is important
• Setting up content marketing pages

Standards Referenced

• Google Search Central — SEO best practices and guidelines
• Schema.org — Structured data vocabulary
• Open Graph Protocol — Social sharing meta tags
• Twitter Card — Twitter sharing markup
• W3C Semantic HTML — Accessibility and SEO semantics
• Web.dev SEO Guidelines — Google's SEO recommendations

Phase Overview

Phase 1: EDUCATE   → Why SEO matters and what we check
Phase 2: SCOPE     → Identify key pages, content types, SEO goals
Phase 3: ANALYZE   → Browser-based SEO validation
Phase 4: REPORT    → Findings with impact assessment and priority
Phase 5: REMEDIATE → Fix guidance + YAML regression tests

---

Phase 1: Educate

Why this matters: 53% of all website traffic comes from organic search (BrightEdge). The first Google result gets 27.6% of all clicks; position 10 gets 2.4% (Backlinko). SPAs often render blank HTML to crawlers, making millions of pages invisible to search. Proper SEO doesn't require tricks — it requires making your content discoverable and understandable by search engines.

This review checks the technical SEO foundation — the things that must be correct before content strategy matters.

---

Phase 2: Scope

Gather context

1. Auto-detect from codebase:

• Framework and rendering strategy (SSR, SSG, CSR, ISR)
• Meta tag management (react-helmet, next/head, vue-meta, etc.)
• Sitemap generation
• robots.txt configuration
• Structured data (JSON-LD, microdata)
• i18n/hreflang setup
• Canonical URL handling
• Route structure and page types

2. Ask the user (one at a time):

• Target URL: Where is the app running? (production preferred for realistic crawling)
• Key pages: Which pages are most important for search? (homepage, product pages, blog posts, landing pages)
• Target keywords: Any primary keywords you want to rank for? (optional, for content evaluation)
• Social sharing important?: Is Open Graph / Twitter Cards needed? (default: yes for public sites)
• Multi-language?: Does the site serve multiple languages? (auto-detected)

3. Identify page types:

• Homepage
• Content pages (blog, docs, about)
• Product/listing pages
• Dynamic pages (search results, user profiles)
• Utility pages (login, 404, terms)

---

Phase 3: Analyze

Open a browser session with new_session using record_evidence: true. For each key page, run all check categories.

Category A: Meta Tags & Head Elements (META)

Browser validation: Use inspect_page to read the DOM. Extract all <head> elements via JavaScript.

Category B: Structured Data (SCHEMA)

Browser validation: Extract JSON-LD scripts via JavaScript. Parse and validate structure. Compare page content against schema claims.

Category C: Open Graph & Social Sharing (OG)

Browser validation: Extract all OG and Twitter meta tags. Verify og:image URL is accessible. Construct a preview representation.

Category D: Crawlability & Indexing (CRAWL)

Browser validation: Navigate to robots.txt, sitemap.xml. Follow internal links. Disable JavaScript to test server-rendered content. Check HTTP status codes.

Category E: Semantic HTML & Content (SEM)

Browser validation: Extract headings, links, images, and semantic elements via JavaScript. Analyze content structure.

Category F: Technical SEO (TECH)

Browser validation: Test redirects, measure performance metrics, check mobile rendering.

---

Phase 4: Report

Generate a structured report saved to shiplight/reports/seo-review-{date}.md:

# SEO Review Report
**Date:** {date}
**URL:** {url}
**Pages reviewed:** {list}
**Rendering:** {SSR/SSG/CSR/ISR}

## Overall Score: {X}/10 | Confidence: {X}%

## Score Breakdown
| Category | Score | Findings |
|----------|-------|----------|
| Meta Tags (META) | 7/10 | 1 high, 2 medium |
| Structured Data (SCHEMA) | 4/10 | 1 critical, 1 high |
| Social Sharing (OG) | 6/10 | 2 high |
| Crawlability (CRAWL) | 5/10 | 1 critical, 1 high |
| Semantic HTML (SEM) | 8/10 | 1 medium |
| Technical SEO (TECH) | 7/10 | 1 high |

## Page-by-Page Summary
| Page | Title | Description | OG Image | Schema | H1 | Score |
|------|-------|-------------|----------|--------|----|-------|
| / | ✅ | ✅ | ❌ missing | ❌ none | ✅ | 6/10 |
| /blog | ✅ | ⚠️ too short | ✅ | ✅ Article | ✅ | 8/10 |

## Findings
(structured findings with evidence and impact)

Confidence Scoring

• 90-100%: Verified in browser — tag present/absent, URL accessible/broken
• 70-89%: Content analysis suggests issue (e.g., thin content, generic alt text)
• 50-69%: Best practice recommendation without clear violation
• Below 50%: Don't report

---

Phase 5: Remediate

1. Fix guidance (example)

#### CRAWL-10: Content not visible without JavaScript
**Impact:** Search engines may not index your content (especially Google Discover, Bing, social crawlers)
**Current:** Client-side rendered React app, empty HTML shell
**Fix:** Implement SSR or SSG:
- Next.js: Use `getServerSideProps` or `getStaticProps`
- Nuxt: Default SSR mode
- Gatsby: Static generation
- Or: Add prerendering service (prerender.io, rendertron)
**Quick win:** Ensure critical content is in initial HTML response

2. YAML regression test

- name: meta-01-title-present
  description: Verify each key page has a unique, properly-sized title tag
  severity: high
  standard: Google-SEO-Guidelines
  steps:
    - URL: /
    - description: Assert the page title is present and 20-60 characters
      js: |
        const title = await page.title();
        if (!title || title.trim() === '') {
          throw new Error('Page has no title');
        }
        if (title.length < 20) {
          throw new Error(`Title too short (${title.length} chars): "${title}"`);
        }
        if (title.length > 60) {
          throw new Error(`Title too long (${title.length} chars): "${title}"`);
        }
        console.log(`Title OK (${title.length} chars): "${title}"`);
    - VERIFY: Page has a descriptive title between 20 and 60 characters

Save all YAML tests to shiplight/tests/seo-review.test.yaml.

---

Depth Levels

• `--quick`: Meta tags + canonical + robots.txt on homepage only. ~2 minutes.
• default: All categories on key pages. ~8-12 minutes.
• `--thorough`: All categories + full site crawl + all page types + content analysis. ~20-30 minutes.

Tips

• Test with JavaScript disabled to see what search engines see (especially for SPAs)
• Use inspect_page DOM output to extract <head> content efficiently
• Check sitemap.xml manually — auto-generated sitemaps often include pages that shouldn't be indexed
• OG image must be an absolute URL — relative URLs don't work for social sharing
• For SPA SEO, the critical question is: "Does the initial HTML contain the content?"
• Close session with close_session and use generate_html_report for evidence