Bmad Testarch Test Review

Test Quality Review

Goal: Review test quality using a comprehensive knowledge base and best-practices validation.

Role: You are the Master Test Architect.

You will continue to operate with your given name, identity, and communication_style, merged with the details of this role description.

Conventions

• Bare paths (e.g. instructions.md) resolve from the skill root.
• {skill-root} resolves to this skill's installed directory (where customize.toml lives).
• {project-root}-prefixed paths resolve from the project working directory.
• {skill-name} resolves to the skill directory's basename.
• Resolve sibling workflow files such as instructions.md, checklist.md, steps-c/..., steps-e/..., steps-v/..., and templates from {skill-root}.

On Activation

Step 1: Resolve the Workflow Block

Run: python3 {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key workflow

If the script fails, resolve the workflow block yourself by reading these three files in base → team → user order and applying the same structural merge rules as the resolver:

1. {skill-root}/customize.toml — defaults 2. {project-root}/_bmad/custom/{skill-name}.toml — team overrides 3. {project-root}/_bmad/custom/{skill-name}.user.toml — personal overrides

Any missing file is skipped. Scalars override, tables deep-merge, arrays of tables keyed by code or id replace matching entries and append new entries, and all other arrays append.

Step 2: Execute Prepend Steps

Execute each entry in {workflow.activation_steps_prepend} in order before proceeding.

Step 3: Load Persistent Facts

Treat every entry in {workflow.persistent_facts} as foundational context you carry for the rest of the workflow run. Entries prefixed file: are paths or globs resolved from {project-root} — expand them and load every matching file in lexical path order as facts. All other entries are facts verbatim.

Step 4: Load Config

Load config from {project-root}/_bmad/tea/config.yaml and resolve:

• user_name
• communication_language

Step 5: Greet the User

Greet {user_name}, speaking in {communication_language}.

Step 6: Execute Append Steps

Execute each entry in {workflow.activation_steps_append} in order.

Activation is complete. Begin the workflow below.

Workflow Architecture

This workflow uses tri-modal step-file architecture:

• Create mode (steps-c/): primary execution flow for new runs and resume continuation
• Validate mode (steps-v/): validation against checklist
• Edit mode (steps-e/): revise existing outputs

Initialization Sequence

1. Mode Determination

"Welcome to the workflow. What would you like to do?"

• [C] Create — Run the workflow from the beginning
• [R] Resume — Resume an interrupted Create workflow
• [V] Validate — Validate existing outputs
• [E] Edit — Edit existing outputs

2. Route to First Step

• If C: Load {skill-root}/steps-c/step-01-load-context.md
• If R: Load {skill-root}/steps-c/step-01b-resume.md (Create-mode continuation)
• If V: Load {skill-root}/steps-v/step-01-validate.md
• If E: Load {skill-root}/steps-e/step-01-assess.md

Test Quality Review - Validation Checklist

Use this checklist to validate that the test quality review workflow completed successfully and all quality criteria were properly evaluated.

---

Prerequisites

Note: test-review is optional and only audits existing tests; it does not generate tests. Coverage analysis is out of scope for this workflow. Use trace for coverage metrics and coverage gate decisions.

Test File Discovery

• [ ] Test file(s) identified for review (single/directory/suite scope)
• [ ] Test files exist and are readable
• [ ] Test framework detected (Playwright, Jest, Cypress, Vitest, etc.)
• [ ] Test framework configuration found (playwright.config.ts, jest.config.js, etc.)

Knowledge Base Loading

• [ ] tea-index.csv loaded successfully
• [ ] test-quality.md loaded (Definition of Done)
• [ ] fixture-architecture.md loaded (Pure function → Fixture patterns)
• [ ] network-first.md loaded (Route intercept before navigate)
• [ ] data-factories.md loaded (Factory patterns)
• [ ] test-levels-framework.md loaded (E2E vs API vs Component vs Unit)
• [ ] All other enabled fragments loaded successfully

Context Gathering

• [ ] Story file discovered or explicitly provided (if available)
• [ ] Test design document discovered or explicitly provided (if available)
• [ ] Acceptance criteria extracted from story (if available)
• [ ] Priority context (P0/P1/P2/P3) extracted from test-design (if available)

---

Process Steps

Step 1: Context Loading

• [ ] Review scope determined (single/directory/suite)
• [ ] Test file paths collected
• [ ] Related artifacts discovered (story, test-design)
• [ ] Knowledge base fragments loaded successfully
• [ ] Quality criteria flags read from workflow variables

Step 2: Test File Parsing

For Each Test File:

• [ ] File read successfully
• [ ] File size measured (lines, KB)
• [ ] File structure parsed (describe blocks, it blocks)
• [ ] Test IDs extracted (if present)
• [ ] Priority markers extracted (if present)
• [ ] Imports analyzed
• [ ] Dependencies identified

Test Structure Analysis:

• [ ] Describe block count calculated
• [ ] It/test block count calculated
• [ ] BDD structure identified (Given-When-Then)
• [ ] Fixture usage detected
• [ ] Data factory usage detected
• [ ] Network interception patterns identified
• [ ] Assertions counted
• [ ] Waits and timeouts cataloged
• [ ] Conditionals (if/else) detected
• [ ] Try/catch blocks detected
• [ ] Shared state or globals detected

Step 3: Quality Criteria Validation

Coverage criteria are intentionally excluded from this checklist.

For Each Enabled Criterion:

BDD Format (if check_given_when_then: true)

• [ ] Given-When-Then structure evaluated
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Violations recorded with line numbers
• [ ] Examples of good/bad patterns noted

Test IDs (if check_test_ids: true)

• [ ] Test ID presence validated
• [ ] Test ID format checked (e.g., 1.3-E2E-001)
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Missing IDs cataloged

Priority Markers (if check_priority_markers: true)

• [ ] P0/P1/P2/P3 classification validated
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Missing priorities cataloged

Hard Waits (if check_hard_waits: true)

• [ ] sleep(), waitForTimeout(), hardcoded delays detected
• [ ] Justification comments checked
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Violations recorded with line numbers and recommended fixes

Determinism (if check_determinism: true)

• [ ] Conditionals (if/else/switch) detected
• [ ] Try/catch abuse detected
• [ ] Random values (Math.random, Date.now) detected
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Violations recorded with recommended fixes

Isolation (if check_isolation: true)

• [ ] Cleanup hooks (afterEach/afterAll) validated
• [ ] Shared state detected
• [ ] Global variable mutations detected
• [ ] Resource cleanup verified
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Violations recorded with recommended fixes

Fixture Patterns (if check_fixture_patterns: true)

• [ ] Fixtures detected (test.extend)
• [ ] Pure functions validated
• [ ] mergeTests usage checked
• [ ] beforeEach complexity analyzed
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Violations recorded with recommended fixes

Data Factories (if check_data_factories: true)

• [ ] Factory functions detected
• [ ] Hardcoded data (magic strings/numbers) detected
• [ ] Faker.js or similar usage validated
• [ ] API-first setup pattern checked
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Violations recorded with recommended fixes

Network-First (if check_network_first: true)

• [ ] page.route() before page.goto() validated
• [ ] Race conditions detected (route after navigate)
• [ ] Network wait patterns checked (interceptNetworkCall preferred over ad hoc waitForResponse)
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Violations recorded with recommended fixes

Assertions (if check_assertions: true)

• [ ] Explicit assertions counted
• [ ] Implicit waits without assertions detected
• [ ] Assertion specificity validated
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Violations recorded with recommended fixes

Test Length (if check_test_length: true)

• [ ] File line count calculated
• [ ] Threshold comparison (≤300 lines ideal)
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Splitting recommendations generated (if >300 lines)

Test Duration (if check_test_duration: true)

• [ ] Test complexity analyzed (as proxy for duration if no execution data)
• [ ] Threshold comparison (≤1.5 min target)
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Optimization recommendations generated

Flakiness Patterns (if check_flakiness_patterns: true)

• [ ] Tight timeouts detected (e.g., { timeout: 1000 })
• [ ] Race conditions detected
• [ ] Timing-dependent assertions detected
• [ ] Retry logic detected
• [ ] Environment-dependent assumptions detected
• [ ] Status assigned (PASS/WARN/FAIL)
• [ ] Violations recorded with recommended fixes

---

Step 4: Quality Score Calculation

Violation Counting:

• [ ] Critical (P0) violations counted
• [ ] High (P1) violations counted
• [ ] Medium (P2) violations counted
• [ ] Low (P3) violations counted
• [ ] Violation breakdown by criterion recorded

Score Calculation:

• [ ] Starting score: 100
• [ ] Critical violations deducted (-10 each)
• [ ] High violations deducted (-5 each)
• [ ] Medium violations deducted (-2 each)
• [ ] Low violations deducted (-1 each)
• [ ] Bonus points added (max +30):
• [ ] Excellent BDD structure (+5 if applicable)
• [ ] Comprehensive fixtures (+5 if applicable)
• [ ] Comprehensive data factories (+5 if applicable)
• [ ] Network-first pattern (+5 if applicable)
• [ ] Perfect isolation (+5 if applicable)
• [ ] All test IDs present (+5 if applicable)
• [ ] Final score calculated: max(0, min(100, Starting - Violations + Bonus))

Quality Grade:

• [ ] Grade assigned based on score:
• 90-100: A+ (Excellent)
• 80-89: A (Good)
• 70-79: B (Acceptable)
• 60-69: C (Needs Improvement)
• <60: F (Critical Issues)

---

Step 5: Review Report Generation

Report Sections Created:

• [ ] Header Section:
• [ ] Test file(s) reviewed listed
• [ ] Review date recorded
• [ ] Review scope noted (single/directory/suite)
• [ ] Quality score and grade displayed

• [ ] Executive Summary:
• [ ] Overall assessment (Excellent/Good/Needs Improvement/Critical)
• [ ] Key strengths listed (3-5 bullet points)
• [ ] Key weaknesses listed (3-5 bullet points)
• [ ] Recommendation stated (Approve/Approve with comments/Request changes/Block)

• [ ] Quality Criteria Assessment:
• [ ] Table with all criteria evaluated
• [ ] Status for each criterion (PASS/WARN/FAIL)
• [ ] Violation count per criterion

• [ ] Critical Issues (Must Fix):
• [ ] P0/P1 violations listed
• [ ] Code location provided for each (file:line)
• [ ] Issue explanation clear
• [ ] Recommended fix provided with code example
• [ ] Knowledge base reference provided

• [ ] Recommendations (Should Fix):
• [ ] P2/P3 violations listed
• [ ] Code location provided for each (file:line)
• [ ] Issue explanation clear
• [ ] Recommended improvement provided with code example
• [ ] Knowledge base reference provided

• [ ] Best Practices Examples (if good patterns found):
• [ ] Good patterns highlighted from tests
• [ ] Knowledge base fragments referenced
• [ ] Examples provided for others to follow

• [ ] Knowledge Base References:
• [ ] All fragments consulted listed
• [ ] Links to detailed guidance provided

---

Step 6: Optional Outputs Generation

Inline Comments (if generate_inline_comments: true):

• [ ] Inline comments generated at violation locations
• [ ] Comment format: // TODO (TEA Review): [Issue] - See test-review-{filename}.md
• [ ] Comments added to test files (no logic changes)
• [ ] Test files remain valid and executable

Quality Badge (if generate_quality_badge: true):

• [ ] Badge created with quality score (e.g., "Test Quality: 87/100 (A)")
• [ ] Badge format suitable for README or documentation
• [ ] Badge saved to output folder

Story Update (if append_to_story: true and story file exists):

• [ ] "Test Quality Review" section created
• [ ] Quality score included
• [ ] Critical issues summarized
• [ ] Link to full review report provided
• [ ] Story file updated successfully

---

Step 7: Save and Notify

Outputs Saved:

• [ ] Review report saved to {output_file}
• [ ] Inline comments written to test files (if enabled)
• [ ] Quality badge saved (if enabled)
• [ ] Story file updated (if enabled)
• [ ] All outputs are valid and readable

Summary Message Generated:

• [ ] Quality score and grade included
• [ ] Critical issue count stated
• [ ] Recommendation provided (Approve/Request changes/Block)
• [ ] Next steps clarified
• [ ] Message displayed to user

---

Output Validation

Review Report Completeness

• [ ] All required sections present
• [ ] No placeholder text or TODOs in report
• [ ] All code locations are accurate (file:line)
• [ ] All code examples are valid and demonstrate fix
• [ ] All knowledge base references are correct

Review Report Accuracy

• [ ] Quality score matches violation breakdown
• [ ] Grade matches score range
• [ ] Violations correctly categorized by severity (P0/P1/P2/P3)
• [ ] Violations correctly attributed to quality criteria
• [ ] No false positives (violations are legitimate issues)
• [ ] No false negatives (critical issues not missed)

Review Report Clarity

• [ ] Executive summary is clear and actionable
• [ ] Issue explanations are understandable
• [ ] Recommended fixes are implementable
• [ ] Code examples are correct and runnable
• [ ] Recommendation (Approve/Request changes) is clear

---

Quality Checks

Knowledge-Based Validation

• [ ] All feedback grounded in knowledge base fragments
• [ ] Recommendations follow proven patterns
• [ ] No arbitrary or opinion-based feedback
• [ ] Knowledge fragment references accurate and relevant

Actionable Feedback

• [ ] Every issue includes recommended fix
• [ ] Every fix includes code example
• [ ] Code examples demonstrate correct pattern
• [ ] Fixes reference knowledge base for more detail

Severity Classification

• [ ] Critical (P0) issues are genuinely critical (hard waits, race conditions, no assertions)
• [ ] High (P1) issues impact maintainability/reliability (missing IDs, hardcoded data)
• [ ] Medium (P2) issues are nice-to-have improvements (long files, missing priorities)
• [ ] Low (P3) issues are minor style/preference (verbose tests)

Context Awareness

• [ ] Review considers project context (some patterns may be justified)
• [ ] Violations with justification comments noted as acceptable
• [ ] Edge cases acknowledged
• [ ] Recommendations are pragmatic, not dogmatic

---

Integration Points

Story File Integration

• [ ] Story file discovered correctly (if available)
• [ ] Acceptance criteria extracted and used for context
• [ ] Test quality section appended to story (if enabled)
• [ ] Link to review report added to story

Test Design Integration

• [ ] Test design document discovered correctly (if available)
• [ ] Priority context (P0/P1/P2/P3) extracted and used
• [ ] Review validates tests align with prioritization
• [ ] Misalignment flagged (e.g., P0 scenario missing tests)

Knowledge Base Integration

• [ ] tea-index.csv loaded successfully
• [ ] All required fragments loaded
• [ ] Fragments applied correctly to validation
• [ ] Fragment references in report are accurate

---

Edge Cases and Special Situations

Empty or Minimal Tests

• [ ] If test file is empty, report notes "No tests found"
• [ ] If test file has only boilerplate, report notes "No meaningful tests"
• [ ] Score reflects lack of content appropriately

Legacy Tests

• [ ] Legacy tests acknowledged in context
• [ ] Review provides practical recommendations for improvement
• [ ] Recognizes that complete refactor may not be feasible
• [ ] Prioritizes critical issues (flakiness) over style

Test Framework Variations

• [ ] Review adapts to test framework (Playwright vs Jest vs Cypress)
• [ ] Framework-specific patterns recognized (e.g., Playwright fixtures)
• [ ] Framework-specific violations detected (e.g., Cypress anti-patterns)
• [ ] Knowledge fragments applied appropriately for framework

Justified Violations

• [ ] Violations with justification comments in code noted as acceptable
• [ ] Justifications evaluated for legitimacy
• [ ] Report acknowledges justified patterns
• [ ] Score not penalized for justified violations

---

Final Validation

Review Completeness

• [ ] All enabled quality criteria evaluated
• [ ] All test files in scope reviewed
• [ ] All violations cataloged
• [ ] All recommendations provided
• [ ] Review report is comprehensive

Review Accuracy

• [ ] Quality score is accurate
• [ ] Violations are correct (no false positives)
• [ ] Critical issues not missed (no false negatives)
• [ ] Code locations are correct
• [ ] Knowledge base references are accurate

Review Usefulness

• [ ] Feedback is actionable
• [ ] Recommendations are implementable
• [ ] Code examples are correct
• [ ] Review helps developer improve tests
• [ ] Review educates on best practices

Workflow Complete

• [ ] All checklist items completed
• [ ] All outputs validated and saved
• [ ] User notified with summary
• [ ] Review ready for developer consumption
• [ ] Follow-up actions identified (if any)

---

Notes

Record any issues, observations, or important context during workflow execution:

• Test Framework: [Playwright, Jest, Cypress, etc.]
• Review Scope: [single file, directory, full suite]
• Quality Score: [0-100 score, letter grade]
• Critical Issues: [Count of P0/P1 violations]
• Recommendation: [Approve / Approve with comments / Request changes / Block]
• Special Considerations: [Legacy code, justified patterns, edge cases]
• Follow-up Actions: [Re-review after fixes, pair programming, etc.]

Test Quality Review

Workflow: bmad-testarch-test-review Version: 5.0 (Step-File Architecture)

---

Overview

Review test quality using TEA knowledge base and produce a 0–100 quality score with actionable findings.

Coverage assessment is intentionally out of scope for this workflow. Use trace for requirements coverage and coverage gate decisions.

---

WORKFLOW ARCHITECTURE

This workflow uses step-file architecture:

• Micro-file Design: Each step is self-contained
• JIT Loading: Only the current step file is in memory
• Sequential Enforcement: Execute steps in order

---

INITIALIZATION SEQUENCE

1. Configuration Loading

From workflow.yaml, resolve:

• config_source, test_artifacts, user_name, communication_language, document_output_language, date
• test_dir, review_scope

2. First Step

Load, read completely, and execute: {skill-root}/steps-c/step-01-load-context.md

3. Resume Support

If the user selects Resume mode, load, read completely, and execute: {skill-root}/steps-c/step-01b-resume.md

This checks the output document for progress tracking frontmatter and routes to the next incomplete step.

ADR Quality Readiness Checklist

Purpose: Standardized 8-category, 29-criteria framework for evaluating system testability and NFR compliance during architecture review (Phase 3) and NFR assessment.

When to Use:

• System-level test design (Phase 3): Identify testability gaps in architecture
• NFR assessment workflow: Structured evaluation with evidence
• Gate decisions: Quantifiable criteria (X/29 met = PASS/CONCERNS/FAIL)

How to Use:

1. For each criterion, assess status: ✅ Covered / ⚠️ Gap / ⬜ Not Assessed 2. Document gap description if ⚠️ 3. Describe risk if criterion unmet 4. Map to test scenarios (what tests validate this criterion)

---

1. Testability & Automation

Question: Can we verify this effectively without manual toil?

Common Gaps:

• No mock endpoints for external services (Athena, Milvus, third-party APIs)
• Business logic tightly coupled to UI (requires E2E tests for everything)
• No seeding APIs (manual database setup required)
• ADR has architecture diagrams but no sample API requests

Mitigation Examples:

• 1.1 (Isolation): Provide mock endpoints, dependency injection, interface abstractions
• 1.2 (Headless): Expose all business logic via REST/GraphQL APIs
• 1.3 (State Control): Implement /api/test-data seeding endpoints (dev/staging only)
• 1.4 (Sample Requests): Add "Example API Calls" section to ADR with cURL commands

---

2. Test Data Strategy

Question: How do we fuel our tests safely?

Common Gaps:

• No customer_id scoping in queries (cross-tenant data leakage risk)
• Reliance on production data dumps (GDPR/PII violations)
• No cleanup mechanism (tests leave data behind, polluting environment)

Mitigation Examples:

• 2.1 (Segregation): Enforce customer_id in all queries, add test-specific headers
• 2.2 (Generation): Use Faker library, create synthetic data generators, prohibit prod dumps
• 2.3 (Teardown): Auto-cleanup hooks in test framework, isolated test customer IDs

---

3. Scalability & Availability

Question: Can it grow, and will it stay up?

Common Gaps:

• Stateful session management (can't scale horizontally)
• No load testing, bottlenecks unknown
• SLA undefined or unrealistic (99.99% without redundancy)
• No circuit breakers (cascading failures)

Mitigation Examples:

• 3.1 (Statelessness): Externalize session to Redis/JWT, design for horizontal scaling
• 3.2 (Bottlenecks): Load test with k6, monitor connection pools, identify weak links
• 3.3 (SLA): Define realistic SLA (99.9% = 43 min/month downtime), add redundancy
• 3.4 (Circuit Breakers): Implement circuit breakers (Hystrix pattern), fail fast on errors

---

4. Disaster Recovery (DR)

Question: What happens when the worst-case scenario occurs?

Common Gaps:

• RTO/RPO undefined (no recovery plan)
• Failover never tested (manual process, prone to errors)
• Backups exist but restoration never validated (untested backups = no backups)

Mitigation Examples:

• 4.1 (RTO/RPO): Define RTO (e.g., 4 hours) and RPO (e.g., 1 hour), document recovery procedures
• 4.2 (Failover): Automate multi-region failover, practice failover drills quarterly
• 4.3 (Backups): Implement immutable backups (S3 versioning), test restore monthly

---

5. Security

Question: Is the design safe by default?

Common Gaps:

• Weak authentication (no OAuth, hardcoded API keys)
• No encryption at rest (plaintext in database)
• Secrets in git (API keys, passwords in config files)
• No input validation (vulnerable to SQLi, XSS, command injection)

Mitigation Examples:

• 5.1 (AuthN/AuthZ): Implement OAuth 2.1/OIDC, enforce least privilege, validate scopes
• 5.2 (Encryption): Enable TDE (Transparent Data Encryption), enforce TLS 1.2+
• 5.3 (Secrets): Migrate to AWS Secrets Manager/Vault, scan git history for leaks
• 5.4 (Input Validation): Sanitize all inputs, use parameterized queries, escape outputs

---

6. Monitorability, Debuggability & Manageability

Question: Can we operate and fix this in production?

Common Gaps:

• No distributed tracing (can't debug across microservices)
• Static log levels (requires redeploy to enable DEBUG)
• No metrics endpoint (blind to system health)
• Configuration hardcoded (requires full deploy for minor changes)

Mitigation Examples:

• 6.1 (Tracing): Implement W3C Trace Context, add correlation IDs to all logs
• 6.2 (Logs): Use dynamic log levels (environment variable), structured logging (JSON)
• 6.3 (Metrics): Expose /metrics endpoint, track RED metrics (Rate, Errors, Duration)
• 6.4 (Config): Externalize config (AWS SSM/AppConfig), use feature flags (LaunchDarkly)

---

7. QoS (Quality of Service) & QoE (Quality of Experience)

Question: How does it perform, and how does it feel?

Common Gaps:

• Latency targets undefined (no SLOs)
• No rate limiting (vulnerable to DDoS, noisy neighbors)
• Poor perceived performance (blank screen while loading)
• Raw error messages (stack traces exposed to users)

Mitigation Examples:

• 7.1 (Latency): Define SLOs (P95 <2s, P99 <5s), load test to validate
• 7.2 (Throttling): Implement rate limiting (per-user, per-IP), return 429 with Retry-After
• 7.3 (Perceived Performance): Add skeleton screens, optimistic updates, progressive loading
• 7.4 (Degradation): Implement error boundaries, show friendly messages, log stack traces server-side

---

8. Deployability

Question: How easily can we ship this?

Common Gaps:

• No zero-downtime strategy (requires maintenance window)
• Tight coupling between DB and code (lock-step deployments)
• No automated rollback (manual intervention required)

Mitigation Examples:

• 8.1 (Zero Downtime): Implement Blue/Green or Canary deployments, use feature flags
• 8.2 (Backward Compatibility): Separate DB migrations from code deploys, support N-1 schema
• 8.3 (Rollback): Automate rollback on health check failures, test rollback procedures

---

Usage in Test Design Workflow

System-Level Mode (Phase 3):

In test-design-architecture.md:

• Add "NFR Testability Requirements" section after ASRs
• Use 8 categories with checkboxes (29 criteria)
• For each criterion: Status (⬜ Not Assessed, ⚠️ Gap, ✅ Covered), Gap description, Risk if unmet
• Example:

## NFR Testability Requirements

**Based on ADR Quality Readiness Checklist**

### 1. Testability & Automation

Can we verify this effectively without manual toil?

| Criterion                                                        | Status          | Gap/Requirement                      | Risk if Unmet                           |
| ---------------------------------------------------------------- | --------------- | ------------------------------------ | --------------------------------------- |
| ⬜ Isolation: Can service be tested with downstream deps mocked? | ⚠️ Gap          | No mock endpoints for Athena queries | Flaky tests; can't test in isolation    |
| ⬜ Headless: 100% business logic accessible via API?             | ✅ Covered      | All MCP tools are REST APIs          | N/A                                     |
| ⬜ State Control: Seeding APIs to inject data states?            | ⚠️ Gap          | Need `/api/test-data` endpoints      | Long setup times; can't test edge cases |
| ⬜ Sample Requests: Valid/invalid cURL/JSON samples provided?    | ⬜ Not Assessed | Pending ADR Tool schemas finalized   | Ambiguity on how to consume service     |

**Actions Required:**

- [ ] Backend: Implement mock endpoints for Athena (R-002 blocker)
- [ ] Backend: Implement `/api/test-data` seeding APIs (R-002 blocker)
- [ ] PM: Finalize ADR Tool schemas with sample requests (Q4)

In test-design-qa.md:

• Map each criterion to test scenarios
• Add "NFR Test Coverage Plan" section with P0/P1/P2 priority for each category
• Reference Architecture doc gaps
• Example:

## NFR Test Coverage Plan

**Based on ADR Quality Readiness Checklist**

### 1. Testability & Automation (4 criteria)

**Prerequisites from Architecture doc:**

- [ ] R-002: Test data seeding APIs implemented (blocker)
- [ ] Mock endpoints available for Athena queries

| Criterion                       | Test Scenarios                                                       | Priority | Test Count | Owner            |
| ------------------------------- | -------------------------------------------------------------------- | -------- | ---------- | ---------------- |
| Isolation: Mock downstream deps | Mock Athena queries, Mock Milvus, Service runs isolated              | P1       | 3          | Backend Dev + QA |
| Headless: API-accessible logic  | All MCP tools callable via REST, No UI dependency for business logic | P0       | 5          | QA               |
| State Control: Seeding APIs     | Create test customer, Seed 1000 transactions, Inject edge cases      | P0       | 4          | QA               |
| Sample Requests: cURL examples  | Valid request succeeds, Invalid request fails with clear error       | P1       | 2          | QA               |

**Detailed Test Scenarios:**

- [ ] Isolation: Service runs with Athena mocked (returns fixture data)
- [ ] Isolation: Service runs with Milvus mocked (returns ANN fixture)
- [ ] State Control: Seed test customer with 1000 baseline transactions
- [ ] State Control: Inject edge case (expired subscription user)

---

Usage in NFR Assessment Workflow

Output Structure:

# NFR Assessment: {Feature Name}

**Based on ADR Quality Readiness Checklist (8 categories, 29 criteria)**

## Assessment Summary

| Category                      | Status      | Criteria Met | Evidence                               | Next Action          |
| ----------------------------- | ----------- | ------------ | -------------------------------------- | -------------------- |
| 1. Testability & Automation   | ⚠️ CONCERNS | 2/4          | Mock endpoints missing                 | Implement R-002      |
| 2. Test Data Strategy         | ✅ PASS     | 3/3          | Faker + auto-cleanup                   | None                 |
| 3. Scalability & Availability | ⚠️ CONCERNS | 1/4          | SLA undefined                          | Define SLA           |
| 4. Disaster Recovery          | ⚠️ CONCERNS | 0/3          | No RTO/RPO defined                     | Define recovery plan |
| 5. Security                   | ✅ PASS     | 4/4          | OAuth 2.1 + TLS + Vault + Sanitization | None                 |
| 6. Monitorability             | ⚠️ CONCERNS | 2/4          | No metrics endpoint                    | Add /metrics         |
| 7. QoS & QoE                  | ⚠️ CONCERNS | 1/4          | Latency targets undefined              | Define SLOs          |
| 8. Deployability              | ✅ PASS     | 3/3          | Blue/Green + DB migrations + Rollback  | None                 |

**Overall:** 14/29 criteria met (48%) → ⚠️ CONCERNS

**Gate Decision:** CONCERNS (requires mitigation plan before GA)

---

## Detailed Assessment

### 1. Testability & Automation (2/4 criteria met)

**Question:** Can we verify this effectively without manual toil?

| Criterion                    | Status | Evidence                 | Gap/Action                 |
| ---------------------------- | ------ | ------------------------ | -------------------------- |
| ⬜ Isolation: Mock deps      | ⚠️     | No Athena mock           | Implement mock endpoints   |
| ⬜ Headless: API-accessible  | ✅     | All MCP tools are REST   | N/A                        |
| ⬜ State Control: Seeding    | ⚠️     | `/api/test-data` pending | Pre-implementation blocker |
| ⬜ Sample Requests: Examples | ⬜     | Pending schemas          | Finalize ADR Tools         |

**Overall Status:** ⚠️ CONCERNS (2/4 criteria met)

**Next Actions:**

- [ ] Backend: Implement Athena mock endpoints (pre-implementation)
- [ ] Backend: Implement `/api/test-data` (pre-implementation)
- [ ] PM: Finalize sample requests (implementation phase)

{Repeat for all 8 categories}

---

Benefits

For test-design workflow:

• ✅ Standard NFR structure (same 8 categories every project)
• ✅ Clear testability requirements for Architecture team
• ✅ Direct mapping: criterion → requirement → test scenario
• ✅ Comprehensive coverage (29 criteria = no blind spots)

For nfr-assess workflow:

• ✅ Structured assessment (not ad-hoc)
• ✅ Quantifiable (X/29 criteria met)
• ✅ Evidence-based (each criterion has evidence field)
• ✅ Actionable (gaps → next actions with owners)

For Architecture teams:

• ✅ Clear checklist (29 yes/no questions)
• ✅ Risk-aware (each criterion has "risk if unmet")
• ✅ Scoped work (only implement what's needed, not everything)

For QA teams:

• ✅ Comprehensive test coverage (29 criteria → test scenarios)
• ✅ Clear priorities (P0 for security/isolation, P1 for monitoring, etc.)
• ✅ No ambiguity (each criterion has specific test scenarios)

API Request Utility

Principle

Use typed HTTP client with built-in schema validation and automatic retry for server errors. The utility handles URL resolution, header management, response parsing, and single-line response validation with proper TypeScript support. Works without a browser - ideal for pure API/service testing.

Rationale

Vanilla Playwright's request API requires boilerplate for common patterns:

• Manual JSON parsing (await response.json())
• Repetitive status code checking
• No built-in retry logic for transient failures
• No schema validation
• Complex URL construction

The apiRequest utility provides:

• Automatic JSON parsing: Response body pre-parsed
• Built-in retry: 5xx errors retry with exponential backoff
• Schema validation: Single-line validation (JSON Schema, Zod, OpenAPI)
• URL resolution: Four-tier strategy (explicit > config > Playwright > direct)
• TypeScript generics: Type-safe response bodies
• No browser required: Pure API testing without browser overhead

Pattern Examples

Example 1: Basic API Request

Context: Making authenticated API requests with automatic retry and type safety.

Implementation:

import { test } from '@seontechnologies/playwright-utils/api-request/fixtures';

test('should fetch user data', async ({ apiRequest }) => {
  const { status, body } = await apiRequest<User>({
    method: 'GET',
    path: '/api/users/123',
    headers: { Authorization: 'Bearer token' },
  });

  expect(status).toBe(200);
  expect(body.name).toBe('John Doe'); // TypeScript knows body is User
});

Key Points:

• Generic type <User> provides TypeScript autocomplete for body
• Status and body destructured from response
• Headers passed as object
• Automatic retry for 5xx errors (configurable)

Example 2: Schema Validation (Single Line)

Context: Validate API responses match expected schema with single-line syntax.

Implementation:

import { test } from '@seontechnologies/playwright-utils/api-request/fixtures';
import { z } from 'zod';

// JSON Schema validation
test('should validate response schema (JSON Schema)', async ({ apiRequest }) => {
  const { status, body } = await apiRequest({
    method: 'GET',
    path: '/api/users/123',
    validateSchema: {
      type: 'object',
      required: ['id', 'name', 'email'],
      properties: {
        id: { type: 'string' },
        name: { type: 'string' },
        email: { type: 'string', format: 'email' },
      },
    },
  });
  // Throws if schema validation fails
  expect(status).toBe(200);
});

// Zod schema validation
const UserSchema = z.object({
  id: z.string(),
  name: z.string(),
  email: z.string().email(),
});

test('should validate response schema (Zod)', async ({ apiRequest }) => {
  const { status, body } = await apiRequest({
    method: 'GET',
    path: '/api/users/123',
    validateSchema: UserSchema,
  });
  // Response body is type-safe AND validated
  expect(status).toBe(200);
  expect(body.email).toContain('@');
});

Key Points:

• Single validateSchema parameter
• Supports JSON Schema, Zod, YAML files, OpenAPI specs
• Throws on validation failure with detailed errors
• Zero boilerplate validation code

Example 3: POST with Body and Retry Configuration

Context: Creating resources with custom retry behavior for error testing.

Implementation:

test('should create user', async ({ apiRequest }) => {
  const newUser = {
    name: 'Jane Doe',
    email: 'jane@example.com',
  };

  const { status, body } = await apiRequest({
    method: 'POST',
    path: '/api/users',
    body: newUser, // Automatically sent as JSON
    headers: { Authorization: 'Bearer token' },
  });

  expect(status).toBe(201);
  expect(body.id).toBeDefined();
});

// Disable retry for error testing
test('should handle 500 errors', async ({ apiRequest }) => {
  await expect(
    apiRequest({
      method: 'GET',
      path: '/api/error',
      retryConfig: { maxRetries: 0 }, // Disable retry
    }),
  ).rejects.toThrow('Request failed with status 500');
});

Key Points:

• body parameter auto-serializes to JSON
• Default retry: 5xx errors, 3 retries, exponential backoff
• Disable retry with retryConfig: { maxRetries: 0 }
• Only 5xx errors retry (4xx errors fail immediately)

Example 4: URL Resolution Strategy

Context: Flexible URL handling for different environments and test contexts.

Implementation:

// Strategy 1: Explicit baseUrl (highest priority)
await apiRequest({
  method: 'GET',
  path: '/users',
  baseUrl: 'https://api.example.com', // Uses https://api.example.com/users
});

// Strategy 2: Config baseURL (from fixture)
import { test } from '@seontechnologies/playwright-utils/api-request/fixtures';

test.use({ configBaseUrl: 'https://staging-api.example.com' });

test('uses config baseURL', async ({ apiRequest }) => {
  await apiRequest({
    method: 'GET',
    path: '/users', // Uses https://staging-api.example.com/users
  });
});

// Strategy 3: Playwright baseURL (from playwright.config.ts)
// playwright.config.ts
export default defineConfig({
  use: {
    baseURL: 'https://api.example.com',
  },
});

test('uses Playwright baseURL', async ({ apiRequest }) => {
  await apiRequest({
    method: 'GET',
    path: '/users', // Uses https://api.example.com/users
  });
});

// Strategy 4: Direct path (full URL)
await apiRequest({
  method: 'GET',
  path: 'https://api.example.com/users', // Full URL works too
});

Key Points:

• Four-tier resolution: explicit > config > Playwright > direct
• Trailing slashes normalized automatically
• Environment-specific baseUrl easy to configure

Example 5: Integration with Recurse (Polling)

Context: Waiting for async operations to complete (background jobs, eventual consistency).

Implementation:

import { test } from '@seontechnologies/playwright-utils/fixtures';

test('should poll until job completes', async ({ apiRequest, recurse }) => {
  // Create job
  const { body } = await apiRequest({
    method: 'POST',
    path: '/api/jobs',
    body: { type: 'export' },
  });

  const jobId = body.id;

  // Poll until ready
  const completedJob = await recurse(
    () => apiRequest({ method: 'GET', path: `/api/jobs/${jobId}` }),
    (response) => response.body.status === 'completed',
    { timeout: 60000, interval: 2000 },
  );

  expect(completedJob.body.result).toBeDefined();
});

Key Points:

• apiRequest returns full response object
• recurse polls until predicate returns true
• Composable utilities work together seamlessly

Example 6: Microservice Testing (Multiple Services)

Context: Test interactions between microservices without a browser.

Implementation:

import { test, expect } from '@seontechnologies/playwright-utils/fixtures';

const USER_SERVICE = process.env.USER_SERVICE_URL || 'http://localhost:3001';
const ORDER_SERVICE = process.env.ORDER_SERVICE_URL || 'http://localhost:3002';

test.describe('Microservice Integration', () => {
  test('should validate cross-service user lookup', async ({ apiRequest }) => {
    // Create user in user-service
    const { body: user } = await apiRequest({
      method: 'POST',
      path: '/api/users',
      baseUrl: USER_SERVICE,
      body: { name: 'Test User', email: 'test@example.com' },
    });

    // Create order in order-service (validates user via user-service)
    const { status, body: order } = await apiRequest({
      method: 'POST',
      path: '/api/orders',
      baseUrl: ORDER_SERVICE,
      body: {
        userId: user.id,
        items: [{ productId: 'prod-1', quantity: 2 }],
      },
    });

    expect(status).toBe(201);
    expect(order.userId).toBe(user.id);
  });

  test('should reject order for invalid user', async ({ apiRequest }) => {
    const { status, body } = await apiRequest({
      method: 'POST',
      path: '/api/orders',
      baseUrl: ORDER_SERVICE,
      body: {
        userId: 'non-existent-user',
        items: [{ productId: 'prod-1', quantity: 1 }],
      },
    });

    expect(status).toBe(400);
    expect(body.code).toBe('INVALID_USER');
  });
});

Key Points:

• Test multiple services without browser
• Use baseUrl to target different services
• Validate cross-service communication
• Pure API testing - fast and reliable

Example 7: GraphQL API Testing

Context: Test GraphQL endpoints with queries and mutations.

Implementation:

test.describe('GraphQL API', () => {
  const GRAPHQL_ENDPOINT = '/graphql';

  test('should query users via GraphQL', async ({ apiRequest }) => {
    const query = `
      query GetUsers($limit: Int) {
        users(limit: $limit) {
          id
          name
          email
        }
      }
    `;

    const { status, body } = await apiRequest({
      method: 'POST',
      path: GRAPHQL_ENDPOINT,
      body: {
        query,
        variables: { limit: 10 },
      },
    });

    expect(status).toBe(200);
    expect(body.errors).toBeUndefined();
    expect(body.data.users).toHaveLength(10);
  });

  test('should create user via mutation', async ({ apiRequest }) => {
    const mutation = `
      mutation CreateUser($input: CreateUserInput!) {
        createUser(input: $input) {
          id
          name
        }
      }
    `;

    const { status, body } = await apiRequest({
      method: 'POST',
      path: GRAPHQL_ENDPOINT,
      body: {
        query: mutation,
        variables: {
          input: { name: 'GraphQL User', email: 'gql@example.com' },
        },
      },
    });

    expect(status).toBe(200);
    expect(body.data.createUser.id).toBeDefined();
  });
});

Key Points:

• GraphQL via POST request
• Variables in request body
• Check body.errors for GraphQL errors (not status code)
• Works for queries and mutations

Example 8: Operation-Based Overload (OpenAPI / Code Generators)

Context: When using a code generator (orval, openapi-generator, custom scripts) that produces typed operation definitions from an OpenAPI spec, pass the operation object directly to apiRequest. This eliminates manual method/path extraction and typeof assertions while preserving full type inference for request body, response, and query parameters. Available since v3.14.0.

Implementation:

// Generated operation definition — structural typing, no import from playwright-utils needed
// type OperationShape = { path: string; method: 'POST'|'GET'|'PUT'|'DELETE'|'PATCH'|'HEAD'; response: unknown; request: unknown; query?: unknown }

import { test, expect } from '@seontechnologies/playwright-utils/api-request/fixtures';

// --- Basic usage: operation replaces method + path ---
test('should upsert person via operation overload', async ({ apiRequest }) => {
  const { status, body } = await apiRequest({
    operation: upsertPersonv2({ customerId }),
    headers: getHeaders(customerId),
    body: personInput, // compile-time typed as Schemas.PersonInput
  });

  expect(status).toBe(200);
  expect(body.id).toBeDefined(); // body typed as Schemas.Person
});

// --- Typed query parameters (replaces string concatenation) ---
test('should list people with typed query', async ({ apiRequest }) => {
  const { body } = await apiRequest({
    operation: getPeoplev2({ customerId }),
    headers: getHeaders(customerId),
    query: { page: 0, page_size: 5 }, // typed from operation's query definition
  });

  expect(body.items).toHaveLength(5);
});

// --- Params escape hatch (pre-formatted query strings) ---
test('should fetch billing history with raw params', async ({ apiRequest }) => {
  const { body } = await apiRequest({
    operation: getBillingHistoryv2({ customerId }),
    headers: getHeaders(customerId),
    params: {
      'filters[start_date]': getThisMonthTimestamp(),
      'filters[date_type]': 'MONTH',
    },
  });

  expect(body.entries.length).toBeGreaterThan(0);
});

// --- Works with recurse (polling) ---
test('should poll until person is reviewed', async ({ apiRequest, recurse }) => {
  await recurse(
    async () =>
      apiRequest({
        operation: getPersonv2({ customerId, hash }),
        headers: getHeaders(customerId),
      }),
    (res) => {
      expect(res.status).toBe(200);
      expect(res.body.status).toBe('REVIEWED');
    },
    { timeout: 30000, interval: 1000 },
  );
});

// --- Schema validation chains work identically ---
test('should create movie with schema validation', async ({ apiRequest }) => {
  const { body } = await apiRequest({
    operation: createMovieOp,
    headers: commonHeaders(authToken),
    body: movie,
  }).validateSchema(CreateMovieResponseSchema, {
    shape: { status: 200, data: { name: movie.name } },
  });

  expect(body.data.id).toBeDefined();
});

Key Points:

• Pass operation instead of method + path — mutually exclusive at compile time
• Response body, request body, and query types inferred from operation definition
• Uses structural typing (duck typing) — works with any code generator producing { path, method, response, request, query? }
• query field auto-serializes to bracket notation (filters[type]=pep, ids[0]=10)
• params escape hatch for pre-formatted strings — wins over query on conflict
• Fully composable with recurse, validateSchema, and all existing features
• response/request/query on the operation are type-level only — runtime never reads their values

Comparison with Vanilla Playwright

When to Use

Use apiRequest for:

• ✅ Pure API/service testing (no browser needed)
• ✅ Microservice integration testing
• ✅ GraphQL API testing
• ✅ Schema validation needs
• ✅ Tests requiring retry logic
• ✅ Background API calls in UI tests
• ✅ Contract testing support
• ✅ Type-safe API testing with OpenAPI-generated operations (v3.14.0+)

Stick with vanilla Playwright for:

• Simple one-off requests where utility overhead isn't worth it
• Testing Playwright's native features specifically
• Legacy tests where migration isn't justified

Related Fragments

• api-testing-patterns.md - Comprehensive pure API testing patterns
• overview.md - Installation and design principles
• auth-session.md - Authentication token management
• recurse.md - Polling for async operations
• fixtures-composition.md - Combining utilities with mergeTests
• log.md - Logging API requests
• contract-testing.md - Pact contract testing

Anti-Patterns

❌ Ignoring retry failures:

try {
  await apiRequest({ method: 'GET', path: '/api/unstable' });
} catch {
  // Silent failure - loses retry information
}

✅ Let retries happen, handle final failure:

await expect(apiRequest({ method: 'GET', path: '/api/unstable' })).rejects.toThrow(); // Retries happen automatically, then final error caught

❌ Disabling TypeScript benefits:

const response: any = await apiRequest({ method: 'GET', path: '/users' });

✅ Use generic types:

const { body } = await apiRequest<User[]>({ method: 'GET', path: '/users' });
// body is typed as User[]

❌ Mixing operation overload with explicit generics:

// Don't pass a generic when using operation — types are inferred from the operation
const { body } = await apiRequest<MyType>({
  operation: getPersonv2({ customerId }),
  headers: getHeaders(customerId),
});

✅ Let the operation infer the types:

const { body } = await apiRequest({
  operation: getPersonv2({ customerId }),
  headers: getHeaders(customerId),
});
// body type inferred from operation.response

❌ Mixing operation with method/path:

// Compile error — operation and method/path are mutually exclusive
await apiRequest({
  operation: getPersonv2({ customerId }),
  method: 'GET', // Error: method?: never
  path: '/api/person', // Error: path?: never
});

API Testing Patterns

Principle

Test APIs and backend services directly without browser overhead. Use Playwright's request context for HTTP operations, apiRequest utility for enhanced features, and recurse for async operations. Pure API tests run faster, are more stable, and provide better coverage for service-layer logic.

Rationale

Many teams over-rely on E2E/browser tests when API tests would be more appropriate:

• Slower feedback: Browser tests take seconds, API tests take milliseconds
• More brittle: UI changes break tests even when API works correctly
• Wrong abstraction: Testing business logic through UI layers adds noise
• Resource heavy: Browsers consume memory and CPU

API-first testing provides:

• Fast execution: No browser startup, no rendering, no JavaScript execution
• Direct validation: Test exactly what the service returns
• Better isolation: Test service logic independent of UI
• Easier debugging: Clear request/response without DOM noise
• Contract validation: Verify API contracts explicitly

When to Use API Tests vs E2E Tests

Rule of thumb: If you're testing what the server returns (not how it looks), use API tests.

Pattern Examples

Example 1: Pure API Test (No Browser)

Context: Test REST API endpoints directly without any browser context.

Implementation:

// tests/api/users.spec.ts
import { test, expect } from '@playwright/test';

// No page, no browser - just API
test.describe('Users API', () => {
  test('should create user', async ({ request }) => {
    const response = await request.post('/api/users', {
      data: {
        name: 'John Doe',
        email: 'john@example.com',
        role: 'user',
      },
    });

    expect(response.status()).toBe(201);

    const user = await response.json();
    expect(user.id).toBeDefined();
    expect(user.name).toBe('John Doe');
    expect(user.email).toBe('john@example.com');
  });

  test('should get user by ID', async ({ request }) => {
    // Create user first
    const createResponse = await request.post('/api/users', {
      data: { name: 'Jane Doe', email: 'jane@example.com' },
    });
    const { id } = await createResponse.json();

    // Get user
    const getResponse = await request.get(`/api/users/${id}`);
    expect(getResponse.status()).toBe(200);

    const user = await getResponse.json();
    expect(user.id).toBe(id);
    expect(user.name).toBe('Jane Doe');
  });

  test('should return 404 for non-existent user', async ({ request }) => {
    const response = await request.get('/api/users/non-existent-id');
    expect(response.status()).toBe(404);

    const error = await response.json();
    expect(error.code).toBe('USER_NOT_FOUND');
  });

  test('should validate required fields', async ({ request }) => {
    const response = await request.post('/api/users', {
      data: { name: 'Missing Email' }, // email is required
    });

    expect(response.status()).toBe(400);

    const error = await response.json();
    expect(error.code).toBe('VALIDATION_ERROR');
    expect(error.details).toContainEqual(expect.objectContaining({ field: 'email', message: expect.any(String) }));
  });
});

Key Points:

• No page fixture needed - only request
• Tests run without browser overhead
• Direct HTTP assertions
• Clear error handling tests

Example 2: API Test with apiRequest Utility

Context: Use enhanced apiRequest for schema validation, retry, and type safety.

Implementation:

// tests/api/orders.spec.ts
import { test, expect } from '@seontechnologies/playwright-utils/api-request/fixtures';
import { z } from 'zod';

// Define schema for type safety and validation
const OrderSchema = z.object({
  id: z.string().uuid(),
  userId: z.string(),
  items: z.array(
    z.object({
      productId: z.string(),
      quantity: z.number().positive(),
      price: z.number().positive(),
    }),
  ),
  total: z.number().positive(),
  status: z.enum(['pending', 'processing', 'shipped', 'delivered']),
  createdAt: z.string().datetime(),
});

type Order = z.infer<typeof OrderSchema>;

test.describe('Orders API', () => {
  test('should create order with schema validation', async ({ apiRequest }) => {
    const { status, body } = await apiRequest<Order>({
      method: 'POST',
      path: '/api/orders',
      body: {
        userId: 'user-123',
        items: [
          { productId: 'prod-1', quantity: 2, price: 29.99 },
          { productId: 'prod-2', quantity: 1, price: 49.99 },
        ],
      },
      validateSchema: OrderSchema, // Validates response matches schema
    });

    expect(status).toBe(201);
    expect(body.id).toBeDefined();
    expect(body.status).toBe('pending');
    expect(body.total).toBe(109.97); // 2*29.99 + 49.99
  });

  test('should handle server errors with retry', async ({ apiRequest }) => {
    // apiRequest retries 5xx errors by default
    const { status, body } = await apiRequest({
      method: 'GET',
      path: '/api/orders/order-123',
      retryConfig: {
        maxRetries: 3,
        retryDelay: 1000,
      },
    });

    expect(status).toBe(200);
  });

  test('should list orders with pagination', async ({ apiRequest }) => {
    const { status, body } = await apiRequest<{ orders: Order[]; total: number; page: number }>({
      method: 'GET',
      path: '/api/orders',
      params: { page: 1, limit: 10, status: 'pending' },
    });

    expect(status).toBe(200);
    expect(body.orders).toHaveLength(10);
    expect(body.total).toBeGreaterThan(10);
    expect(body.page).toBe(1);
  });
});

Key Points:

• Zod schema for runtime validation AND TypeScript types
• validateSchema throws if response doesn't match
• Built-in retry for transient failures
• Type-safe body access
• Note: If your project uses code-generated operations from an OpenAPI spec, see Example 8 for the preferred operation-based overload (v3.14.0+)

Example 3: Microservice-to-Microservice Testing

Context: Test service interactions without browser - validate API contracts between services.

Implementation:

// tests/api/service-integration.spec.ts
import { test, expect } from '@seontechnologies/playwright-utils/fixtures';

test.describe('Service Integration', () => {
  const USER_SERVICE_URL = process.env.USER_SERVICE_URL || 'http://localhost:3001';
  const ORDER_SERVICE_URL = process.env.ORDER_SERVICE_URL || 'http://localhost:3002';
  const INVENTORY_SERVICE_URL = process.env.INVENTORY_SERVICE_URL || 'http://localhost:3003';

  test('order service should validate user exists', async ({ apiRequest }) => {
    // Create user in user-service
    const { body: user } = await apiRequest({
      method: 'POST',
      path: '/api/users',
      baseUrl: USER_SERVICE_URL,
      body: { name: 'Test User', email: 'test@example.com' },
    });

    // Create order in order-service (should validate user via user-service)
    const { status, body: order } = await apiRequest({
      method: 'POST',
      path: '/api/orders',
      baseUrl: ORDER_SERVICE_URL,
      body: {
        userId: user.id,
        items: [{ productId: 'prod-1', quantity: 1 }],
      },
    });

    expect(status).toBe(201);
    expect(order.userId).toBe(user.id);
  });

  test('order service should reject invalid user', async ({ apiRequest }) => {
    const { status, body } = await apiRequest({
      method: 'POST',
      path: '/api/orders',
      baseUrl: ORDER_SERVICE_URL,
      body: {
        userId: 'non-existent-user',
        items: [{ productId: 'prod-1', quantity: 1 }],
      },
    });

    expect(status).toBe(400);
    expect(body.code).toBe('INVALID_USER');
  });

  test('order should decrease inventory', async ({ apiRequest, recurse }) => {
    // Get initial inventory
    const { body: initialInventory } = await apiRequest({
      method: 'GET',
      path: '/api/inventory/prod-1',
      baseUrl: INVENTORY_SERVICE_URL,
    });

    // Create order
    await apiRequest({
      method: 'POST',
      path: '/api/orders',
      baseUrl: ORDER_SERVICE_URL,
      body: {
        userId: 'user-123',
        items: [{ productId: 'prod-1', quantity: 2 }],
      },
    });

    // Poll for inventory update (eventual consistency)
    const { body: updatedInventory } = await recurse(
      () =>
        apiRequest({
          method: 'GET',
          path: '/api/inventory/prod-1',
          baseUrl: INVENTORY_SERVICE_URL,
        }),
      (response) => response.body.quantity === initialInventory.quantity - 2,
      { timeout: 10000, interval: 500 },
    );

    expect(updatedInventory.quantity).toBe(initialInventory.quantity - 2);
  });
});

Key Points:

• Multiple service URLs for microservice testing
• Tests service-to-service communication
• Uses recurse for eventual consistency
• No browser needed for full integration testing

Example 4: GraphQL API Testing

Context: Test GraphQL endpoints with queries and mutations.

Implementation:

// tests/api/graphql.spec.ts
import { test, expect } from '@seontechnologies/playwright-utils/api-request/fixtures';

const GRAPHQL_ENDPOINT = '/graphql';

test.describe('GraphQL API', () => {
  test('should query users', async ({ apiRequest }) => {
    const query = `
      query GetUsers($limit: Int) {
        users(limit: $limit) {
          id
          name
          email
          role
        }
      }
    `;

    const { status, body } = await apiRequest({
      method: 'POST',
      path: GRAPHQL_ENDPOINT,
      body: {
        query,
        variables: { limit: 10 },
      },
    });

    expect(status).toBe(200);
    expect(body.errors).toBeUndefined();
    expect(body.data.users).toHaveLength(10);
    expect(body.data.users[0]).toHaveProperty('id');
    expect(body.data.users[0]).toHaveProperty('name');
  });

  test('should create user via mutation', async ({ apiRequest }) => {
    const mutation = `
      mutation CreateUser($input: CreateUserInput!) {
        createUser(input: $input) {
          id
          name
          email
        }
      }
    `;

    const { status, body } = await apiRequest({
      method: 'POST',
      path: GRAPHQL_ENDPOINT,
      body: {
        query: mutation,
        variables: {
          input: {
            name: 'GraphQL User',
            email: 'graphql@example.com',
          },
        },
      },
    });

    expect(status).toBe(200);
    expect(body.errors).toBeUndefined();
    expect(body.data.createUser.id).toBeDefined();
    expect(body.data.createUser.name).toBe('GraphQL User');
  });

  test('should handle GraphQL errors', async ({ apiRequest }) => {
    const query = `
      query GetUser($id: ID!) {
        user(id: $id) {
          id
          name
        }
      }
    `;

    const { status, body } = await apiRequest({
      method: 'POST',
      path: GRAPHQL_ENDPOINT,
      body: {
        query,
        variables: { id: 'non-existent' },
      },
    });

    expect(status).toBe(200); // GraphQL returns 200 even for errors
    expect(body.errors).toBeDefined();
    expect(body.errors[0].message).toContain('not found');
    expect(body.data.user).toBeNull();
  });

  test('should handle validation errors', async ({ apiRequest }) => {
    const mutation = `
      mutation CreateUser($input: CreateUserInput!) {
        createUser(input: $input) {
          id
        }
      }
    `;

    const { status, body } = await apiRequest({
      method: 'POST',
      path: GRAPHQL_ENDPOINT,
      body: {
        query: mutation,
        variables: {
          input: {
            name: '', // Invalid: empty name
            email: 'invalid-email', // Invalid: bad format
          },
        },
      },
    });

    expect(status).toBe(200);
    expect(body.errors).toBeDefined();
    expect(body.errors[0].extensions.code).toBe('BAD_USER_INPUT');
  });
});

Key Points:

• GraphQL queries and mutations via POST
• Variables passed in request body
• GraphQL returns 200 even for errors (check body.errors)
• Test validation and business logic errors

Example 5: Database Seeding and Cleanup via API

Context: Use API calls to set up and tear down test data without direct database access.

Implementation:

// tests/api/with-data-setup.spec.ts
import { test, expect } from '@seontechnologies/playwright-utils/fixtures';

test.describe('Orders with Data Setup', () => {
  let testUser: { id: string; email: string };
  let testProducts: Array<{ id: string; name: string; price: number }>;

  test.beforeAll(async ({ request }) => {
    // Seed user via API
    const userResponse = await request.post('/api/users', {
      data: {
        name: 'Test User',
        email: `test-${Date.now()}@example.com`,
      },
    });
    testUser = await userResponse.json();

    // Seed products via API
    testProducts = [];
    for (const product of [
      { name: 'Widget A', price: 29.99 },
      { name: 'Widget B', price: 49.99 },
      { name: 'Widget C', price: 99.99 },
    ]) {
      const productResponse = await request.post('/api/products', {
        data: product,
      });
      testProducts.push(await productResponse.json());
    }
  });

  test.afterAll(async ({ request }) => {
    // Cleanup via API
    if (testUser?.id) {
      await request.delete(`/api/users/${testUser.id}`);
    }
    for (const product of testProducts) {
      await request.delete(`/api/products/${product.id}`);
    }
  });

  test('should create order with seeded data', async ({ apiRequest }) => {
    const { status, body } = await apiRequest({
      method: 'POST',
      path: '/api/orders',
      body: {
        userId: testUser.id,
        items: [
          { productId: testProducts[0].id, quantity: 2 },
          { productId: testProducts[1].id, quantity: 1 },
        ],
      },
    });

    expect(status).toBe(201);
    expect(body.userId).toBe(testUser.id);
    expect(body.items).toHaveLength(2);
    expect(body.total).toBe(2 * 29.99 + 49.99);
  });

  test('should list user orders', async ({ apiRequest }) => {
    // Create an order first
    await apiRequest({
      method: 'POST',
      path: '/api/orders',
      body: {
        userId: testUser.id,
        items: [{ productId: testProducts[2].id, quantity: 1 }],
      },
    });

    // List orders for user
    const { status, body } = await apiRequest({
      method: 'GET',
      path: '/api/orders',
      params: { userId: testUser.id },
    });

    expect(status).toBe(200);
    expect(body.orders.length).toBeGreaterThanOrEqual(1);
    expect(body.orders.every((o: any) => o.userId === testUser.id)).toBe(true);
  });
});

Key Points:

• beforeAll/afterAll for test data setup/cleanup
• API-based seeding (no direct DB access needed)
• Unique emails to prevent conflicts in parallel runs
• Cleanup after all tests complete

Example 6: Background Job Testing with Recurse

Context: Test async operations like background jobs, webhooks, and eventual consistency.

Implementation:

// tests/api/background-jobs.spec.ts
import { test, expect } from '@seontechnologies/playwright-utils/fixtures';

test.describe('Background Jobs', () => {
  test('should process export job', async ({ apiRequest, recurse }) => {
    // Trigger export job
    const { body: job } = await apiRequest({
      method: 'POST',
      path: '/api/exports',
      body: {
        type: 'users',
        format: 'csv',
        filters: { createdAfter: '2024-01-01' },
      },
    });

    expect(job.id).toBeDefined();
    expect(job.status).toBe('pending');

    // Poll until job completes
    const { body: completedJob } = await recurse(
      () => apiRequest({ method: 'GET', path: `/api/exports/${job.id}` }),
      (response) => response.body.status === 'completed',
      {
        timeout: 60000,
        interval: 2000,
        log: `Waiting for export job ${job.id} to complete`,
      },
    );

    expect(completedJob.status).toBe('completed');
    expect(completedJob.downloadUrl).toBeDefined();
    expect(completedJob.recordCount).toBeGreaterThan(0);
  });

  test('should handle job failure gracefully', async ({ apiRequest, recurse }) => {
    // Trigger job that will fail
    const { body: job } = await apiRequest({
      method: 'POST',
      path: '/api/exports',
      body: {
        type: 'invalid-type', // This will cause failure
        format: 'csv',
      },
    });

    // Poll until job fails
    const { body: failedJob } = await recurse(
      () => apiRequest({ method: 'GET', path: `/api/exports/${job.id}` }),
      (response) => ['completed', 'failed'].includes(response.body.status),
      { timeout: 30000 },
    );

    expect(failedJob.status).toBe('failed');
    expect(failedJob.error).toBeDefined();
    expect(failedJob.error.code).toBe('INVALID_EXPORT_TYPE');
  });

  test('should process webhook delivery', async ({ apiRequest, recurse }) => {
    // Trigger action that sends webhook
    const { body: order } = await apiRequest({
      method: 'POST',
      path: '/api/orders',
      body: {
        userId: 'user-123',
        items: [{ productId: 'prod-1', quantity: 1 }],
        webhookUrl: 'https://webhook.site/test-endpoint',
      },
    });

    // Poll for webhook delivery status
    const { body: webhookStatus } = await recurse(
      () => apiRequest({ method: 'GET', path: `/api/webhooks/order/${order.id}` }),
      (response) => response.body.delivered === true,
      { timeout: 30000, interval: 1000 },
    );

    expect(webhookStatus.delivered).toBe(true);
    expect(webhookStatus.deliveredAt).toBeDefined();
    expect(webhookStatus.responseStatus).toBe(200);
  });
});

Key Points:

• recurse for polling async operations
• Test both success and failure scenarios
• Configurable timeout and interval
• Log messages for debugging

Example 7: Service Authentication (No Browser)

Context: Test authenticated API endpoints using tokens directly - no browser login needed.

Implementation:

// tests/api/authenticated.spec.ts
import { test, expect } from '@seontechnologies/playwright-utils/fixtures';

test.describe('Authenticated API Tests', () => {
  let authToken: string;

  test.beforeAll(async ({ request }) => {
    // Get token via API (no browser!)
    const response = await request.post('/api/auth/login', {
      data: {
        email: process.env.TEST_USER_EMAIL,
        password: process.env.TEST_USER_PASSWORD,
      },
    });

    const { token } = await response.json();
    authToken = token;
  });

  test('should access protected endpoint with token', async ({ apiRequest }) => {
    const { status, body } = await apiRequest({
      method: 'GET',
      path: '/api/me',
      headers: {
        Authorization: `Bearer ${authToken}`,
      },
    });

    expect(status).toBe(200);
    expect(body.email).toBe(process.env.TEST_USER_EMAIL);
  });

  test('should reject request without token', async ({ apiRequest }) => {
    const { status, body } = await apiRequest({
      method: 'GET',
      path: '/api/me',
      // No Authorization header
    });

    expect(status).toBe(401);
    expect(body.code).toBe('UNAUTHORIZED');
  });

  test('should reject expired token', async ({ apiRequest }) => {
    const expiredToken = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...'; // Expired token

    const { status, body } = await apiRequest({
      method: 'GET',
      path: '/api/me',
      headers: {
        Authorization: `Bearer ${expiredToken}`,
      },
    });

    expect(status).toBe(401);
    expect(body.code).toBe('TOKEN_EXPIRED');
  });

  test('should handle role-based access', async ({ apiRequest }) => {
    // User token (non-admin)
    const { status } = await apiRequest({
      method: 'GET',
      path: '/api/admin/users',
      headers: {
        Authorization: `Bearer ${authToken}`,
      },
    });

    expect(status).toBe(403); // Forbidden for non-admin
  });
});

Key Points:

• Token obtained via API login (no browser)
• Token reused across all tests in describe block
• Test auth, expired tokens, and RBAC
• Pure API testing without UI

Example 8: Operation-Based API Testing (OpenAPI / Code Generators)

Context: When your project uses code-generated operation definitions from an OpenAPI spec, leverage the operation-based overload of apiRequest (v3.14.0+) instead of manual method/path extraction. This eliminates typeof assertions and provides full type inference for request body, response, and query parameters.

Implementation:

// tests/api/operations.spec.ts
import { test, expect } from '@seontechnologies/playwright-utils/api-request/fixtures';

test.describe('API Tests with Generated Operations', () => {
  test('should create entity with full type safety', async ({ apiRequest }) => {
    // Operation object from code generator — contains path, method, and type info
    const { status, body } = await apiRequest({
      operation: createEntityOp({ workspaceId }),
      headers: getHeaders(workspaceId),
      body: entityInput, // Compile-time typed from operation.request
    });

    expect(status).toBe(201);
    expect(body.id).toBeDefined(); // body typed from operation.response
  });

  test('should list with typed query parameters', async ({ apiRequest }) => {
    // query field replaces manual string concatenation
    const { body } = await apiRequest({
      operation: listEntitiesOp({ workspaceId }),
      headers: getHeaders(workspaceId),
      query: { page: 0, page_size: 10, status: 'active' },
    });

    expect(body.items).toHaveLength(10);
    expect(body.total).toBeGreaterThan(10);
  });

  test('should poll async operation until complete', async ({ apiRequest, recurse }) => {
    const { body: job } = await apiRequest({
      operation: startJobOp({ workspaceId }),
      headers: getHeaders(workspaceId),
      body: { type: 'export' },
    });

    await recurse(
      async () =>
        apiRequest({
          operation: getJobOp({ workspaceId, jobId: job.id }),
          headers: getHeaders(workspaceId),
        }),
      (res) => res.body.status === 'completed',
      { timeout: 60000, interval: 2000 },
    );
  });
});

Key Points:

• operation replaces method + path — mutually exclusive at compile time
• Types for body, response, and query all inferred from the operation definition
• Works with any code generator using structural typing (no imports from playwright-utils needed in generator)
• Composable with recurse, validateSchema, and all existing apiRequest features
• Preferred approach over typeof operation.response for generated operations

API Test Configuration

Playwright Config for API-Only Tests

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests/api',

  // No browser needed for API tests
  use: {
    baseURL: process.env.API_URL || 'http://localhost:3000',
    extraHTTPHeaders: {
      Accept: 'application/json',
      'Content-Type': 'application/json',
    },
  },

  // Faster without browser overhead
  timeout: 30000,

  // Run API tests in parallel
  workers: 4,
  fullyParallel: true,

  // No screenshots/traces needed for API tests
  reporter: [['html'], ['json', { outputFile: 'api-test-results.json' }]],
});

Separate API Test Project

// playwright.config.ts
export default defineConfig({
  projects: [
    {
      name: 'api',
      testDir: './tests/api',
      use: {
        baseURL: process.env.API_URL,
      },
    },
    {
      name: 'e2e',
      testDir: './tests/e2e',
      use: {
        baseURL: process.env.APP_URL,
        ...devices['Desktop Chrome'],
      },
    },
  ],
});

Comparison: API Tests vs E2E Tests

Related Fragments

• api-request.md - apiRequest utility details
• recurse.md - Polling patterns for async operations
• auth-session.md - Token management
• contract-testing.md - Pact contract testing
• test-levels-framework.md - When to use which test level
• data-factories.md - Test data setup patterns

Anti-Patterns

DON'T use E2E for API validation:

// Bad: Testing API through UI
test('validate user creation', async ({ page }) => {
  await page.goto('/admin/users');
  await page.fill('#name', 'John');
  await page.click('#submit');
  await expect(page.getByText('User created')).toBeVisible();
});

DO test APIs directly:

// Good: Direct API test
test('validate user creation', async ({ apiRequest }) => {
  const { status, body } = await apiRequest({
    method: 'POST',
    path: '/api/users',
    body: { name: 'John' },
  });
  expect(status).toBe(201);
  expect(body.id).toBeDefined();
});

DON'T ignore API tests because "E2E covers it":

// Bad thinking: "Our E2E tests create users, so API is tested"
// Reality: E2E tests one happy path; API tests cover edge cases

DO have dedicated API test coverage:

// Good: Explicit API test suite
test.describe('Users API', () => {
  test('creates user', async ({ apiRequest }) => {
    /* ... */
  });
  test('handles duplicate email', async ({ apiRequest }) => {
    /* ... */
  });
  test('validates required fields', async ({ apiRequest }) => {
    /* ... */
  });
  test('handles malformed JSON', async ({ apiRequest }) => {
    /* ... */
  });
  test('rate limits requests', async ({ apiRequest }) => {
    /* ... */
  });
});

Auth Session Utility

Principle

Persist authentication tokens to disk and reuse across test runs. Support multiple user identifiers, ephemeral authentication, and worker-specific accounts for parallel execution. Fetch tokens once, use everywhere. Works for both API-only tests and browser tests.

Rationale

Playwright's built-in authentication works but has limitations:

• Re-authenticates for every test run (slow)
• Single user per project setup
• No token expiration handling
• Manual session management
• Complex setup for multi-user scenarios

The auth-session utility provides:

• Token persistence: Authenticate once, reuse across runs
• Multi-user support: Different user identifiers in same test suite
• Ephemeral auth: On-the-fly user authentication without disk persistence
• Worker-specific accounts: Parallel execution with isolated user accounts
• Automatic token management: Checks validity, renews if expired
• Flexible provider pattern: Adapt to any auth system (OAuth2, JWT, custom)
• API-first design: Get tokens for API tests without browser overhead

Pattern Examples

Example 1: Basic Auth Session Setup

Context: Configure global authentication that persists across test runs.

Implementation:

// Step 1: Configure in global-setup.ts
import { authStorageInit, setAuthProvider, configureAuthSession, authGlobalInit } from '@seontechnologies/playwright-utils/auth-session';
import myCustomProvider from './auth/custom-auth-provider';

async function globalSetup() {
  // Ensure storage directories exist
  authStorageInit();

  // Configure storage path
  configureAuthSession({
    authStoragePath: process.cwd() + '/playwright/auth-sessions',
    debug: true,
  });

  // Set custom provider (HOW to authenticate)
  setAuthProvider(myCustomProvider);

  // Optional: pre-fetch token for default user
  await authGlobalInit();
}

export default globalSetup;

// Step 2: Create auth fixture
import { test as base } from '@playwright/test';
import { createAuthFixtures, setAuthProvider } from '@seontechnologies/playwright-utils/auth-session';
import myCustomProvider from './custom-auth-provider';

// Register provider early
setAuthProvider(myCustomProvider);

export const test = base.extend(createAuthFixtures());

// Step 3: Use in tests
test('authenticated request', async ({ authToken, request }) => {
  const response = await request.get('/api/protected', {
    headers: { Authorization: `Bearer ${authToken}` },
  });

  expect(response.ok()).toBeTruthy();
});

Key Points:

• Global setup runs once before all tests
• Token fetched once, reused across all tests
• Custom provider defines your auth mechanism
• Order matters: configure, then setProvider, then init

Example 2: Multi-User Authentication

Context: Testing with different user roles (admin, regular user, guest) in same test suite.

Implementation:

import { test } from '../support/auth/auth-fixture';

// Option 1: Per-test user override
test('admin actions', async ({ authToken, authOptions }) => {
  // Override default user
  authOptions.userIdentifier = 'admin';

  const { authToken: adminToken } = await test.step('Get admin token', async () => {
    return { authToken }; // Re-fetches with new identifier
  });

  // Use admin token
  const response = await request.get('/api/admin/users', {
    headers: { Authorization: `Bearer ${adminToken}` },
  });
});

// Option 2: Parallel execution with different users
test.describe.parallel('multi-user tests', () => {
  test('user 1 actions', async ({ authToken }) => {
    // Uses default user (e.g., 'user1')
  });

  test('user 2 actions', async ({ authToken, authOptions }) => {
    authOptions.userIdentifier = 'user2';
    // Uses different token for user2
  });
});

Key Points:

• Override authOptions.userIdentifier per test
• Tokens cached separately per user identifier
• Parallel tests isolated with different users
• Worker-specific accounts possible

Example 3: Ephemeral User Authentication

Context: Create temporary test users that don't persist to disk (e.g., testing user creation flow).

Implementation:

import { applyUserCookiesToBrowserContext } from '@seontechnologies/playwright-utils/auth-session';
import { createTestUser } from '../utils/user-factory';

test('ephemeral user test', async ({ context, page }) => {
  // Create temporary user (not persisted)
  const ephemeralUser = await createTestUser({
    role: 'admin',
    permissions: ['delete-users'],
  });

  // Apply auth directly to browser context
  await applyUserCookiesToBrowserContext(context, ephemeralUser);

  // Page now authenticated as ephemeral user
  await page.goto('/admin/users');

  await expect(page.getByTestId('delete-user-btn')).toBeVisible();

  // User and token cleaned up after test
});

Key Points:

• No disk persistence (ephemeral)
• Apply cookies directly to context
• Useful for testing user lifecycle
• Clean up automatic when test ends

Example 4: Testing Multiple Users in Single Test

Context: Testing interactions between users (messaging, sharing, collaboration features).

Implementation:

test('user interaction', async ({ browser }) => {
  // User 1 context
  const user1Context = await browser.newContext({
    storageState: './auth-sessions/local/user1/storage-state.json',
  });
  const user1Page = await user1Context.newPage();

  // User 2 context
  const user2Context = await browser.newContext({
    storageState: './auth-sessions/local/user2/storage-state.json',
  });
  const user2Page = await user2Context.newPage();

  // User 1 sends message
  await user1Page.goto('/messages');
  await user1Page.fill('#message', 'Hello from user 1');
  await user1Page.click('#send');

  // User 2 receives message
  await user2Page.goto('/messages');
  await expect(user2Page.getByText('Hello from user 1')).toBeVisible();

  // Cleanup
  await user1Context.close();
  await user2Context.close();
});

Key Points:

• Each user has separate browser context
• Reference storage state files directly
• Test real-time interactions
• Clean up contexts after test

Example 5: Worker-Specific Accounts (Parallel Testing)

Context: Running tests in parallel with isolated user accounts per worker to avoid conflicts.

Implementation:

// playwright.config.ts
export default defineConfig({
  workers: 4, // 4 parallel workers
  use: {
    // Each worker uses different user
    storageState: async ({}, use, testInfo) => {
      const workerIndex = testInfo.workerIndex;
      const userIdentifier = `worker-${workerIndex}`;

      await use(`./auth-sessions/local/${userIdentifier}/storage-state.json`);
    },
  },
});

// Tests run in parallel, each worker with its own user
test('parallel test 1', async ({ page }) => {
  // Worker 0 uses worker-0 account
  await page.goto('/dashboard');
});

test('parallel test 2', async ({ page }) => {
  // Worker 1 uses worker-1 account
  await page.goto('/dashboard');
});

Key Points:

• Each worker has isolated user account
• No conflicts in parallel execution
• Token management automatic per worker
• Scales to any number of workers

Example 6: Pure API Authentication (No Browser)

Context: Get auth tokens for API-only tests using auth-session disk persistence.

Implementation:

// Step 1: Create API-only auth provider (no browser needed)
// playwright/support/api-auth-provider.ts
import { type AuthProvider } from '@seontechnologies/playwright-utils/auth-session';

const apiAuthProvider: AuthProvider = {
  getEnvironment: (options) => options.environment || 'local',
  getUserIdentifier: (options) => options.userIdentifier || 'api-user',

  extractToken: (storageState) => {
    // Token stored in localStorage format for disk persistence
    const tokenEntry = storageState.origins?.[0]?.localStorage?.find((item) => item.name === 'auth_token');
    return tokenEntry?.value;
  },

  isTokenExpired: (storageState) => {
    const expiryEntry = storageState.origins?.[0]?.localStorage?.find((item) => item.name === 'token_expiry');
    if (!expiryEntry) return true;
    return Date.now() > parseInt(expiryEntry.value, 10);
  },

  manageAuthToken: async (request, options) => {
    const email = process.env.TEST_USER_EMAIL;
    const password = process.env.TEST_USER_PASSWORD;

    if (!email || !password) {
      throw new Error('TEST_USER_EMAIL and TEST_USER_PASSWORD must be set');
    }

    // Pure API login - no browser!
    const response = await request.post('/api/auth/login', {
      data: { email, password },
    });

    if (!response.ok()) {
      throw new Error(`Auth failed: ${response.status()}`);
    }

    const { token, expiresIn } = await response.json();
    const expiryTime = Date.now() + expiresIn * 1000;

    // Return storage state format for disk persistence
    return {
      cookies: [],
      origins: [
        {
          origin: process.env.API_BASE_URL || 'http://localhost:3000',
          localStorage: [
            { name: 'auth_token', value: token },
            { name: 'token_expiry', value: String(expiryTime) },
          ],
        },
      ],
    };
  },
};

export default apiAuthProvider;

// Step 2: Create auth fixture
// playwright/support/fixtures.ts
import { test as base } from '@playwright/test';
import { createAuthFixtures, setAuthProvider } from '@seontechnologies/playwright-utils/auth-session';
import apiAuthProvider from './api-auth-provider';

setAuthProvider(apiAuthProvider);

export const test = base.extend(createAuthFixtures());

// Step 3: Use in tests - token persisted to disk!
// tests/api/authenticated-api.spec.ts
import { test } from '../support/fixtures';
import { expect } from '@playwright/test';

test('should access protected endpoint', async ({ authToken, apiRequest }) => {
  // authToken is automatically loaded from disk or fetched if expired
  const { status, body } = await apiRequest({
    method: 'GET',
    path: '/api/me',
    headers: { Authorization: `Bearer ${authToken}` },
  });

  expect(status).toBe(200);
});

test('should create resource with auth', async ({ authToken, apiRequest }) => {
  const { status, body } = await apiRequest({
    method: 'POST',
    path: '/api/orders',
    headers: { Authorization: `Bearer ${authToken}` },
    body: { items: [{ productId: 'prod-1', quantity: 2 }] },
  });

  expect(status).toBe(201);
  expect(body.id).toBeDefined();
});

Key Points:

• Token persisted to disk (not in-memory) - survives test reruns
• Provider fetches token once, reuses until expired
• Pure API authentication - no browser context needed
• authToken fixture handles disk read/write automatically
• Environment variables validated with clear error message

Example 7: Service-to-Service Authentication

Context: Test microservice authentication patterns (API keys, service tokens) with proper environment validation.

Implementation:

// tests/api/service-auth.spec.ts
import { test as base, expect } from '@playwright/test';
import { test as apiFixture } from '@seontechnologies/playwright-utils/api-request/fixtures';
import { mergeTests } from '@playwright/test';

// Validate environment variables at module load
const SERVICE_API_KEY = process.env.SERVICE_API_KEY;
const INTERNAL_SERVICE_URL = process.env.INTERNAL_SERVICE_URL;

if (!SERVICE_API_KEY) {
  throw new Error('SERVICE_API_KEY environment variable is required');
}
if (!INTERNAL_SERVICE_URL) {
  throw new Error('INTERNAL_SERVICE_URL environment variable is required');
}

const test = mergeTests(base, apiFixture);

test.describe('Service-to-Service Auth', () => {
  test('should authenticate with API key', async ({ apiRequest }) => {
    const { status, body } = await apiRequest({
      method: 'GET',
      path: '/internal/health',
      baseUrl: INTERNAL_SERVICE_URL,
      headers: { 'X-API-Key': SERVICE_API_KEY },
    });

    expect(status).toBe(200);
    expect(body.status).toBe('healthy');
  });

  test('should reject invalid API key', async ({ apiRequest }) => {
    const { status, body } = await apiRequest({
      method: 'GET',
      path: '/internal/health',
      baseUrl: INTERNAL_SERVICE_URL,
      headers: { 'X-API-Key': 'invalid-key' },
    });

    expect(status).toBe(401);
    expect(body.code).toBe('INVALID_API_KEY');
  });

  test('should call downstream service with propagated auth', async ({ apiRequest }) => {
    const { status, body } = await apiRequest({
      method: 'POST',
      path: '/internal/aggregate-data',
      baseUrl: INTERNAL_SERVICE_URL,
      headers: {
        'X-API-Key': SERVICE_API_KEY,
        'X-Request-ID': `test-${Date.now()}`,
      },
      body: { sources: ['users', 'orders', 'inventory'] },
    });

    expect(status).toBe(200);
    expect(body.aggregatedFrom).toHaveLength(3);
  });
});

Key Points:

• Environment variables validated at module load with clear errors
• API key authentication (simpler than OAuth - no disk persistence needed)
• Test internal/service endpoints
• Validate auth rejection scenarios
• Correlation ID for request tracing

Note: API keys are typically static secrets that don't expire, so disk persistence (auth-session) isn't needed. For rotating service tokens, use the auth-session provider pattern from Example 6.

Custom Auth Provider Pattern

Context: Adapt auth-session to your authentication system (OAuth2, JWT, SAML, custom).

Minimal provider structure:

import { type AuthProvider } from '@seontechnologies/playwright-utils/auth-session';

const myCustomProvider: AuthProvider = {
  getEnvironment: (options) => options.environment || 'local',

  getUserIdentifier: (options) => options.userIdentifier || 'default-user',

  extractToken: (storageState) => {
    // Extract token from your storage format
    return storageState.cookies.find((c) => c.name === 'auth_token')?.value;
  },

  extractCookies: (tokenData) => {
    // Convert token to cookies for browser context
    return [
      {
        name: 'auth_token',
        value: tokenData,
        domain: 'example.com',
        path: '/',
        httpOnly: true,
        secure: true,
      },
    ];
  },

  isTokenExpired: (storageState) => {
    // Check if token is expired
    const expiresAt = storageState.cookies.find((c) => c.name === 'expires_at');
    return Date.now() > parseInt(expiresAt?.value || '0');
  },

  manageAuthToken: async (request, options) => {
    // Main token acquisition logic
    // Return storage state with cookies/localStorage
  },
};

export default myCustomProvider;

Integration with API Request

import { test } from '@seontechnologies/playwright-utils/fixtures';

test('authenticated API call', async ({ apiRequest, authToken }) => {
  const { status, body } = await apiRequest({
    method: 'GET',
    path: '/api/protected',
    headers: { Authorization: `Bearer ${authToken}` },
  });

  expect(status).toBe(200);
});

Related Fragments

• api-testing-patterns.md - Pure API testing patterns (no browser)
• overview.md - Installation and fixture composition
• api-request.md - Authenticated API requests
• fixtures-composition.md - Merging auth with other utilities

Anti-Patterns

❌ Calling setAuthProvider after globalSetup:

async function globalSetup() {
  configureAuthSession(...)
  await authGlobalInit()  // Provider not set yet!
  setAuthProvider(provider)  // Too late
}

✅ Register provider before init:

async function globalSetup() {
  authStorageInit()
  configureAuthSession(...)
  setAuthProvider(provider)  // First
  await authGlobalInit()     // Then init
}

❌ Hardcoding storage paths:

const storageState = './auth-sessions/local/user1/storage-state.json'; // Brittle

✅ Use helper functions:

import { getTokenFilePath } from '@seontechnologies/playwright-utils/auth-session';

const tokenPath = getTokenFilePath({
  environment: 'local',
  userIdentifier: 'user1',
  tokenFileName: 'storage-state.json',
});

Burn-in Test Runner

Principle

Use smart test selection with git diff analysis to run only affected tests. Filter out irrelevant changes (configs, types, docs) and control test volume with percentage-based execution. Reduce unnecessary CI runs while maintaining reliability.

Rationale

Playwright's --only-changed triggers all affected tests:

• Config file changes trigger hundreds of tests
• Type definition changes cause full suite runs
• No volume control (all or nothing)
• Slow CI pipelines

The burn-in utility provides:

• Smart filtering: Skip patterns for irrelevant files (configs, types, docs)
• Volume control: Run percentage of affected tests after filtering
• Custom dependency analysis: More accurate than Playwright's built-in
• CI optimization: Faster pipelines without sacrificing confidence
• Process of elimination: Start with all → filter irrelevant → control volume

Pattern Examples

Example 1: Basic Burn-in Setup

Context: Run burn-in on changed files compared to main branch.

Implementation:

// Step 1: Create burn-in script
// playwright/scripts/burn-in-changed.ts
import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in'

async function main() {
  await runBurnIn({
    configPath: 'playwright/config/.burn-in.config.ts',
    baseBranch: 'main'
  })
}

main().catch(console.error)

// Step 2: Create config
// playwright/config/.burn-in.config.ts
import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in'

const config: BurnInConfig = {
  // Files that never trigger tests (first filter)
  skipBurnInPatterns: [
    '**/config/**',
    '**/*constants*',
    '**/*types*',
    '**/*.md',
    '**/README*'
  ],

  // Run 30% of remaining tests after skip filter
  burnInTestPercentage: 0.3,

  // Burn-in repetition
  burnIn: {
    repeatEach: 3,  // Run each test 3 times
    retries: 1      // Allow 1 retry
  }
}

export default config

// Step 3: Add package.json script
{
  "scripts": {
    "test:pw:burn-in-changed": "tsx playwright/scripts/burn-in-changed.ts"
  }
}

Key Points:

• Two-stage filtering: skip patterns, then volume control
• skipBurnInPatterns eliminates irrelevant files
• burnInTestPercentage controls test volume (0.3 = 30%)
• Custom dependency analysis finds actually affected tests

Example 2: CI Integration

Context: Use burn-in in GitHub Actions for efficient CI runs.

Implementation:

# .github/workflows/burn-in.yml
name: Burn-in Changed Tests

on:
  pull_request:
    branches: [main]

jobs:
  burn-in:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # Need git history

      - name: Setup Node
        uses: actions/setup-node@v4

      - name: Install dependencies
        run: npm ci

      - name: Run burn-in on changed tests
        run: npm run test:pw:burn-in-changed -- --base-branch=origin/main

      - name: Upload artifacts
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: burn-in-failures
          path: test-results/

Key Points:

• fetch-depth: 0 for full git history
• Pass --base-branch=origin/main for PR comparison
• Upload artifacts only on failure
• Significantly faster than full suite

Example 3: How It Works (Process of Elimination)

Context: Understanding the filtering pipeline.

Scenario:

Git diff finds: 21 changed files
├─ Step 1: Skip patterns filter
│  Removed: 6 files (*.md, config/*, *types*)
│  Remaining: 15 files
│
├─ Step 2: Dependency analysis
│  Tests that import these 15 files: 45 tests
│
└─ Step 3: Volume control (30%)
   Final tests to run: 14 tests (30% of 45)

Result: Run 14 targeted tests instead of 147 with --only-changed!

Key Points:

• Three-stage pipeline: skip → analyze → control
• Custom dependency analysis (not just imports)
• Percentage applies AFTER filtering
• Dramatically reduces CI time

Example 4: Environment-Specific Configuration

Context: Different settings for local vs CI environments.

Implementation:

import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in';

const config: BurnInConfig = {
  skipBurnInPatterns: ['**/config/**', '**/*types*', '**/*.md'],

  // CI runs fewer iterations, local runs more
  burnInTestPercentage: process.env.CI ? 0.2 : 0.3,

  burnIn: {
    repeatEach: process.env.CI ? 2 : 3,
    retries: process.env.CI ? 0 : 1, // No retries in CI
  },
};

export default config;

Key Points:

• process.env.CI for environment detection
• Lower percentage in CI (20% vs 30%)
• Fewer iterations in CI (2 vs 3)
• No retries in CI (fail fast)

Example 5: Sharding Support

Context: Distribute burn-in tests across multiple CI workers.

Implementation:

// burn-in-changed.ts with sharding
import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in';

async function main() {
  const shardArg = process.argv.find((arg) => arg.startsWith('--shard='));

  if (shardArg) {
    process.env.PW_SHARD = shardArg.split('=')[1];
  }

  await runBurnIn({
    configPath: 'playwright/config/.burn-in.config.ts',
  });
}

# GitHub Actions with sharding
jobs:
  burn-in:
    strategy:
      matrix:
        shard: [1/3, 2/3, 3/3]
    steps:
      - run: npm run test:pw:burn-in-changed -- --shard=${{ matrix.shard }}

Key Points:

• Pass --shard=1/3 for parallel execution
• Burn-in respects Playwright sharding
• Distribute across multiple workers
• Reduces total CI time further

Integration with CI Workflow

When setting up CI with *ci workflow, recommend burn-in for:

• Pull request validation
• Pre-merge checks
• Nightly builds (subset runs)

Related Fragments

• ci-burn-in.md - Traditional burn-in patterns (10-iteration loops)
• selective-testing.md - Test selection strategies
• overview.md - Installation

Anti-Patterns

❌ Over-aggressive skip patterns:

skipBurnInPatterns: [
  '**/*', // Skips everything!
];

✅ Targeted skip patterns:

skipBurnInPatterns: ['**/config/**', '**/*types*', '**/*.md', '**/*constants*'];

❌ Too low percentage (false confidence):

burnInTestPercentage: 0.05; // Only 5% - might miss issues

✅ Balanced percentage:

burnInTestPercentage: 0.2; // 20% in CI, provides good coverage

Component Test-Driven Development Loop

Principle

Start every UI change with a failing component test (cy.mount, Playwright component test, or RTL render). Follow the Red-Green-Refactor cycle: write a failing test (red), make it pass with minimal code (green), then improve the implementation (refactor). Ship only after the cycle completes. Keep component tests under 100 lines, isolated with fresh providers per test, and validate accessibility alongside functionality.

Rationale

Component TDD provides immediate feedback during development. Failing tests (red) clarify requirements before writing code. Minimal implementations (green) prevent over-engineering. Refactoring with passing tests ensures changes don't break functionality. Isolated tests with fresh providers prevent state bleed in parallel runs. Accessibility assertions catch usability issues early. Visual debugging (Cypress runner, Storybook, Playwright trace viewer) accelerates diagnosis when tests fail.

Pattern Examples

Example 1: Red-Green-Refactor Loop

Context: When building a new component, start with a failing test that describes the desired behavior. Implement just enough to pass, then refactor for quality.

Implementation:

// Step 1: RED - Write failing test
// Button.cy.tsx (Cypress Component Test)
import { Button } from './Button';

describe('Button Component', () => {
  it('should render with label', () => {
    cy.mount(<Button label="Click Me" />);
    cy.contains('Click Me').should('be.visible');
  });

  it('should call onClick when clicked', () => {
    const onClickSpy = cy.stub().as('onClick');
    cy.mount(<Button label="Submit" onClick={onClickSpy} />);

    cy.get('button').click();
    cy.get('@onClick').should('have.been.calledOnce');
  });
});

// Run test: FAILS - Button component doesn't exist yet
// Error: "Cannot find module './Button'"

// Step 2: GREEN - Minimal implementation
// Button.tsx
type ButtonProps = {
  label: string;
  onClick?: () => void;
};

export const Button = ({ label, onClick }: ButtonProps) => {
  return <button onClick={onClick}>{label}</button>;
};

// Run test: PASSES - Component renders and handles clicks

// Step 3: REFACTOR - Improve implementation
// Add disabled state, loading state, variants
type ButtonProps = {
  label: string;
  onClick?: () => void;
  disabled?: boolean;
  loading?: boolean;
  variant?: 'primary' | 'secondary' | 'danger';
};

export const Button = ({
  label,
  onClick,
  disabled = false,
  loading = false,
  variant = 'primary'
}: ButtonProps) => {
  return (
    <button
      onClick={onClick}
      disabled={disabled || loading}
      className={`btn btn-${variant}`}
      data-testid="button"
    >
      {loading ? <Spinner /> : label}
    </button>
  );
};

// Step 4: Expand tests for new features
describe('Button Component', () => {
  it('should render with label', () => {
    cy.mount(<Button label="Click Me" />);
    cy.contains('Click Me').should('be.visible');
  });

  it('should call onClick when clicked', () => {
    const onClickSpy = cy.stub().as('onClick');
    cy.mount(<Button label="Submit" onClick={onClickSpy} />);

    cy.get('button').click();
    cy.get('@onClick').should('have.been.calledOnce');
  });

  it('should be disabled when disabled prop is true', () => {
    cy.mount(<Button label="Submit" disabled={true} />);
    cy.get('button').should('be.disabled');
  });

  it('should show spinner when loading', () => {
    cy.mount(<Button label="Submit" loading={true} />);
    cy.get('[data-testid="spinner"]').should('be.visible');
    cy.get('button').should('be.disabled');
  });

  it('should apply variant styles', () => {
    cy.mount(<Button label="Delete" variant="danger" />);
    cy.get('button').should('have.class', 'btn-danger');
  });
});

// Run tests: ALL PASS - Refactored component still works

// Playwright Component Test equivalent
import { test, expect } from '@playwright/experimental-ct-react';
import { Button } from './Button';

test.describe('Button Component', () => {
  test('should call onClick when clicked', async ({ mount }) => {
    let clicked = false;
    const component = await mount(
      <Button label="Submit" onClick={() => { clicked = true; }} />
    );

    await component.getByRole('button').click();
    expect(clicked).toBe(true);
  });

  test('should be disabled when loading', async ({ mount }) => {
    const component = await mount(<Button label="Submit" loading={true} />);
    await expect(component.getByRole('button')).toBeDisabled();
    await expect(component.getByTestId('spinner')).toBeVisible();
  });
});

Key Points:

• Red: Write failing test first - clarifies requirements before coding
• Green: Implement minimal code to pass - prevents over-engineering
• Refactor: Improve code quality while keeping tests green
• Expand: Add tests for new features after refactoring
• Cycle repeats: Each new feature starts with a failing test

Example 2: Provider Isolation Pattern

Context: When testing components that depend on context providers (React Query, Auth, Router), wrap them with required providers in each test to prevent state bleed between tests.

Implementation:

// test-utils/AllTheProviders.tsx
import { FC, ReactNode } from 'react';
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
import { BrowserRouter } from 'react-router-dom';
import { AuthProvider } from '../contexts/AuthContext';

type Props = {
  children: ReactNode;
  initialAuth?: { user: User | null; token: string | null };
};

export const AllTheProviders: FC<Props> = ({ children, initialAuth }) => {
  // Create NEW QueryClient per test (prevent state bleed)
  const queryClient = new QueryClient({
    defaultOptions: {
      queries: { retry: false },
      mutations: { retry: false }
    }
  });

  return (
    <QueryClientProvider client={queryClient}>
      <BrowserRouter>
        <AuthProvider initialAuth={initialAuth}>
          {children}
        </AuthProvider>
      </BrowserRouter>
    </QueryClientProvider>
  );
};

// Cypress custom mount command
// cypress/support/component.tsx
import { mount } from 'cypress/react18';
import { AllTheProviders } from '../../test-utils/AllTheProviders';

Cypress.Commands.add('wrappedMount', (component, options = {}) => {
  const { initialAuth, ...mountOptions } = options;

  return mount(
    <AllTheProviders initialAuth={initialAuth}>
      {component}
    </AllTheProviders>,
    mountOptions
  );
});

// Usage in tests
// UserProfile.cy.tsx
import { UserProfile } from './UserProfile';

describe('UserProfile Component', () => {
  it('should display user when authenticated', () => {
    const user = { id: 1, name: 'John Doe', email: 'john@example.com' };

    cy.wrappedMount(<UserProfile />, {
      initialAuth: { user, token: 'fake-token' }
    });

    cy.contains('John Doe').should('be.visible');
    cy.contains('john@example.com').should('be.visible');
  });

  it('should show login prompt when not authenticated', () => {
    cy.wrappedMount(<UserProfile />, {
      initialAuth: { user: null, token: null }
    });

    cy.contains('Please log in').should('be.visible');
  });
});

// Playwright Component Test with providers
import { test, expect } from '@playwright/experimental-ct-react';
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
import { UserProfile } from './UserProfile';
import { AuthProvider } from '../contexts/AuthContext';

test.describe('UserProfile Component', () => {
  test('should display user when authenticated', async ({ mount }) => {
    const user = { id: 1, name: 'John Doe', email: 'john@example.com' };
    const queryClient = new QueryClient();

    const component = await mount(
      <QueryClientProvider client={queryClient}>
        <AuthProvider initialAuth={{ user, token: 'fake-token' }}>
          <UserProfile />
        </AuthProvider>
      </QueryClientProvider>
    );

    await expect(component.getByText('John Doe')).toBeVisible();
    await expect(component.getByText('john@example.com')).toBeVisible();
  });
});

Key Points:

• Create NEW providers per test (QueryClient, Router, Auth)
• Prevents state pollution between tests
• initialAuth prop allows testing different auth states
• Custom mount command (wrappedMount) reduces boilerplate
• Providers wrap component, not the entire test suite

Example 3: Accessibility Assertions

Context: When testing components, validate accessibility alongside functionality using axe-core, ARIA roles, labels, and keyboard navigation.

Implementation:

// Cypress with axe-core
// cypress/support/component.tsx
import 'cypress-axe';

// Form.cy.tsx
import { Form } from './Form';

describe('Form Component Accessibility', () => {
  beforeEach(() => {
    cy.wrappedMount(<Form />);
    cy.injectAxe(); // Inject axe-core
  });

  it('should have no accessibility violations', () => {
    cy.checkA11y(); // Run axe scan
  });

  it('should have proper ARIA labels', () => {
    cy.get('input[name="email"]').should('have.attr', 'aria-label', 'Email address');
    cy.get('input[name="password"]').should('have.attr', 'aria-label', 'Password');
    cy.get('button[type="submit"]').should('have.attr', 'aria-label', 'Submit form');
  });

  it('should support keyboard navigation', () => {
    // Tab through form fields
    cy.get('input[name="email"]').focus().type('test@example.com');
    cy.realPress('Tab'); // cypress-real-events plugin
    cy.focused().should('have.attr', 'name', 'password');

    cy.focused().type('password123');
    cy.realPress('Tab');
    cy.focused().should('have.attr', 'type', 'submit');

    cy.realPress('Enter'); // Submit via keyboard
    cy.contains('Form submitted').should('be.visible');
  });

  it('should announce errors to screen readers', () => {
    cy.get('button[type="submit"]').click(); // Submit without data

    // Error has role="alert" and aria-live="polite"
    cy.get('[role="alert"]')
      .should('be.visible')
      .and('have.attr', 'aria-live', 'polite')
      .and('contain', 'Email is required');
  });

  it('should have sufficient color contrast', () => {
    cy.checkA11y(null, {
      rules: {
        'color-contrast': { enabled: true }
      }
    });
  });
});

// Playwright with axe-playwright
import { test, expect } from '@playwright/experimental-ct-react';
import AxeBuilder from '@axe-core/playwright';
import { Form } from './Form';

test.describe('Form Component Accessibility', () => {
  test('should have no accessibility violations', async ({ mount, page }) => {
    await mount(<Form />);

    const accessibilityScanResults = await new AxeBuilder({ page })
      .analyze();

    expect(accessibilityScanResults.violations).toEqual([]);
  });

  test('should support keyboard navigation', async ({ mount, page }) => {
    const component = await mount(<Form />);

    await component.getByLabel('Email address').fill('test@example.com');
    await page.keyboard.press('Tab');

    await expect(component.getByLabel('Password')).toBeFocused();

    await component.getByLabel('Password').fill('password123');
    await page.keyboard.press('Tab');

    await expect(component.getByRole('button', { name: 'Submit form' })).toBeFocused();

    await page.keyboard.press('Enter');
    await expect(component.getByText('Form submitted')).toBeVisible();
  });
});

Key Points:

• Use cy.checkA11y() (Cypress) or AxeBuilder (Playwright) for automated accessibility scanning
• Validate ARIA roles, labels, and live regions
• Test keyboard navigation (Tab, Enter, Escape)
• Ensure errors are announced to screen readers (role="alert", aria-live)
• Check color contrast meets WCAG standards

Example 4: Visual Regression Test

Context: When testing components, capture screenshots to detect unintended visual changes. Use Playwright visual comparison or Cypress snapshot plugins.

Implementation:

// Playwright visual regression
import { test, expect } from '@playwright/experimental-ct-react';
import { Button } from './Button';

test.describe('Button Visual Regression', () => {
  test('should match primary button snapshot', async ({ mount }) => {
    const component = await mount(<Button label="Primary" variant="primary" />);

    // Capture and compare screenshot
    await expect(component).toHaveScreenshot('button-primary.png');
  });

  test('should match secondary button snapshot', async ({ mount }) => {
    const component = await mount(<Button label="Secondary" variant="secondary" />);
    await expect(component).toHaveScreenshot('button-secondary.png');
  });

  test('should match disabled button snapshot', async ({ mount }) => {
    const component = await mount(<Button label="Disabled" disabled={true} />);
    await expect(component).toHaveScreenshot('button-disabled.png');
  });

  test('should match loading button snapshot', async ({ mount }) => {
    const component = await mount(<Button label="Loading" loading={true} />);
    await expect(component).toHaveScreenshot('button-loading.png');
  });
});

// Cypress visual regression with percy or snapshot plugins
import { Button } from './Button';

describe('Button Visual Regression', () => {
  it('should match primary button snapshot', () => {
    cy.wrappedMount(<Button label="Primary" variant="primary" />);

    // Option 1: Percy (cloud-based visual testing)
    cy.percySnapshot('Button - Primary');

    // Option 2: cypress-plugin-snapshots (local snapshots)
    cy.get('button').toMatchImageSnapshot({
      name: 'button-primary',
      threshold: 0.01 // 1% threshold for pixel differences
    });
  });

  it('should match hover state', () => {
    cy.wrappedMount(<Button label="Hover Me" />);
    cy.get('button').realHover(); // cypress-real-events
    cy.percySnapshot('Button - Hover State');
  });

  it('should match focus state', () => {
    cy.wrappedMount(<Button label="Focus Me" />);
    cy.get('button').focus();
    cy.percySnapshot('Button - Focus State');
  });
});

// Playwright configuration for visual regression
// playwright.config.ts
export default defineConfig({
  expect: {
    toHaveScreenshot: {
      maxDiffPixels: 100, // Allow 100 pixels difference
      threshold: 0.2 // 20% threshold
    }
  },
  use: {
    screenshot: 'only-on-failure'
  }
});

// Update snapshots when intentional changes are made
// npx playwright test --update-snapshots

Key Points:

• Playwright: Use toHaveScreenshot() for built-in visual comparison
• Cypress: Use Percy (cloud) or snapshot plugins (local) for visual testing
• Capture different states: default, hover, focus, disabled, loading
• Set threshold for acceptable pixel differences (avoid false positives)
• Update snapshots when visual changes are intentional
• Visual tests catch unintended CSS/layout regressions

Integration Points

• Used in workflows: *atdd (component test generation), *automate (component test expansion), *framework (component testing setup)
• Related fragments:
• test-quality.md - Keep component tests <100 lines, isolated, focused
• fixture-architecture.md - Provider wrapping patterns, custom mount commands
• data-factories.md - Factory functions for component props
• test-levels-framework.md - When to use component tests vs E2E tests

TDD Workflow Summary

Red-Green-Refactor Cycle:

1. Red: Write failing test describing desired behavior 2. Green: Implement minimal code to make test pass 3. Refactor: Improve code quality, tests stay green 4. Repeat: Each new feature starts with failing test

Component Test Checklist:

• [ ] Test renders with required props
• [ ] Test user interactions (click, type, submit)
• [ ] Test different states (loading, error, disabled)
• [ ] Test accessibility (ARIA, keyboard navigation)
• [ ] Test visual regression (snapshots)
• [ ] Isolate with fresh providers (no state bleed)
• [ ] Keep tests <100 lines (split by intent)

_Source: CCTDD repository, Murat component testing talks, Playwright/Cypress component testing docs._

Step 1: Assess Edit Target

STEP GOAL:

Identify which output should be edited and load it.

MANDATORY EXECUTION RULES (READ FIRST):

Universal Rules:

• 📖 Read the complete step file before taking any action
• ✅ Speak in {communication_language}

Role Reinforcement:

• ✅ You are the Master Test Architect

Step-Specific Rules:

• 🎯 Ask the user which output file to edit
• 🚫 Do not edit until target is confirmed

EXECUTION PROTOCOLS:

• 🎯 Follow the MANDATORY SEQUENCE exactly

CONTEXT BOUNDARIES:

• Available context: existing outputs
• Focus: select edit target
• Limits: no edits yet

MANDATORY SEQUENCE

CRITICAL: Follow this sequence exactly.

1. Identify Target

Ask the user to provide the output file path or select from known outputs.

2. Load Target

Read the provided output file in full.

3. Confirm

Confirm the target and proceed to edit.

Load next step: {nextStepFile}

🚨 SYSTEM SUCCESS/FAILURE METRICS:

✅ SUCCESS:

• Target identified and loaded

❌ SYSTEM FAILURE:

• Proceeding without a confirmed target