Testing

Testing

This skill provides comprehensive testing capabilities including test strategy, automation setup, Test-Driven Development (TDD), test writing best practices, coverage analysis, CI/CD integration, and web application testing with Playwright.

When to Use This Skill

• When setting up test infrastructure for a project
• When creating test strategies and test plans
• When writing unit, integration, or E2E tests
• When implementing TDD/test-first development
• When analyzing test coverage and quality
• When integrating tests into CI/CD pipelines
• When testing web applications with Playwright
• When debugging test failures or improving test reliability
• When writing test fixtures, mock data, or factory functions
• When mocking external dependencies (APIs, databases, file systems)
• When organizing test file structure and test suites
• When testing async code, Promises, or event-driven behavior
• When implementing snapshot tests for UI components
• When configuring test coverage thresholds

What This Skill Does

1. Test Strategy: Designs comprehensive testing strategies (unit, integration, E2E) 2. Test Automation: Sets up test frameworks and automation tools 3. TDD Methodology: Implements Test-Driven Development workflows (Red-Green-Refactor) 4. Test Writing: Writes focused, maintainable tests with proper patterns 5. Coverage Analysis: Analyzes and improves test coverage 6. CI/CD Integration: Integrates tests into continuous integration pipelines 7. Web App Testing: Tests web applications using Playwright 8. Test Quality: Improves test reliability and maintainability

Test Strategy

Test Pyramid

Recommended Distribution:

• Unit Tests: 70% - Fast, isolated, test individual functions
• Integration Tests: 20% - Test component interactions
• E2E Tests: 10% - Test complete user workflows

Test Types:

• Functional tests (happy path, edge cases, error handling)
• Non-functional tests (performance, security, accessibility)
• Regression tests (prevent breaking changes)
• Smoke tests (critical path verification)

Framework Selection

JavaScript/TypeScript:

• Jest, Vitest, Mocha for unit/integration
• Playwright, Cypress for E2E
• React Testing Library for component testing

Python:

• pytest for unit/integration
• Selenium, Playwright for E2E
• unittest for standard library testing

Java:

• JUnit for unit tests
• TestNG for integration
• Selenium for E2E

Go:

• Built-in testing package
• Testify for assertions

Rust:

• Built-in test framework
• Cargo test for running tests

Test-Driven Development (TDD)

TDD is a design technique, not just a testing technique. It produces better-designed, more maintainable code through small, disciplined steps.

Core Principle

Write tests before code. Always. TDD forces you to think about:

• What behavior do I need?
• How will I know it works?
• What's the simplest implementation?

The Three Laws (Never Violate)

1. Write NO production code without a failing test first 2. Write only enough test to demonstrate one failure 3. Write only enough code to pass that test

Red-Green-Refactor Cycle

Phase 1: RED - Write Failing Test

1. Write ONE test that defines desired behavior 2. Run test - verify it FAILS 3. Verify it fails for the RIGHT reason (not syntax error) 4. DO NOT write implementation yet

Phase 2: GREEN - Minimal Implementation

1. Write MINIMAL code to make test pass 2. Resist urge to add extra features 3. Run test - verify it PASSES 4. If test still fails, fix implementation (not test)

Phase 3: REFACTOR - Clean Code

1. Remove code duplication (DRY) 2. Improve naming for clarity 3. Extract complex logic into functions 4. Run ALL tests - must stay green throughout 5. Check test coverage on changed lines

After REFACTOR, start new RED phase for next behavior.

Test Writing Patterns

Arrange-Act-Assert (AAA)

Structure:

1. Arrange: Set up test data and conditions 2. Act: Execute the code being tested 3. Assert: Verify the expected outcome

Example:

describe('UserService', () => {
  it('should create user with valid data', async () => {
    // Arrange
    const userData = { email: 'test@example.com', name: 'Test User' };

    // Act
    const result = await userService.createUser(userData);

    // Assert
    expect(result).toHaveProperty('id');
    expect(result.email).toBe(userData.email);
  });
});

Given-When-Then (BDD Style)

Structure:

1. Given: Initial context/preconditions 2. When: Action/event that triggers behavior 3. Then: Expected outcome

Test Organization

File Structure:

project/
├── src/
│   └── components/
│       └── User.jsx
├── tests/
│   ├── unit/
│   │   └── User.test.jsx
│   ├── integration/
│   │   └── UserAPI.test.js
│   └── e2e/
│       └── user-flow.spec.js
├── jest.config.js
└── playwright.config.js

Coverage Analysis

Coverage Goals

Recommended Thresholds:

• Lines: 80%+
• Functions: 80%+
• Branches: 80%+
• Statements: 80%+

Critical Paths:

• Always aim for 100% coverage on critical business logic
• Authentication and authorization
• Payment processing
• Data validation

Coverage Gaps

Common Gaps:

• Error handling paths
• Edge cases
• Boundary conditions
• Integration points

Improvement Strategies:

• Identify untested code paths
• Add tests for error scenarios
• Test edge cases and boundaries
• Increase integration test coverage

CI/CD Integration

Test Pipeline

Stages:

1. Unit Tests: Fast feedback, run on every commit 2. Integration Tests: Run on pull requests 3. E2E Tests: Run before merging to main 4. Performance Tests: Run on main branch

Quality Gates:

• All tests must pass
• Coverage must meet threshold
• No critical security issues
• Performance benchmarks met

Web Application Testing with Playwright

Helper Scripts

This skill includes Python helper scripts in scripts/:

• `with_server.py` - Manages server lifecycle (supports multiple servers). Always run with --help first to see usage.

  # Single server
  python scripts/with_server.py --server "npm run dev" --port 5173 -- python your_automation.py

  # Multiple servers (e.g., backend + frontend)
  python scripts/with_server.py \
    --server "cd backend && python server.py" --port 3000 \
    --server "cd frontend && npm run dev" --port 5173 \
    -- python your_automation.py

Decision Tree: Choosing Your Approach

User task → Is it static HTML?
    ├─ Yes → Read HTML file directly to identify selectors
    │         ├─ Success → Write Playwright script using selectors
    │         └─ Fails/Incomplete → Treat as dynamic (below)
    │
    └─ No (dynamic webapp) → Is the server already running?
        ├─ No → Run: python scripts/with_server.py --help
        │        Then use the helper + write simplified Playwright script
        │
        └─ Yes → Reconnaissance-then-action:
            1. Navigate and wait for networkidle
            2. Take screenshot or inspect DOM
            3. Identify selectors from rendered state
            4. Execute actions with discovered selectors

Playwright Best Practices

• Use bundled scripts as black boxes - Use --help to see usage, then invoke directly
• Use sync_playwright() for synchronous scripts
• Always close the browser when done
• Use descriptive selectors: text=, role=, CSS selectors, or IDs
• Add appropriate waits: page.wait_for_selector() or page.wait_for_timeout()
• CRITICAL: Wait for page.wait_for_load_state('networkidle') before inspection on dynamic apps

Example: Basic Playwright Script

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto('http://localhost:5173')
    page.wait_for_load_state('networkidle')  # CRITICAL: Wait for JS to execute
    # ... your automation logic
    browser.close()

Examples

See examples/ directory for:

• element_discovery.py - Discovering buttons, links, and inputs on a page
• static_html_automation.py - Using file:// URLs for local HTML
• console_logging.py - Capturing console logs during automation

Reference Files

For detailed testing patterns and workflows, load reference files as needed:

• `references/framework_workflows.md` - Framework-specific TDD workflows and examples for Python (pytest), JavaScript (Jest, Vitest), Java (JUnit), Go, Rust
• `references/test_patterns.md` - Common test patterns, test organization, naming conventions, test doubles (mocks, stubs, spies), parametrization, and anti-patterns
• `references/webapp_testing.md` - Web application testing patterns, Playwright best practices, and E2E testing strategies
• `references/TESTING_REPORT.template.md` - Test quality report template with coverage metrics, audit findings, and recommendations

When working with specific frameworks or need detailed patterns, load the appropriate reference file.

Best Practices

Test Quality

1. Isolation: Tests should be independent and runnable in any order 2. Deterministic: Tests should produce consistent results 3. Fast: Unit tests should run quickly (< 100ms each) 4. Clear: Test names should describe what they test 5. Maintainable: Tests should be easy to update when code changes

TDD Best Practices

1. One Behavior Per Test: Each test verifies ONE behavior 2. Descriptive Names: Test names describe the behavior being tested 3. Independent Tests: Tests don't depend on each other 4. Fast Tests: Mock external dependencies to keep tests fast 5. Clear Assertions: Assertions clearly show what's being verified

Common Mistakes to Avoid

• ❌ Writing multiple tests at once (write one test at a time)
• ❌ Skipping refactor phase (always refactor after green)
• ❌ Implementation before test (delete code and start with test)
• ❌ Over-engineering in GREEN (simplest thing that passes)
• ❌ Writing test that passes immediately (must fail first)

Test Maintenance

• Review and update tests when requirements change
• Remove obsolete tests
• Refactor tests to reduce duplication
• Keep test data factories up to date
• Monitor test execution time

Integration with Other Skills

• debugging: Use when tests fail unexpectedly
• code-review: TDD produces code that's easier to review
• dead-code-removal: Tests help identify unused code
• performance: Use for performance testing strategies

Meta-Principle

TDD is a DESIGN technique, not a testing technique.

The cycle never changes: RED → GREEN → REFACTOR → Repeat

Writing tests first forces you to think about:
- What behavior do I need?
- How will I know it works?
- What's the simplest implementation?

This produces better-designed, more maintainable code.

Framework-Specific TDD Workflows

Detailed TDD workflows and examples for different testing frameworks.

Python - pytest

Setup

Installation:

pip install pytest pytest-cov pytest-mock
# or
uv add pytest pytest-cov pytest-mock

Configuration (`pyproject.toml`):

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py", "*_test.py"]
python_functions = ["test_*"]
addopts = "-v --tb=short"

TDD Workflow Example

RED Phase:

# tests/test_calculator.py
def test_add_two_numbers():
    result = add(2, 3)
    assert result == 5

$ pytest tests/test_calculator.py::test_add_two_numbers
FAILED - NameError: name 'add' is not defined

GREEN Phase:

# src/calculator.py
def add(a, b):
    return 5  # Minimal implementation

$ pytest tests/test_calculator.py::test_add_two_numbers
PASSED in 0.01s

REFACTOR Phase:

# src/calculator.py
def add(a, b):
    return a + b  # Proper implementation

$ pytest
PASSED - All tests passing

Common Patterns

Fixtures:

@pytest.fixture
def sample_user():
    return User(name="Test", email="test@example.com")

def test_user_greeting(sample_user):
    assert sample_user.greet() == "Hello, Test"

Parametrization:

@pytest.mark.parametrize("a,b,expected", [
    (1, 2, 3),
    (0, 0, 0),
    (-1, 1, 0),
])
def test_add(a, b, expected):
    assert add(a, b) == expected

Mocking:

def test_api_call(mocker):
    mock_get = mocker.patch('requests.get')
    mock_get.return_value.json.return_value = {'data': 'test'}

    result = fetch_data()
    assert result == {'data': 'test'}

JavaScript/TypeScript - Jest

Setup

Installation:

npm install --save-dev jest @types/jest

Configuration (`package.json`):

{
  "scripts": {
    "test": "jest",
    "test:watch": "jest --watch"
  },
  "jest": {
    "testMatch": ["**/__tests__/**/*.js", "**/*.test.js"],
    "coverageDirectory": "coverage"
  }
}

TDD Workflow Example

RED Phase:

// __tests__/calculator.test.js
test('adds two numbers', () => {
  const result = add(2, 3);
  expect(result).toBe(5);
});

$ npm test
FAIL - ReferenceError: add is not defined

GREEN Phase:

// src/calculator.js
function add(a, b) {
  return 5; // Minimal implementation
}

$ npm test
PASS - 1 passed

REFACTOR Phase:

// src/calculator.js
function add(a, b) {
  return a + b; // Proper implementation
}

$ npm test
PASS - All tests passing

Common Patterns

Setup/Teardown:

beforeEach(() => {
  user = new User('Test');
});

afterEach(() => {
  user.cleanup();
});

Mocking:

jest.mock('./api');

test('fetches data', async () => {
  api.fetchData.mockResolvedValue({ data: 'test' });
  const result = await fetchData();
  expect(result).toEqual({ data: 'test' });
});

Snapshots:

test('renders component', () => {
  const component = render(<MyComponent />);
  expect(component).toMatchSnapshot();
});

JavaScript/TypeScript - Vitest

Setup

Installation:

npm install --save-dev vitest

Configuration (`vite.config.ts`):

import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    globals: true,
    environment: 'node',
  },
});

TDD Workflow

Similar to Jest, but with Vitest syntax:

import { describe, it, expect } from 'vitest';

describe('calculator', () => {
  it('adds two numbers', () => {
    expect(add(2, 3)).toBe(5);
  });
});

Java - JUnit

Setup

Maven (`pom.xml`):

<dependencies>
  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>5.9.2</version>
    <scope>test</scope>
  </dependency>
</dependencies>

TDD Workflow Example

RED Phase:

// src/test/java/CalculatorTest.java
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;

class CalculatorTest {
    @Test
    void testAddTwoNumbers() {
        int result = Calculator.add(2, 3);
        assertEquals(5, result);
    }
}

GREEN Phase:

// src/main/java/Calculator.java
public class Calculator {
    public static int add(int a, int b) {
        return 5; // Minimal implementation
    }
}

REFACTOR Phase:

public class Calculator {
    public static int add(int a, int b) {
        return a + b; // Proper implementation
    }
}

Common Patterns

Parameterized Tests:

@ParameterizedTest
@ValueSource(ints = {1, 2, 3})
void testIsPositive(int number) {
    assertTrue(number > 0);
}

Mocking (Mockito):

@Mock
private UserRepository userRepository;

@Test
void testGetUser() {
    when(userRepository.findById(1)).thenReturn(new User(1, "Test"));
    User user = userService.getUser(1);
    assertEquals("Test", user.getName());
}

Go - testing

Setup

Go testing is built-in, no setup needed.

TDD Workflow Example

RED Phase:

// calculator_test.go
package main

import "testing"

func TestAdd(t *testing.T) {
    result := Add(2, 3)
    if result != 5 {
        t.Errorf("Expected 5, got %d", result)
    }
}

$ go test
FAIL - undefined: Add

GREEN Phase:

// calculator.go
package main

func Add(a, b int) int {
    return 5 // Minimal implementation
}

$ go test
PASS

REFACTOR Phase:

func Add(a, b int) int {
    return a + b // Proper implementation
}

$ go test
PASS

Common Patterns

Table-Driven Tests:

func TestAdd(t *testing.T) {
    tests := []struct {
        a, b, expected int
    }{
        {1, 2, 3},
        {0, 0, 0},
        {-1, 1, 0},
    }

    for _, tt := range tests {
        result := Add(tt.a, tt.b)
        if result != tt.expected {
            t.Errorf("Add(%d, %d) = %d; expected %d", tt.a, tt.b, result, tt.expected)
        }
    }
}

Subtests:

func TestAdd(t *testing.T) {
    t.Run("positive numbers", func(t *testing.T) {
        result := Add(2, 3)
        if result != 5 {
            t.Errorf("Expected 5, got %d", result)
        }
    })

    t.Run("negative numbers", func(t *testing.T) {
        result := Add(-1, -2)
        if result != -3 {
            t.Errorf("Expected -3, got %d", result)
        }
    })
}

Rust - cargo test

Setup

Rust testing is built-in, no setup needed.

TDD Workflow Example

RED Phase:

// src/lib.rs
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_add() {
        assert_eq!(add(2, 3), 5);
    }
}

$ cargo test
FAIL - cannot find function `add` in this scope

GREEN Phase:

// src/lib.rs
pub fn add(a: i32, b: i32) -> i32 {
    5 // Minimal implementation
}

$ cargo test
PASS

REFACTOR Phase:

pub fn add(a: i32, b: i32) -> i32 {
    a + b // Proper implementation
}

$ cargo test
PASS

Common Patterns

Integration Tests:

// tests/integration_test.rs
use my_crate::add;

#[test]
fn test_add_integration() {
    assert_eq!(add(2, 3), 5);
}

Property-Based Testing (proptest):

use proptest::prelude::*;

proptest! {
    #[test]
    fn test_add_commutative(a in 0i32..1000, b in 0i32..1000) {
        assert_eq!(add(a, b), add(b, a));
    }
}

Running Tests

Python (pytest)

pytest                    # Run all tests
pytest path/to/test.py    # Run specific file
pytest -k "test_name"     # Run by name pattern
pytest -v                 # Verbose output
pytest --cov              # With coverage

JavaScript (Jest)

npm test                  # Run all tests
npm test -- file.test.js  # Run specific file
npm test -- --watch       # Watch mode
npm test -- --coverage    # With coverage

Java (Maven)

mvn test                  # Run all tests
mvn test -Dtest=TestClass # Run specific class

Go

go test                   # Run all tests
go test ./...             # Run all tests recursively
go test -v                # Verbose output
go test -cover            # With coverage

Rust

cargo test                # Run all tests
cargo test test_name      # Run specific test
cargo test -- --nocapture # Show output

TDD Patterns Reference

Common patterns, conventions, and anti-patterns for Test-Driven Development across different languages and frameworks.

Test Organization Patterns

File Structure

Python (pytest):

project/
├── src/
│   └── calculator.py
├── tests/
│   ├── conftest.py          # Shared fixtures
│   └── test_calculator.py
└── pyproject.toml

JavaScript/TypeScript (Jest/Vitest):

project/
├── src/
│   └── calculator.js
├── __tests__/
│   └── calculator.test.js
└── package.json

Java (JUnit):

project/
├── src/
│   └── main/java/Calculator.java
└── src/
    └── test/java/CalculatorTest.java

Go:

project/
├── calculator.go
└── calculator_test.go

Rust:

project/
├── src/
│   └── lib.rs
└── tests/
    └── integration_test.rs

Naming Conventions

Test Files

Python:

• test_*.py or *_test.py
• Example: test_calculator.py, calculator_test.py

JavaScript/TypeScript:

• *.test.js, *.test.ts
• *.spec.js, *.spec.ts
• Example: calculator.test.js

Java:

• *Test.java
• Example: CalculatorTest.java

Go:

• *_test.go
• Example: calculator_test.go

Rust:

• *_test.rs (unit tests in same file)
• tests/*.rs (integration tests)
• Example: calculator_test.rs

Test Functions/Methods

General Pattern:

• Start with test_ or Test prefix
• Describe behavior being tested
• Use descriptive names: test_user_can_login_with_valid_credentials

Examples:

• test_calculate_total_with_multiple_items
• test_login_fails_with_invalid_password
• test_user_cannot_access_admin_without_permission

Test Structure Patterns

AAA Pattern (Arrange-Act-Assert)

Structure:

def test_behavior():
    # Arrange: Set up test data
    user = User(name="Alice")

    # Act: Execute the code
    result = user.greet()

    # Assert: Verify outcome
    assert result == "Hello, Alice"

Given-When-Then (BDD)

Structure:

def test_behavior():
    # Given: Initial context
    user = create_user(role="admin")

    # When: Action occurs
    has_access = user.can_access("/admin")

    # Then: Expected outcome
    assert has_access is True

Setup/Teardown Pattern

Python (pytest fixtures):

@pytest.fixture
def user():
    user = User(name="Test")
    yield user
    user.cleanup()  # Teardown

JavaScript (Jest):

beforeEach(() => {
  user = new User('Test');
});

afterEach(() => {
  user.cleanup();
});

Test Doubles Patterns

Mock Pattern

When to Use:

• External API calls
• Database operations
• File system operations
• Expensive operations

Python (pytest-mock):

def test_api_call(mocker):
    mock_request = mocker.patch('requests.get')
    mock_request.return_value.json.return_value = {'data': 'test'}

    result = fetch_data()
    assert result == {'data': 'test'}

JavaScript (Jest):

jest.mock('./api');
test('api call', () => {
  api.fetchData.mockResolvedValue({ data: 'test' });
  const result = await fetchData();
  expect(result).toEqual({ data: 'test' });
});

Stub Pattern

When to Use:

• Replace dependencies with simple return values
• Control test data

Example:

def test_with_stub():
    stub_db = StubDatabase()
    stub_db.set_return_value('get_user', User(id=1))

    service = UserService(stub_db)
    user = service.get_user(1)
    assert user.id == 1

Spy Pattern

When to Use:

• Verify method calls without replacing behavior
• Count invocations

Example:

def test_logs_message(mocker):
    spy_log = mocker.spy(logging, 'info')

    process_data()

    spy_log.assert_called_once_with('Processing started')

Parametrization Patterns

Data-Driven Tests

Python (pytest):

@pytest.mark.parametrize("input,expected", [
    ("hello", "HELLO"),
    ("world", "WORLD"),
    ("", ""),
])
def test_uppercase(input, expected):
    assert input.upper() == expected

JavaScript (Jest):

test.each([
  ['hello', 'HELLO'],
  ['world', 'WORLD'],
  ['', ''],
])('uppercase %s', (input, expected) => {
  expect(input.toUpperCase()).toBe(expected);
});

Test Categories

Unit Tests

• Test individual functions/methods
• Fast execution (< 1ms per test)
• No external dependencies
• Mock all external calls

Integration Tests

• Test component interactions
• May use real dependencies (database, APIs)
• Slower than unit tests
• Test workflows, not individual functions

End-to-End Tests

• Test complete user workflows
• Use real systems
• Slowest tests
• Fewest in number (test pyramid)

Anti-Patterns

❌ Testing Implementation Details

Bad:

def test_internal_state():
    assert user._internal_counter == 5  # Testing private attribute

Good:

def test_behavior():
    assert user.is_ready()  # Testing public behavior

❌ Over-Mocking

Bad:

def test_calculation(mocker):
    mocker.patch('math.add')
    mocker.patch('math.multiply')
    # Mocking everything, testing nothing

Good:

def test_calculation():
    result = calculate(2, 3)
    assert result == 6  # Test actual behavior

❌ Test Interdependence

Bad:

# Test 1
def test_create_user():
    user = User.create()  # Creates global state
    assert user.id == 1

# Test 2
def test_get_user():
    user = User.get(1)  # Depends on test 1
    assert user is not None

Good:

# Each test is independent
def test_create_user():
    user = User.create()
    assert user.id is not None

def test_get_user():
    user = User.create()  # Creates its own data
    retrieved = User.get(user.id)
    assert retrieved.id == user.id

❌ Testing Multiple Things

Bad:

def test_user():
    user = User.create()
    assert user.name == "Test"
    assert user.email == "test@example.com"
    assert user.is_active is True
    assert user.can_login() is True
    # Too many assertions, unclear what's being tested

Good:

def test_user_creation():
    user = User.create(name="Test")
    assert user.name == "Test"

def test_user_email():
    user = User.create(email="test@example.com")
    assert user.email == "test@example.com"

def test_user_can_login():
    user = User.create()
    assert user.can_login() is True

❌ Slow Tests

Bad:

def test_api_call():
    result = real_api_call()  # Slow network call
    assert result is not None

Good:

def test_api_call(mocker):
    mock_api = mocker.patch('api.call')
    mock_api.return_value = {'data': 'test'}

    result = fetch_data()
    assert result == {'data': 'test'}

Test Quality Indicators

Good Tests

• ✅ Fast (< 1ms for unit tests)
• ✅ Independent (no shared state)
• ✅ Repeatable (same result every time)
• ✅ Self-validating (clear pass/fail)
• ✅ Timely (written before code)
• ✅ Clear names (describe behavior)
• ✅ One assertion per behavior

Bad Tests

• ❌ Slow (> 100ms for unit tests)
• ❌ Dependent on other tests
• ❌ Flaky (sometimes pass, sometimes fail)
• ❌ Unclear what's being tested
• ❌ Written after code
• ❌ Vague names (test1, test_function)
• ❌ Multiple unrelated assertions

TDD Workflow Patterns

Micro-Cycle Pattern

1. RED: Write smallest failing test 2. GREEN: Write minimal code to pass 3. REFACTOR: Clean up code 4. Repeat: Next smallest test

Feature Cycle Pattern

1. RED: Write test for feature 2. GREEN: Implement feature 3. REFACTOR: Improve implementation 4. RED: Write test for edge case 5. GREEN: Handle edge case 6. REFACTOR: Clean up

Bug Fix Pattern

1. RED: Write test that reproduces bug 2. GREEN: Fix bug (test now passes) 3. REFACTOR: Improve fix if needed 4. Verify: Ensure no regressions

Pytest Test Quality Report

Generated: {{TIMESTAMP}} Project: {{PROJECT_PATH}} Branch: {{GIT_BRANCH}} Python: {{PYTHON_VERSION}} | Pytest: {{PYTEST_VERSION}}

---

Executive Summary

---

Test Execution Summary

Command Executed:

{{PYTEST_COMMAND}}

Duration: {{DURATION}}

Test Output

<details> <summary>Click to expand full pytest output</summary>

{{PYTEST_OUTPUT}}

</details>

---

Coverage Summary

Overall Coverage: {{COVERAGE_PERCENT}}% Threshold: {{COVERAGE_THRESHOLD}}% Status: {{COVERAGE_STATUS}}

Coverage by Module

{{COVERAGE_TABLE}}

Files Below Threshold

{{FILES_BELOW_THRESHOLD}}

Uncovered Critical Code

{{UNCOVERED_CRITICAL_CODE}}

---

Audit Findings

Total Issues: {{TOTAL_ISSUES}} (Critical: {{CRITICAL_COUNT}}, High: {{HIGH_COUNT}}, Medium: {{MEDIUM_COUNT}}, Low: {{LOW_COUNT}})

CRITICAL (Must Fix)

{{CRITICAL_ISSUES}}

HIGH (Should Fix)

{{HIGH_ISSUES}}

MEDIUM (Improvements)

{{MEDIUM_ISSUES}}

LOW (Optional)

{{LOW_ISSUES}}

---

Test File Inventory

{{TEST_FILE_TABLE}}

---

Recommendations

Immediate Actions (Critical/High)

{{IMMEDIATE_ACTIONS}}

Short-term Improvements (Medium)

{{SHORT_TERM_ACTIONS}}

Future Enhancements (Low)

{{FUTURE_ACTIONS}}

---

Coverage Details

Missing Coverage by File

{{MISSING_COVERAGE_DETAILS}}

Suggested Tests to Add

{{SUGGESTED_TESTS}}

---

Test Quality Metrics

---

Appendix

A. Failed Test Details

{{FAILED_TEST_DETAILS}}

B. Slow Tests (>1s)

{{SLOW_TESTS}}

C. Coverage Report (Raw)

<details> <summary>Click to expand coverage JSON</summary>

{{COVERAGE_JSON}}

</details>

---

Report Location: {{REPORT_PATH}} Next Report: Run /tdd-pytest:report to regenerate

Web Application Testing

Comprehensive guide for testing web applications using Playwright and browser automation.

Playwright Basics

Installation

pip install playwright
playwright install chromium

Basic Script Structure

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto('http://localhost:5173')
    page.wait_for_load_state('networkidle')  # CRITICAL: Wait for JS
    # ... your test logic
    browser.close()

Server Management

Using with_server.py Helper

The scripts/with_server.py helper manages server lifecycle automatically:

Single server:

python scripts/with_server.py --server "npm run dev" --port 5173 -- python your_test.py

Multiple servers:

python scripts/with_server.py \
  --server "cd backend && python server.py" --port 3000 \
  --server "cd frontend && npm run dev" --port 5173 \
  -- python your_test.py

Always run with `--help` first to see current usage:

python scripts/with_server.py --help

Decision Tree

User task → Is it static HTML?
    ├─ Yes → Read HTML file directly to identify selectors
    │         ├─ Success → Write Playwright script using selectors
    │         └─ Fails/Incomplete → Treat as dynamic (below)
    │
    └─ No (dynamic webapp) → Is the server already running?
        ├─ No → Run: python scripts/with_server.py --help
        │        Then use the helper + write simplified Playwright script
        │
        └─ Yes → Reconnaissance-then-action:
            1. Navigate and wait for networkidle
            2. Take screenshot or inspect DOM
            3. Identify selectors from rendered state
            4. Execute actions with discovered selectors

Reconnaissance-Then-Action Pattern

Step 1: Inspect Rendered DOM

# Navigate and wait for full load
page.goto('http://localhost:5173')
page.wait_for_load_state('networkidle')

# Take screenshot for visual reference
page.screenshot(path='/tmp/inspect.png', full_page=True)

# Get page content
content = page.content()

# Discover elements
buttons = page.locator('button').all()
links = page.locator('a[href]').all()
inputs = page.locator('input, textarea, select').all()

Step 2: Identify Selectors

Use multiple selector strategies:

• text= - Match by visible text
• role= - Match by ARIA role
• CSS selectors - Standard CSS selectors
• IDs - Element IDs

Best Practice: Prefer text= and role= over CSS selectors for better maintainability.

Step 3: Execute Actions

# Click button by text
page.click('text=Submit')

# Fill form field
page.fill('#email', 'test@example.com')

# Select dropdown
page.select_option('#country', 'US')

# Wait for element
page.wait_for_selector('.success-message')

Common Patterns

Element Discovery

# Discover all buttons
buttons = page.locator('button').all()
for i, button in enumerate(buttons):
    text = button.inner_text() if button.is_visible() else "[hidden]"
    print(f"  [{i}] {text}")

# Discover links
links = page.locator('a[href]').all()
for link in links:
    text = link.inner_text().strip()
    href = link.get_attribute('href')
    print(f"  - {text} -> {href}")

# Discover input fields
inputs = page.locator('input, textarea, select').all()
for input_elem in inputs:
    name = input_elem.get_attribute('name') or input_elem.get_attribute('id')
    input_type = input_elem.get_attribute('type') or 'text'
    print(f"  - {name} ({input_type})")

Static HTML Testing

import os

html_file_path = os.path.abspath('path/to/your/file.html')
file_url = f'file://{html_file_path}'

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    # Navigate to local HTML file
    page.goto(file_url)

    # Interact with elements
    page.click('text=Click Me')
    page.fill('#name', 'John Doe')

    browser.close()

Console Log Capture

console_logs = []

def handle_console_message(msg):
    console_logs.append(f"[{msg.type}] {msg.text}")
    print(f"Console: [{msg.type}] {msg.text}")

page.on("console", handle_console_message)

# Navigate and interact
page.goto(url)
page.wait_for_load_state('networkidle')
page.click('text=Dashboard')

# Save logs
with open('console.log', 'w') as f:
    f.write('\n'.join(console_logs))

Best Practices

Waiting Strategies

Always wait for networkidle on dynamic apps:

page.goto(url)
page.wait_for_load_state('networkidle')  # CRITICAL

Wait for specific elements:

page.wait_for_selector('.loading-spinner', state='hidden')
page.wait_for_selector('.success-message', state='visible')

Wait for navigation:

with page.expect_navigation():
    page.click('text=Submit')

Selector Best Practices

Prefer semantic selectors:

# Good: Text-based selector
page.click('text=Submit Form')

# Good: Role-based selector
page.click('role=button[name="Submit"]')

# Acceptable: CSS selector
page.click('#submit-button')

# Avoid: Fragile CSS selectors
page.click('div.container > div.row > div.col > button.btn-primary')

Error Handling

try:
    page.goto(url)
    page.wait_for_load_state('networkidle', timeout=10000)
except Exception as e:
    print(f"Error loading page: {e}")
    page.screenshot(path='error.png')
    raise

Screenshots

# Full page screenshot
page.screenshot(path='full_page.png', full_page=True)

# Element screenshot
element = page.locator('.widget')
element.screenshot(path='widget.png')

# Screenshot on failure
try:
    page.click('text=Submit')
except Exception:
    page.screenshot(path='failure.png')
    raise

Common Pitfalls

❌ Don't Inspect Before networkidle

Bad:

page.goto(url)
content = page.content()  # Too early! JS hasn't executed

Good:

page.goto(url)
page.wait_for_load_state('networkidle')  # Wait first
content = page.content()  # Now safe

❌ Don't Use Hardcoded Timeouts

Bad:

page.click('text=Submit')
time.sleep(5)  # Arbitrary wait

Good:

page.click('text=Submit')
page.wait_for_selector('.success-message')  # Wait for actual condition

❌ Don't Forget to Close Browser

Bad:

browser = p.chromium.launch()
page = browser.new_page()
# ... test code
# Browser never closed!

Good:

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    # ... test code
    browser.close()  # Automatically closed by context manager

Examples

See examples/ directory for complete examples:

• element_discovery.py - Discovering buttons, links, and inputs
• static_html_automation.py - Testing static HTML files
• console_logging.py - Capturing console logs during automation

Integration with Test Frameworks

With pytest

import pytest
from playwright.sync_api import sync_playwright

@pytest.fixture(scope="module")
def browser():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        yield browser
        browser.close()

def test_login(browser):
    page = browser.new_page()
    page.goto('http://localhost:5173')
    page.wait_for_load_state('networkidle')
    page.fill('#email', 'test@example.com')
    page.fill('#password', 'password')
    page.click('text=Login')
    page.wait_for_selector('.dashboard')
    assert page.locator('.dashboard').is_visible()
    page.close()

With unittest

import unittest
from playwright.sync_api import sync_playwright

class WebAppTest(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.playwright = sync_playwright().start()
        cls.browser = cls.playwright.chromium.launch(headless=True)

    @classmethod
    def tearDownClass(cls):
        cls.browser.close()
        cls.playwright.stop()

    def setUp(self):
        self.page = self.browser.new_page()

    def tearDown(self):
        self.page.close()

    def test_homepage_loads(self):
        self.page.goto('http://localhost:5173')
        self.page.wait_for_load_state('networkidle')
        self.assertIn('Welcome', self.page.content())