Test Automation Expert

Test Automation Expert

Comprehensive testing guidance from unit to E2E. Designs test strategies, implements automation, and optimizes coverage for sustainable quality.

When to Use

Use for:

• Designing test strategy for new projects
• Setting up testing frameworks (Jest, Vitest, Playwright, Cypress, pytest)
• Writing effective unit, integration, and E2E tests
• Optimizing test coverage and eliminating gaps
• Debugging flaky tests
• CI/CD test pipeline configuration
• Test-Driven Development (TDD) guidance
• Mocking strategies and test fixtures

Do NOT use for:

• Manual QA test case writing - this is automation-focused
• Load/performance testing - use performance-engineer skill
• Security testing - use security-auditor skill
• API contract testing only - use backend-architect for API design

Test Pyramid Philosophy

         /\
        /  \      E2E Tests (10%)
       /----\     - Critical user journeys
      /      \    - Cross-browser validation
     /--------\
    /          \  Integration Tests (20%)
   /            \ - API contracts
  /--------------\- Component interactions
 /                \
/------------------\ Unit Tests (70%)
                    - Fast, isolated, deterministic
                    - Business logic validation

Distribution Guidelines

Framework Selection

JavaScript/TypeScript

Python

Decision Tree: Framework Selection

New project?
├── Yes → Using Vite?
│   ├── Yes → Vitest
│   └── No → Jest or Vitest (both work)
└── No → What exists?
    ├── Jest → Keep Jest (migration cost rarely worth it)
    ├── Mocha → Consider migration to Vitest
    └── Nothing → Vitest (modern default)

Need E2E?
├── Cross-browser critical → Playwright
├── Developer experience priority → Cypress
└── Both → Playwright (more flexible)

Unit Testing Patterns

Good Unit Test Anatomy

describe('UserService', () => {
  describe('validateEmail', () => {
    // Arrange-Act-Assert pattern
    it('should accept valid email formats', () => {
      // Arrange
      const validEmails = ['user@example.com', 'name+tag@domain.co'];

      // Act & Assert
      validEmails.forEach(email => {
        expect(validateEmail(email)).toBe(true);
      });
    });

    it('should reject invalid email formats', () => {
      // Arrange
      const invalidEmails = ['invalid', '@missing.com', 'no@tld'];

      // Act & Assert
      invalidEmails.forEach(email => {
        expect(validateEmail(email)).toBe(false);
      });
    });

    // Edge cases explicitly tested
    it('should handle empty string', () => {
      expect(validateEmail('')).toBe(false);
    });

    it('should handle null/undefined', () => {
      expect(validateEmail(null)).toBe(false);
      expect(validateEmail(undefined)).toBe(false);
    });
  });
});

Mocking Strategies

// ✅ Good: Mock at boundaries
jest.mock('../services/api', () => ({
  fetchUser: jest.fn()
}));

// ✅ Good: Explicit mock setup per test
beforeEach(() => {
  fetchUser.mockReset();
});

it('handles user not found', async () => {
  fetchUser.mockRejectedValue(new NotFoundError());
  await expect(getUser(123)).rejects.toThrow('User not found');
});

// ❌ Bad: Mocking implementation details
jest.mock('../utils/internal-helper'); // Don't mock internals

Test Isolation Checklist

• [ ] Each test can run independently
• [ ] No shared mutable state between tests
• [ ] Database/API state reset between tests
• [ ] No test order dependencies
• [ ] Parallel execution safe

Integration Testing Patterns

API Integration Test

describe('POST /api/users', () => {
  let app;
  let db;

  beforeAll(async () => {
    db = await createTestDatabase();
    app = createApp({ db });
  });

  afterAll(async () => {
    await db.close();
  });

  beforeEach(async () => {
    await db.clear();
  });

  it('creates user with valid data', async () => {
    const response = await request(app)
      .post('/api/users')
      .send({ name: 'Test', email: 'test@example.com' })
      .expect(201);

    expect(response.body).toMatchObject({
      id: expect.any(String),
      name: 'Test',
      email: 'test@example.com'
    });

    // Verify side effects
    const dbUser = await db.users.findById(response.body.id);
    expect(dbUser).toBeDefined();
  });

  it('rejects duplicate email', async () => {
    await db.users.create({ name: 'Existing', email: 'test@example.com' });

    await request(app)
      .post('/api/users')
      .send({ name: 'New', email: 'test@example.com' })
      .expect(409);
  });
});

Component Integration (React)

import { render, screen, waitFor } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { UserProfile } from './UserProfile';
import { UserProvider } from '../context/UserContext';

describe('UserProfile integration', () => {
  it('loads and displays user data', async () => {
    render(
      <UserProvider>
        <UserProfile userId="123" />
      </UserProvider>
    );

    // Verify loading state
    expect(screen.getByRole('progressbar')).toBeInTheDocument();

    // Wait for data
    await waitFor(() => {
      expect(screen.getByText('John Doe')).toBeInTheDocument();
    });

    // Verify loaded state
    expect(screen.queryByRole('progressbar')).not.toBeInTheDocument();
  });
});

E2E Testing Patterns

Playwright Best Practices

import { test, expect } from '@playwright/test';

test.describe('Checkout Flow', () => {
  test.beforeEach(async ({ page }) => {
    // Seed test data via API
    await page.request.post('/api/test/seed', {
      data: { scenario: 'checkout-ready' }
    });
  });

  test('complete purchase with credit card', async ({ page }) => {
    await page.goto('/cart');

    // Use accessible selectors
    await page.getByRole('button', { name: 'Proceed to checkout' }).click();

    // Fill payment form
    await page.getByLabel('Card number').fill('4242424242424242');
    await page.getByLabel('Expiry').fill('12/25');
    await page.getByLabel('CVC').fill('123');

    // Complete purchase
    await page.getByRole('button', { name: 'Pay now' }).click();

    // Verify success
    await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
    await expect(page.getByText(/Order #\d+/)).toBeVisible();
  });

  test('shows error for declined card', async ({ page }) => {
    await page.goto('/checkout');

    // Use test card that triggers decline
    await page.getByLabel('Card number').fill('4000000000000002');
    await page.getByLabel('Expiry').fill('12/25');
    await page.getByLabel('CVC').fill('123');

    await page.getByRole('button', { name: 'Pay now' }).click();

    await expect(page.getByRole('alert')).toContainText('Card declined');
  });
});

Flaky Test Detection & Prevention

Common Causes: 1. Race conditions in async operations 2. Time-dependent tests 3. Shared state between tests 4. Network variability 5. Animation/transition timing

Fixes:

// ❌ Bad: Fixed timeout
await page.waitForTimeout(2000);

// ✅ Good: Wait for specific condition
await expect(page.getByText('Loaded')).toBeVisible();

// ❌ Bad: Checking exact time
expect(new Date()).toEqual(specificDate);

// ✅ Good: Mock time
jest.useFakeTimers();
jest.setSystemTime(new Date('2024-01-15'));

// ❌ Bad: Depending on animation completion
await page.click('.button');
expect(await page.isVisible('.modal')).toBe(true);

// ✅ Good: Wait for animation
await page.click('.button');
await expect(page.locator('.modal')).toBeVisible();

Coverage Optimization

What to Measure

Coverage Configuration

// vitest.config.js
export default defineConfig({
  test: {
    coverage: {
      provider: 'v8',
      reporter: ['text', 'json', 'html'],
      exclude: [
        'node_modules/',
        'test/',
        '**/*.d.ts',
        '**/*.config.*',
        '**/index.ts', // barrel files
      ],
      thresholds: {
        branches: 75,
        functions: 80,
        lines: 80,
        statements: 80
      }
    }
  }
});

Finding Coverage Gaps

# Generate detailed coverage report
npx vitest run --coverage

# Find untested files
npx vitest run --coverage --reporter=json | jq '.coverageMap | to_entries | map(select(.value.s | values | any(. == 0))) | .[].key'

CI/CD Integration

GitHub Actions

name: Tests
on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npm test -- --coverage
      - uses: codecov/codecov-action@v4

  e2e-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npm run test:e2e
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/

Test Parallelization

// vitest.config.js - parallel by default
export default defineConfig({
  test: {
    pool: 'threads',
    poolOptions: {
      threads: {
        singleThread: false
      }
    }
  }
});

// playwright.config.js
export default defineConfig({
  workers: process.env.CI ? 2 : undefined,
  fullyParallel: true
});

Anti-Patterns

Anti-Pattern: Testing Implementation Details

What it looks like:

// ❌ Testing internal state
expect(component.state.isLoading).toBe(true);

// ❌ Testing private methods
expect(service._calculateHash()).toBe('abc123');

Why wrong: Couples tests to implementation, breaks on refactors

Instead:

// ✅ Test observable behavior
expect(screen.getByRole('progressbar')).toBeInTheDocument();

// ✅ Test public interface
expect(service.getHash()).toBe('abc123');

Anti-Pattern: Over-Mocking

What it looks like:

// ❌ Mocking everything
jest.mock('../utils/format');
jest.mock('../utils/validate');
jest.mock('../utils/transform');

Why wrong: Tests pass even when real code is broken

Instead: Mock only at system boundaries (APIs, databases, external services)

Anti-Pattern: Flaky Acceptance

What it looks like: "That test is just flaky, skip it"

Why wrong: Flaky tests indicate real problems (race conditions, timing issues)

Instead: Fix the flakiness or quarantine while fixing

Anti-Pattern: Coverage Theater

What it looks like:

// ❌ Testing for coverage, not behavior
it('covers the function', () => {
  myFunction();
  // No assertions!
});

Why wrong: 100% coverage with 0% confidence

Instead: Every test should assert meaningful behavior

Quick Commands

# Run all tests
npm test

# Run with coverage
npm test -- --coverage

# Run specific file
npm test -- src/utils/format.test.ts

# Run in watch mode
npm test -- --watch

# Run E2E tests
npx playwright test

# Run E2E with UI
npx playwright test --ui

# Debug E2E test
npx playwright test --debug

# Update snapshots
npm test -- -u

Reference Files

• references/test-strategy.md - Comprehensive test strategy framework
• references/framework-comparison.md - Detailed framework comparison
• references/coverage-patterns.md - Coverage optimization techniques
• references/ci-integration.md - CI/CD pipeline configurations

---

Covers: Test strategy | Unit testing | Integration testing | E2E testing | Coverage | CI/CD | Flaky test debugging

Use with: security-auditor (security tests) | performance-engineer (load tests) | code-reviewer (test quality)

Changelog

All notable changes to the test-automation-expert skill will be documented in this file.

[1.0.0] - 2024-12-08

Added

• Initial release of test-automation-expert skill
• Comprehensive SKILL.md with test pyramid philosophy
• Framework selection guidance (Jest, Vitest, Playwright, Cypress, pytest)
• Unit testing patterns with mocking strategies
• Integration testing patterns for APIs and components
• E2E testing best practices with Playwright
• Flaky test detection and prevention guide
• Coverage optimization strategies
• CI/CD integration examples (GitHub Actions)
• Anti-patterns documentation
• Reference files:
• references/test-strategy.md - Test strategy framework
• references/framework-comparison.md - Framework comparison matrix
• references/coverage-patterns.md - Coverage techniques
• references/ci-integration.md - CI/CD configurations

Technical Decisions

• Chose Vitest as recommended modern default over Jest
• Playwright recommended over Cypress for cross-browser needs
• Test pyramid distribution: 70% unit, 20% integration, 10% E2E
• Coverage thresholds: 80% lines, 75% branches, 90% functions

References

• Based on testing best practices from Testing Library, Playwright, and Vitest documentation
• Anti-patterns derived from common issues in production codebases

CI/CD Test Integration

Complete configurations for integrating tests into continuous integration pipelines.

GitHub Actions

Complete Test Workflow

# .github/workflows/test.yml
name: Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

env:
  NODE_VERSION: '20'
  CI: true

jobs:
  # ============================================
  # Unit & Integration Tests
  # ============================================
  unit-tests:
    name: Unit Tests
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run unit tests
        run: npm test -- --coverage --maxWorkers=2

      - name: Upload coverage
        uses: codecov/codecov-action@v4
        with:
          files: ./coverage/lcov.info
          flags: unittests
          fail_ci_if_error: true

      - name: Archive coverage report
        uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: coverage/
          retention-days: 7

  # ============================================
  # E2E Tests
  # ============================================
  e2e-tests:
    name: E2E Tests
    runs-on: ubuntu-latest
    needs: unit-tests # Only run if unit tests pass

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium firefox

      - name: Build application
        run: npm run build

      - name: Run E2E tests
        run: npx playwright test
        env:
          BASE_URL: http://localhost:3000

      - name: Upload Playwright report
        uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 7

      - name: Upload test results
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results
          path: test-results/
          retention-days: 7

  # ============================================
  # Component Tests (Storybook)
  # ============================================
  component-tests:
    name: Component Tests
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0 # For Chromatic

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run Chromatic
        uses: chromaui/action@latest
        with:
          projectToken: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}
          buildScriptName: build-storybook
          onlyChanged: true

  # ============================================
  # Test Summary
  # ============================================
  test-summary:
    name: Test Summary
    runs-on: ubuntu-latest
    needs: [unit-tests, e2e-tests]
    if: always()

    steps:
      - name: Check test results
        run: |
          if [ "${{ needs.unit-tests.result }}" == "failure" ] || \
             [ "${{ needs.e2e-tests.result }}" == "failure" ]; then
            echo "Tests failed!"
            exit 1
          fi
          echo "All tests passed!"

Matrix Testing (Multiple Versions)

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        node: [18, 20, 22]
        exclude:
          - os: windows-latest
            node: 18

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
      - run: npm ci
      - run: npm test

Parallel E2E with Sharding

e2e-tests:
  runs-on: ubuntu-latest
  strategy:
    fail-fast: false
    matrix:
      shard: [1, 2, 3, 4]

  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
    - run: npm ci
    - run: npx playwright install --with-deps

    - name: Run tests (shard ${{ matrix.shard }}/4)
      run: npx playwright test --shard=${{ matrix.shard }}/4

GitLab CI

Complete Pipeline

# .gitlab-ci.yml
stages:
  - install
  - test
  - e2e
  - report

variables:
  NODE_VERSION: "20"
  npm_config_cache: "$CI_PROJECT_DIR/.npm"

cache:
  key:
    files:
      - package-lock.json
  paths:
    - .npm/
    - node_modules/

# ============================================
# Install Stage
# ============================================
install:
  stage: install
  image: node:${NODE_VERSION}
  script:
    - npm ci
  artifacts:
    paths:
      - node_modules/
    expire_in: 1 hour

# ============================================
# Test Stage
# ============================================
unit-tests:
  stage: test
  image: node:${NODE_VERSION}
  needs: [install]
  script:
    - npm test -- --coverage
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    when: always
    paths:
      - coverage/
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

lint:
  stage: test
  image: node:${NODE_VERSION}
  needs: [install]
  script:
    - npm run lint
    - npm run typecheck

# ============================================
# E2E Stage
# ============================================
e2e-tests:
  stage: e2e
  image: mcr.microsoft.com/playwright:v1.40.0-jammy
  needs: [unit-tests]
  script:
    - npm ci
    - npm run build
    - npx playwright test
  artifacts:
    when: always
    paths:
      - playwright-report/
      - test-results/
    expire_in: 1 week

# ============================================
# Report Stage
# ============================================
pages:
  stage: report
  needs: [unit-tests, e2e-tests]
  script:
    - mkdir public
    - cp -r coverage/ public/coverage
    - cp -r playwright-report/ public/e2e
  artifacts:
    paths:
      - public
  only:
    - main

CircleCI

Complete Config

# .circleci/config.yml
version: 2.1

orbs:
  node: circleci/node@5.1.0
  browser-tools: circleci/browser-tools@1.4.6

executors:
  node-executor:
    docker:
      - image: cimg/node:20.10
    working_directory: ~/project

commands:
  install-deps:
    steps:
      - restore_cache:
          keys:
            - npm-deps-{{ checksum "package-lock.json" }}
      - run: npm ci
      - save_cache:
          key: npm-deps-{{ checksum "package-lock.json" }}
          paths:
            - node_modules

jobs:
  # ============================================
  # Unit Tests
  # ============================================
  unit-tests:
    executor: node-executor
    steps:
      - checkout
      - install-deps
      - run:
          name: Run unit tests
          command: npm test -- --coverage --ci
      - store_test_results:
          path: test-results
      - store_artifacts:
          path: coverage
          destination: coverage
      - run:
          name: Upload coverage
          command: bash <(curl -s https://codecov.io/bash)

  # ============================================
  # E2E Tests
  # ============================================
  e2e-tests:
    executor: node-executor
    parallelism: 4
    steps:
      - checkout
      - install-deps
      - run:
          name: Install Playwright
          command: npx playwright install --with-deps
      - run:
          name: Build app
          command: npm run build
      - run:
          name: Run E2E tests
          command: |
            SHARD="$((${CIRCLE_NODE_INDEX}+1))"
            npx playwright test --shard=${SHARD}/${CIRCLE_NODE_TOTAL}
      - store_artifacts:
          path: playwright-report
          destination: playwright-report
      - store_test_results:
          path: test-results

  # ============================================
  # Integration Tests (with services)
  # ============================================
  integration-tests:
    docker:
      - image: cimg/node:20.10
      - image: postgres:15
        environment:
          POSTGRES_DB: test
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
      - image: redis:7
    steps:
      - checkout
      - install-deps
      - run:
          name: Wait for services
          command: dockerize -wait tcp://localhost:5432 -wait tcp://localhost:6379 -timeout 30s
      - run:
          name: Run integration tests
          command: npm run test:integration
          environment:
            DATABASE_URL: postgres://test:test@localhost:5432/test
            REDIS_URL: redis://localhost:6379

workflows:
  test:
    jobs:
      - unit-tests
      - integration-tests:
          requires:
            - unit-tests
      - e2e-tests:
          requires:
            - unit-tests

Jenkins

Jenkinsfile

// Jenkinsfile
pipeline {
    agent {
        docker {
            image 'node:20'
        }
    }

    environment {
        CI = 'true'
        npm_config_cache = "${WORKSPACE}/.npm"
    }

    stages {
        stage('Install') {
            steps {
                sh 'npm ci'
            }
        }

        stage('Test') {
            parallel {
                stage('Unit Tests') {
                    steps {
                        sh 'npm test -- --coverage --ci'
                    }
                    post {
                        always {
                            junit 'junit.xml'
                            publishHTML([
                                reportDir: 'coverage/lcov-report',
                                reportFiles: 'index.html',
                                reportName: 'Coverage Report'
                            ])
                        }
                    }
                }

                stage('Lint') {
                    steps {
                        sh 'npm run lint'
                    }
                }
            }
        }

        stage('E2E Tests') {
            agent {
                docker {
                    image 'mcr.microsoft.com/playwright:v1.40.0-jammy'
                }
            }
            steps {
                sh 'npm ci'
                sh 'npm run build'
                sh 'npx playwright test'
            }
            post {
                always {
                    publishHTML([
                        reportDir: 'playwright-report',
                        reportFiles: 'index.html',
                        reportName: 'Playwright Report'
                    ])
                }
            }
        }
    }

    post {
        failure {
            slackSend(
                color: 'danger',
                message: "Build Failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}"
            )
        }
    }
}

Test Caching Strategies

npm/Node.js

# GitHub Actions
- uses: actions/cache@v4
  with:
    path: |
      ~/.npm
      node_modules
    key: npm-${{ hashFiles('package-lock.json') }}
    restore-keys: npm-

# Playwright browsers
- uses: actions/cache@v4
  with:
    path: ~/.cache/ms-playwright
    key: playwright-${{ hashFiles('package-lock.json') }}

pytest

- uses: actions/cache@v4
  with:
    path: |
      ~/.cache/pip
      .pytest_cache
    key: pytest-${{ hashFiles('requirements.txt') }}

Flaky Test Handling

Retry Configuration

# GitHub Actions
- name: Run tests with retry
  uses: nick-fields/retry@v2
  with:
    max_attempts: 3
    timeout_minutes: 10
    command: npm test

# Playwright built-in
- name: Run E2E with retries
  run: npx playwright test --retries=2

Quarantine Flaky Tests

// Mark flaky tests for tracking
test.describe('Flaky Feature', () => {
  test.fixme('sometimes fails due to race condition', async () => {
    // Test code
  });
});

// Or skip in CI only
test('flaky test', async () => {
  test.skip(process.env.CI, 'Flaky in CI - JIRA-123');
});

Notifications

Slack Integration

# GitHub Actions
- name: Notify Slack on failure
  if: failure()
  uses: slackapi/slack-github-action@v1
  with:
    payload: |
      {
        "text": "❌ Tests failed on ${{ github.ref }}",
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "*Test Failure*\n• Repository: ${{ github.repository }}\n• Branch: ${{ github.ref }}\n• Commit: ${{ github.sha }}"
            }
          }
        ]
      }
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

Performance Optimization

Test Splitting

# Dynamic test splitting based on timing
- name: Run tests with timing
  run: |
    npx jest --listTests --json > tests.json
    npx jest $(cat tests.json | jq -r '.[] | select(. | test("unit"))' | head -n $(($(cat tests.json | jq length) / 4)))

Selective Testing

# Only run affected tests
- uses: dorny/paths-filter@v2
  id: changes
  with:
    filters: |
      src:
        - 'src/**'
      tests:
        - 'tests/**'

- name: Run tests if source changed
  if: steps.changes.outputs.src == 'true'
  run: npm test

Coverage Optimization Patterns

Strategies for meaningful test coverage that catches bugs without slowing development.

Understanding Coverage Metrics

Types of Coverage

Why Branch Coverage Matters Most

function processOrder(order) {
  if (order.isPriority && order.total > 100) {  // 4 branches!
    applyDiscount(order);
  }
  return order;
}

// Line coverage: 100% with just one test
// Branch coverage: Only 25% - missing 3 combinations!

// Need tests for:
// 1. isPriority=true, total>100 (discount applied)
// 2. isPriority=true, total<=100 (no discount)
// 3. isPriority=false, total&gt;100 (no discount)
// 4. isPriority=false, total<=100 (no discount)

Coverage Configuration

Vitest

// vitest.config.ts
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    coverage: {
      provider: 'v8', // or 'istanbul'
      reporter: ['text', 'json', 'html', 'lcov'],
      reportsDirectory: './coverage',

      // What to include
      include: ['src/**/*.{ts,tsx}'],

      // What to exclude
      exclude: [
        'node_modules/',
        'src/**/*.d.ts',
        'src/**/*.test.{ts,tsx}',
        'src/**/*.spec.{ts,tsx}',
        'src/**/*.stories.{ts,tsx}',
        'src/**/index.ts',      // Barrel files
        'src/**/types.ts',      // Type definitions
        'src/**/*.config.*',    // Config files
        'src/mocks/**',         // Test mocks
        'src/__fixtures__/**',  // Test fixtures
      ],

      // Thresholds
      thresholds: {
        global: {
          branches: 75,
          functions: 80,
          lines: 80,
          statements: 80,
        },
        // Per-file thresholds
        'src/utils/**': {
          branches: 90,
          functions: 95,
        },
        'src/components/**': {
          branches: 70,
          lines: 75,
        },
      },

      // Fail if coverage drops
      watermarks: {
        lines: [70, 80],
        functions: [70, 80],
        branches: [70, 75],
        statements: [70, 80],
      },
    },
  },
});

Jest

// jest.config.js
module.exports = {
  collectCoverage: true,
  coverageDirectory: 'coverage',
  coverageReporters: ['text', 'lcov', 'html'],

  collectCoverageFrom: [
    'src/**/*.{js,jsx,ts,tsx}',
    '!src/**/*.d.ts',
    '!src/**/*.stories.*',
    '!src/**/*.test.*',
    '!src/**/index.{js,ts}',
  ],

  coverageThreshold: {
    global: {
      branches: 75,
      functions: 80,
      lines: 80,
      statements: 80,
    },
    './src/utils/': {
      branches: 90,
      functions: 95,
    },
  },
};

pytest

# pyproject.toml
[tool.coverage.run]
branch = true
source = ["src"]
omit = [
    "tests/*",
    "**/__init__.py",
    "**/conftest.py",
    "**/*_test.py",
]
parallel = true

[tool.coverage.report]
fail_under = 80
show_missing = true
skip_covered = false
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "raise NotImplementedError",
    "if TYPE_CHECKING:",
    "if __name__ == .__main__.:",
    "@abstractmethod",
]

[tool.coverage.html]
directory = "coverage_html"

Finding Coverage Gaps

Command Line Analysis

# Vitest - show uncovered lines
npx vitest run --coverage

# Jest - detailed report
npx jest --coverage --coverageReporters=text-summary

# Find files with low coverage
npx vitest run --coverage --reporter=json | \
  jq '.coverageMap | to_entries |
      map(select(.value.s | values | map(select(. == 0)) | length > 0)) |
      .[].key'

# pytest - show missing lines
pytest --cov=src --cov-report=term-missing

HTML Report Navigation

1. Generate HTML report: npx vitest run --coverage 2. Open coverage/index.html 3. Sort by "Branches" or "Lines" (ascending) 4. Click into files with low coverage 5. Red highlights = uncovered code

Identifying Critical Gaps

Priority order for fixing gaps:

1. Error handling paths - Often untested but critical

   try {
     await api.call();
   } catch (error) {
     // This branch often untested!
     logger.error(error);
     throw new AppError('API failed');
   }

2. Edge cases in conditionals

   if (value === null || value === undefined) {
     // Null/undefined often missed
   }

3. Async error paths

   promise.catch(error => {
     // Rejection handlers often untested
   });

4. Default switch cases

   switch (status) {
     case 'active': return 'green';
     case 'pending': return 'yellow';
     default: return 'gray'; // Often untested
   }

Coverage Anti-Patterns

Anti-Pattern: Coverage Without Assertions

// ❌ Bad: Executes code but tests nothing
it('covers the function', () => {
  processOrder({ id: 1 });
  // No expect()!
});

// ✅ Good: Meaningful assertions
it('processes valid order', () => {
  const result = processOrder({ id: 1, items: [{ price: 10 }] });
  expect(result.total).toBe(10);
  expect(result.status).toBe('processed');
});

Anti-Pattern: Testing Private Internals

// ❌ Bad: Testing implementation detail
it('sets internal flag', () => {
  const service = new UserService();
  service._processUser(user);
  expect(service._isProcessed).toBe(true);
});

// ✅ Good: Test observable behavior
it('marks user as active after processing', () => {
  const service = new UserService();
  const result = service.processUser(user);
  expect(result.status).toBe('active');
});

Anti-Pattern: Excessive Mocking

// ❌ Bad: Everything mocked, testing nothing real
jest.mock('./database');
jest.mock('./logger');
jest.mock('./validator');
jest.mock('./formatter');

it('processes data', () => {
  // All real code is mocked out!
});

// ✅ Good: Mock only boundaries
jest.mock('./database'); // External system

it('validates and saves data', () => {
  // Real validation, real formatting
  // Only DB is mocked
});

Effective Coverage Strategies

Strategy 1: Test Behaviors, Not Lines

// Instead of testing each line...
describe('OrderCalculator', () => {
  // Test the behaviors users care about
  it('calculates subtotal from items', () => {});
  it('applies percentage discount', () => {});
  it('applies fixed discount', () => {});
  it('calculates tax after discounts', () => {});
  it('rounds total to 2 decimal places', () => {});
});

Strategy 2: Boundary Value Testing

// Test at boundaries, not random values
describe('validateAge', () => {
  it('rejects age below minimum (17)', () => {
    expect(validateAge(17)).toBe(false);
  });

  it('accepts age at minimum (18)', () => {
    expect(validateAge(18)).toBe(true);
  });

  it('accepts age at maximum (120)', () => {
    expect(validateAge(120)).toBe(true);
  });

  it('rejects age above maximum (121)', () => {
    expect(validateAge(121)).toBe(false);
  });
});

Strategy 3: Error Path Testing

describe('fetchUser', () => {
  it('returns user on success', async () => {
    mockApi.get.mockResolvedValue({ data: { id: 1 } });
    const user = await fetchUser(1);
    expect(user.id).toBe(1);
  });

  // Explicitly test each error case
  it('throws NotFoundError for 404', async () => {
    mockApi.get.mockRejectedValue({ status: 404 });
    await expect(fetchUser(999)).rejects.toThrow(NotFoundError);
  });

  it('throws NetworkError for timeout', async () => {
    mockApi.get.mockRejectedValue({ code: 'ETIMEDOUT' });
    await expect(fetchUser(1)).rejects.toThrow(NetworkError);
  });

  it('retries on 503 before failing', async () => {
    mockApi.get
      .mockRejectedValueOnce({ status: 503 })
      .mockRejectedValueOnce({ status: 503 })
      .mockResolvedValue({ data: { id: 1 } });

    const user = await fetchUser(1);
    expect(mockApi.get).toHaveBeenCalledTimes(3);
  });
});

Strategy 4: Parameterized Tests

// Test multiple cases efficiently
describe.each([
  ['valid@email.com', true],
  ['user+tag@domain.co', true],
  ['name@subdomain.domain.org', true],
  ['invalid', false],
  ['@missing.com', false],
  ['spaces @domain.com', false],
  ['', false],
  [null, false],
])('validateEmail(%s)', (email, expected) => {
  it(`returns ${expected}`, () => {
    expect(validateEmail(email)).toBe(expected);
  });
});

Strategy 5: Coverage-Driven Refactoring

When you find untestable code, refactor it:

// ❌ Hard to test: side effects mixed with logic
function processOrder(order) {
  const discount = order.customer.tier === 'gold' ? 0.1 : 0;
  const total = order.items.reduce((sum, i) => sum + i.price, 0);
  const finalTotal = total * (1 - discount);

  database.save({ ...order, total: finalTotal }); // Side effect!
  emailService.send(order.customer.email, finalTotal); // Side effect!

  return finalTotal;
}

// ✅ Testable: Pure calculation, separate side effects
function calculateOrderTotal(order) {
  const discount = getCustomerDiscount(order.customer);
  const subtotal = calculateSubtotal(order.items);
  return applyDiscount(subtotal, discount);
}

function processOrder(order) {
  const total = calculateOrderTotal(order);
  await saveOrder({ ...order, total });
  await notifyCustomer(order.customer, total);
  return total;
}

// Now you can unit test calculateOrderTotal with 100% coverage
// And integration test processOrder separately

Coverage in CI/CD

GitHub Actions

- name: Run tests with coverage
  run: npm test -- --coverage --coverageReporters=json-summary

- name: Check coverage thresholds
  run: |
    COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
    if (( $(echo "$COVERAGE < 80" | bc -l) )); then
      echo "Coverage $COVERAGE% is below 80%"
      exit 1
    fi

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v4
  with:
    files: ./coverage/lcov.info
    fail_ci_if_error: true

Coverage Diff on PRs

- name: Coverage diff
  uses: ArtiomTr/jest-coverage-report-action@v2
  with:
    threshold: 80
    annotations: coverage

Coverage Reporting Tools

Realistic Targets

By Project Type

Incremental Improvement

Week 1: Establish baseline, add CI check Week 2: Target +5% on lowest files Week 4: Hit 60% if starting low Week 8: Hit 70% overall Week 12: Hit 80% stable target

Testing Framework Comparison

Detailed comparison of modern testing frameworks to help select the right tools.

Unit Testing Frameworks

JavaScript/TypeScript

When to Choose Each

Choose Jest when:

• Existing React project with CRA or Next.js
• Team already knows Jest
• Need extensive plugin ecosystem
• Snapshot testing is primary strategy

Choose Vitest when:

• New project, especially with Vite
• Performance is priority
• Native ESM/TypeScript needed
• Want Jest-compatible API with modern DX

Choose Mocha when:

• Need maximum flexibility
• Custom reporting requirements
• Legacy project compatibility
• Specific assertion library preference

Configuration Examples

Jest (jest.config.js):

module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'jsdom',
  setupFilesAfterEnv: ['<rootDir>/jest.setup.js'],
  moduleNameMapper: {
    '^@/(.*)$': '<rootDir>/src/$1',
    '\\.(css|less|scss)$': 'identity-obj-proxy'
  },
  collectCoverageFrom: [
    'src/**/*.{ts,tsx}',
    '!src/**/*.d.ts'
  ],
  coverageThreshold: {
    global: {
      branches: 75,
      functions: 80,
      lines: 80
    }
  }
};

Vitest (vitest.config.ts):

import { defineConfig } from 'vitest/config';
import react from '@vitejs/plugin-react';

export default defineConfig({
  plugins: [react()],
  test: {
    environment: 'jsdom',
    setupFiles: ['./vitest.setup.ts'],
    globals: true,
    coverage: {
      provider: 'v8',
      reporter: ['text', 'json', 'html'],
      exclude: ['node_modules/', 'test/']
    }
  },
  resolve: {
    alias: {
      '@': '/src'
    }
  }
});

E2E Testing Frameworks

Comparison Matrix

When to Choose Each

Choose Playwright when:

• Cross-browser testing is critical
• Need WebKit/Safari testing
• API testing alongside E2E
• Multiple language support needed
• Complex scenarios (multiple tabs, auth)

Choose Cypress when:

• Developer experience is priority
• React/Vue/Angular component testing
• Team new to E2E testing
• Single browser is acceptable
• Real-time debugging needed

Choose Selenium when:

• Legacy infrastructure exists
• Need language flexibility
• Complex enterprise requirements
• Grid-based parallel execution

Configuration Examples

Playwright (playwright.config.ts):

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './e2e',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 2 : undefined,
  reporter: [
    ['html'],
    ['junit', { outputFile: 'results.xml' }]
  ],
  use: {
    baseURL: 'http://localhost:3000',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure'
  },
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] }
    },
    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] }
    },
    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] }
    },
    {
      name: 'mobile-chrome',
      use: { ...devices['Pixel 5'] }
    }
  ],
  webServer: {
    command: 'npm run dev',
    url: 'http://localhost:3000',
    reuseExistingServer: !process.env.CI
  }
});

Cypress (cypress.config.ts):

import { defineConfig } from 'cypress';

export default defineConfig({
  e2e: {
    baseUrl: 'http://localhost:3000',
    viewportWidth: 1280,
    viewportHeight: 720,
    video: false,
    screenshotOnRunFailure: true,
    retries: {
      runMode: 2,
      openMode: 0
    },
    setupNodeEvents(on, config) {
      // Task plugins
      on('task', {
        seedDatabase(data) {
          return db.seed(data);
        }
      });
    }
  },
  component: {
    devServer: {
      framework: 'react',
      bundler: 'vite'
    }
  }
});

Python Testing Frameworks

Comparison

pytest Configuration

pyproject.toml:

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py", "*_test.py"]
python_functions = ["test_*"]
addopts = [
    "-v",
    "--strict-markers",
    "--cov=src",
    "--cov-report=term-missing",
    "--cov-report=html",
    "--cov-fail-under=80"
]
markers = [
    "slow: marks tests as slow",
    "integration: marks tests as integration tests",
    "e2e: marks tests as end-to-end tests"
]
filterwarnings = [
    "error",
    "ignore::DeprecationWarning"
]

[tool.coverage.run]
branch = true
source = ["src"]
omit = ["tests/*", "**/__init__.py"]

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "if TYPE_CHECKING:",
    "raise NotImplementedError"
]

conftest.py:

import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

@pytest.fixture(scope="session")
def engine():
    return create_engine("sqlite:///:memory:")

@pytest.fixture(scope="function")
def db_session(engine):
    Session = sessionmaker(bind=engine)
    session = Session()
    yield session
    session.rollback()
    session.close()

@pytest.fixture
def client(app):
    return app.test_client()

@pytest.fixture
def auth_headers(client):
    response = client.post('/login', json={
        'username': 'test',
        'password': 'test123'
    })
    token = response.json['token']
    return {'Authorization': f'Bearer {token}'}

Component Testing

React Testing Library vs Enzyme

React Testing Library (Recommended):

import { render, screen, waitFor } from '@testing-library/react';
import userEvent from '@testing-library/user-event';

test('submits form with user data', async () => {
  const user = userEvent.setup();
  const onSubmit = jest.fn();

  render(<LoginForm onSubmit={onSubmit} />);

  await user.type(screen.getByLabelText(/email/i), 'test@example.com');
  await user.type(screen.getByLabelText(/password/i), 'password123');
  await user.click(screen.getByRole('button', { name: /sign in/i }));

  await waitFor(() => {
    expect(onSubmit).toHaveBeenCalledWith({
      email: 'test@example.com',
      password: 'password123'
    });
  });
});

API Testing

Framework Options

Supertest Example:

import request from 'supertest';
import app from '../src/app';

describe('Users API', () => {
  it('GET /users returns list', async () => {
    const response = await request(app)
      .get('/api/users')
      .set('Authorization', `Bearer ${token}`)
      .expect('Content-Type', /json/)
      .expect(200);

    expect(response.body).toHaveLength(10);
    expect(response.body[0]).toHaveProperty('id');
  });
});

Decision Matrix

Quick Selection Guide

Need to test...
│
├── Unit/Business Logic
│   ├── JavaScript/TypeScript
│   │   ├── Using Vite → Vitest
│   │   ├── Existing Jest → Keep Jest
│   │   └── New project → Vitest
│   └── Python → pytest
│
├── React Components
│   ├── User behavior → React Testing Library
│   └── With stories → Storybook + Chromatic
│
├── API Endpoints
│   ├── Node.js → Supertest
│   ├── With E2E → Playwright API
│   └── Python → pytest + requests
│
└── E2E/Browser
    ├── Cross-browser critical → Playwright
    ├── Developer experience → Cypress
    └── Legacy/Enterprise → Selenium

Test Strategy Framework

A comprehensive guide to building effective test strategies for modern applications.

Test Strategy Document Template

1. Scope Definition

In Scope:

• [ ] Unit tests for business logic
• [ ] Integration tests for API contracts
• [ ] E2E tests for critical user journeys
• [ ] Component tests for UI elements
• [ ] Accessibility testing
• [ ] Visual regression testing

Out of Scope:

• [ ] Performance/load testing (separate strategy)
• [ ] Security testing (separate strategy)
• [ ] Manual exploratory testing

2. Test Pyramid Implementation

Target Distribution:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
E2E:         ████████░░░░░░░░░░░░░░░░░░░░░░  10%
Integration: ████████████████░░░░░░░░░░░░░░  20%
Unit:        ████████████████████████████████ 70%
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

3. Critical Path Identification

Tier 1 - Must Never Break (100% coverage):

• User authentication flow
• Payment/checkout process
• Core data mutations
• Security-critical operations

Tier 2 - Important (90% coverage):

• User profile management
• Search and filtering
• Navigation flows
• Form submissions

Tier 3 - Nice to Have (70% coverage):

• Edge cases in UI
• Administrative features
• Logging and analytics

4. Test Data Strategy

Approaches: 1. Factory Pattern - Generate test data programmatically 2. Fixtures - Static test data files 3. Seeding - Database seeding scripts 4. Mocking - API response mocks

Data Isolation:

// Each test gets fresh data
beforeEach(async () => {
  await db.truncate(['users', 'orders']);
  await seedTestData();
});

5. Environment Strategy

6. Ownership Model

Risk-Based Testing

Risk Assessment Matrix

Impact
  ↑
  │  ┌─────────┬─────────┐
H │  │ MEDIUM  │  HIGH   │  ← Comprehensive testing
  │  │ Priority│ Priority│
  │  ├─────────┼─────────┤
M │  │   LOW   │ MEDIUM  │  ← Standard testing
  │  │ Priority│ Priority│
  │  ├─────────┼─────────┤
L │  │  MINIMAL│   LOW   │  ← Basic coverage
  │  │ Priority│ Priority│
  └──┴─────────┴─────────┴──→
        Low      High    Probability

Coverage by Risk Level

Testing Quadrants

        Business-Facing
              ↑
   Q2 │          │ Q3
Functional    │    Exploratory
  Tests       │    Testing
              │
──────────────┼──────────────→ Manual
              │
   Q1 │          │ Q4
   Unit       │   Performance
   Tests      │   Security
              │
        Technology-Facing
              ↓
         Automated

Q1 (Technology/Automated): Unit tests, component tests Q2 (Business/Automated): Functional tests, API tests Q3 (Business/Manual): Exploratory testing, usability Q4 (Technology/Tools): Performance, security, load

Test Maintenance Strategy

Keeping Tests Healthy

1. Regular Review Cycles

• Weekly: Flaky test triage
• Monthly: Coverage gaps analysis
• Quarterly: Strategy review

2. Test Debt Tracking

   # In test file headers
   // @tech-debt: Needs refactor when API v2 ships
   // @flaky: Intermittent timeout - tracking in JIRA-123
   // @skip-reason: Blocked by feature flag removal

3. Deletion Criteria

• Test for removed feature
• Duplicate coverage
• Permanently flaky with no fix path
• Testing implementation details

Metrics to Track

Test Documentation Standards

Test Naming Convention

// Pattern: should_[expected]_when_[condition]
it('should_return_user_when_id_exists', () => {});
it('should_throw_error_when_id_invalid', () => {});

// Or: describe behavior
describe('UserService', () => {
  describe('getUser', () => {
    it('returns user for valid ID', () => {});
    it('throws NotFoundError for unknown ID', () => {});
  });
});

Test File Organization

src/
├── components/
│   ├── Button/
│   │   ├── Button.tsx
│   │   ├── Button.test.tsx      # Unit tests
│   │   └── Button.stories.tsx   # Visual tests
├── services/
│   ├── user.ts
│   └── user.test.ts
tests/
├── integration/
│   ├── api/
│   │   └── users.test.ts
│   └── setup.ts
├── e2e/
│   ├── checkout.spec.ts
│   └── auth.spec.ts
└── fixtures/
    └── users.json

Continuous Improvement

Retrospective Questions

1. What tests caught real bugs this sprint? 2. What bugs escaped to production - why no test? 3. Which tests are frequently skipped/ignored? 4. What's the test run time trend? 5. Are developers writing tests first (TDD) or after?

Improvement Actions