
Playwright Pro
- 117 installs
- 451 repo stars
- Updated July 21, 2026
- borghei/claude-skills
Author resilient Playwright E2E suites—selectors, fixtures, CI parallelization, trace debugging, and flake reduction—for web apps and extensions before release.
About
Playwright-pro equips Claude to design stable end-to-end browser tests with smart selectors, fixture patterns, parallel CI execution, trace-based debugging, and anti-flake practices for SaaS, extension, and CLI-backed web surfaces.
- E2E test authoring
- Selector strategies
- CI parallel runs
- Trace debugging
- Flake mitigation
Playwright Pro by the numbers
- 117 all-time installs (skills.sh)
- Ranked #957 of 2,153 Testing & QA skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/borghei/claude-skills --skill playwright-proAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 117 |
|---|---|
| repo stars | ★ 451 |
| Last updated | July 21, 2026 |
| Repository | borghei/claude-skills ↗ |
What it does
Author resilient Playwright E2E suites—selectors, fixtures, CI parallelization, trace debugging, and flake reduction—for web apps and extensions before release.
Files
Playwright Pro
Production-grade end-to-end testing with Playwright. Generate tests from user stories, implement the Page Object pattern for maintainability, apply the correct locator strategy for resilient tests, diagnose and fix flaky tests, migrate from Cypress or Selenium, integrate with CI/CD, run visual regression tests, and perform accessibility audits. Enforces the 10 golden rules that eliminate 90% of E2E test failures.
Core Capabilities
- Resilient authoring —
getByRole/getByLabel/getByTextlocator priority, web-first auto-retrying assertions, and the 10 golden rules. - Page Object Model — centralize locators, expose user-intent methods, generate POM classes from HTML/selectors.
- Test generation — produce specs from user stories with happy-path and error-path coverage.
- Flaky-test diagnosis — trace analysis, headed/debug runs, race-condition and shared-state fixes.
- Migration — Cypress/Selenium → Playwright command mapping and auth-setup patterns.
- CI integration — GitHub Actions, sharding, conditional browser installs, trace/screenshot artifacts.
- Visual regression & accessibility —
toHaveScreenshotbaselines and Axe WCAG AA gates.
When to Use
- Writing or organizing E2E tests for web apps.
- Fixing flaky tests or diagnosing CI-only failures.
- Migrating a suite from Cypress or Selenium.
- Adding visual regression or WCAG accessibility gates.
- Setting up Playwright CI integration and reporting.
Sub-Skills
This skill uses compound sub-skill architecture. Each sub-skill in skills/ handles a specific workflow:
| Sub-Skill | File | Purpose |
|---|---|---|
| Init | skills/init.md | Bootstrap Playwright in a project -- install, configure, create first test |
| Generate | skills/generate.md | Generate test files from user stories or page descriptions |
| Fix | skills/fix.md | Diagnose and fix failing or flaky tests using trace analysis |
| Migrate | skills/migrate.md | Migrate from Cypress or Selenium to Playwright |
| Review | skills/review.md | Audit test quality, coverage gaps, and flaky test indicators |
| Report | skills/report.md | Generate execution reports from Playwright JSON output |
| Coverage | skills/coverage.md | Map tests to user stories, identify coverage gaps |
| BrowserStack | skills/browserstack.md | BrowserStack cloud integration for cross-browser testing |
| TestRail | skills/testrail.md | TestRail integration for test case management |
Sub-skill flow: Init → Generate → Review → Fix (if needed); Coverage → Generate (fill gaps); Report → BrowserStack / TestRail.
Tools
| Tool | Purpose | Command |
|---|---|---|
test_generator.py | Generate Playwright test code from user story descriptions | python scripts/test_generator.py --story "..." --page LoginPage --output tests/ |
flaky_detector.py | Analyze multiple CI runs to detect flaky test patterns | python scripts/flaky_detector.py --results-dir results/ --runs 10 --threshold 0.05 |
coverage_mapper.py | Map tests to user flows and identify coverage gaps | python scripts/coverage_mapper.py --tests tests/ --flows flows.json --gaps-only |
page_object_generator.py | Generate Page Object classes from HTML or selector lists | python scripts/page_object_generator.py --html page.html --name LoginPage --route /login |
test_analyzer.py | Scan test files for anti-patterns and quality issues | python scripts/test_analyzer.py tests/ --severity high |
test_report_parser.py | Parse Playwright JSON reports into summaries | python scripts/test_report_parser.py report.json --top-slow 10 |
References
Load the reference that matches the task — keep this file lean and pull detail on demand:
- [references/core-rules-and-patterns.md](references/core-rules-and-patterns.md) — the 10 golden rules, locator priority,
playwright.config.ts, Page Object Model classes, test generation from user stories, and shared-auth setup. Read when authoring tests or configuring a project. - [references/playwright-patterns.md](references/playwright-patterns.md) — locator decision tree, web-first vs non-retrying assertion patterns, custom/worker fixtures, network mocking, anti-pattern quick reference, and CI sharding patterns. Read when choosing locators, mocking APIs, or building fixtures.
- [references/diagnosis-migration-and-ci.md](references/diagnosis-migration-and-ci.md) — flaky-test causes/fixes and diagnosis commands, Cypress→Playwright migration table, GitHub Actions CI, visual regression, accessibility testing, common pitfalls, best practices, troubleshooting table, and the success-criteria bar. Read when fixing flaky tests, migrating, wiring CI, or auditing a suite.
Scope & Limitations
This skill covers:
- End-to-end test authoring, organization, and maintenance with Playwright
- Page Object Model architecture and locator strategy best practices
- Flaky test diagnosis, CI integration, and trace-based debugging
- Visual regression testing and WCAG accessibility auditing
This skill does NOT cover:
- Unit testing or component testing in isolation (see
engineering/testing-strategyfor test pyramid guidance) - API contract testing or load/performance testing (see
api-test-suite-builderfor API-focused testing) - Test data management, database seeding, or factory patterns for test fixtures
- Mobile native app testing (Appium, Detox); this skill targets web browsers only
Integration Points
| Skill | Integration | Data Flow |
|---|---|---|
ci-cd-pipeline-builder | E2E tests run as a pipeline stage after build and unit tests | Pipeline config triggers playwright test; artifacts (traces, screenshots) upload on failure |
api-test-suite-builder | API tests validate backend contracts; Playwright tests validate UI flows end-to-end | API test results confirm endpoint stability before E2E suite runs against the same environment |
pr-review-expert | PR reviews check for test coverage on UI changes and flag missing E2E specs | Review checklist references Playwright Pro golden rules; flags waitForTimeout or raw CSS selectors |
performance-profiler | Performance budgets complement E2E tests to catch regressions | Profiler identifies slow pages; Playwright tests add networkidle waits or performance assertions for flagged routes |
observability-designer | Test failures feed into observability dashboards for flake tracking | CI test results export JSON reports; observability pipelines ingest pass/fail/flake metrics over time |
release-manager | E2E suite is a release gate; green suite required before deployment proceeds | Release workflow calls Playwright CI job; blocks release tag creation on any test failure |
Core Rules & Patterns
Read this when authoring tests: the golden rules, locator priority, base config, Page Object Model, test generation from stories, and shared-auth setup.
10 Golden Rules
These rules are non-negotiable. Following them eliminates 90% of E2E test failures.
1. `getByRole()` over CSS/XPath — resilient to markup changes 2. Never `page.waitForTimeout()` — use web-first assertions instead 3. `expect(locator)` auto-retries; `expect(await locator.textContent())` does NOT 4. Isolate every test — no shared state between tests 5. `baseURL` in config — zero hardcoded URLs in tests 6. Retries: 2 in CI, 0 locally — retries mask flakiness in dev 7. Traces: `'on-first-retry'` — rich debugging without slowdown 8. Fixtures over globals — test.extend() for shared setup 9. One behavior per test — multiple related assertions are fine 10. Mock external services only — never mock your own app
Locator Priority (Most to Least Preferred)
1. getByRole('button', { name: 'Submit' }) — semantic, accessible
2. getByLabel('Email address') — form fields with labels
3. getByText('Welcome back') — visible text content
4. getByPlaceholder('Enter your email') — inputs with placeholder
5. getByTestId('submit-button') — when no semantic option exists
6. page.locator('.submit-btn') — CSS as last resort
7. page.locator('//button[@type="submit"]') — XPath: avoid entirelyWhy This Order Matters
// FRAGILE: breaks when CSS class changes
await page.locator('.btn-primary-lg').click();
// FRAGILE: breaks when DOM structure changes
await page.locator('div > form > button:nth-child(2)').click();
// RESILIENT: survives refactors, tests what users see
await page.getByRole('button', { name: 'Create account' }).click();Configuration
playwright.config.ts
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
testDir: './tests/e2e',
fullyParallel: true,
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 4 : undefined,
reporter: process.env.CI
? [['html'], ['github'], ['json', { outputFile: 'test-results.json' }]]
: [['html']],
use: {
baseURL: process.env.BASE_URL || 'http://localhost:3000',
trace: 'on-first-retry',
screenshot: 'only-on-failure',
video: 'retain-on-failure',
},
projects: [
// Auth setup: runs once, shares state with all tests
{ name: 'setup', testMatch: /.*\.setup\.ts/ },
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
dependencies: ['setup'],
},
{
name: 'firefox',
use: { ...devices['Desktop Firefox'] },
dependencies: ['setup'],
},
{
name: 'mobile-chrome',
use: { ...devices['Pixel 5'] },
dependencies: ['setup'],
},
],
webServer: {
command: 'pnpm dev',
url: 'http://localhost:3000',
reuseExistingServer: !process.env.CI,
timeout: 30000,
},
});Page Object Pattern
// pages/login.page.ts
import { type Page, type Locator, expect } from '@playwright/test';
export class LoginPage {
readonly page: Page;
readonly emailInput: Locator;
readonly passwordInput: Locator;
readonly submitButton: Locator;
readonly errorMessage: Locator;
readonly forgotPasswordLink: Locator;
constructor(page: Page) {
this.page = page;
this.emailInput = page.getByLabel('Email address');
this.passwordInput = page.getByLabel('Password');
this.submitButton = page.getByRole('button', { name: 'Sign in' });
this.errorMessage = page.getByRole('alert');
this.forgotPasswordLink = page.getByRole('link', { name: 'Forgot password?' });
}
async goto() {
await this.page.goto('/login');
}
async login(email: string, password: string) {
await this.emailInput.fill(email);
await this.passwordInput.fill(password);
await this.submitButton.click();
}
async expectError(message: string) {
await expect(this.errorMessage).toContainText(message);
}
async expectRedirectToDashboard() {
await expect(this.page).toHaveURL(/\/dashboard/);
}
}// pages/dashboard.page.ts
import { type Page, type Locator, expect } from '@playwright/test';
export class DashboardPage {
readonly page: Page;
readonly heading: Locator;
readonly projectList: Locator;
readonly createProjectButton: Locator;
constructor(page: Page) {
this.page = page;
this.heading = page.getByRole('heading', { name: 'Dashboard' });
this.projectList = page.getByRole('list', { name: 'Projects' });
this.createProjectButton = page.getByRole('button', { name: 'New project' });
}
async expectLoaded() {
await expect(this.heading).toBeVisible();
}
async getProjectCount() {
return this.projectList.getByRole('listitem').count();
}
async createProject(name: string) {
await this.createProjectButton.click();
await this.page.getByLabel('Project name').fill(name);
await this.page.getByRole('button', { name: 'Create' }).click();
}
}Test Generation from User Stories
Given a user story, generate tests following this pattern:
// tests/e2e/auth/login.spec.ts
import { test, expect } from '@playwright/test';
import { LoginPage } from '../../pages/login.page';
import { DashboardPage } from '../../pages/dashboard.page';
test.describe('User Login', () => {
let loginPage: LoginPage;
test.beforeEach(async ({ page }) => {
loginPage = new LoginPage(page);
await loginPage.goto();
});
test('successful login redirects to dashboard', async ({ page }) => {
await loginPage.login('user@example.com', 'password123');
const dashboard = new DashboardPage(page);
await dashboard.expectLoaded();
});
test('shows error for invalid credentials', async () => {
await loginPage.login('user@example.com', 'wrongpassword');
await loginPage.expectError('Invalid email or password');
});
test('shows validation error for empty email', async () => {
await loginPage.login('', 'password123');
await loginPage.expectError('Email is required');
});
test('shows validation error for invalid email format', async () => {
await loginPage.login('not-an-email', 'password123');
await loginPage.expectError('Enter a valid email');
});
test('forgot password link navigates to reset page', async ({ page }) => {
await loginPage.forgotPasswordLink.click();
await expect(page).toHaveURL(/\/forgot-password/);
});
});Authentication Setup (Shared State)
// tests/e2e/auth.setup.ts
import { test as setup, expect } from '@playwright/test';
const authFile = 'playwright/.auth/user.json';
setup('authenticate as test user', async ({ page }) => {
await page.goto('/login');
await page.getByLabel('Email address').fill('test@example.com');
await page.getByLabel('Password').fill('password123');
await page.getByRole('button', { name: 'Sign in' }).click();
// Wait for redirect to confirm login succeeded
await expect(page).toHaveURL(/\/dashboard/);
// Save authentication state for reuse
await page.context().storageState({ path: authFile });
});Diagnosis, Migration, CI & Quality
Read this when diagnosing flaky tests, migrating from Cypress, wiring CI, adding visual regression or accessibility checks, or auditing a suite against the quality bar.
Flaky Test Diagnosis
Common Causes and Fixes
| Symptom | Cause | Fix |
|---|---|---|
waitForTimeout(2000) in test | Timing-dependent | Replace with await expect(locator).toBeVisible() |
| Test passes locally, fails in CI | Race condition | Add web-first assertion before interaction |
| Element not found after navigation | Page not loaded | await page.waitForURL('/expected-path') |
| Stale element reference | DOM re-rendered | Use Playwright locators (auto-retry) |
| Different data between runs | Shared test state | Isolate with test.beforeEach setup |
| Flaky on slow CI runners | Insufficient timeout | Increase expect timeout, not waitForTimeout |
Diagnosis Commands
# Run with trace on every test (not just retries)
npx playwright test --trace on
# Run a specific flaky test 10 times
for i in $(seq 1 10); do npx playwright test tests/e2e/checkout.spec.ts; done
# Show test timeline
npx playwright test --trace on
npx playwright show-trace test-results/*/trace.zip
# Run in headed mode for visual debugging
npx playwright test --headed --retries 0
# Debug a specific test interactively
npx playwright test --debug tests/e2e/checkout.spec.tsCypress to Playwright Migration
| Cypress | Playwright |
|---|---|
cy.visit('/path') | await page.goto('/path') |
cy.get('.selector') | page.locator('.selector') |
cy.contains('text') | page.getByText('text') |
cy.get('[data-testid="x"]') | page.getByTestId('x') |
cy.intercept('GET', '/api/*') | await page.route('/api/*', ...) |
cy.wait('@alias') | await page.waitForResponse('/api/*') |
cy.should('be.visible') | await expect(locator).toBeVisible() |
cy.should('have.text', 'x') | await expect(locator).toHaveText('x') |
cy.fixture('data.json') | JSON.parse(fs.readFileSync(...)) |
beforeEach(() => { cy.login() }) | Auth setup project + storageState |
CI Integration
GitHub Actions
name: E2E Tests
on: [push, pull_request]
jobs:
e2e:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: 'pnpm'
- run: pnpm install --frozen-lockfile
- run: pnpm exec playwright install --with-deps chromium
- run: pnpm build # build the app first
- run: pnpm exec playwright test
env:
BASE_URL: http://localhost:3000
- uses: actions/upload-artifact@v4
if: failure()
with:
name: playwright-report
path: playwright-report/
retention-days: 7Visual Regression Testing
// tests/e2e/visual/dashboard.spec.ts
import { test, expect } from '@playwright/test';
test('dashboard matches visual snapshot', async ({ page }) => {
await page.goto('/dashboard');
await page.waitForLoadState('networkidle');
// Full page screenshot comparison
await expect(page).toHaveScreenshot('dashboard.png', {
maxDiffPixels: 50, // allow small rendering differences
});
});
test('project card component snapshot', async ({ page }) => {
await page.goto('/dashboard');
const card = page.getByTestId('project-card').first();
await expect(card).toHaveScreenshot('project-card.png', {
maxDiffPixelRatio: 0.01,
});
});# Generate/update baseline screenshots
npx playwright test --update-snapshots
# Run visual comparison
npx playwright test tests/e2e/visual/Accessibility Testing
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
test('login page has no accessibility violations', async ({ page }) => {
await page.goto('/login');
const results = await new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
.analyze();
expect(results.violations).toEqual([]);
});
test('dashboard meets WCAG AA standards', async ({ page }) => {
await page.goto('/dashboard');
const results = await new AxeBuilder({ page })
.exclude('.third-party-widget') // exclude elements you don't control
.analyze();
// Log violations for debugging
for (const violation of results.violations) {
console.log(`${violation.impact}: ${violation.description}`);
for (const node of violation.nodes) {
console.log(` - ${node.html}`);
}
}
expect(results.violations.filter(v => v.impact === 'critical')).toEqual([]);
});Common Pitfalls
- `page.waitForTimeout(N)` — the single most common cause of flaky tests; use web-first assertions
- CSS selectors as primary strategy — breaks on every refactor; use role/label/text locators
- Shared state between tests — one test's data pollutes another; isolate with proper setup/teardown
- No trace configuration — debugging CI failures without traces wastes hours; enable
on-first-retry - Testing third-party services — mock external APIs; only test your own application
- Running all browsers in development — test Chromium locally, full matrix in CI only
- No page objects — duplicate locators across tests create maintenance nightmares
Best Practices
1. Page Object per page/component — centralize locators, expose user-intent methods 2. Web-first assertions everywhere — expect(locator).toBeVisible() auto-retries, waitForTimeout does not 3. Auth as a setup project — authenticate once, reuse storageState across all tests 4. One behavior per test — keeps failures isolated and test names meaningful 5. Run in CI with `--retries 2` — but investigate any test that needs retries locally 6. Trace + screenshot on failure — upload as CI artifacts for post-mortem debugging 7. Visual regression for critical UI — catch unintended visual changes automatically 8. Accessibility tests in the suite — WCAG compliance as a regression gate, not an afterthought
Troubleshooting
| Problem | Cause | Solution |
|---|---|---|
locator.click() times out | Element hidden behind overlay/modal or not yet rendered | Add await expect(locator).toBeVisible() before clicking; check for z-index overlays blocking the target |
| Visual snapshots fail on every CI run | Different OS font rendering between local and CI | Generate baseline screenshots inside CI (Linux), not locally (macOS); use maxDiffPixelRatio: 0.02 for tolerance |
| Auth setup project runs but tests get 401 | storageState path mismatch or expired token | Verify storageState path in playwright.config.ts matches the setup script; confirm tokens don't expire mid-suite |
page.route() intercept never triggers | Route pattern doesn't match the actual request URL or glob syntax error | Log requests with page.on('request', r => console.log(r.url())) to verify the exact URL; use ** glob for subpaths |
| Tests pass individually but fail when run together | Shared mutable state or port collision between parallel workers | Ensure test isolation via test.beforeEach setup; use unique test data per worker with test.info().parallelIndex |
toHaveScreenshot() throws "missing baseline" | Snapshot file not committed to version control | Run npx playwright test --update-snapshots and commit the generated files under the __screenshots__ directory |
| Accessibility audit returns false positives | Axe scanning third-party embedded widgets or iframes | Use .exclude('.third-party-widget') or .include('#app-root') to scope the audit to your own markup |
Success Criteria
- Flaky test rate below 2% — measured over a rolling 7-day window across all CI runs; any test exceeding 5% flake rate is quarantined and fixed within 48 hours
- E2E suite completes within 10 minutes — full cross-browser matrix (Chromium, Firefox, mobile) on CI with 4 parallel workers; alert if wall-clock time exceeds threshold
- 100% of critical user journeys covered — login, checkout, onboarding, and core CRUD workflows each have dedicated spec files with happy-path and primary error-path tests
- Zero `waitForTimeout()` calls in the codebase — enforced via ESLint rule or grep-based CI check; all waits use web-first assertions
- Page Object coverage for every tested page — no raw locators in spec files; all element access goes through page object classes with user-intent method names
- WCAG AA compliance gate passing — accessibility tests run on every PR; zero critical or serious violations allowed to merge
- Visual regression baselines reviewed on every UI PR — screenshot diffs attached to PR as artifacts; baseline updates require explicit reviewer approval
Playwright Patterns Reference
Locator Strategy Decision Tree
Need to find an element?
├── Does it have a semantic role? (button, link, heading, textbox)
│ └── YES → getByRole('role', { name: 'visible text' })
├── Is it a labeled form field?
│ └── YES → getByLabel('Label text')
├── Does it have visible text content?
│ └── YES → getByText('Text content')
├── Does it have a placeholder?
│ └── YES → getByPlaceholder('Placeholder text')
├── Does it have a data-testid?
│ └── YES → getByTestId('test-id')
└── Last resort
└── page.locator('css-selector')Assertion Patterns
Web-First (Auto-Retrying) Assertions
These assertions automatically retry until the condition is met or the timeout expires:
await expect(locator).toBeVisible();
await expect(locator).toBeHidden();
await expect(locator).toBeEnabled();
await expect(locator).toBeDisabled();
await expect(locator).toHaveText('exact text');
await expect(locator).toContainText('partial text');
await expect(locator).toHaveValue('input value');
await expect(locator).toHaveAttribute('name', 'value');
await expect(locator).toHaveCount(5);
await expect(locator).toHaveClass(/active/);
await expect(page).toHaveURL(/\/dashboard/);
await expect(page).toHaveTitle('Page Title');Non-Retrying (Avoid in Tests)
These evaluate once and do NOT retry:
// BAD: Does not auto-retry
expect(await locator.textContent()).toBe('text');
expect(await locator.isVisible()).toBe(true);
expect(await page.title()).toBe('Title');Fixture Patterns
Custom Test Fixture
import { test as base, expect } from '@playwright/test';
import { LoginPage } from '../pages/login.page';
type Fixtures = {
loginPage: LoginPage;
authenticatedPage: Page;
};
export const test = base.extend<Fixtures>({
loginPage: async ({ page }, use) => {
const loginPage = new LoginPage(page);
await loginPage.goto();
await use(loginPage);
},
authenticatedPage: async ({ browser }, use) => {
const context = await browser.newContext({
storageState: 'playwright/.auth/user.json',
});
const page = await context.newPage();
await use(page);
await context.close();
},
});Worker-Scoped Fixture (Shared Across Tests)
export const test = base.extend<{}, { dbConnection: Connection }>({
dbConnection: [async ({}, use) => {
const conn = await createConnection();
await use(conn);
await conn.close();
}, { scope: 'worker' }],
});Network Mocking Patterns
Mock API Response
await page.route('/api/users', async (route) => {
await route.fulfill({
status: 200,
contentType: 'application/json',
body: JSON.stringify([{ id: 1, name: 'Test User' }]),
});
});Intercept and Modify Response
await page.route('/api/config', async (route) => {
const response = await route.fetch();
const json = await response.json();
json.featureFlag = true;
await route.fulfill({ response, body: JSON.stringify(json) });
});Wait for API Call
const responsePromise = page.waitForResponse('/api/submit');
await page.getByRole('button', { name: 'Submit' }).click();
const response = await responsePromise;
expect(response.status()).toBe(200);Anti-Pattern Quick Reference
| Anti-Pattern | Fix |
|---|---|
page.waitForTimeout(N) | await expect(locator).toBeVisible() |
expect(await el.textContent()) | await expect(el).toHaveText() |
page.locator('.class-name') | page.getByRole('button', { name: '...' }) |
{ force: true } | Fix underlying visibility issue |
Global let page variable | Use Playwright fixtures |
page.goto('http://localhost:3000/path') | page.goto('/path') with baseURL |
for loop running test N times | Use test.describe.configure({ retries: N }) |
| Shared mutable state between tests | test.beforeEach setup or fixtures |
CI Configuration Patterns
Sharding (Split Tests Across Workers)
strategy:
matrix:
shard: [1/4, 2/4, 3/4, 4/4]
steps:
- run: npx playwright test --shard=${{ matrix.shard }}Merge Shard Reports
npx playwright merge-reports --reporter html ./all-blob-reportsConditional Browser Installation
# Install only what you need
npx playwright install --with-deps chromium # CI: fast
npx playwright install --with-deps # Full: all browsers#!/usr/bin/env python3
"""Map Playwright tests to user stories and identify coverage gaps.
Reads a user flows definition file and scans Playwright test files to
determine which flows are covered by tests and which have gaps. Produces
a coverage matrix with gap analysis and recommendations.
Usage:
python coverage_mapper.py --tests ./tests/e2e/ --flows flows.json
python coverage_mapper.py --tests ./tests/e2e/ --flows flows.json --json
python coverage_mapper.py --tests ./tests/e2e/ --flows flows.json --gaps-only
Expected flows.json format:
[
{
"id": "AUTH-001",
"name": "User login with email",
"priority": "critical",
"keywords": ["login", "sign in", "email", "password", "authenticate"],
"pages": ["/login", "/dashboard"],
"steps": ["Navigate to login", "Enter email", "Enter password", "Click sign in", "See dashboard"]
}
]
"""
import argparse
import json
import os
import re
import sys
from collections import defaultdict
from pathlib import Path
def load_flows(path):
"""Load user flow definitions from JSON file."""
with open(path, "r") as f:
flows = json.load(f)
for flow in flows:
flow.setdefault("priority", "medium")
flow.setdefault("keywords", [])
flow.setdefault("pages", [])
flow.setdefault("steps", [])
return flows
def scan_test_files(test_dir):
"""Scan Playwright test files and extract test metadata."""
test_dir = Path(test_dir)
test_files = []
patterns = ["**/*.spec.ts", "**/*.test.ts", "**/*.spec.js", "**/*.test.js"]
found_files = set()
for pattern in patterns:
found_files.update(test_dir.glob(pattern))
for tf in sorted(found_files):
try:
content = tf.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
# Extract test names
test_names = re.findall(r"test\s*\(\s*['\"](.+?)['\"]", content)
# Extract describe block names
describe_names = re.findall(r"test\.describe\s*\(\s*['\"](.+?)['\"]", content)
# Extract URLs navigated to
urls = re.findall(r"\.goto\s*\(\s*['\"](.+?)['\"]", content)
urls += re.findall(r"toHaveURL\s*\(\s*['\"/](.+?)['\"/]", content)
# Extract Page Object imports
page_objects = re.findall(r"import\s*\{.*?(\w+Page)\w*.*?\}\s*from", content)
# Extract all significant words for keyword matching
words = set(re.findall(r"[a-zA-Z]{3,}", content.lower()))
test_files.append({
"path": str(tf.relative_to(test_dir) if tf.is_relative_to(test_dir) else tf),
"test_names": test_names,
"describe_names": describe_names,
"urls": urls,
"page_objects": page_objects,
"word_set": words,
"test_count": len(test_names),
})
return test_files
def compute_match_score(flow, test_file):
"""Compute a match score between a flow and a test file.
Returns a score from 0.0 to 1.0 indicating how likely the test
covers the flow.
"""
score = 0.0
max_score = 0.0
# Keyword matching (weight: 0.4)
max_score += 0.4
if flow["keywords"]:
keyword_matches = sum(
1 for kw in flow["keywords"]
if kw.lower() in test_file["word_set"]
)
keyword_ratio = keyword_matches / len(flow["keywords"])
score += 0.4 * keyword_ratio
# URL matching (weight: 0.3)
max_score += 0.3
if flow["pages"]:
url_matches = 0
for flow_url in flow["pages"]:
flow_url_clean = flow_url.strip("/").lower()
for test_url in test_file["urls"]:
test_url_clean = test_url.strip("/").lower()
if flow_url_clean in test_url_clean or test_url_clean in flow_url_clean:
url_matches += 1
break
url_ratio = url_matches / len(flow["pages"])
score += 0.3 * url_ratio
# Test name matching (weight: 0.2)
max_score += 0.2
flow_name_words = set(re.findall(r"[a-zA-Z]{3,}", flow["name"].lower()))
all_test_text = " ".join(test_file["test_names"] + test_file["describe_names"]).lower()
test_text_words = set(re.findall(r"[a-zA-Z]{3,}", all_test_text))
if flow_name_words:
name_overlap = len(flow_name_words & test_text_words) / len(flow_name_words)
score += 0.2 * name_overlap
# Flow ID in test name (weight: 0.1)
max_score += 0.1
flow_id = flow["id"].lower()
if flow_id in all_test_text.lower():
score += 0.1
return round(score / max_score if max_score > 0 else 0, 3)
def map_coverage(flows, test_files, threshold=0.3):
"""Map flows to tests and identify gaps."""
coverage = []
for flow in flows:
matches = []
for tf in test_files:
score = compute_match_score(flow, tf)
if score >= threshold:
matches.append({
"file": tf["path"],
"score": score,
"test_count": tf["test_count"],
"test_names": tf["test_names"][:5], # Limit for readability
})
matches.sort(key=lambda m: -m["score"])
status = "covered" if matches else "gap"
best_score = matches[0]["score"] if matches else 0.0
coverage.append({
"flow_id": flow["id"],
"flow_name": flow["name"],
"priority": flow["priority"],
"status": status,
"best_match_score": best_score,
"matching_tests": matches[:3], # Top 3 matches
"total_matching_files": len(matches),
})
return coverage
def compute_summary(coverage):
"""Compute summary statistics from coverage results."""
total = len(coverage)
covered = sum(1 for c in coverage if c["status"] == "covered")
gaps = total - covered
by_priority = defaultdict(lambda: {"total": 0, "covered": 0, "gaps": 0})
for c in coverage:
p = c["priority"]
by_priority[p]["total"] += 1
if c["status"] == "covered":
by_priority[p]["covered"] += 1
else:
by_priority[p]["gaps"] += 1
return {
"total_flows": total,
"covered": covered,
"gaps": gaps,
"coverage_pct": round(covered / total * 100, 1) if total else 0,
"by_priority": dict(by_priority),
}
def format_human(coverage, summary, test_file_count):
"""Format results for human-readable output."""
output = []
output.append("=" * 70)
output.append("TEST COVERAGE MAPPER")
output.append("=" * 70)
output.append("")
output.append(f" User flows: {summary['total_flows']}")
output.append(f" Test files: {test_file_count}")
output.append(f" Covered: {summary['covered']} ({summary['coverage_pct']}%)")
output.append(f" Gaps: {summary['gaps']}")
output.append("")
# By priority
priority_order = ["critical", "high", "medium", "low"]
output.append("COVERAGE BY PRIORITY")
output.append("-" * 70)
for p in priority_order:
stats = summary["by_priority"].get(p)
if stats:
pct = round(stats["covered"] / stats["total"] * 100) if stats["total"] else 0
bar_filled = int(pct / 5)
bar = "#" * bar_filled + "." * (20 - bar_filled)
output.append(f" {p.upper():<10} [{bar}] {pct:>3}% ({stats['covered']}/{stats['total']})")
output.append("")
# Coverage matrix
output.append("COVERAGE MATRIX")
output.append("-" * 70)
output.append(f" {'ID':<12} {'Priority':<10} {'Status':<10} {'Score':>6} Flow Name")
output.append(f" {'─' * 12} {'─' * 10} {'─' * 10} {'─' * 6} {'─' * 25}")
for c in coverage:
status_mark = "COVERED" if c["status"] == "covered" else "GAP"
output.append(
f" {c['flow_id']:<12} {c['priority']:<10} {status_mark:<10} "
f"{c['best_match_score']:>5.0%} {c['flow_name'][:35]}"
)
if c["matching_tests"]:
for mt in c["matching_tests"][:2]:
output.append(f" {'':>42} -> {mt['file']} ({mt['test_count']} tests)")
output.append("")
# Gaps detail
gaps = [c for c in coverage if c["status"] == "gap"]
if gaps:
output.append("COVERAGE GAPS (need tests)")
output.append("-" * 70)
for g in sorted(gaps, key=lambda x: {"critical": 0, "high": 1, "medium": 2, "low": 3}.get(x["priority"], 9)):
output.append(f" [{g['priority'].upper()}] {g['flow_id']}: {g['flow_name']}")
output.append("")
# Verdict
output.append("=" * 70)
critical_gaps = [g for g in gaps if g["priority"] == "critical"]
if critical_gaps:
output.append(f"VERDICT: CRITICAL GAPS - {len(critical_gaps)} critical flows have no test coverage")
elif gaps:
output.append(f"VERDICT: GAPS EXIST - {len(gaps)} flows need tests ({summary['coverage_pct']}% covered)")
else:
output.append("VERDICT: FULL COVERAGE - All defined user flows have test coverage")
output.append("=" * 70)
return "\n".join(output)
def main():
parser = argparse.ArgumentParser(
description="Map Playwright tests to user stories and identify coverage gaps.",
epilog="Example: python coverage_mapper.py --tests ./tests/e2e/ --flows flows.json",
)
parser.add_argument("--tests", required=True, help="Path to Playwright test directory")
parser.add_argument("--flows", required=True, help="Path to user flows JSON definition file")
parser.add_argument("--threshold", type=float, default=0.3, help="Match score threshold (default: 0.3)")
parser.add_argument("--gaps-only", action="store_true", help="Only show flows with coverage gaps")
parser.add_argument("--json", action="store_true", dest="json_output", help="Output as JSON")
args = parser.parse_args()
if not os.path.isdir(args.tests):
print(f"Error: Test directory '{args.tests}' not found.", file=sys.stderr)
sys.exit(1)
if not os.path.isfile(args.flows):
print(f"Error: Flows file '{args.flows}' not found.", file=sys.stderr)
sys.exit(1)
flows = load_flows(args.flows)
test_files = scan_test_files(args.tests)
coverage = map_coverage(flows, test_files, args.threshold)
if args.gaps_only:
coverage = [c for c in coverage if c["status"] == "gap"]
summary = compute_summary(coverage)
if args.json_output:
print(json.dumps({"summary": summary, "coverage": coverage}, indent=2))
else:
print(format_human(coverage, summary, len(test_files)))
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""Detect flaky Playwright tests by analyzing multiple test result files.
Scans a directory of Playwright JSON result files from multiple CI runs and
identifies tests that pass intermittently. Produces a flakiness report with
stability scores and recommended actions.
Usage:
python flaky_detector.py --results-dir ./test-results/ --runs 10
python flaky_detector.py --results-dir ./test-results/ --runs 10 --threshold 0.9
python flaky_detector.py --results-dir ./test-results/ --runs 10 --json
"""
import argparse
import json
import os
import sys
from collections import defaultdict
from pathlib import Path
def load_result_files(results_dir, max_runs):
"""Load Playwright JSON result files from a directory.
Expects files named like: results-001.json, test-results-20260315.json, etc.
Sorts by modification time to get the most recent runs.
"""
result_dir = Path(results_dir)
if not result_dir.is_dir():
print(f"Error: '{results_dir}' is not a directory.", file=sys.stderr)
sys.exit(1)
json_files = sorted(
result_dir.glob("*.json"),
key=lambda f: f.stat().st_mtime,
reverse=True,
)[:max_runs]
runs = []
for jf in json_files:
try:
with open(jf, "r") as f:
data = json.load(f)
runs.append({"file": str(jf), "data": data})
except (json.JSONDecodeError, OSError) as e:
print(f"Warning: Skipping {jf}: {e}", file=sys.stderr)
return runs
def extract_test_outcomes(report):
"""Extract test outcomes from a Playwright JSON report."""
outcomes = {}
def walk_suites(suites, parent=""):
for suite in suites:
title = suite.get("title", "")
path = f"{parent} > {title}" if parent else title
for spec in suite.get("specs", []):
spec_title = spec.get("title", "")
full_title = f"{path} > {spec_title}".strip(" > ")
for test in spec.get("tests", []):
project = test.get("projectName", "default")
test_key = f"[{project}] {full_title}"
status = test.get("status", "unknown")
results = test.get("results", [])
attempt_count = len(results)
# Determine effective status
if status == "flaky":
effective = "flaky"
elif status in ("expected", "passed"):
# Check if it needed retries
attempt_statuses = [r.get("status", "") for r in results]
if attempt_count > 1 and any(s in ("failed", "timedOut") for s in attempt_statuses[:-1]):
effective = "flaky"
else:
effective = "passed"
elif status in ("unexpected", "failed"):
effective = "failed"
elif status == "skipped":
effective = "skipped"
else:
effective = status
duration = sum(r.get("duration", 0) for r in results)
outcomes[test_key] = {
"status": effective,
"attempts": attempt_count,
"duration_ms": duration,
}
if "suites" in suite:
walk_suites(suite["suites"], path)
if "suites" in report:
walk_suites(report["suites"])
return outcomes
def compute_flakiness(runs):
"""Compute flakiness scores across multiple runs."""
test_history = defaultdict(lambda: {
"pass_count": 0,
"fail_count": 0,
"flaky_count": 0,
"skip_count": 0,
"total_runs": 0,
"durations": [],
})
for run in runs:
outcomes = extract_test_outcomes(run["data"])
for test_key, outcome in outcomes.items():
history = test_history[test_key]
history["total_runs"] += 1
if outcome["status"] == "passed":
history["pass_count"] += 1
elif outcome["status"] == "failed":
history["fail_count"] += 1
elif outcome["status"] == "flaky":
history["flaky_count"] += 1
elif outcome["status"] == "skipped":
history["skip_count"] += 1
if outcome["duration_ms"] > 0:
history["durations"].append(outcome["duration_ms"])
# Compute stability scores
results = []
for test_key, history in test_history.items():
non_skipped = history["total_runs"] - history["skip_count"]
if non_skipped == 0:
stability = 1.0
else:
# Stability = consistent pass or consistent fail (not mixed)
pass_rate = history["pass_count"] / non_skipped
# A test is stable if it always passes or always fails
stability = max(pass_rate, 1.0 - pass_rate - (history["flaky_count"] / non_skipped))
stability = max(0.0, min(1.0, stability))
durations = history["durations"]
duration_variance = 0.0
if len(durations) >= 2:
mean_dur = sum(durations) / len(durations)
if mean_dur > 0:
duration_variance = (
sum((d - mean_dur) ** 2 for d in durations) / len(durations)
) ** 0.5 / mean_dur # coefficient of variation
is_flaky = (
history["flaky_count"] > 0
or (history["pass_count"] > 0 and history["fail_count"] > 0)
)
results.append({
"test": test_key,
"total_runs": history["total_runs"],
"pass_count": history["pass_count"],
"fail_count": history["fail_count"],
"flaky_count": history["flaky_count"],
"skip_count": history["skip_count"],
"stability_score": round(stability, 3),
"duration_cv": round(duration_variance, 3),
"avg_duration_ms": round(sum(durations) / len(durations)) if durations else 0,
"is_flaky": is_flaky,
})
results.sort(key=lambda r: r["stability_score"])
return results
def classify_action(test_result, threshold):
"""Classify recommended action for a test."""
if test_result["stability_score"] >= threshold:
return "OK"
elif test_result["stability_score"] >= 0.5:
return "INVESTIGATE"
elif test_result["fail_count"] > test_result["pass_count"]:
return "QUARANTINE"
else:
return "FIX_URGENTLY"
def format_human(test_results, threshold, runs_analyzed):
"""Format results for human-readable output."""
output = []
output.append("=" * 70)
output.append("FLAKY TEST DETECTOR")
output.append("=" * 70)
output.append("")
output.append(f" Runs analyzed: {runs_analyzed}")
output.append(f" Tests tracked: {len(test_results)}")
output.append(f" Stability threshold: {threshold:.0%}")
flaky_tests = [t for t in test_results if t["is_flaky"]]
stable_tests = [t for t in test_results if not t["is_flaky"]]
output.append(f" Flaky tests: {len(flaky_tests)}")
output.append(f" Stable tests: {len(stable_tests)}")
if flaky_tests:
flaky_rate = len(flaky_tests) / len(test_results) * 100 if test_results else 0
output.append(f" Flaky rate: {flaky_rate:.1f}%")
output.append("")
if flaky_tests:
output.append("FLAKY TESTS (sorted by stability, lowest first)")
output.append("-" * 70)
output.append(f" {'Stability':>10} {'P':>4} {'F':>4} {'Fl':>4} {'Action':<15} Test")
output.append(f" {'─' * 10} {'─' * 4} {'─' * 4} {'─' * 4} {'─' * 15} {'─' * 30}")
for t in flaky_tests:
action = classify_action(t, threshold)
output.append(
f" {t['stability_score']:>10.1%} "
f"{t['pass_count']:>4} {t['fail_count']:>4} {t['flaky_count']:>4} "
f"{action:<15} {t['test'][:50]}"
)
output.append("")
# Duration variance warning
high_variance = [t for t in flaky_tests if t["duration_cv"] > 0.5]
if high_variance:
output.append("HIGH DURATION VARIANCE (may indicate timing issues)")
output.append("-" * 70)
for t in sorted(high_variance, key=lambda x: -x["duration_cv"]):
output.append(
f" CV={t['duration_cv']:.2f} "
f"avg={t['avg_duration_ms']}ms "
f"{t['test'][:50]}"
)
output.append("")
# Verdict
output.append("=" * 70)
if not flaky_tests:
output.append("VERDICT: STABLE - No flaky tests detected")
elif any(classify_action(t, threshold) == "QUARANTINE" for t in flaky_tests):
output.append("VERDICT: CRITICAL - Tests need quarantine")
elif any(classify_action(t, threshold) == "FIX_URGENTLY" for t in flaky_tests):
output.append("VERDICT: URGENT - Flaky tests need immediate attention")
else:
output.append("VERDICT: INVESTIGATE - Some tests show intermittent behavior")
output.append("=" * 70)
return "\n".join(output)
def main():
parser = argparse.ArgumentParser(
description="Detect flaky Playwright tests by analyzing multiple test result files.",
epilog="Example: python flaky_detector.py --results-dir ./test-results/ --runs 10",
)
parser.add_argument("--results-dir", required=True, help="Directory containing Playwright JSON result files")
parser.add_argument("--runs", type=int, default=10, help="Number of recent runs to analyze (default: 10)")
parser.add_argument("--threshold", type=float, default=0.95, help="Stability threshold (default: 0.95)")
parser.add_argument("--json", action="store_true", dest="json_output", help="Output as JSON")
args = parser.parse_args()
runs = load_result_files(args.results_dir, args.runs)
if not runs:
print(f"Error: No JSON result files found in '{args.results_dir}'.", file=sys.stderr)
sys.exit(1)
test_results = compute_flakiness(runs)
if args.json_output:
result = {
"runs_analyzed": len(runs),
"total_tests": len(test_results),
"flaky_count": sum(1 for t in test_results if t["is_flaky"]),
"stability_threshold": args.threshold,
"flaky_tests": [t for t in test_results if t["is_flaky"]],
"all_tests": test_results,
}
print(json.dumps(result, indent=2))
else:
print(format_human(test_results, args.threshold, len(runs)))
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""Generate Playwright Page Object Model classes from HTML or selector lists.
Parses HTML files to extract interactive elements (inputs, buttons, links,
headings) and generates TypeScript Page Object classes following Playwright
best practices: semantic locators, user-intent methods, and proper typing.
Usage:
python page_object_generator.py --html page.html --name LoginPage
python page_object_generator.py --selectors selectors.txt --name DashboardPage
python page_object_generator.py --html page.html --name LoginPage --json
"""
import argparse
import json
import os
import re
import sys
from html.parser import HTMLParser
from pathlib import Path
class ElementExtractor(HTMLParser):
"""Extract interactive elements from HTML for Page Object generation."""
INTERACTIVE_TAGS = {"input", "button", "a", "select", "textarea", "form"}
LANDMARK_TAGS = {"h1", "h2", "h3", "h4", "h5", "h6", "nav", "main", "header", "footer"}
def __init__(self):
super().__init__()
self.elements = []
self._current_tag = None
self._current_attrs = {}
self._current_text = ""
self._capture_text = False
def handle_starttag(self, tag, attrs):
attr_dict = dict(attrs)
tag_lower = tag.lower()
if tag_lower in self.INTERACTIVE_TAGS or tag_lower in self.LANDMARK_TAGS:
self._current_tag = tag_lower
self._current_attrs = attr_dict
self._current_text = ""
self._capture_text = True
if tag_lower == "input" or (tag_lower == "img"):
# Self-closing: record immediately
if tag_lower == "input":
self._record_element(tag_lower, attr_dict, "")
def handle_data(self, data):
if self._capture_text:
self._current_text += data.strip()
def handle_endtag(self, tag):
tag_lower = tag.lower()
if tag_lower == self._current_tag and self._capture_text:
self._record_element(tag_lower, self._current_attrs, self._current_text)
self._capture_text = False
self._current_tag = None
def _record_element(self, tag, attrs, text):
element = {
"tag": tag,
"text": text,
"id": attrs.get("id", ""),
"name": attrs.get("name", ""),
"type": attrs.get("type", ""),
"role": attrs.get("role", ""),
"aria_label": attrs.get("aria-label", ""),
"placeholder": attrs.get("placeholder", ""),
"data_testid": attrs.get("data-testid", ""),
"href": attrs.get("href", ""),
"class": attrs.get("class", ""),
"for": attrs.get("for", ""),
"label_text": attrs.get("aria-label", "") or attrs.get("placeholder", ""),
}
self.elements.append(element)
def parse_selector_file(filepath):
"""Parse a text file of selectors (one per line) into elements.
Format per line: locator_type:value:property_name
Examples:
role:button:Submit:submitButton
label:Email address:emailInput
testid:nav-menu:navMenu
text:Welcome back:welcomeText
"""
elements = []
content = Path(filepath).read_text(encoding="utf-8")
for line_num, line in enumerate(content.strip().split("\n"), 1):
line = line.strip()
if not line or line.startswith("#"):
continue
parts = line.split(":")
if len(parts) < 3:
print(f"Warning: Line {line_num} skipped (format: type:value:propertyName): {line}", file=sys.stderr)
continue
loc_type = parts[0].strip().lower()
value = parts[1].strip()
prop_name = parts[2].strip()
role_name = parts[3].strip() if len(parts) > 3 else ""
elements.append({
"locator_type": loc_type,
"value": value,
"property_name": prop_name,
"role_name": role_name,
})
return elements
def to_camel_case(text):
"""Convert text to camelCase for property names."""
text = re.sub(r"[^a-zA-Z0-9\s]", " ", text)
words = text.strip().split()
if not words:
return "element"
return words[0].lower() + "".join(w.capitalize() for w in words[1:])
def determine_locator(element):
"""Determine the best Playwright locator strategy for an element.
Priority: getByRole > getByLabel > getByText > getByPlaceholder > getByTestId > CSS
"""
tag = element.get("tag", "")
text = element.get("text", "")
aria_label = element.get("aria_label", "")
placeholder = element.get("placeholder", "")
data_testid = element.get("data_testid", "")
el_type = element.get("type", "")
el_id = element.get("id", "")
el_name = element.get("name", "")
role = element.get("role", "")
# Determine ARIA role from tag
role_map = {
"button": "button",
"a": "link",
"h1": "heading",
"h2": "heading",
"h3": "heading",
"h4": "heading",
"h5": "heading",
"h6": "heading",
"nav": "navigation",
"select": "combobox",
"textarea": "textbox",
}
if el_type == "checkbox":
inferred_role = "checkbox"
elif el_type == "radio":
inferred_role = "radio"
elif tag == "input" and el_type in ("text", "email", "password", "search", "tel", "url", ""):
inferred_role = "textbox"
elif tag == "input" and el_type == "submit":
inferred_role = "button"
else:
inferred_role = role or role_map.get(tag, "")
display_name = aria_label or text
# 1. getByRole (preferred)
if inferred_role and display_name:
return f"page.getByRole('{inferred_role}', {{ name: '{_escape(display_name)}' }})"
if inferred_role and not display_name and tag in ("nav", "main", "header", "footer"):
return f"page.getByRole('{inferred_role}')"
# 2. getByLabel (form fields)
if tag in ("input", "textarea", "select") and aria_label:
return f"page.getByLabel('{_escape(aria_label)}')"
# 3. getByText
if text and tag not in ("input", "textarea", "select"):
return f"page.getByText('{_escape(text)}')"
# 4. getByPlaceholder
if placeholder:
return f"page.getByPlaceholder('{_escape(placeholder)}')"
# 5. getByTestId
if data_testid:
return f"page.getByTestId('{_escape(data_testid)}')"
# 6. CSS fallback
if el_id:
return f"page.locator('#{_escape(el_id)}')"
if el_name:
return f"page.locator('[name=\"{_escape(el_name)}\"]')"
return f"page.locator('{tag}')"
def _escape(s):
"""Escape single quotes in strings for TypeScript output."""
return s.replace("'", "\\'")
def generate_property_name(element, seen_names):
"""Generate a unique camelCase property name for an element."""
tag = element.get("tag", "")
text = element.get("text", "") or element.get("aria_label", "") or element.get("placeholder", "")
data_testid = element.get("data_testid", "")
el_name = element.get("name", "")
el_id = element.get("id", "")
base = text or data_testid or el_name or el_id or tag
name = to_camel_case(base)
# Add type suffix for clarity
suffix_map = {"input": "Input", "button": "Button", "a": "Link", "select": "Select", "textarea": "TextArea"}
tag_suffix = suffix_map.get(tag, "")
if tag_suffix and not name.lower().endswith(tag_suffix.lower()):
name = name + tag_suffix
# Deduplicate
original = name
counter = 2
while name in seen_names:
name = f"{original}{counter}"
counter += 1
seen_names.add(name)
return name
def generate_page_object(class_name, elements, page_path, source_type="html"):
"""Generate TypeScript Page Object class code."""
lines = []
seen_names = set()
properties = []
if source_type == "selectors":
for el in elements:
prop_name = el["property_name"]
loc_type = el["locator_type"]
value = el["value"]
role_name = el.get("role_name", "")
if loc_type == "role" and role_name:
locator = f"page.getByRole('{value}', {{ name: '{_escape(role_name)}' }})"
elif loc_type == "role":
locator = f"page.getByRole('{value}')"
elif loc_type == "label":
locator = f"page.getByLabel('{_escape(value)}')"
elif loc_type == "text":
locator = f"page.getByText('{_escape(value)}')"
elif loc_type == "placeholder":
locator = f"page.getByPlaceholder('{_escape(value)}')"
elif loc_type == "testid":
locator = f"page.getByTestId('{_escape(value)}')"
else:
locator = f"page.locator('{_escape(value)}')"
properties.append({"name": prop_name, "locator": locator})
else:
for el in elements:
prop_name = generate_property_name(el, seen_names)
locator = determine_locator(el)
properties.append({"name": prop_name, "locator": locator})
# Build TypeScript class
lines.append(f"// {_filename_from_class(class_name)}")
lines.append(f"import {{ type Page, type Locator, expect }} from '@playwright/test';")
lines.append("")
lines.append(f"export class {class_name} {{")
lines.append(f" readonly page: Page;")
for prop in properties:
lines.append(f" readonly {prop['name']}: Locator;")
lines.append("")
lines.append(f" constructor(page: Page) {{")
lines.append(f" this.page = page;")
for prop in properties:
lines.append(f" this.{prop['name']} = {prop['locator']};")
lines.append(f" }}")
# goto method
lines.append("")
lines.append(f" async goto() {{")
lines.append(f" await this.page.goto('{page_path}');")
lines.append(f" }}")
# expectLoaded method
lines.append("")
lines.append(f" async expectLoaded() {{")
if properties:
lines.append(f" await expect(this.{properties[0]['name']}).toBeVisible();")
else:
lines.append(f" await expect(this.page).toHaveURL(/{page_path.replace('/', '\\\\/')}/);" )
lines.append(f" }}")
lines.append(f"}}")
lines.append("")
return "\n".join(lines), properties
def _filename_from_class(class_name):
"""Convert ClassName to class-name.page.ts."""
name = re.sub(r"(?<!^)(?=[A-Z])", "-", class_name).lower()
if not name.endswith("-page"):
name += ".page"
else:
name = name.rsplit("-page", 1)[0] + ".page"
return name + ".ts"
def format_human(class_name, code, properties, source):
"""Format output for human consumption."""
output = []
output.append("=" * 70)
output.append("PAGE OBJECT GENERATOR")
output.append("=" * 70)
output.append("")
output.append(f" Class: {class_name}")
output.append(f" Source: {source}")
output.append(f" Properties: {len(properties)}")
output.append(f" File: {_filename_from_class(class_name)}")
output.append("")
output.append("-" * 70)
output.append("GENERATED CODE")
output.append("-" * 70)
output.append("")
output.append(code)
output.append("-" * 70)
output.append("")
output.append("LOCATOR STRATEGY BREAKDOWN")
output.append("-" * 70)
strategy_counts = {"getByRole": 0, "getByLabel": 0, "getByText": 0,
"getByPlaceholder": 0, "getByTestId": 0, "locator (CSS)": 0}
for prop in properties:
loc = prop["locator"]
if "getByRole" in loc:
strategy_counts["getByRole"] += 1
elif "getByLabel" in loc:
strategy_counts["getByLabel"] += 1
elif "getByText" in loc:
strategy_counts["getByText"] += 1
elif "getByPlaceholder" in loc:
strategy_counts["getByPlaceholder"] += 1
elif "getByTestId" in loc:
strategy_counts["getByTestId"] += 1
else:
strategy_counts["locator (CSS)"] += 1
for strategy, count in strategy_counts.items():
if count > 0:
bar = "#" * count
output.append(f" {strategy:<20} {count:>3} {bar}")
css_count = strategy_counts["locator (CSS)"]
total = len(properties) or 1
if css_count > 0:
output.append("")
pct = css_count / total * 100
output.append(f" WARNING: {pct:.0f}% of locators use CSS fallback. Consider adding")
output.append(f" aria-label, role, or data-testid attributes to improve resilience.")
return "\n".join(output)
def main():
parser = argparse.ArgumentParser(
description="Generate Playwright Page Object Model classes from HTML or selector lists.",
epilog="Example: python page_object_generator.py --html login.html --name LoginPage --route /login",
)
source_group = parser.add_mutually_exclusive_group(required=True)
source_group.add_argument("--html", help="Path to HTML file to extract elements from")
source_group.add_argument("--selectors", help="Path to selector list file (type:value:name per line)")
parser.add_argument("--name", required=True, help="Name for the generated Page Object class (e.g., LoginPage)")
parser.add_argument("--route", default="/", help="Page route for goto() method (default: /)")
parser.add_argument("--json", action="store_true", dest="json_output", help="Output results as JSON")
args = parser.parse_args()
if args.html:
source_path = Path(args.html)
if not source_path.exists():
print(f"Error: HTML file '{args.html}' not found.", file=sys.stderr)
sys.exit(1)
html_content = source_path.read_text(encoding="utf-8")
extractor = ElementExtractor()
extractor.feed(html_content)
elements = extractor.elements
if not elements:
print("Warning: No interactive elements found in the HTML.", file=sys.stderr)
code, properties = generate_page_object(args.name, elements, args.route, source_type="html")
source = str(source_path)
else:
source_path = Path(args.selectors)
if not source_path.exists():
print(f"Error: Selector file '{args.selectors}' not found.", file=sys.stderr)
sys.exit(1)
elements = parse_selector_file(source_path)
if not elements:
print("Error: No valid selectors found in file.", file=sys.stderr)
sys.exit(1)
code, properties = generate_page_object(args.name, elements, args.route, source_type="selectors")
source = str(source_path)
if args.json_output:
result = {
"class_name": args.name,
"filename": _filename_from_class(args.name),
"route": args.route,
"source": source,
"properties_count": len(properties),
"properties": properties,
"code": code,
}
print(json.dumps(result, indent=2))
else:
print(format_human(args.name, code, properties, source))
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""Analyze Playwright test files for anti-patterns and quality issues.
Scans Playwright test files (.spec.ts, .test.ts) for common anti-patterns
including hardcoded waits, missing assertions, fragile selectors, flaky
indicators, and violations of the 10 golden rules.
Usage:
python test_analyzer.py path/to/tests/
python test_analyzer.py path/to/tests/ --severity high
python test_analyzer.py path/to/tests/ --json
"""
import argparse
import json
import os
import re
import sys
from collections import defaultdict
from pathlib import Path
ANTI_PATTERNS = [
{
"id": "WAIT_TIMEOUT",
"name": "Hardcoded waitForTimeout",
"pattern": r"waitForTimeout\s*\(",
"severity": "high",
"message": "Replace waitForTimeout() with web-first assertions like expect(locator).toBeVisible()",
"rule": "Rule #2: Never page.waitForTimeout()",
},
{
"id": "WAIT_SLEEP",
"name": "setTimeout / sleep pattern",
"pattern": r"(?:setTimeout|sleep)\s*\(",
"severity": "high",
"message": "Avoid manual sleep/setTimeout; use Playwright auto-waiting assertions",
"rule": "Rule #2: Never page.waitForTimeout()",
},
{
"id": "CSS_SELECTOR",
"name": "Raw CSS selector as primary locator",
"pattern": r"page\.locator\s*\(\s*['\"][\.\#\[]",
"severity": "medium",
"message": "Prefer getByRole(), getByLabel(), getByText() over CSS selectors",
"rule": "Rule #1: getByRole() over CSS/XPath",
},
{
"id": "XPATH_SELECTOR",
"name": "XPath selector usage",
"pattern": r"page\.locator\s*\(\s*['\"]\/\/",
"severity": "high",
"message": "XPath selectors are fragile; use semantic locators instead",
"rule": "Rule #1: getByRole() over CSS/XPath",
},
{
"id": "HARDCODED_URL",
"name": "Hardcoded URL in test",
"pattern": r"(?:goto|navigate)\s*\(\s*['\"]https?://",
"severity": "medium",
"message": "Use baseURL in playwright.config.ts; pass relative paths to goto()",
"rule": "Rule #5: baseURL in config",
},
{
"id": "NO_ASSERTION",
"name": "Test block without expect()",
"pattern": None,
"severity": "high",
"message": "Test has no assertions; every test should verify expected behavior",
"rule": "Rule #9: One behavior per test",
},
{
"id": "SNAPSHOT_NO_AWAIT",
"name": "Non-awaited locator value in expect",
"pattern": r"expect\s*\(\s*await\s+\w+\.(?:textContent|innerText|innerHTML|getAttribute|inputValue)\s*\(",
"severity": "high",
"message": "Use expect(locator).toHaveText() instead of expect(await locator.textContent()); the latter does not auto-retry",
"rule": "Rule #3: expect(locator) auto-retries",
},
{
"id": "SHARED_STATE",
"name": "Mutable shared variable between tests",
"pattern": r"(?:let|var)\s+\w+\s*[=;]\s*$",
"severity": "medium",
"message": "Shared mutable state between tests causes flakiness; use fixtures or beforeEach",
"rule": "Rule #4: Isolate every test",
},
{
"id": "FORCE_CLICK",
"name": "Force click bypassing actionability",
"pattern": r"\.click\s*\(\s*\{\s*force\s*:\s*true",
"severity": "medium",
"message": "force:true bypasses actionability checks; fix the underlying visibility/overlay issue instead",
"rule": "Rule #2: Never page.waitForTimeout()",
},
{
"id": "NTH_CHILD",
"name": "Fragile nth-child / nth-of-type selector",
"pattern": r"(?:nth-child|nth-of-type)\s*\(",
"severity": "medium",
"message": "nth-child selectors break on DOM changes; use semantic locators or data-testid",
"rule": "Rule #1: getByRole() over CSS/XPath",
},
{
"id": "PAGE_WAIT_SELECTOR",
"name": "Deprecated waitForSelector usage",
"pattern": r"waitForSelector\s*\(",
"severity": "low",
"message": "waitForSelector is rarely needed; use web-first assertions (expect) for waiting",
"rule": "Rule #2: Never page.waitForTimeout()",
},
{
"id": "GLOBAL_PAGE",
"name": "Global page variable outside fixtures",
"pattern": r"(?:^|\n)\s*(?:let|var|const)\s+page\s*[=:]",
"severity": "low",
"message": "Use Playwright fixtures (test.extend) instead of global page variables",
"rule": "Rule #8: Fixtures over globals",
},
{
"id": "MULTIPLE_GOTO",
"name": "Multiple goto() calls in single test",
"pattern": None,
"severity": "low",
"message": "Multiple navigations in one test may indicate testing multiple behaviors; consider splitting",
"rule": "Rule #9: One behavior per test",
},
]
SEVERITY_ORDER = {"high": 0, "medium": 1, "low": 2}
def find_test_files(path):
"""Recursively find Playwright test files."""
test_files = []
p = Path(path)
if p.is_file():
test_files.append(p)
else:
for ext_pattern in ["**/*.spec.ts", "**/*.test.ts", "**/*.spec.js", "**/*.test.js"]:
test_files.extend(p.glob(ext_pattern))
return sorted(set(test_files))
def extract_test_blocks(content):
"""Extract individual test blocks with their line ranges."""
blocks = []
lines = content.split("\n")
# Find test(...) or it(...) calls
test_pattern = re.compile(r"^\s*(?:test|it)\s*\(\s*['\"](.+?)['\"]")
brace_depth = 0
current_test = None
start_line = 0
for i, line in enumerate(lines, 1):
match = test_pattern.match(line)
if match and current_test is None:
current_test = match.group(1)
start_line = i
brace_depth = 0
if current_test is not None:
brace_depth += line.count("{") - line.count("}")
if brace_depth <= 0 and "{" in content[sum(len(l) + 1 for l in lines[:start_line - 1]):]:
block_content = "\n".join(lines[start_line - 1:i])
blocks.append({
"name": current_test,
"start_line": start_line,
"end_line": i,
"content": block_content,
})
current_test = None
return blocks
def check_no_assertion(test_block):
"""Check if a test block contains no expect() calls."""
return "expect(" not in test_block["content"] and "expect (" not in test_block["content"]
def check_multiple_goto(test_block):
"""Check if a test block has multiple goto() calls."""
goto_count = len(re.findall(r"\.goto\s*\(", test_block["content"]))
return goto_count > 1
def analyze_file(filepath):
"""Analyze a single test file for anti-patterns."""
findings = []
try:
content = filepath.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError) as e:
return [{"file": str(filepath), "error": str(e)}]
lines = content.split("\n")
test_blocks = extract_test_blocks(content)
# Regex-based pattern checks
for ap in ANTI_PATTERNS:
if ap["pattern"] is None:
continue
regex = re.compile(ap["pattern"])
for i, line in enumerate(lines, 1):
if regex.search(line):
findings.append({
"file": str(filepath),
"line": i,
"code": line.strip(),
"anti_pattern": ap["id"],
"name": ap["name"],
"severity": ap["severity"],
"message": ap["message"],
"rule": ap["rule"],
})
# Structural checks on test blocks
for block in test_blocks:
if check_no_assertion(block):
ap = next(a for a in ANTI_PATTERNS if a["id"] == "NO_ASSERTION")
findings.append({
"file": str(filepath),
"line": block["start_line"],
"code": f"test('{block['name']}', ...)",
"anti_pattern": ap["id"],
"name": ap["name"],
"severity": ap["severity"],
"message": ap["message"],
"rule": ap["rule"],
})
if check_multiple_goto(block):
ap = next(a for a in ANTI_PATTERNS if a["id"] == "MULTIPLE_GOTO")
findings.append({
"file": str(filepath),
"line": block["start_line"],
"code": f"test('{block['name']}', ...)",
"anti_pattern": ap["id"],
"name": ap["name"],
"severity": ap["severity"],
"message": ap["message"],
"rule": ap["rule"],
})
return findings
def format_human(findings, stats):
"""Format findings as human-readable output."""
output = []
output.append("=" * 70)
output.append("PLAYWRIGHT TEST ANALYZER")
output.append("=" * 70)
output.append("")
if not findings:
output.append("No anti-patterns detected. Your tests look clean!")
return "\n".join(output)
# Group by file
by_file = defaultdict(list)
for f in findings:
by_file[f["file"]].append(f)
for filepath, file_findings in sorted(by_file.items()):
output.append(f"File: {filepath}")
output.append("-" * 70)
sorted_findings = sorted(file_findings, key=lambda x: SEVERITY_ORDER.get(x["severity"], 9))
for finding in sorted_findings:
sev = finding["severity"].upper()
output.append(f" [{sev}] Line {finding['line']}: {finding['name']}")
output.append(f" Code: {finding['code'][:80]}")
output.append(f" Fix: {finding['message']}")
output.append(f" Ref: {finding['rule']}")
output.append("")
output.append("")
# Summary
output.append("=" * 70)
output.append("SUMMARY")
output.append("=" * 70)
output.append(f" Files scanned: {stats['files_scanned']}")
output.append(f" Total findings: {stats['total_findings']}")
output.append(f" High severity: {stats['high']}")
output.append(f" Medium severity: {stats['medium']}")
output.append(f" Low severity: {stats['low']}")
output.append("")
if stats["high"] > 0:
output.append(" VERDICT: NEEDS ATTENTION - High severity issues found")
elif stats["medium"] > 0:
output.append(" VERDICT: ACCEPTABLE - Consider fixing medium issues")
else:
output.append(" VERDICT: GOOD - Only minor suggestions")
return "\n".join(output)
def main():
parser = argparse.ArgumentParser(
description="Analyze Playwright test files for anti-patterns and quality issues.",
epilog="Example: python test_analyzer.py ./tests/e2e/ --severity medium",
)
parser.add_argument("path", help="Path to test file or directory containing test files")
parser.add_argument("--json", action="store_true", dest="json_output", help="Output results as JSON")
parser.add_argument(
"--severity",
choices=["high", "medium", "low"],
default="low",
help="Minimum severity to report (default: low, shows all)",
)
args = parser.parse_args()
target = Path(args.path)
if not target.exists():
print(f"Error: Path '{args.path}' does not exist.", file=sys.stderr)
sys.exit(1)
test_files = find_test_files(target)
if not test_files:
print(f"No test files (.spec.ts/.test.ts) found in '{args.path}'.", file=sys.stderr)
sys.exit(1)
all_findings = []
for tf in test_files:
all_findings.extend(analyze_file(tf))
# Filter by severity
min_sev = SEVERITY_ORDER[args.severity]
filtered = [f for f in all_findings if SEVERITY_ORDER.get(f.get("severity", "low"), 9) <= min_sev]
stats = {
"files_scanned": len(test_files),
"total_findings": len(filtered),
"high": sum(1 for f in filtered if f.get("severity") == "high"),
"medium": sum(1 for f in filtered if f.get("severity") == "medium"),
"low": sum(1 for f in filtered if f.get("severity") == "low"),
}
if args.json_output:
result = {"stats": stats, "findings": filtered}
print(json.dumps(result, indent=2))
else:
print(format_human(filtered, stats))
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""Generate Playwright test code from user story descriptions.
Parses structured user story input (JSON or plain text) and generates
Playwright spec files with Page Object references, proper assertions,
and test isolation following the 10 golden rules.
Usage:
python test_generator.py --story "User can log in with email and password" --page LoginPage --route /login
python test_generator.py --stories stories.json --output ./tests/e2e/generated/
python test_generator.py --story "User can add item to cart" --page CartPage --route /cart --json
"""
import argparse
import json
import os
import re
import sys
from pathlib import Path
# Maps common actions to Playwright assertion patterns
ACTION_PATTERNS = {
"log in": {"action": "fill + click", "assertions": ["toHaveURL", "toBeVisible"]},
"sign up": {"action": "fill + click", "assertions": ["toHaveURL", "toBeVisible"]},
"register": {"action": "fill + click", "assertions": ["toHaveURL", "toBeVisible"]},
"submit": {"action": "click", "assertions": ["toBeVisible", "toHaveText"]},
"navigate": {"action": "goto + click", "assertions": ["toHaveURL"]},
"search": {"action": "fill + click", "assertions": ["toBeVisible", "toHaveCount"]},
"delete": {"action": "click + confirm", "assertions": ["not.toBeVisible", "toHaveCount"]},
"edit": {"action": "fill + click", "assertions": ["toHaveText", "toBeVisible"]},
"add": {"action": "click + fill", "assertions": ["toBeVisible", "toHaveCount"]},
"remove": {"action": "click", "assertions": ["not.toBeVisible"]},
"upload": {"action": "setInputFiles", "assertions": ["toBeVisible"]},
"download": {"action": "click + waitForDownload", "assertions": ["toBeTruthy"]},
"filter": {"action": "click + select", "assertions": ["toHaveCount"]},
"sort": {"action": "click", "assertions": ["toHaveText"]},
}
VALIDATION_SCENARIOS = [
{"name": "empty required field", "approach": "leave field empty, submit", "assertion": "error message visible"},
{"name": "invalid format", "approach": "enter malformed data", "assertion": "validation error shown"},
{"name": "boundary value", "approach": "enter min/max boundary", "assertion": "accepted or rejected correctly"},
{"name": "duplicate entry", "approach": "submit same data twice", "assertion": "duplicate error or idempotent success"},
]
def to_kebab_case(text):
"""Convert text to kebab-case for file names."""
text = re.sub(r"[^a-zA-Z0-9\s]", "", text)
return re.sub(r"\s+", "-", text.strip().lower())
def to_camel_case(text):
"""Convert text to camelCase."""
words = re.sub(r"[^a-zA-Z0-9\s]", " ", text).strip().split()
if not words:
return "element"
return words[0].lower() + "".join(w.capitalize() for w in words[1:])
def detect_action(story):
"""Detect the primary action from a user story."""
story_lower = story.lower()
for action_key, pattern in ACTION_PATTERNS.items():
if action_key in story_lower:
return action_key, pattern
return "interact", {"action": "click", "assertions": ["toBeVisible"]}
def extract_nouns(story):
"""Extract likely UI element nouns from a story."""
# Remove common stop words and extract meaningful nouns
stop_words = {
"user", "can", "should", "be", "able", "to", "the", "a", "an", "with",
"and", "or", "on", "in", "for", "from", "by", "as", "is", "are", "was",
"when", "then", "given", "that", "their", "my", "i", "they",
}
words = re.findall(r"[a-zA-Z]+", story.lower())
nouns = [w for w in words if w not in stop_words and len(w) > 2]
return nouns
def generate_test_scenarios(story, action_key, pattern):
"""Generate test scenarios from a user story."""
scenarios = []
# Happy path
scenarios.append({
"name": f"successfully {action_key}s",
"type": "happy_path",
"description": f"Verify that the user can {action_key} under normal conditions",
"assertions": pattern["assertions"],
})
# Validation scenarios
if pattern["action"] in ("fill + click", "click + fill"):
for vs in VALIDATION_SCENARIOS[:3]: # Top 3 validation scenarios
scenarios.append({
"name": f"shows error for {vs['name']}",
"type": "validation",
"description": vs["approach"],
"assertions": ["toBeVisible"], # Error message visible
})
# Negative path
scenarios.append({
"name": f"handles {action_key} failure gracefully",
"type": "negative",
"description": f"Verify error handling when {action_key} fails",
"assertions": ["toBeVisible"], # Error state visible
})
return scenarios
def generate_spec_code(story, page_name, route, scenarios):
"""Generate TypeScript spec file content."""
describe_name = story.rstrip(".")
page_var = to_camel_case(page_name)
page_file = re.sub(r"(?<!^)(?=[A-Z])", "-", page_name).lower()
if not page_file.endswith("-page"):
page_file += ".page"
lines = []
lines.append(f"import {{ test, expect }} from '@playwright/test';")
lines.append(f"import {{ {page_name} }} from '../../pages/{page_file}';")
lines.append("")
lines.append(f"test.describe('{describe_name}', () => {{")
lines.append(f" let {page_var}: {page_name};")
lines.append("")
lines.append(f" test.beforeEach(async ({{ page }}) => {{")
lines.append(f" {page_var} = new {page_name}(page);")
lines.append(f" await {page_var}.goto();")
lines.append(f" }});")
for scenario in scenarios:
lines.append("")
lines.append(f" test('{scenario['name']}', async ({{ page }}) => {{")
lines.append(f" // TODO: Implement - {scenario['description']}")
if scenario["type"] == "happy_path":
lines.append(f" // Arrange: Set up test data")
lines.append(f" // Act: Perform the {story.lower().split('can')[-1].strip() if 'can' in story.lower() else 'action'}")
lines.append(f" // Assert: Verify expected outcome")
for assertion in scenario["assertions"]:
if assertion == "toHaveURL":
lines.append(f" await expect(page).toHaveURL(/\\/{route.strip('/')}/);")
else:
lines.append(f" // await expect(locator).{assertion}();")
elif scenario["type"] == "validation":
lines.append(f" // Act: {scenario['description']}")
lines.append(f" // Assert: Error message is displayed")
lines.append(f" // await expect({page_var}.errorMessage).toBeVisible();")
else:
lines.append(f" // Arrange: Set up failure condition")
lines.append(f" // Act: Attempt the action")
lines.append(f" // Assert: Error state is shown to user")
lines.append(f" // await expect({page_var}.errorMessage).toBeVisible();")
lines.append(f" }});")
lines.append(f"}});")
lines.append("")
return "\n".join(lines)
def process_story(story, page_name, route):
"""Process a single user story into test scenarios and code."""
action_key, pattern = detect_action(story)
scenarios = generate_test_scenarios(story, action_key, pattern)
code = generate_spec_code(story, page_name, route, scenarios)
nouns = extract_nouns(story)
return {
"story": story,
"detected_action": action_key,
"page_name": page_name,
"route": route,
"scenario_count": len(scenarios),
"scenarios": scenarios,
"suggested_elements": nouns,
"code": code,
}
def load_stories_file(path):
"""Load stories from a JSON file.
Expected format:
[
{"story": "User can log in", "page": "LoginPage", "route": "/login"},
...
]
"""
with open(path, "r") as f:
return json.load(f)
def format_human(results):
"""Format results for human-readable output."""
output = []
output.append("=" * 70)
output.append("PLAYWRIGHT TEST GENERATOR")
output.append("=" * 70)
for result in results:
output.append("")
output.append(f"Story: {result['story']}")
output.append(f"Action: {result['detected_action']}")
output.append(f"Page: {result['page_name']} ({result['route']})")
output.append(f"Scenarios: {result['scenario_count']}")
output.append("")
for s in result["scenarios"]:
marker = {"happy_path": "+", "validation": "~", "negative": "x"}
output.append(f" [{marker.get(s['type'], '?')}] {s['name']}")
output.append("")
output.append("-" * 70)
output.append("GENERATED CODE")
output.append("-" * 70)
output.append(result["code"])
if result["suggested_elements"]:
output.append("Suggested Page Object elements: " + ", ".join(result["suggested_elements"][:8]))
output.append("")
return "\n".join(output)
def main():
parser = argparse.ArgumentParser(
description="Generate Playwright test code from user story descriptions.",
epilog="Example: python test_generator.py --story 'User can log in' --page LoginPage --route /login",
)
source = parser.add_mutually_exclusive_group(required=True)
source.add_argument("--story", help="Single user story text")
source.add_argument("--stories", help="Path to JSON file with multiple stories")
parser.add_argument("--page", help="Page Object class name (required with --story)")
parser.add_argument("--route", default="/", help="Page route (default: /)")
parser.add_argument("--output", help="Output directory for generated spec files")
parser.add_argument("--json", action="store_true", dest="json_output", help="Output as JSON")
args = parser.parse_args()
results = []
if args.story:
if not args.page:
print("Error: --page is required when using --story", file=sys.stderr)
sys.exit(1)
results.append(process_story(args.story, args.page, args.route))
else:
stories_path = Path(args.stories)
if not stories_path.exists():
print(f"Error: Stories file '{args.stories}' not found.", file=sys.stderr)
sys.exit(1)
stories = load_stories_file(stories_path)
for s in stories:
results.append(process_story(s["story"], s["page"], s.get("route", "/")))
# Write output files if --output specified
if args.output:
out_dir = Path(args.output)
out_dir.mkdir(parents=True, exist_ok=True)
for result in results:
filename = to_kebab_case(result["story"])[:50] + ".spec.ts"
filepath = out_dir / filename
filepath.write_text(result["code"], encoding="utf-8")
print(f"Written: {filepath}", file=sys.stderr)
if args.json_output:
print(json.dumps(results, indent=2))
else:
print(format_human(results))
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""Parse Playwright JSON test reports and generate summaries with flaky test detection.
Reads Playwright's JSON reporter output and produces a summary including
pass/fail/skip counts, duration statistics, flaky test identification
(tests that passed on retry), slowest tests, and failure details.
Usage:
python test_report_parser.py test-results.json
python test_report_parser.py test-results.json --flaky-threshold 5
python test_report_parser.py test-results.json --json
python test_report_parser.py test-results.json --top-slow 10
"""
import argparse
import json
import sys
from collections import defaultdict
from datetime import timedelta
from pathlib import Path
def parse_report(filepath):
"""Parse a Playwright JSON report file."""
try:
content = Path(filepath).read_text(encoding="utf-8")
return json.loads(content)
except (OSError, json.JSONDecodeError) as e:
print(f"Error reading report: {e}", file=sys.stderr)
sys.exit(1)
def extract_test_results(report):
"""Extract individual test results from the report structure.
Playwright JSON reports can have different structures depending on version.
This handles both the nested (suites) and flat formats.
"""
results = []
def _walk_suites(suites, parent_path=""):
for suite in suites:
suite_title = suite.get("title", "")
current_path = f"{parent_path} > {suite_title}" if parent_path else suite_title
# Process specs in this suite
for spec in suite.get("specs", []):
spec_title = spec.get("title", "")
full_title = f"{current_path} > {spec_title}" if current_path else spec_title
spec_file = spec.get("file", suite.get("file", ""))
for test in spec.get("tests", []):
project = test.get("projectName", test.get("projectId", "unknown"))
status = test.get("status", test.get("expectedStatus", "unknown"))
annotations = test.get("annotations", [])
# Collect all results (attempts) for this test
attempts = test.get("results", [])
durations = [r.get("duration", 0) for r in attempts]
total_duration = sum(durations)
attempt_count = len(attempts)
# Determine flakiness: passed on retry means flaky
attempt_statuses = [r.get("status", "unknown") for r in attempts]
is_flaky = (
status == "expected"
and attempt_count > 1
and any(s in ("failed", "timedOut") for s in attempt_statuses[:-1])
and attempt_statuses[-1] == "passed"
)
# Also check explicit flaky status
if test.get("status") == "flaky" or status == "flaky":
is_flaky = True
# Extract error details from failed attempts
errors = []
for r in attempts:
if r.get("status") in ("failed", "timedOut"):
error_msg = r.get("error", {})
if isinstance(error_msg, dict):
msg = error_msg.get("message", "")
snippet = error_msg.get("snippet", "")
elif isinstance(error_msg, str):
msg = error_msg
snippet = ""
else:
msg = str(error_msg) if error_msg else ""
snippet = ""
if msg:
errors.append({"message": msg[:500], "snippet": snippet[:300]})
# Determine final status
if is_flaky:
final_status = "flaky"
elif status in ("expected", "passed"):
final_status = "passed"
elif status in ("unexpected", "failed"):
final_status = "failed"
elif status in ("skipped",):
final_status = "skipped"
elif status == "timedOut":
final_status = "timedOut"
else:
# Fallback: check last attempt
last_status = attempt_statuses[-1] if attempt_statuses else "unknown"
final_status = last_status
results.append({
"title": full_title.strip(" > "),
"file": spec_file,
"project": project,
"status": final_status,
"duration_ms": total_duration,
"attempts": attempt_count,
"is_flaky": is_flaky,
"errors": errors,
"annotations": annotations,
})
# Recurse into nested suites
if "suites" in suite:
_walk_suites(suite["suites"], current_path)
# Handle top-level structure
if "suites" in report:
_walk_suites(report["suites"])
elif "specs" in report:
_walk_suites([report])
return results
def compute_stats(results):
"""Compute summary statistics from test results."""
total = len(results)
passed = sum(1 for r in results if r["status"] == "passed")
failed = sum(1 for r in results if r["status"] == "failed")
flaky = sum(1 for r in results if r["is_flaky"])
skipped = sum(1 for r in results if r["status"] == "skipped")
timed_out = sum(1 for r in results if r["status"] == "timedOut")
durations = [r["duration_ms"] for r in results if r["duration_ms"] > 0]
total_duration = sum(durations)
avg_duration = total_duration / len(durations) if durations else 0
max_duration = max(durations) if durations else 0
min_duration = min(durations) if durations else 0
p90_duration = sorted(durations)[int(len(durations) * 0.9)] if durations else 0
# Per-project breakdown
by_project = defaultdict(lambda: {"passed": 0, "failed": 0, "flaky": 0, "skipped": 0, "total": 0})
for r in results:
proj = r["project"]
by_project[proj]["total"] += 1
if r["is_flaky"]:
by_project[proj]["flaky"] += 1
elif r["status"] == "passed":
by_project[proj]["passed"] += 1
elif r["status"] == "failed":
by_project[proj]["failed"] += 1
elif r["status"] == "skipped":
by_project[proj]["skipped"] += 1
# Flaky rate
non_skipped = total - skipped
flaky_rate = (flaky / non_skipped * 100) if non_skipped > 0 else 0
pass_rate = (passed / non_skipped * 100) if non_skipped > 0 else 0
return {
"total": total,
"passed": passed,
"failed": failed,
"flaky": flaky,
"skipped": skipped,
"timed_out": timed_out,
"pass_rate": round(pass_rate, 1),
"flaky_rate": round(flaky_rate, 1),
"total_duration_ms": total_duration,
"avg_duration_ms": round(avg_duration),
"max_duration_ms": max_duration,
"min_duration_ms": min_duration,
"p90_duration_ms": p90_duration,
"by_project": dict(by_project),
}
def format_duration(ms):
"""Format milliseconds as human-readable duration."""
if ms < 1000:
return f"{ms}ms"
seconds = ms / 1000
if seconds < 60:
return f"{seconds:.1f}s"
minutes = int(seconds // 60)
remaining_secs = seconds % 60
return f"{minutes}m {remaining_secs:.0f}s"
def format_human(results, stats, top_slow, flaky_threshold):
"""Format results as human-readable output."""
output = []
output.append("=" * 70)
output.append("PLAYWRIGHT TEST REPORT SUMMARY")
output.append("=" * 70)
output.append("")
# Overall stats
output.append("RESULTS")
output.append("-" * 70)
total_bar_width = 40
pass_pct = stats["passed"] / stats["total"] if stats["total"] else 0
fail_pct = stats["failed"] / stats["total"] if stats["total"] else 0
flaky_pct = stats["flaky"] / stats["total"] if stats["total"] else 0
pass_bar = int(pass_pct * total_bar_width)
fail_bar = int(fail_pct * total_bar_width)
flaky_bar = int(flaky_pct * total_bar_width)
skip_bar = total_bar_width - pass_bar - fail_bar - flaky_bar
bar = "+" * pass_bar + "~" * flaky_bar + "x" * fail_bar + "." * max(0, skip_bar)
output.append(f" [{bar}]")
output.append(f" + passed ~ flaky x failed . skipped")
output.append("")
output.append(f" Total: {stats['total']}")
output.append(f" Passed: {stats['passed']} ({stats['pass_rate']}%)")
output.append(f" Failed: {stats['failed']}")
output.append(f" Flaky: {stats['flaky']} ({stats['flaky_rate']}% flaky rate)")
output.append(f" Skipped: {stats['skipped']}")
if stats["timed_out"] > 0:
output.append(f" Timed out: {stats['timed_out']}")
output.append("")
# Duration stats
output.append("DURATION")
output.append("-" * 70)
output.append(f" Total: {format_duration(stats['total_duration_ms'])}")
output.append(f" Average: {format_duration(stats['avg_duration_ms'])}")
output.append(f" P90: {format_duration(stats['p90_duration_ms'])}")
output.append(f" Slowest: {format_duration(stats['max_duration_ms'])}")
output.append("")
# Per-project breakdown
if stats["by_project"]:
output.append("BY PROJECT")
output.append("-" * 70)
for project, pstats in sorted(stats["by_project"].items()):
status_parts = []
if pstats["passed"]:
status_parts.append(f"{pstats['passed']} passed")
if pstats["failed"]:
status_parts.append(f"{pstats['failed']} failed")
if pstats["flaky"]:
status_parts.append(f"{pstats['flaky']} flaky")
if pstats["skipped"]:
status_parts.append(f"{pstats['skipped']} skipped")
status_str = ", ".join(status_parts)
output.append(f" {project:<20} {pstats['total']:>4} tests ({status_str})")
output.append("")
# Flaky tests
flaky_tests = [r for r in results if r["is_flaky"]]
if flaky_tests:
output.append("FLAKY TESTS")
output.append("-" * 70)
for ft in sorted(flaky_tests, key=lambda x: -x["attempts"]):
output.append(f" [{ft['project']}] {ft['title']}")
output.append(f" Attempts: {ft['attempts']} | Duration: {format_duration(ft['duration_ms'])}")
if ft["errors"]:
output.append(f" Last error: {ft['errors'][-1]['message'][:100]}")
output.append("")
if stats["flaky_rate"] > flaky_threshold:
output.append(f" WARNING: Flaky rate ({stats['flaky_rate']}%) exceeds threshold ({flaky_threshold}%)")
output.append(f" Action: Quarantine flaky tests and fix within 48 hours")
output.append("")
# Failed tests
failed_tests = [r for r in results if r["status"] == "failed" or r["status"] == "timedOut"]
if failed_tests:
output.append("FAILED TESTS")
output.append("-" * 70)
for ft in failed_tests:
status_label = "TIMEOUT" if ft["status"] == "timedOut" else "FAILED"
output.append(f" [{status_label}] [{ft['project']}] {ft['title']}")
output.append(f" File: {ft['file']}")
output.append(f" Duration: {format_duration(ft['duration_ms'])}")
if ft["errors"]:
error = ft["errors"][-1]
# Truncate for readability
msg_lines = error["message"].split("\n")
for msg_line in msg_lines[:3]:
output.append(f" Error: {msg_line.strip()[:100]}")
output.append("")
# Slowest tests
sorted_by_duration = sorted(results, key=lambda x: -x["duration_ms"])
slow_tests = sorted_by_duration[:top_slow]
if slow_tests:
output.append(f"TOP {top_slow} SLOWEST TESTS")
output.append("-" * 70)
for i, st in enumerate(slow_tests, 1):
dur = format_duration(st["duration_ms"])
output.append(f" {i:>2}. {dur:>8} [{st['project']}] {st['title']}")
output.append("")
# Verdict
output.append("=" * 70)
if stats["failed"] > 0:
output.append("VERDICT: FAILED - Fix failing tests before merging")
elif stats["flaky_rate"] > flaky_threshold:
output.append(f"VERDICT: UNSTABLE - Flaky rate ({stats['flaky_rate']}%) exceeds {flaky_threshold}% threshold")
elif stats["total_duration_ms"] > 600000:
output.append(f"VERDICT: SLOW - Suite took {format_duration(stats['total_duration_ms'])} (target: <10min)")
else:
output.append("VERDICT: PASSED")
output.append("=" * 70)
return "\n".join(output)
def main():
parser = argparse.ArgumentParser(
description="Parse Playwright JSON test reports and generate summary with flaky test detection.",
epilog="Example: python test_report_parser.py test-results.json --flaky-threshold 3",
)
parser.add_argument("report", help="Path to Playwright JSON report file")
parser.add_argument("--json", action="store_true", dest="json_output", help="Output results as JSON")
parser.add_argument(
"--flaky-threshold",
type=float,
default=5.0,
help="Flaky rate percentage threshold for warnings (default: 5.0)",
)
parser.add_argument(
"--top-slow",
type=int,
default=5,
help="Number of slowest tests to display (default: 5)",
)
args = parser.parse_args()
report_path = Path(args.report)
if not report_path.exists():
print(f"Error: Report file '{args.report}' not found.", file=sys.stderr)
sys.exit(1)
report = parse_report(report_path)
results = extract_test_results(report)
if not results:
print("Warning: No test results found in the report.", file=sys.stderr)
# Still produce output with zeros
results = []
stats = compute_stats(results)
if args.json_output:
output = {
"stats": stats,
"flaky_tests": [r for r in results if r["is_flaky"]],
"failed_tests": [r for r in results if r["status"] in ("failed", "timedOut")],
"slowest_tests": sorted(results, key=lambda x: -x["duration_ms"])[:args.top_slow],
"flaky_threshold": args.flaky_threshold,
"threshold_exceeded": stats["flaky_rate"] > args.flaky_threshold,
}
print(json.dumps(output, indent=2))
else:
print(format_human(results, stats, args.top_slow, args.flaky_threshold))
if __name__ == "__main__":
main()
Sub-Skill: BrowserStack Integration
Parent: playwright-pro Trigger: "BrowserStack", "cloud testing", "cross-browser cloud"
Purpose
Configure Playwright to run tests on BrowserStack's cloud infrastructure for cross-browser and cross-device testing at scale.
Workflow
Step 1: Install BrowserStack SDK
pnpm add -D browserstack-node-sdkStep 2: Configure BrowserStack
Create browserstack.yml in project root:
userName: ${BROWSERSTACK_USERNAME}
accessKey: ${BROWSERSTACK_ACCESS_KEY}
platforms:
- os: Windows
osVersion: 11
browserName: Chrome
browserVersion: latest
- os: OS X
osVersion: Sonoma
browserName: Safari
browserVersion: latest
- deviceName: iPhone 15
osVersion: 17
browserName: Safari
- deviceName: Samsung Galaxy S24
osVersion: 14.0
browserName: Chrome
parallelsPerPlatform: 2
projectName: "My Project E2E"
buildName: "Build ${BUILD_NUMBER}"
debug: true
networkLogs: trueStep 3: Update Playwright Config
Add BrowserStack-specific project entries or use the SDK's automatic browser provisioning. The SDK wraps playwright test and routes browser connections through BrowserStack.
Step 4: Run on BrowserStack
npx browserstack-node-sdk playwright testStep 5: CI Integration
Add BrowserStack credentials as CI secrets and run the cloud suite on a schedule or for release candidates (not every PR -- too slow and expensive).
# GitHub Actions
- run: npx browserstack-node-sdk playwright test
env:
BROWSERSTACK_USERNAME: ${{ secrets.BROWSERSTACK_USERNAME }}
BROWSERSTACK_ACCESS_KEY: ${{ secrets.BROWSERSTACK_ACCESS_KEY }}Best Practices
1. Run cloud tests on a schedule (nightly) or for release candidates, not every commit 2. Keep a fast local suite (Chromium only) for PR feedback loops 3. Use BrowserStack for Safari/IE/mobile-specific validation 4. Set debug: true and networkLogs: true for failure investigation 5. Tag builds with commit SHA for traceability
Inputs
| Input | Required | Description |
|---|---|---|
| BrowserStack credentials | Yes | Username and access key |
| Target platforms | No | Defaults to Chrome + Safari + mobile |
| Parallel count | No | Defaults to 2 per platform |
Outputs
browserstack.ymlconfiguration- Updated CI workflow with cloud testing step
- Test results on BrowserStack dashboard
Sub-Skill: Test Coverage Analysis
Parent: playwright-pro Trigger: "analyze test coverage", "map tests to user flows", "coverage gaps"
Purpose
Map Playwright tests to user stories and requirements. Identify coverage gaps by comparing what is tested against what should be tested based on critical user flows.
Workflow
Step 1: Define User Flows
Enumerate critical user journeys. Each flow has:
- ID: Unique identifier (e.g.,
AUTH-001) - Name: Human-readable name (e.g., "User login with email")
- Priority: Critical / High / Medium / Low
- Steps: Ordered list of user actions
Step 2: Inventory Tests
Scan the test directory and extract:
- Test file paths and describe block names
- Individual test names
- Page Objects referenced
- URLs navigated to
Step 3: Map Tests to Flows
Use coverage_mapper.py to match tests against flows:
python scripts/coverage_mapper.py --tests ./tests/e2e/ --flows flows.jsonMatching heuristics:
- Test name contains flow keywords
- Test navigates to flow-related URLs
- Test uses Page Objects for flow pages
- Test assertions match expected flow outcomes
Step 4: Identify Gaps
For each critical flow without test coverage:
- Generate a recommended test outline
- Estimate effort to implement
- Prioritize by flow criticality
Step 5: Report
Output a coverage matrix:
| Flow | Priority | Tests | Status |
|---|---|---|---|
| AUTH-001 Login | Critical | login.spec.ts (3 tests) | Covered |
| AUTH-002 Password reset | Critical | -- | GAP |
| CHECKOUT-001 Purchase | Critical | checkout.spec.ts (5 tests) | Covered |
Inputs
| Input | Required | Description |
|---|---|---|
| Test directory | Yes | Path to Playwright test files |
| Flows definition | Yes | JSON file defining critical user flows |
Outputs
- Coverage matrix (flows vs tests)
- Gap list with recommended test outlines
- Coverage percentage by priority tier
Sub-Skill: Fix Failing Tests
Parent: playwright-pro Trigger: "fix flaky test", "Playwright test failing", "debug e2e failure"
Purpose
Diagnose and fix failing or flaky Playwright tests. Uses trace analysis, error pattern matching, and the 10 golden rules to identify root causes and apply fixes.
Workflow
Step 1: Reproduce and Classify
Run the failing test with full diagnostics:
npx playwright test <test-file> --trace on --retries 3 --reporter=listClassify the failure:
- Deterministic failure: Fails every time (likely a real bug or broken locator)
- Flaky failure: Passes sometimes (timing, state isolation, or environment issue)
- Environment failure: Only fails in CI (font rendering, network, resource constraints)
Step 2: Analyze Trace
Open the trace viewer to inspect the failure point:
npx playwright show-trace test-results/<test>/trace.zipCheck for:
- Element not found at the moment of interaction
- Network request timing vs assertion timing
- Console errors or uncaught exceptions
- Screenshot showing unexpected UI state
Step 3: Match Error Pattern
| Error Pattern | Root Cause | Fix |
|---|---|---|
waiting for locator timeout | Element not rendered or selector wrong | Add await expect(locator).toBeVisible() before interaction |
strict mode violation | Multiple elements match locator | Make locator more specific with .first(), .nth(), or tighter role/name |
Target closed | Page navigated during action | Add waitForURL() or waitForLoadState() before assertion |
net::ERR_CONNECTION_REFUSED | Dev server not running | Check webServer config in playwright.config.ts |
toHaveScreenshot mismatch | Visual diff from baseline | Update snapshot or fix the CSS regression |
Step 4: Apply Fix
Apply the appropriate fix from the pattern table. After fixing: 1. Run the test 5 times locally to confirm stability 2. Run with --retries 0 to ensure it passes without retries 3. If the fix involved adding a wait, verify it uses web-first assertions (not waitForTimeout)
Step 5: Prevent Recurrence
- Add the failure pattern to team knowledge base
- If the root cause is a common anti-pattern, flag it for the test analyzer
Inputs
| Input | Required | Description |
|---|---|---|
| Test file or name | Yes | The failing test to diagnose |
| Error message | No | The error output (speeds up diagnosis) |
| CI vs local | No | Where the failure occurs |
Outputs
- Root cause diagnosis with evidence
- Applied fix with explanation
- Stability verification (5x pass confirmation)
Sub-Skill: Test Generation
Parent: playwright-pro Trigger: "generate tests for", "write e2e tests", "create Playwright tests from"
Purpose
Generate Playwright test files from user stories, page descriptions, or API endpoint lists. Produces spec files with Page Objects, proper assertions, and test isolation.
Workflow
Step 1: Gather Input
Accept one of:
- User story: "As a user, I can log in with email and password"
- Page description: "Login page with email field, password field, submit button, forgot password link"
- URL + selectors: Crawl a page and extract interactive elements
Step 2: Identify Test Scenarios
For each input, derive:
- Happy path: Primary success flow
- Validation errors: Empty fields, invalid formats, boundary values
- Edge cases: Network failure, timeout, concurrent sessions
- Negative paths: Wrong credentials, expired tokens, unauthorized access
Step 3: Generate Page Object
If no Page Object exists for the target page, generate one using the locator priority from the parent skill (getByRole > getByLabel > getByText > getByTestId > CSS).
Step 4: Generate Spec File
test.describe('Feature Name', () => {
test.beforeEach(async ({ page }) => {
// Navigate and set up
});
test('happy path description', async ({ page }) => {
// Arrange, Act, Assert
});
test('validation: empty required field', async ({ page }) => {
// Test validation behavior
});
});Step 5: Validate Generated Tests
- Every test has at least one
expect()assertion - No
waitForTimeout()calls - All locators use semantic strategies
- Tests are isolated (no shared mutable state)
Rules
1. One behavior per test -- multiple related assertions within a behavior are fine 2. Use Page Object methods, never raw locators in spec files 3. Name tests as "expected behavior when condition" (e.g., "redirects to dashboard after valid login") 4. Group related tests in test.describe() blocks 5. Use test.beforeEach() for common setup, never shared variables
Inputs
| Input | Required | Description |
|---|---|---|
| Source | Yes | User story, page description, or URL |
| Page name | Yes | Target page for Page Object naming |
| Auth required | No | Whether tests need authenticated state |
Outputs
- Page Object class (if new)
- Spec file with happy path + error path tests
- Suggested additional test scenarios
Sub-Skill: Project Initialization
Parent: playwright-pro Trigger: "set up Playwright", "initialize e2e tests", "add Playwright to project"
Purpose
Bootstrap a Playwright testing environment from scratch in an existing project. Handles installation, configuration, directory structure, first test, and CI integration.
Workflow
Step 1: Install Dependencies
# Detect package manager
pnpm add -D @playwright/test
pnpm exec playwright install --with-deps chromium firefoxStep 2: Generate Configuration
Create playwright.config.ts following the golden rules:
baseURLfrom environment variable with localhost fallbackretries: 2in CI,0locallytrace: 'on-first-retry'screenshot: 'only-on-failure'- Projects: chromium + firefox + mobile-chrome
- Auth setup project with
storageState
Step 3: Create Directory Structure
tests/
├── e2e/
│ ├── auth.setup.ts # Shared authentication
│ ├── smoke.spec.ts # First smoke test
│ └── __fixtures__/ # Test data
├── pages/ # Page Objects
│ └── base.page.ts # Base page class
└── helpers/ # Test utilities
└── test-data.ts # Data factory helpersStep 4: Write Smoke Test
Generate a minimal smoke test that validates the app loads, the page title is correct, and no console errors appear. This confirms the setup works end-to-end.
Step 5: Add to CI
Add a GitHub Actions job or extend existing pipeline with the Playwright step. Include artifact upload for reports on failure.
Step 6: Verify
npx playwright test tests/e2e/smoke.spec.ts --reporter=listInputs
| Input | Required | Description |
|---|---|---|
| Package manager | No | Auto-detected (pnpm > npm > yarn) |
| Base URL | No | Defaults to http://localhost:3000 |
| Browsers | No | Defaults to chromium + firefox + mobile |
Outputs
playwright.config.tsconfigured per golden rules- Directory structure with first smoke test
- CI workflow step (if requested)
- Confirmation the smoke test passes
Sub-Skill: Migration from Cypress/Selenium
Parent: playwright-pro Trigger: "migrate from Cypress", "convert Selenium tests", "switch to Playwright"
Purpose
Migrate an existing Cypress or Selenium test suite to Playwright. Handles API translation, pattern modernization, and validation of migrated tests.
Workflow
Step 1: Audit Existing Suite
Inventory the current test suite:
- Count of test files and test cases
- Frameworks used (Cypress, Selenium WebDriver, Protractor, TestCafe)
- Custom commands and plugins in use
- CI integration points
- Test data management approach
Step 2: Translate API Calls
Apply the translation table from the parent SKILL.md. Key mappings:
Cypress to Playwright:
cy.visit()->page.goto()cy.get()->page.locator()(then upgrade togetByRole)cy.contains()->page.getByText()cy.intercept()->page.route()cy.should('be.visible')->expect(locator).toBeVisible()- Custom commands -> Page Object methods or fixtures
Selenium to Playwright:
driver.findElement(By.css())->page.locator()driver.findElement(By.xpath())->page.getByRole()(eliminate XPath)WebDriverWait-> web-first assertions (auto-retry)driver.get()->page.goto()
Step 3: Modernize Patterns
During migration, upgrade patterns: 1. Replace CSS/XPath selectors with semantic locators 2. Replace explicit waits with web-first assertions 3. Extract Page Objects from inline locators 4. Replace custom retry logic with Playwright's built-in retries 5. Convert fixture files to proper test data management
Step 4: Validate Migration
For each migrated test: 1. Run and confirm it passes 2. Compare behavior with the original test 3. Check that no waitForTimeout was introduced 4. Verify locator strategy follows priority order
Step 5: Remove Old Framework
After all tests are migrated and passing: 1. Remove old framework dependencies 2. Update CI configuration 3. Remove old config files (cypress.config.js, etc.) 4. Update documentation and contributing guides
Inputs
| Input | Required | Description |
|---|---|---|
| Source framework | Yes | Cypress, Selenium, Protractor, or TestCafe |
| Test directory | Yes | Path to existing test files |
| Priority | No | Migrate all at once or incrementally |
Outputs
- Migrated Playwright test files
- New Page Object classes
- Updated playwright.config.ts
- Migration report (tests migrated, patterns upgraded, issues found)
Sub-Skill: Test Execution Reports
Parent: playwright-pro Trigger: "generate test report", "summarize test results", "parse Playwright report"
Purpose
Generate human-readable test execution reports from Playwright JSON output. Includes pass/fail summary, duration stats, flaky test detection, and trend analysis across runs.
Workflow
Step 1: Collect Results
Ensure Playwright is configured to output JSON:
// playwright.config.ts
reporter: [
['html'],
['json', { outputFile: 'test-results.json' }],
],Step 2: Parse Single Run
Use the test_report_parser.py script:
python scripts/test_report_parser.py test-results.json --flaky-threshold 3This produces:
- Pass/fail/flaky/skip counts with visual bar
- Duration breakdown (total, average, P90, slowest)
- Per-project breakdown (chromium, firefox, mobile)
- Flaky test list with attempt counts
- Failed test details with error messages
- Top N slowest tests
Step 3: Trend Analysis (Multi-Run)
When multiple result files are available, compare across runs:
- Is the flaky rate trending up or down?
- Are specific tests getting slower over time?
- Are new failures appearing in specific browsers?
Step 4: Generate Verdict
| Condition | Verdict |
|---|---|
| All tests pass, flaky < threshold | PASSED |
| All pass but flaky > threshold | UNSTABLE |
| Any test fails | FAILED |
| Suite > 10 minutes | SLOW |
Step 5: Distribute
Output formats:
- Terminal: Human-readable summary (default)
- JSON: Machine-parseable for CI integration
- Markdown: For PR comments or Slack notifications
Inputs
| Input | Required | Description |
|---|---|---|
| Report file | Yes | Playwright JSON report |
| Flaky threshold | No | Percentage threshold (default: 5%) |
| Top slow count | No | Number of slowest tests to show (default: 5) |
Outputs
- Formatted test execution summary
- Flaky test warnings (if above threshold)
- Verdict (PASSED / UNSTABLE / FAILED / SLOW)
Sub-Skill: Test Quality Review
Parent: playwright-pro Trigger: "review test quality", "audit Playwright tests", "check test coverage gaps"
Purpose
Review a Playwright test suite for quality issues, coverage gaps, anti-patterns, and flaky test indicators. Produces an actionable report with prioritized fixes.
Workflow
Step 1: Run Static Analysis
Use the test_analyzer.py script to scan for anti-patterns:
python scripts/test_analyzer.py ./tests/e2e/ --severity lowThis catches: waitForTimeout, CSS/XPath selectors, missing assertions, force:true clicks, hardcoded URLs.
Step 2: Assess Coverage
Map tests to critical user journeys:
- Login / Authentication
- Core CRUD operations
- Checkout / Payment (if applicable)
- Onboarding flow
- Error handling paths
Flag any critical journey without dedicated tests.
Step 3: Check Flaky Indicators
Review recent CI runs for:
- Tests that needed retries to pass
- Tests with inconsistent durations (>2x variation)
- Tests that were recently disabled or skipped
Use flaky_detector.py on CI results:
python scripts/flaky_detector.py --results-dir ./test-results/ --runs 10Step 4: Review Architecture
- Every tested page has a Page Object class
- No raw locators in spec files
test.describe()groups are logical- Fixtures used for shared setup (not global variables)
- Auth setup project configured (not per-test login)
Step 5: Generate Report
Produce a prioritized report: 1. Critical: Tests with no assertions, hardcoded waits, flaky tests above 5% rate 2. High: Missing coverage for critical user journeys 3. Medium: CSS selectors that should be semantic, missing Page Objects 4. Low: Style inconsistencies, test naming conventions
Inputs
| Input | Required | Description |
|---|---|---|
| Test directory | Yes | Path to Playwright test files |
| CI results | No | Recent test result JSON files for flaky analysis |
| User journeys | No | List of critical flows to check coverage against |
Outputs
- Anti-pattern report with line-level findings
- Coverage gap analysis
- Flaky test list with stability scores
- Prioritized fix recommendations
Sub-Skill: TestRail Integration
Parent: playwright-pro Trigger: "TestRail", "test management", "sync test results to TestRail"
Purpose
Integrate Playwright test execution with TestRail for test case management, run tracking, and reporting. Sync test results automatically from CI.
Workflow
Step 1: Install Reporter
pnpm add -D playwright-testrail-reporterStep 2: Configure Reporter
Add TestRail reporter to playwright.config.ts:
reporter: [
['html'],
['playwright-testrail-reporter', {
host: process.env.TESTRAIL_HOST,
username: process.env.TESTRAIL_USERNAME,
password: process.env.TESTRAIL_API_KEY,
projectId: 1,
suiteId: 1,
runName: `Automated Run - ${new Date().toISOString()}`,
includeAllInTestRun: false,
}],
],Step 3: Map Tests to TestRail Cases
Add TestRail case IDs to test titles or annotations:
// Option A: In test title
test('C12345 - User can log in with valid credentials', async ({ page }) => {
// ...
});
// Option B: Via annotation
test('User can log in', async ({ page }) => {
test.info().annotations.push({ type: 'testrail', description: 'C12345' });
// ...
});Step 4: Sync Results
When tests run in CI, the reporter automatically: 1. Creates a new test run in TestRail (or updates existing) 2. Maps Playwright test results to TestRail case IDs 3. Uploads pass/fail status with execution time 4. Attaches failure messages and screenshots
Step 5: CI Configuration
- run: pnpm exec playwright test
env:
TESTRAIL_HOST: ${{ secrets.TESTRAIL_HOST }}
TESTRAIL_USERNAME: ${{ secrets.TESTRAIL_USERNAME }}
TESTRAIL_API_KEY: ${{ secrets.TESTRAIL_API_KEY }}Best Practices
1. Use TestRail case IDs consistently (prefix C + number) 2. Map every automated test to a TestRail case for full traceability 3. Keep manual-only test cases separate from automated ones in TestRail 4. Use TestRail milestones to group runs by release 5. Review unmapped tests periodically -- they indicate coverage drift
Inputs
| Input | Required | Description |
|---|---|---|
| TestRail credentials | Yes | Host, username, API key |
| Project and suite IDs | Yes | TestRail project/suite to sync to |
| Case ID mapping | Yes | Test-to-case ID mapping (in titles or annotations) |
Outputs
- TestRail reporter configured in Playwright
- Test runs created and synced automatically
- Pass/fail results with failure details in TestRail