Frontend Testing

Frontend Testing

Unlock reliable confidence fast: enable safe refactors by choosing the right test layer, making the app observable, and eliminating nondeterminism so failures are actionable.

Philosophy: Confidence Per Minute

Frontend tests fail for two reasons: the product is broken, or the test is lying. Your job is to maximize signal and minimize “test is lying”.

Before writing a test, ask:

• What user risk am I covering (money, progression, auth, data loss, “can’t start” crashes)?
• What’s the narrowest layer that catches this bug class (pure logic vs UI vs full browser)?
• What nondeterminism exists (time, RNG, async loading, network, animations, fonts, GPU)?
• What “ready” signal can I wait on besides setTimeout?
• What should a failure print/screenshot so it’s diagnosable in CI?

Core principles: 1. Test the contract, not the implementation: assert stable user-meaningful outcomes and public seams. 2. Prefer determinism over retries: make time/RNG/network controllable; remove flake at the source. 3. Observe like a debugger: console errors, network failures, screenshots, and state dumps on failure. 4. One critical flow first: a reliable smoke test beats 50 flaky tests.

Workflow Decision Tree

Pick the test type by the cheapest layer that provides the needed confidence:

• Unit tests (fastest): pure functions, reducers, validators, math, pathfinding, deterministic simulation steps.
• Component/integration tests (medium): UI behavior with mocked IO (React Testing Library / Vue Testing Library / Testing Library DOM).
• E2E tests (slowest, highest confidence): critical user flows across routing, storage, real bundling/runtime.
• Visual regression (specialized): layout/pixel regressions; for canvas/WebGL, only after locking determinism.
• A11y checks: great for DOM UIs; limited value for pure canvas unless you expose accessible DOM overlays.

Quick Start (Any Project)

1. Define 1 smoke flow: “page loads → user can start → one key action works”. 2. Choose runner:

• Prefer Playwright for browser E2E + screenshots.
• Prefer Testing Library for DOM component behavior.
• Prefer unit tests for logic you can run without a browser.

3. Add a “ready” signal in the app (DOM marker, window flag, or game event) and wait on that. 4. Fail loudly: treat console errors and failed requests as test failures. 5. Stabilize: seed RNG, freeze time, fix viewport/DPR, disable animations, and remove network variability.

Playwright Patterns (Especially Useful For Games)

Use Playwright when you need “real browser” confidence:

• Drive input via mouse/keyboard/touch; treat the canvas like the user does.
• Add a test seam: expose a small, stable test API on window (read-only state + a few commands).
• Prefer waitForFunction-style readiness over sleep; gate on “scene ready” / “assets loaded” / “first frame rendered”.
• For screenshots: lock viewport, device scale factor, fonts, and animation timing.
• For 9-slice / canvas UI regressions: add a dedicated UI harness scene/page and assert via targeted screenshots (see references/phaser-canvas-testing.md).

If using the Playwright MCP tools (browser automation inside Codex), follow the same mindset:

• Use browser_console_messages and browser_network_requests to catch silent failures.
• Use browser_evaluate to assert window.__TEST__ state and to set up deterministic mode.
• Use browser_take_screenshot for visual assertions after determinism is enforced.

Reconnaissance-Then-Action (Borrowed From Real Debugging)

When a UI is dynamic, don’t guess selectors—recon first, then act:

Quick decision guide:

Task → Is it static HTML (no JS runtime needed)?
  ├─ Yes → read the HTML to find stable selectors/content, then automate
  └─ No  → treat as dynamic: run the app, wait for readiness, then inspect rendered state

1. Navigate and wait for readiness:

• For many webapps: wait for a meaningful “loaded” element (preferred).
• networkidle can help for SPAs, but avoid it if the app uses websockets/polling.

2. Capture evidence (what the user actually sees):

• screenshot (full page for DOM; targeted for canvas)
• console errors + failed requests

3. Discover selectors from the rendered state:

• prefer role/text/label selectors over brittle CSS

4. Execute actions using discovered selectors and re-check state.

Common pitfall: ❌ Inspect/interact before the app is ready. ✅ Wait on an explicit ready signal (DOM marker or window.__TEST__.ready), not a sleep.

Server Lifecycle Helper (Playwright E2E)

When the dev server isn’t already running, use the bundled helper as a black box:

• Run python scripts/with_server.py --help first.
• Start one (or multiple) servers, wait for their ports, then run your test command.

Example:

python scripts/with_server.py --server "npm run dev" --port 5173 -- npm test

Flake Reduction Checklist

• Replace sleeps with explicit readiness conditions.
• Control time (Date.now, timers), RNG, and animation loops.
• Make network deterministic (mock, record/replay, or run against a seeded local backend).
• Eliminate “first-run” differences (asset caches, fonts) or warm them explicitly.
• Lock environment: viewport, DPR, locale/timezone, and rendering settings.

Anti-Patterns to Avoid

❌ Testing the wrong layer: E2E tests for pure logic. Better: unit tests for logic; reserve E2E for integration contracts.

❌ Testing implementation details: asserting DOM structure/classnames or internal engine objects. Better: assert user-meaningful outputs (text, navigation, score/HP changes) or a small stable test seam.

❌ Sleep-driven tests: wait 2s then click. Better: wait on explicit readiness (DOM marker, event, window flag).

❌ Uncontrolled randomness: RNG/time-based behaviors in assertions. Better: seed RNG, freeze time, and assert stable invariants.

❌ Pixel snapshots without determinism (especially canvas/WebGL). Better: add deterministic mode first; then screenshot selectively.

❌ Snapshot explosion: hundreds of snapshots that no one can interpret. Better: keep snapshots targeted (critical screens); prefer specific assertions for behavior.

❌ Retries as a strategy: “just bump retries in CI”. Better: fix readiness and determinism; use retries only as temporary guardrails.

Variation Guidance (Prevent One-Size-Fits-All)

Vary the approach based on:

• UI type: DOM app vs canvas/WebGL game vs hybrid.
• Risk: core revenue/progression flows get E2E first; edge UI polish gets component tests.
• CI constraints: headless-only, limited GPU, slow CPUs, no audio devices.
• Test seam availability: if you can add a stable window.__TEST__ API, assert state; if not, stick to black-box input/output.

Remember

You can make almost any frontend (including canvas/WebGL games) testable by adding a tiny, stable seam for readiness + state. This skill is meant to empower creative, high-signal testing rather than cargo-cult checklists. Aim for tests that are boring to maintain: deterministic, explicit about readiness, and rich in failure evidence. One reliable smoke test is the foundation; everything else compounds from there.

Bundled Resources

Read these only when needed:

• references/playwright-mcp-cheatsheet.md: patterns for using Playwright MCP tools for assertions, waiting, and diagnostics.
• references/phaser-canvas-testing.md: deterministic mode + hooks for Phaser/canvas/WebGL games.
• references/flake-reduction.md: deeper flake triage and stabilization tactics.

Use these scripts as black boxes (run --help first; don’t read source unless you must):

• scripts/with_server.py: start/wait/stop one or more dev servers around a test command.
• scripts/imgdiff.py: lightweight screenshot diff helper (requires pip install pillow).

Flake Reduction (Frontend)

First classify the flake

1. Readiness flake: app not ready when test interacts (most common). 2. Timing flake: animation/transition/physics changes outcome by milliseconds. 3. Environment flake: viewport/DPR/fonts/GPU differences change rendering. 4. Data flake: network/backend state changes. 5. Concurrency flake: tests interfere via shared storage, ports, global state.

Triage workflow

• Re-run locally with the same flags as CI (headless, same viewport, same env vars).
• Capture evidence on failure:
• console messages
• network requests with status codes
• screenshot at failure moment
• a “state dump” from a stable test seam (e.g., window.__TEST__.state())

Fix patterns (in order of leverage)

Readiness

• Add explicit “ready” signals; wait on them.
• Avoid broad waitForTimeout; prefer waitForFunction or DOM conditions.

Determinism

• Seed RNG, control time, and remove animation variability.
• For canvas/WebGL: fixed timestep mode for tests.

Isolation

• Reset storage between tests.
• Use unique test data; avoid shared accounts/state.
• Avoid depending on test execution order.

Environment

• Lock viewport/DPR/locale/timezone.
• Ensure fonts are installed/loaded deterministically (or use default system fonts).

Temporary guardrails

• Retries are acceptable only as a short-term measure with a tracked follow-up.

Phaser / Canvas / WebGL Testing Notes

Why canvas/WebGL tests get flaky

Common nondeterminism sources:

• variable frame times (CPU load, headless rendering)
• time-based movement/physics without fixed timestep
• RNG for loot/spawns/AI
• async asset loading and “first frame” races
• font loading differences affecting layout (if mixed DOM+canvas)

The fix is not “more retries”; it’s deterministic mode + explicit readiness.

Deterministic mode (recommended pattern)

When ?test=1 (or a build-time flag) is enabled:

• seed RNG from a known value (and expose it)
• freeze or control time (fixed timestep) for simulation-sensitive assertions
• disable camera shake, screen flash, particles, and audio if they introduce nondeterminism
• ensure assets are preloaded before the first interactive frame

Add a minimal test seam

Expose a stable API (avoid leaking internal Phaser objects unless wrapped):

• window.__TEST__ = { ready: false, seed, sceneKey, state: () => ({...}) }
• set ready = true only after:
• preload completed
• first scene created
• first render tick occurred (optional but helpful for screenshot timing)

If you need to expose entities, expose IDs + essential fields (x/y/hp/state), not sprite instances.

What to assert (avoid brittle assertions)

Prefer invariants that match player-visible behavior:

• “player can start” (scene key, UI state)
• “pressing attack spawns hitbox and reduces enemy HP”
• “collecting coin increments score”

Avoid:

• raw pixel-perfect sprite positions unless you fixed dt and RNG
• asserting on ordering of internally iterated arrays/maps

Screenshot testing: make it reliable

Before comparing screenshots:

• fix viewport size + device scale factor
• fix RNG seed + fixed dt
• wait on __TEST__.ready and optionally a __TEST__.frameCount >= N
• keep snapshots targeted (menus/first scene), not every frame

---

UI slicing regressions (nine-slice / ribbons / bars)

Canvas UI bugs (9-slice seams, padded-frame “side bars”, segmented ribbons, transparent HUD bases) are easiest to catch with a purpose-built UI harness scene rather than trying to reproduce in the full game flow.

Recommended harness pattern

Create a dedicated test page/scene (e.g., test.html) that:

• loads only UI assets
• renders each element on multiple backdrops (dark UI background + “world green” + paper/wood) to expose transparency problems
• renders raw frames and assembled output side-by-side
• supports keyboard toggles (1..N) and a programmatic seam via window.__TEST__.commands.showTest(n)

This is especially high-signal for 9-slice work because the failure mode is visual (gaps/bands), and the correct output is hard to assert via internal state alone.

What to render (minimum set)

• Raw frames: all 9 frames of a 3×3 sheet (catches loader frameWidth/spacing errors)
• Assembled panel: several target sizes (catches trim/overlap math issues)
• Ribbons/banners: show “raw crop+scale” and “stitched multi-slice” (catches internal transparent gutters)
• Bars: base + track + fill (cropped) over a “world” backdrop (catches transparent-window issues)

Playwright screenshot workflow (practical)

1. Start a static server for public/ 2. page.goto('/test.html') 3. await page.waitForFunction(() => window.__TEST__?.ready) 4. Switch modes: await page.evaluate(() => window.__TEST__.commands.showTest(5)) 5. Screenshot: page.screenshot({ fullPage: true })

For CI: store the screenshots and diff them with scripts/imgdiff.py (after making viewport/DPR deterministic).

Playwright MCP Cheatsheet (Codex)

This reference is for using the Playwright MCP tools during frontend testing tasks (especially canvas/WebGL games).

Mental Model

• Use MCP to reproduce a user flow and collect evidence: console, network, screenshots, and state.
• Prefer explicit readiness over time-based waits.
• Treat any console error (or failed asset request) as a product failure unless explicitly allowed.

High-signal tool patterns

Navigate + wait for app readiness

• mcp__playwright__browser_navigate to the URL.
• mcp__playwright__browser_wait_for when a specific text appears (DOM UIs).
• For canvas apps, prefer mcp__playwright__browser_evaluate to poll a window readiness flag:
• Example readiness seam (recommended in app code): window.__TEST__ = { ready: true }
• For DOM inspection, mcp__playwright__browser_snapshot is often higher-signal than raw HTML, and is safer than “guessing selectors”.

Assert state (white-box)

• mcp__playwright__browser_evaluate to read window.__TEST__:
• window.__TEST__.sceneKey, window.__TEST__.score, window.__TEST__.entities, etc.

Drive input

• mcp__playwright__browser_click / mcp__playwright__browser_drag for pointer flows.
• mcp__playwright__browser_press_key for keyboard-driven movement/combat.
• mcp__playwright__browser_type for text fields (menus, names, chats).

Fail fast on “silent” errors

• mcp__playwright__browser_console_messages and fail on any error-level messages.
• mcp__playwright__browser_network_requests and fail on non-2xx/3xx for required assets.

Visual evidence

• mcp__playwright__browser_take_screenshot for a deterministic frame (lock viewport/DPR first).
• If you need pixel diffs in-repo, use scripts/imgdiff.py.

Recommended “test seams” to add to the app

Minimal, stable, and read-only is best:

• window.__TEST__.ready (boolean)
• window.__TEST__.version (string)
• window.__TEST__.state() (returns JSON-serializable state snapshot)
• window.__TEST__.commands (optional small set of commands like reset(), seed(n), skipIntro())

Avoid exposing raw engine objects unless you also provide stable wrappers.