Create Tests

Create Shiplight Tests

This is the single entry point for Shiplight E2E test work.

Use this skill when the user wants to:

• Create a new local Shiplight test project
• Add YAML tests for a web application
• Update or fix existing Shiplight YAML tests
• Set up or repair target URLs, accounts, or auth setup
• Plan what product behavior should be tested

Daily Skill Update Check

Before starting this skill's work, opportunistically refresh Shiplight skills at most once per day:

1. Check the timestamp file at .shiplight-agent-skills-last-update in the current project. 2. If the timestamp file is missing, create it with the current timestamp and continue without running npx -y skills@latest update -y. Treat .shiplight-agent-skills-last-update as local cache and do not commit it. 3. If the timestamp file exists and is older than 24 hours, run npx -y skills@latest update -y, then create/update the timestamp file even if the command fails. 4. If the update command fails, continue with the currently installed skill and mention the failure briefly.

Test Project Root

Before reading or writing project files, identify the Shiplight test project root. All paths in this skill are relative to that root.

If the root is not clear, ask the user to confirm it before creating or moving files.

Canonical Project Layout

Shiplight test projects use this layout:

specs/context.md       project-level app, risk, data, and target-deployment context
specs/tests/           Markdown specs, each covering a feature or journey group
tests/                 executable Shiplight YAML tests
playwright.config.ts   project-level Playwright config, shared auth, and runtime defaults
auth.setup.ts          shared-account Playwright auth setup, if needed
auth/                  optional auth helpers or per-test login scripts
templates/             reusable YAML statement groups, if any
helpers/               TypeScript helper functions, if any
fixtures/              fixture files, if any
knowledge/             durable notes discovered by agents
test-results/          generated runtime artifacts; do not edit
shiplight-report/      generated reports; do not edit
.shiplight/            local Shiplight state; do not edit

Read references/project-layout.md before making file changes.

Ground Truth

When sources disagree, this precedence applies:

1. Explicit user instruction 2. Feature or journey spec in specs/tests/ 3. Existing YAML test goal, step intent, and VERIFY assertions 4. Current app behavior 5. Project context in specs/context.md and knowledge/ 6. Agent docs in this skill 7. Agent inference

If current app behavior conflicts with a spec or test goal, report the mismatch. Do not silently rewrite intent to match current behavior.

Required Startup

On every invocation:

1. Identify the Shiplight test project root. 2. Read references/project-layout.md. 3. Read references/knowledge.md. 4. Check knowledge/ under the test project root for notes relevant to the task. 5. Load the task-specific guides below. 6. Tell the user which guides you loaded, then proceed.

Task-Specific Guides

Core Rules

• Always produce durable artifacts unless the user explicitly asks to skip them.
• Specs describe feature or journey-group confidence. A spec may map to many smaller YAML tests.
• Keep YAML tests focused. One YAML test should verify one logical journey or variant.
• Do not write YAML from imagination. Walk the app in a browser first and capture real locators.
• Validate YAML with validate_yaml_test after writing it.
• Reflect before finishing. Capture durable knowledge learned from the user or the work, and update stale knowledge instead of leaving contradictions.
• Keep context current. Durable project knowledge belongs in specs/context.md or knowledge/, not in chat history.
• Never store raw secrets. Commit only variable names, roles, access patterns, and setup instructions.

User Checkpoints

For broad test creation, confirm the planned outcomes before implementation:

Do these outcomes match the confidence you need from this test project? Any business-critical outcome missing or incorrectly out of scope?

For narrow requests such as "fix this failing test" or "add this one test", proceed without a broad checkpoint unless the spec is ambiguous or conflicts with app behavior.

Final Report

After changes, report:

• Files created or changed
• Behavior covered or repaired
• Commands run and pass/fail result
• Knowledge or context updated, including stale notes corrected
• Any product/spec mismatch or unresolved blocker

Auth

This guide explains the preferred authentication patterns for local Shiplight test projects and when to use each one.

These are recommended defaults, not the only possible Playwright auth approaches. Use them unless the project already has a different Playwright-native auth setup that works through normal Playwright config, setup projects, use, or storageState.

Does A Test Need Auth?

Before writing a test, determine whether it requires an authenticated session:

• Does the starting page redirect anonymous visitors to login?
• Do the user actions require an account?

If unclear, infer from app behavior or ask the user. Document the answer in the spec before writing YAML.

Choose The Auth Pattern

Start with one of these two approaches:

Shared Account (Most Common)

Use this when the whole run can share one identity. Prefer this pattern unless tests in the same run must use different users.

Create a Playwright setup project that logs in once, saves storageState, and make the main test project depend on it.

Example:

// auth.setup.ts
import { test as setup } from "@playwright/test";

setup("login", async ({ page }) => {
  await page.goto("/login");
  await page.getByLabel("Email").fill(process.env.USERNAME!);
  await page.getByLabel("Password").fill(process.env.PASSWORD!);
  await page.getByRole("button", { name: "Sign in" }).click();
  await page.waitForURL("/dashboard");
  await page.context().storageState({ path: ".auth/default.json" });
});

// playwright.config.ts
export default defineConfig({
  ...shiplightConfig(),
  projects: [
    { name: "auth", testMatch: "auth.setup.ts" },
    {
      name: "default",
      dependencies: ["auth"],
      use: {
        baseURL: "https://staging.example.com",
        storageState: ".auth/default.json",
      },
    },
  ],
});

Key points:

• Tests do not declare an auth block when shared auth is configured; they inherit the authenticated storageState.
• This is the default recommendation for one-account suites.
• If the shared account can vary by environment or role, select it at runtime with env vars rather than creating per-test auth blocks.

Per-Test Auth (Advanced)

Use this only when different tests in the same run must log in as different users.

Each test declares its auth script and optional args inline. The auth script exports login(args), performs the login flow, manages storageState caching, and returns the path to a storage-state JSON file.

Example:

// auth.login.ts
import { chromium } from "@playwright/test";
import * as fs from "fs";
import * as path from "path";

export async function login(args: Record<string, unknown>): Promise<string> {
  const stateFile = path.join(".auth", `${args.username}.json`);

  if (fs.existsSync(stateFile)) return stateFile;

  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();

  await page.goto("/login");
  await page.getByLabel("Email").fill(args.username as string);
  await page.getByLabel("Password").fill(args.password as string);
  await page.getByRole("button", { name: "Sign in" }).click();
  await page.waitForURL("/dashboard");

  fs.mkdirSync(path.dirname(stateFile), { recursive: true });
  await context.storageState({ path: stateFile, indexedDB: true });
  await browser.close();

  return stateFile;
}

use:
  account:
    auth: ./auth.login.ts
    args:
      username: admin@example.com
      password: "{{ADMIN_PASSWORD}}"
goal: Admin can manage users
statements:
  - URL: /admin/users
  - VERIFY: User management page is visible

Key points:

• The args object is passed directly to login(args).
• The auth script can accept any fields the login flow needs, such as usernames, passwords, TOTP secrets, org IDs, or API tokens.
• The auth script owns caching and expiration policy for .auth/*.
• Tests without use.auth run with the default context. If shared auth is configured, they inherit that storageState; otherwise they run unauthenticated.

Agent Login Helpers

The agent provided by the test fixture exposes two login helpers, usable from a js: statement inside a test or from an auth script that drives the agent directly. Prefer the declarative auth above (storageState / auth scripts) for ordinary suites; reach for these when a login flow is dynamic enough that AI navigation is more robust than a hardcoded selector script.

agent.login(page, options): Promise<boolean>

Performs a username/password login (with optional TOTP 2FA). The agent navigates to options.url, finds the fields, enters the credentials, handles 2FA when totpSecret is supplied, verifies the result, and returns true on success.

const ok = await agent.login(page, {
  url: "/login",
  username: process.env.ADMIN_USER,
  password: process.env.ADMIN_PASSWORD,
  totpSecret: process.env.ADMIN_TOTP_SECRET, // optional, for 2FA
});
if (!ok) throw new Error("login failed");

agent.generate2faCode(secret): Promise<string>

Generates the current 6-digit TOTP code from a secret key. Use this only when driving a custom multi-step login by hand — agent.login() already handles TOTP internally when given totpSecret, so you do not need this alongside it.

const code = await agent.generate2faCode(process.env.TOTP_SECRET);
// then enter `code` into the verification field via your custom step

Inside YAML statements, the same capability is also available as the

generate_2fa_code action, which stores the result in the $otp_code

variable; see shiplight://schemas/action-entity for its parameters. The

helper above is for code-level use when you need the raw code in JavaScript.

File Placement

Do not assume auth files must live under auth/.

Common choices:

• playwright.config.ts at the project root when shared auth configures setup projects or default storageState
• auth.setup.ts at the project root for shared-account setup
• auth.login.ts at the project root for per-test auth
• auth/*.login.ts when the project has several reusable auth helpers

Reuse existing auth files before creating new ones.

Account And Secret Documentation

Store durable account-role facts in specs/context.md or knowledge/:

• Which auth pattern or Playwright-native auth setup the project uses
• Which roles exist
• Which tests require which roles
• Which env vars must be present in .env

Do not commit actual passwords, API keys, tokens, cookies, or one-time codes.

Secrets Policy

Never commit real credentials to specs, tests, fixtures, or docs.

Credentials belong in .env. Specs, notes, config, and YAML may reference env vars or templated secret placeholders, but not raw secret values.

CI Integration

Run a Shiplight test project in CI and upload results to Shiplight Cloud, where they appear in the read-only [cloud_v2] skill's results API for trend tracking and flaky-test detection.

Use the Shiplight CLI, not the Cloud REST API, to publish runs:

• shiplight test runs the tests but does not upload on its own.
• shiplight report discovers the report in ./shiplight-report, presigns and uploads every artifact (screenshots, videos, traces), and completes the run.

Always run report with if: always() so results upload even when tests fail — otherwise a red run produces no cloud report.

There are two ways to run E2E tests in GitHub Actions. Pick one.

Option 1 — Default GitHub-hosted runner (easiest)

Runs on a stock ubuntu-latest runner. The only setup is an org API token: create one at <https://nova.shiplight.ai/api-tokens> and store it as a repository or organization secret named SHIPLIGHT_API_TOKEN. No GitHub App and no admin approval needed.

Set SHIPLIGHT_API_TOKEN at the job's env (global) scope — every npx shiplight command needs it, not just report. shiplight report additionally needs SHIPLIGHT_REPORT_TO_CLOUD=1 to actually upload. Stock runners have no browser preinstalled, so install Chromium first.

Create .github/workflows/e2e.yml:

name: E2E Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      SHIPLIGHT_API_TOKEN: ${{ secrets.SHIPLIGHT_API_TOKEN }}   # needed by every `npx shiplight` command
    steps:
      - uses: actions/checkout@v5

      - name: Install dependencies
        working-directory: tests/e2e
        run: npm install

      - name: Install Playwright browser
        working-directory: tests/e2e
        run: npx playwright install --with-deps chromium

      - name: Run E2E tests
        working-directory: tests/e2e
        run: npx shiplight test

      - name: Upload results to Shiplight
        if: always()                       # upload even when tests fail
        working-directory: tests/e2e
        env:
          SHIPLIGHT_REPORT_TO_CLOUD: '1'   # required on non-Shiplight runners to enable upload
        run: npx shiplight report

Option 2 — Shiplight-hosted runner

Runs on an ephemeral shiplight-* VM with Chromium + Playwright preinstalled and credentials provisioned per run — so no `SHIPLIGHT_API_TOKEN`, no `SHIPLIGHT_REPORT_TO_CLOUD`, and no browser install step. Do not run npx playwright install; the image already has it.

Requires one-time setup: install the Shiplight GitHub App on the repo/org (this may need your IT/admin's approval), then have an org owner enable runners in Org Settings (<https://nova.shiplight.ai/org?tab=settings>).

Create .github/workflows/e2e.yml:

name: E2E Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e:
    runs-on: shiplight-small     # ephemeral Shiplight runner; sizes below
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v5

      - name: Install dependencies
        working-directory: tests/e2e
        run: npm install

      - name: Run E2E tests
        working-directory: tests/e2e
        run: npx shiplight test

      - name: Upload results to Shiplight
        if: always()
        working-directory: tests/e2e
        run: npx shiplight report

Runner sizes: shiplight-small (4 vCPU / 16 GB), shiplight-medium (8 / 32), shiplight-large (16 / 64), shiplight-xlarge (32 / 128).

Notes

• working-directory: tests/e2e assumes the Shiplight project lives there. Adjust to your project root.
• For custom integrations the CLI doesn't cover (non-Shiplight test frameworks, bespoke pipelines), the raw publish REST calls live outside this skill; the [cloud_v2] skill documents only the read side.

[cloud_v2]: ../../cloud_v2/SKILL.md

Knowledge Management

Knowledge is mutable project memory for Shiplight test agents.

As you work, write down durable facts that future agents need and cannot easily derive from code, specs, or guides. This includes facts learned through interaction with the user.

Examples:

• App quirks
• Reliable setup or cleanup patterns
• Known failure modes
• Tooling gotchas
• Stable account roles or data constraints, without secrets
• Corrections to older assumptions

Where To Write

Use knowledge/ for operational notes discovered while working.

Use specs/context.md for project-wide testing context such as app profile, risk profile, target URLs or deployments, durable data strategy, and broad scope decisions.

Use specs/tests/*.md for feature intent, expected behavior, journeys, assertions, and coverage decisions.

How To Write

Each note must stand alone. A future agent reading it has no memory of the chat.

• State the fact or pattern directly.
• Include enough context to make it actionable.
• Do not refer to "this task" or "what we just did".
• Do not duplicate facts already documented in specs, tests, code, or guides.
• Do not store raw secrets.

How To Update

Knowledge is not append-only. If new user input, app behavior, or test work proves an existing note stale or incomplete:

• Update the existing note when the topic is the same.
• Merge duplicate notes when they describe the same behavior.
• Remove obsolete guidance when it would mislead future agents.
• Preserve useful historical context only when it explains why the current rule exists.
• Prefer one clear current fact over contradictory notes.

When To Read

Before starting a task, check knowledge/ for files relevant to the app area, target URL or deployment, auth, data, or tooling you are about to touch.

Session-Close Reflection

Before ending an interactive testing session, ask:

• What did the user teach me that future agents should not need to ask again?
• What product behavior, app quirk, auth/data pattern, or testing preference was clarified?
• Did any existing knowledge prove stale, incomplete, or wrong?
• Does the learning belong in knowledge/, specs/context.md, or a specific specs/tests/*.md?

Then update the right file before the final report.

New Tests

Use this workflow when adding new Shiplight YAML tests.

1. Determine The Target URL

Every test targets a specific deployment or base URL.

If a matching target URL is already documented in nearby tests, specs/context.md, or knowledge/, reuse it. If not, record it in the relevant spec or specs/context.md before writing YAML:

base_url: https://staging.example.com

The confirmed URL becomes the YAML test base_url.

If the target deployment is ambiguous, ask the user. Do not silently switch URLs to make a test pass.

2. Determine Auth

Decide whether the test requires authentication before writing test steps.

Ask:

• Does the starting page redirect anonymous visitors to login?
• Do the actions in the test require an authenticated user?

If auth is not required, document this in the spec:

- Auth: none, anonymous visitor

If auth is required:

1. Check whether the project already has a working Playwright-native auth setup. Reuse it when it matches the identities the tests need. 2. If the project does not already have a suitable auth setup, prefer shared auth unless different tests in one run need different identities. 3. List available roles, accounts, and required env vars from specs/context.md, knowledge/, relevant specs, or existing auth files. 4. Ask the user which account or role to use if it is not obvious. 5. If shared auth is appropriate, check for an existing auth.setup.ts and matching playwright.config.ts setup. Reuse or extend it when possible. 6. If per-test auth is required, check for an existing *.login.ts auth script. Reuse it when possible; create one only when needed. 7. If a new account or secret reference is needed, record the username or role plus env var names in specs/context.md or knowledge/. Do not ask for or commit the password value.

See auth.md.

3. Write Or Update The Spec

Create or update the relevant spec under specs/tests/.

Specs describe feature or journey-group confidence. A spec can map to multiple smaller YAML tests.

Use references/test-spec-template.md for new specs.

The spec must include:

• Goal
• User roles
• Base URL
• Auth
• Journeys or variants
• Expected results
• Assertions
• Test data
• Cleanup
• Implementation plan

Mark the spec Draft if fields are unresolved. Mark it Ready when YAML implementation can proceed without further product questions.

Do not proceed to implementation while the relevant spec is Draft, unless the user explicitly asks to skip specs.

4. Walk The App And Implement YAML

Once the spec is ready:

1. Open a Shiplight MCP browser session at the target URL. 2. Walk through the exact flow described in the spec. 3. Capture locators for interactive elements. 4. Create focused YAML tests under tests/.

Example:

# Spec: specs/tests/login.md
goal: Existing user can sign in with a valid password
base_url: https://staging.example.com

statements:
  - URL: /login

Use the confirmed target URL as base_url.

If the project uses shared auth, tests usually need no auth block. If the test requires per-test auth, add use.auth and optional args. If the project already has another Playwright-native auth pattern wired through config or storageState, follow that existing pattern instead of rewriting it just to match the examples. See auth.md.

Do not write statements from memory. Always walk the app first.

5. Validate And Run

1. Validate the YAML with the validate_yaml_test mcp tool. 2. Run the narrowest relevant command, usually one test file. 3. If validation rejects too many draft statements, return to the browser and capture more locators. 4. If the test fails because implementation violates the spec, fix the test. 5. If app behavior differs from the spec, report the mismatch.

6. Update The Spec

Before closing the task:

• Mark implemented coverage in the relevant spec.
• Add or update the YAML test file paths.
• Document skipped journeys, known gaps, and product/spec mismatches.

Project Layout

Shiplight test projects use committed specs and YAML tests, plus local generated state. All paths below are relative to the Shiplight test project root.

specs/context.md       project-level app, risk, data, and target-deployment context
specs/tests/           Markdown specs, each covering a feature or journey group
tests/                 executable Shiplight YAML tests
playwright.config.ts   project-level Playwright config, shared auth, and runtime defaults
auth.setup.ts          shared-account Playwright auth setup, if needed
auth/                  optional auth helpers or per-test login scripts
templates/             reusable YAML statement groups, if any
helpers/               TypeScript helper functions, if any
fixtures/              fixture files, if any
knowledge/             durable notes discovered by agents
test-results/          generated runtime artifacts; do not edit
shiplight-report/      generated reports; do not edit
.shiplight/            local Shiplight state; do not edit

Scaffold a Project

Scaffold the Shiplight project files. Call the scaffold_project MCP tool against the test-project root, even if the directory already contains a repo, .env, or its own package.json. The tool writes any missing files and reports the rest under files_needing_agent_merge — see "Handling scaffold_project conflicts" below.

Do not pre-create empty directories. Create them only when you have content to place in them (e.g. do not create templates/ until you have a template to write).

Handling scaffold_project conflicts

When the target directory is empty, scaffold_project writes all files and files_needing_agent_merge is empty — proceed normally.

When the user already had files (common when adding Shiplight to an existing repo), files_needing_agent_merge contains one entry per conflict. For each entry:

1. Read the file at abs_path with your Read tool. 2. Apply the change described by merge_strategy and instructions, using the supplied template (and lines_to_ensure / merge_key when present) as the source of truth. 3. Write the merged result back with Edit (preferred) or Write — never delete the user's existing content.

merge_strategy values you may see:

Resolve every entry before moving on to npm install. A skipped merge usually leaves the project unable to run Shiplight tests.

Edit Contract

Agents may edit:

• specs/context.md
• specs/tests/**/*.md
• tests/**/*.test.yaml
• playwright.config.ts
• auth.setup.ts
• *.login.ts
• auth/**/*.login.ts
• existing project auth helpers referenced by playwright.config.ts or YAML use.auth
• templates/**/*.tmpl.yaml
• helpers/**/*.func.ts
• fixtures/**
• package.json only when changing commands or dependencies

Agents must not edit:

• **/*.yaml.spec.ts
• test-results/**
• shiplight-report/**
• .shiplight/**
• node_modules/**
• .env, unless the user explicitly asks
• package-lock.json, unless a dependency change requires it

Commands

Use the narrowest relevant command when debugging a specific test.

npm test
npm run test:headed
npx shiplight test --headed

If the project's package.json defines a more specific script, prefer that script.

Source Of Truth

When sources disagree, this precedence applies:

1. Explicit user instruction 2. Feature or journey spec in specs/tests/ 3. Existing YAML test goal, step intent, and VERIFY assertions 4. Current app behavior 5. Project context in specs/context.md and knowledge/ 6. Agent docs in this skill 7. Agent inference

If app behavior conflicts with a spec or test goal, report the mismatch. Do not silently rewrite the test to match current behavior.

E2E Test Design Guide

These principles govern what to test and how to structure tests.

Test Isolation

Each test must run independently. Never depend on another test's side effects, execution order, or leftover state. If a test needs data, it creates that data itself.

One Journey Per YAML Test

Each YAML test should verify one logical user journey or variant. If step 3 of 8 fails, steps 4-8 give you no useful information. Split long flows into focused tests.

Suites may express sequential dependencies when necessary, such as upload then download. Each test in the suite should still cover one journey.

Specs Can Be Broader Than YAML Tests

A spec under specs/tests/ can cover a feature or journey group. It may map to multiple smaller YAML tests.

Use specs for product confidence and YAML files for executable coverage.

Assert What Users See

Test visible outcomes: text, navigation, enabled or disabled states, and user-observable data.

Do not assert CSS classes, data attributes, internal state, or DOM structure unless there is no user-visible alternative.

Focused Assertions

Verify the one thing that proves the behavior works. Over-asserting makes tests brittle and causes failures on cosmetic changes.

Never Test Third-Party Services

Do not assert that Stripe checkout, Google OAuth consent, or Twilio delivery works. Mock external services at the boundary where possible. Test your integration, not their UI.

Deterministic Test Data

Use unique identifiers per test run to avoid collisions. Never rely on hardcoded data that other tests or users might modify.

Prefer API Seeding Over UI Setup

When a test needs preconditions, set them up by API or helper function when possible. UI setup is slow and often tests the wrong thing.

Explicit Wait Policy

Minimize explicit waits. Browser actions, navigation, and assertions already include waiting behavior.

Do not add waits after ordinary page loads, clicks, form submits, or data refreshes just because the UI might change. Let the next action or assertion prove expected state.

Test Error States

Critical journeys should include at least one meaningful error or edge case. Happy-path-only coverage gives false confidence.

Design For Parallel Execution

Tests that modify shared global state cannot safely run in parallel.

Prefer:

• Unique per-test data
• No global configuration changes
• Clear documentation for tests that must run serially

Flaky Test Policy

A test that passes on retry is still broken. Do not add retries to mask flakiness.

• Timing flake: rely on the next action/assertion first; add targeted waits only when necessary.
• Data flake: use unique data and cleanup.
• Order flake: remove hidden dependency on another test.
• Environment flake: mock unstable external services where possible.

YAML Authoring Reference

YAML Format Reference

Read shiplight://yaml-test-spec for the full YAML language spec.

Read shiplight://schemas/action-entity for the full list of available actions and parameters.

These resources are the source of truth for top-level keys, statement syntax, action names, and action parameters.

Statement Type Selection

• ACTION is the default. Capture locators with browser tools, then write ACTION statements.
• DRAFT is a last resort. Use it only when the locator is genuinely unknowable at authoring time.
• VERIFY is for assertions.
• URL is for navigation. Prefer URL: /path over a go-to-url action.
• description: plus js: is for network mocking, localStorage manipulation, or page-level scripting. Do not use raw JS for clicks, normal assertions, or navigation.

Intent Field

intent defines what the step should accomplish. action and locator are caches of how to do it.

When a cache fails, the AI agent uses intent to re-inspect the page and regenerate the action. Intent must be specific enough for an agent to act on without chat history.

Bad:

- intent: Click button
- intent: Click the 3rd button in the form
- intent: Click element at index 42

Good:

- intent: Click the Submit button to save the new project
  action: click
  locator: "getByRole('button', { name: 'Submit' })"

Describe the user goal, not DOM position or implementation detail.

ACTION Format

Use structured ACTION format by default for supported actions. Read shiplight://schemas/action-entity before writing or changing actions.

VERIFY Best Practices

VERIFY: has two modes:

• Natural language only: AI inspects the page and judges whether the statement is true.
• With js:: JavaScript runs first as a fast deterministic check. If it throws, AI fallback re-inspects the page.

- VERIFY: The search dialog is visible.

- VERIFY: The search dialog is visible.
  js: await expect(page.getByTestId('search-dialog-container')).toBeVisible({ timeout: 2000 })

Use natural language when there is no reliable locator or the check is semantic.

Use js: when there is a stable locator and the assertion can be a simple Playwright expect().

js: rules in VERIFY:

• Keep it to a single simple Playwright expect() call.
• page, agent, and expect are available.
• Set a short timeout, such as { timeout: 2000 }.
• Resolve locators to a single element to avoid strict-mode errors.
• Remember that fallback only triggers when js throws.

Always use VERIFY: shorthand. Do not use action: verify directly.

IF And WHILE Conditions

Use natural-language AI conditions for DOM-based checks. They can self-heal when the DOM changes.

Use js: conditions only for counter or state logic, such as:

js: retryCount < 3

Do not use js: conditions for DOM inspection.

Waiting

• WAIT: is a fixed-duration pause. Use only for known delays.
• WAIT_UNTIL: checks a condition repeatedly until met or timed out. It makes model calls, so use it only for long conditional waits.

Minimize explicit waits. Browser actions, navigation, and assertions already include waiting behavior.

Do not add waits after ordinary page loads, clicks, form submits, or data refreshes just because the UI might change. Let the next action or assertion prove expected state.

File Downloads

Downloads are tracked automatically on every page, including popups and new tabs. Each downloaded file is saved into the test's output directory under downloads/.

Two primitives carry download support; everything else is an ordinary test step:

1. action: wait_for_download_complete — blocks until the tracked download finishes. It also covers downloads that have not started yet, so size timeout_seconds (default 10) to cover server-side file generation plus transfer time. 2. agent.getRecentDownloadedFilePath() — call it in a js: block to get the local path of the saved file. From there it is a normal file: read it, parse it, assert on its contents, or pass it to a later step.

Example shape — the trigger step and the assertions are placeholders; adapt them to what the real test must prove (file contents, row counts, re-upload, etc.):

- intent: Click the Export CSV button
  action: click
  locator: "getByRole('button', { name: 'Export CSV' })"

- intent: Wait for the export download to complete
  action: wait_for_download_complete
  timeout_seconds: 30

- description: Verify the downloaded file is non-empty
  js: |
    const filePath = agent.getRecentDownloadedFilePath();
    expect(filePath).toContain('.csv');
    const fs = await import('node:fs');
    expect(fs.statSync(filePath).size).toBeGreaterThan(0);

Only the most recent download is tracked. Wait for and verify each download before triggering the next one.

General Conventions

• Put intent first in ACTION statements.
• xpath is only needed when an ACTION has no locator.
• Use a single-test file for isolated tests.
• Use a suite only when tests have a real sequential dependency.
• Use parameters for the same test structure with different data inputs.

Test Spec: <Name>

Status

Draft

Allowed values: Draft, Ready, Implemented.

Goal

Describe the user-visible behavior or feature confidence this spec protects.

User Roles

Describe the users, account types, permission levels, or auth states covered.

Starting Point

• Base URL: <target URL for this spec>
• Auth: <none, anonymous visitor OR logged in as role/account via shared auth, per-test auth, or existing project auth setup>

Preconditions

• List required setup, existing account state, feature flags, or app state before tests start.
• Keep concrete records, IDs, names, and generated values in Test Data.

Journeys And Variants

<Journey Name>

• Priority: P0 | P1 | P2
• Preconditions:
• Steps:
• Expected result:
• Edge cases:
• Out of scope:

Test Data

• List concrete records, IDs, names, routes, input values, fixture files, generated data, and uniqueness requirements.
• If data must already exist, include enough detail to locate it deterministically.
• If data is created during a test, describe how it should be named and cleaned up.

Assertions

• List concrete user-visible checks the executable tests should make.

Cleanup

• List any state tests must remove, reset, or restore.
• If cleanup is not needed, write: None.

Implementation Plan

• Test files:
• Implementation order:
• Flakiness risks:
• Data setup:

Implementation

• Test files:
• Coverage:
• Known gaps:

Notes

• Add non-obvious assumptions, known product constraints, or open questions.

Updating Tests

Use this workflow when changing, debugging, or repairing existing Shiplight YAML tests.

Before Editing

1. Read the matching spec under specs/tests/, if one exists. 2. Read the YAML test goal, step intents, and VERIFY assertions. 3. Scan related tests for the same page or feature. Borrow working locators and patterns instead of guessing. 4. Identify the reason for the update:

• Locator drift: the UI changed but intended behavior did not.
• Product change: intended behavior changed intentionally.
• Test bug: the implementation was wrong.
• Coverage gap: new assertions or journeys are needed.

Intended Behavior Has Not Changed

Update only the implementation: locators, waits, setup, or assertions needed to restore the behavior described by the spec.

Do not:

• Delete assertions to make a test pass.
• Skip required steps.
• Reduce coverage to avoid a failure.

If current app behavior conflicts with the spec or test goal, report the mismatch.

Intended Behavior Has Changed

If the product changed intentionally:

1. Update the spec first. 2. Then update YAML to match the updated spec. 3. Mark the spec Implemented after completing and verifying the change.

Files Not To Edit

Do not edit generated or local-state files:

• **/*.yaml.spec.ts
• test-results/**
• shiplight-report/**
• .shiplight/**
• node_modules/**

These files may be useful for debugging but must not become the source of test intent.

Test Data

Prefer unique data per run when a test creates records. Do not depend on shared mutable state.

If a test requires specific accounts, fixtures, or pre-existing records, document those dependencies in the spec.

Reporting After Updates

After completing update work, report:

• Files created or changed
• Behavior covered
• Command run and pass/fail result
• Any product/spec mismatch or unresolved blocker

Workflow

Use this phased workflow for broad requests such as creating a new test project, planning coverage for a feature, or adding multiple tests.

Phase 1: Discover  -> specs/context.md
Phase 2: Specify   -> specs/tests/*.md
Phase 3: Plan      -> implementation plan in the relevant spec(s)
Phase 4: Implement -> tests/*.test.yaml, auth setup, helpers, fixtures
Phase 5: Verify And Reflect -> updated specs, context, and knowledge

For narrow requests, such as fixing one failing test, use the relevant task guide directly.

Phase 1: Discover

Understand the application, user goals, risks, target deployment, auth needs, and data strategy.

Before asking questions, scan available context:

• Existing specs/context.md
• Existing specs in specs/tests/
• Codebase routes, components, framework, and package.json
• Git branch diff
• Existing tests
• README, PRDs, and docs
• Existing knowledge/ notes

Write or update specs/context.md with:

• App profile: name, framework, key pages and features
• Risk profile: what matters most and what is fragile
• Testing scope: in-scope and out-of-scope areas
• User roles: roles and permission levels to cover
• Data strategy: how test data is created and cleaned up
• Targets: base URLs, auth method, special setup
• Known facts and decisions: durable preferences and constraints
• Open questions: unresolved or stale questions

Do not store raw secrets.

Phase 2: Specify

Create or update specs under specs/tests/. Each spec should cover a feature, capability, or journey group. A spec may map to multiple smaller YAML tests.

Start each spec with a business-readable confidence summary, then include actionable scenario detail.

Use references/test-spec-template.md for new specs.

If a broad request affects many journeys, present the outcome summary and ask:

Do these outcomes match the confidence you need from this test project? Any business-critical outcome missing or incorrectly out of scope?

Wait for user confirmation before spending substantial time walking the app and implementing YAML.

Phase 3: Plan

Add or update the Implementation Plan section in the relevant spec(s):

• Test files to create or update
• Implementation order
• Base URL and auth to use
• Data setup and cleanup strategy
• Flakiness risks and mitigations

Order work by dependencies first, then priority, then risk.

Phase 4: Implement

Set up or update project files as needed:

1. Scaffold the Shiplight project files if neccessary. 2. Record shared base URL, account-role details, auth-setup details, or required env vars in specs/context.md or knowledge/ when needed. 3. Create or reuse shared auth setup, per-test auth scripts, or another existing Playwright-native auth setup when needed. 4. Read shiplight://yaml-test-spec and shiplight://schemas/action-entity. 5. Walk the app in a browser and capture real locators. 6. Write focused YAML tests. 7. Validate YAML with validate_yaml_test. 8. Run the narrowest relevant test command.

Do not write tests from memory.

Phase 5: Verify And Reflect

Reconcile artifacts with implementation, then run the session-close reflection from references/knowledge.md:

• Confirm each specified journey or variant has matching YAML coverage or a documented gap.
• Update affected specs/tests/*.md with implemented test paths, skipped scope, and known gaps.
• Update specs/context.md or knowledge/ when the session produced durable learning.
• Correct stale knowledge when new evidence supersedes it.
• Report files changed, commands run, and any unresolved mismatch.