Scraper Builder

Scraper Builder

Generate complete, runnable web scraper projects using the PageObject pattern with Playwright and TypeScript. This skill produces site-specific scrapers with typed data extraction, Docker deployment, and optional agent-browser integration for automated site analysis.

When to Use This Skill

Use this skill when:

• Building a site-specific web scraper for data extraction
• Generating PageObject classes for a target website
• Scaffolding a complete scraper project with Docker support
• Using agent-browser to analyze a site and auto-generate selectors
• Creating reusable scraping components (pagination, data tables)

Do NOT use this skill when:

• Building API clients (use HTTP client libraries directly)
• Writing QA/E2E test suites (use Playwright test runner with test-focused patterns)
• Mass crawling or spidering entire domains (use Crawlee or Scrapy)
• Scraping sites that require authentication bypass or CAPTCHA solving

Core Principles

1. PageObject Encapsulation

Each page on the target site maps to one PageObject class. Locators are defined in the constructor, and scraping logic lives in methods. Page objects never contain assertions or business logic — they extract and return data.

2. Selector Resilience

Prefer selectors in this order: data-testid > id > semantic HTML (role, aria-label) > structured CSS classes > text content. Avoid positional selectors (nth-child) and layout-dependent paths. See references/playwright-selectors.md for the full hierarchy.

3. Composition Over Inheritance

Reusable UI patterns (pagination, data tables, search bars) are modeled as component classes that page objects compose via properties. Only BasePage uses inheritance — everything else composes.

4. Typed Data Extraction

All scraped data flows through Zod schemas for validation. This catches selector drift (when a site changes its markup) at extraction time rather than downstream. See assets/templates/data-schema.ts.md.

5. Docker-First Deployment

Generated projects include a Dockerfile using Microsoft's official Playwright images and a docker-compose.yml with volume mounts for output data and debug screenshots. This ensures consistent browser environments across machines.

Generation Modes

Mode 1: Agent-Browser Analysis

Use agent-browser to navigate the target site, capture accessibility tree snapshots, and automatically discover selectors. This is the preferred mode when the agent has access to the agent-browser CLI.

Prerequisites: If agent-browser is not already installed, add it as a skill first:

npx skills add vercel-labs/agent-browser

Workflow:

# 1. Open the target page
agent-browser open https://example.com/products

# 2. Capture interactive snapshot with element references
agent-browser snapshot -i --json > snapshot.json

# 3. Capture scoped sections for focused analysis
agent-browser snapshot -i --json -s "main" > main-content.json
agent-browser snapshot -i --json -s "nav" > navigation.json

# 4. Test dynamic behavior (pagination, load-more)
agent-browser click @e3
agent-browser wait --load networkidle
agent-browser snapshot -i --json > after-click.json

# 5. Close when done
agent-browser close

What the agent does with snapshots:

1. Parse element references (@e1, @e2, etc.) and their roles 2. Group elements by semantic purpose (navigation, data display, forms, actions) 3. Map data elements to fields (title, price, image, etc.) 4. Generate PageObject classes with discovered selectors 5. Identify pagination and dynamic loading patterns

See references/agent-browser-workflow.md for the complete workflow reference.

Mode 2: Manual Description

The user describes the target site's page structure and the agent maps it to page objects. The agent asks structured questions:

1. What pages to scrape? — List of URLs or page types 2. What data to extract? — Field names and expected types per page 3. How is data paginated? — Numbered pages, load-more, infinite scroll, or single page 4. What selectors are known? — Any CSS selectors, data-testid values, or XPath the user already knows

The agent then:

• Matches the description to a site archetype from data/site-archetypes.json
• Proposes a page object map with class names and responsibilities
• Generates code after the user confirms the plan

Mode 3: Full Project Scaffold

Generate a complete runnable project in one operation using the scaffolder script:

deno run --allow-read --allow-write scripts/scaffold-scraper-project.ts \
  --name "my-scraper" \
  --url "https://example.com" \
  --pages "ProductListing,ProductDetail" \
  --fields "title,price,image_url,description"

This produces a project with all source files, configuration, Docker setup, and an entry point ready to run. See the Scripts Reference section for full options.

Quick Reference

Generation Process

Follow this sequence when generating a scraper:

Step 1: Gather Requirements

Ask the user for:

• Target site URL(s)
• Data fields to extract
• Number of pages/items expected
• Output format preference (JSON, CSV, both)
• Whether Docker deployment is needed

Step 2: Analyze the Site

Use Mode 1 (agent-browser) or Mode 2 (manual description) to understand:

• Page structure and navigation flow
• Data element locations and selector strategies
• Pagination or infinite scroll patterns
• Dynamic content loading behavior

Step 3: Design the Page Object Map

Create a plan listing:

• Each PageObject class and its URL pattern
• Component classes needed (Pagination, DataTable, etc.)
• Data schema fields and types per page
• The scraper's navigation flow between pages

Step 4: Present the Plan

Show the user the page object map before generating code. Include class names, field names, and the execution flow. Wait for confirmation.

Step 5: Generate Code

Use the templates in assets/templates/ as the foundation:

• base-page.ts.md — BasePage abstract class
• page-object.ts.md — Site-specific page object
• component.ts.md — Reusable components
• scraper-runner.ts.md — Orchestrator
• data-schema.ts.md — Zod validation schemas

Step 6: Deliver

Provide the complete project with:

• All source files
• Configuration files from assets/configs/
• A README explaining how to run it
• Docker setup (unless explicitly excluded)

Code Patterns

BasePage

Abstract class providing navigate(), waitForPageLoad(), screenshot(), and getText() helpers. All page objects extend this.

export abstract class BasePage {
  constructor(protected readonly page: Page) {}
  async navigate(url: string): Promise<void> { /* ... */ }
  async screenshot(name: string): Promise<void> { /* ... */ }
}

See: assets/templates/base-page.ts.md

PageObject

Site-specific class with locators as readonly properties, scrape methods returning typed data, and navigation methods for multi-page flows.

export class ProductListingPage extends BasePage {
  readonly productCards: Locator;
  readonly nextButton: Locator;
  async scrapeProducts(): Promise<Product[]> { /* ... */ }
  async goToNextPage(): Promise<boolean> { /* ... */ }
}

See: assets/templates/page-object.ts.md

Component

Reusable UI pattern (Pagination, DataTable) that receives a parent locator scope and provides extraction methods.

export class Pagination {
  constructor(private page: Page, private scope: Locator) {}
  async hasNextPage(): Promise<boolean> { /* ... */ }
  async goToNext(): Promise<void> { /* ... */ }
}

See: assets/templates/component.ts.md

ScraperRunner

Orchestrator that launches the browser, creates page objects, iterates through pages, collects data, validates with schemas, and writes output.

export class SiteScraper {
  async run(): Promise<void> {
    const browser = await chromium.launch();
    const page = await browser.newPage();
    // navigate, scrape, validate, write
  }
}

See: assets/templates/scraper-runner.ts.md

DataSchema

Zod schemas that validate scraped records, catching selector drift and malformed data at extraction time.

export const ProductSchema = z.object({
  title: z.string().min(1),
  price: z.number().positive(),
});

See: assets/templates/data-schema.ts.md

Anti-Patterns

See references/anti-patterns.md for the extended catalog with examples and fixes.

Scripts Reference

scaffold-scraper-project.ts

Generate a complete scraper project:

deno run --allow-read --allow-write scripts/scaffold-scraper-project.ts [options]

Options:
  --name <name>       Project name (required)
  --path <path>       Target directory (default: ./)
  --url <url>         Target site base URL
  --pages <pages>     Comma-separated page names (e.g., ProductListing,ProductDetail)
  --fields <fields>   Comma-separated data fields (e.g., title,price,rating)
  --no-docker         Skip Docker setup
  --no-validation     Skip Zod validation setup
  --json              Output as JSON
  -h, --help          Show help

Examples:
  # Scaffold a product scraper
  deno run --allow-read --allow-write scripts/scaffold-scraper-project.ts \
    --name "shop-scraper" --url "https://shop.example.com" \
    --pages "ProductListing,ProductDetail" --fields "title,price,image_url"

  # Minimal scraper without Docker
  deno run --allow-read --allow-write scripts/scaffold-scraper-project.ts \
    --name "blog-scraper" --no-docker

generate-page-object.ts

Generate a single PageObject class for an existing project:

deno run --allow-read --allow-write scripts/generate-page-object.ts [options]

Options:
  --name <name>           Class name (required)
  --url <url>             Page URL (for documentation comment)
  --fields <fields>       Comma-separated data fields
  --selectors <json>      JSON map of field to selector
  --with-pagination       Include pagination methods
  --output <path>         Output file path (default: stdout)
  --json                  Output as JSON
  -h, --help              Show help

Examples:
  # Generate a page object with known selectors
  deno run --allow-read --allow-write scripts/generate-page-object.ts \
    --name "ProductListing" --url "https://shop.example.com/products" \
    --fields "title,price,rating" \
    --selectors '{"title":".product-title","price":".product-price","rating":".star-rating"}' \
    --with-pagination --output src/pages/ProductListingPage.ts

  # Quick generation to stdout
  deno run --allow-read scripts/generate-page-object.ts \
    --name "SearchResults" --fields "title,url,snippet"

Templates & References

Templates (assets/templates/)

Configs (assets/configs/)

References (references/)

Examples (assets/examples/)

Data Files (data/)

Example Interaction

User: "I need a scraper for an online bookstore. I want to get book titles, authors, prices, and ratings from the catalog pages."

Agent workflow:

1. Checks site-archetypes.json — matches ecommerce archetype 2. Proposes page object map:

• BookListingPage — catalog with pagination
• BookDetailPage — individual book page (if detail scraping needed)
• Pagination component — shared pagination handler

3. Presents the plan with field mapping:

• title → [itemprop="name"] or .book-title
• author → [itemprop="author"] or .book-author
• price → [itemprop="price"] or .price
• rating → .star-rating or [data-rating]

4. After confirmation, generates using the scaffold script or manual code generation 5. Delivers project with Docker setup and Zod schemas for Book type

Integration

This skill connects to:

• typescript-best-practices — TypeScript coding patterns used in generated code
• devcontainer — Development container setup for the generated project
• agent-browser — Site analysis and selector discovery (external tool)

What You Do NOT Do

This skill does NOT:

• Bypass authentication or login walls
• Solve CAPTCHAs or bot detection
• Generate JavaScript-only output (always TypeScript)
• Produce crawlers that spider entire domains
• Create scrapers that violate robots.txt
• Handle rate-limited APIs (use HTTP clients for API work)
• Generate test suites (use Playwright test patterns for QA)

Docker Compose Template

Docker Compose configuration for running the scraper with volume mounts for output data and debug screenshots.

Template

services:
  scraper:
    build: .
    environment:
      - NODE_ENV=production
      - BASE_URL=${BASE_URL:-https://example.com}
      - HEADLESS=true
      - MAX_PAGES=${MAX_PAGES:-10}
      - REQUEST_DELAY=${REQUEST_DELAY:-1000}
      - OUTPUT_DIR=/app/data
    volumes:
      # Persist scraped data on the host
      - ./data:/app/data
      # Persist debug screenshots on the host
      - ./screenshots:/app/screenshots
    # Prevent runaway browser processes from consuming all resources
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: '2'

Usage

# Build and run
docker compose up --build

# Run with custom URL
BASE_URL=https://shop.example.com docker compose up

# Run in background
docker compose up -d

# View logs
docker compose logs -f scraper

# One-off run (remove container after)
docker compose run --rm scraper

Customization Notes

• Environment variables: Override at runtime via shell exports or a .env file in the project root. Docker Compose automatically reads .env.
• Resource limits: Adjust memory and cpus based on target site complexity. Heavy JavaScript sites need more memory.
• Shared memory: If Chromium crashes with out-of-memory errors, add shm_size: '512mb' under the service.
• Networking: Add a networks section if the scraper needs to communicate with other services (proxy, database).
• Restart policy: Add restart: on-failure for scheduled/recurring scrapes.

See Also

• dockerfile.md — Dockerfile template
• ../../references/docker-setup.md — Full Docker setup guide

Dockerfile Template

Multi-stage Dockerfile using Microsoft's official Playwright image with all browser dependencies pre-installed.

Template

# ============================================
# Stage 1: Build
# ============================================
FROM mcr.microsoft.com/playwright:v1.48.0-jammy AS builder

WORKDIR /app

# Copy dependency manifests first for Docker cache optimization
COPY package*.json ./
RUN npm ci

# Copy source and compile TypeScript
COPY tsconfig.json ./
COPY src/ ./src/
RUN npx tsc

# ============================================
# Stage 2: Production
# ============================================
FROM mcr.microsoft.com/playwright:v1.48.0-jammy

WORKDIR /app

# Install production dependencies only
COPY package*.json ./
RUN npm ci --omit=dev

# Copy compiled output from builder
COPY --from=builder /app/dist ./dist

# Copy .env.example as reference (actual .env should be mounted or set via compose)
COPY .env.example ./.env.example

# Create output directories
RUN mkdir -p data screenshots

# Run as the non-root user included in the Playwright image
USER pwuser

CMD ["node", "dist/index.js"]

Customization Notes

• Playwright version: Always match the image tag version (v1.48.0) to the playwright package version in package.json. Mismatched versions cause browser launch failures.
• Base OS: jammy = Ubuntu 22.04. Use noble for Ubuntu 24.04.
• Single-stage alternative: For simpler setups, remove the build stage and use tsx to run TypeScript directly (add tsx to production dependencies).
• Environment variables: Pass via docker-compose.yml or docker run -e. Do not bake secrets into the image.
• Shared memory: If Chromium crashes, add --shm-size=512m to docker run or shm_size: '512mb' in compose.

See Also

• docker-compose.yml.md — Compose configuration
• ../../references/docker-setup.md — Full Docker setup guide

Package.json Template

Package configuration with Playwright, Zod, and tsx as core dependencies.

Template

{
  "name": "{{project-name}}",
  "version": "0.1.0",
  "private": true,
  "type": "module",
  "scripts": {
    "start": "tsx src/index.ts",
    "build": "tsc",
    "dev": "tsx watch src/index.ts",
    "scrape": "tsx src/index.ts",
    "typecheck": "tsc --noEmit"
  },
  "dependencies": {
    "playwright": "^1.48.0",
    "zod": "^3.23.0",
    "dotenv": "^16.4.0"
  },
  "devDependencies": {
    "typescript": "^5.6.0",
    "tsx": "^4.19.0",
    "@types/node": "^22.0.0"
  }
}

Customization Notes

• `playwright` not `@playwright/test`: Scrapers use the library package, not the test runner. The library package is smaller and doesn't include test runner overhead.
• `tsx`: TypeScript execute — runs .ts files directly without a build step. Used for development and can be used in production for simpler setups.
• `zod`: Remove from dependencies if --no-validation was used during scaffolding.
• `dotenv`: Loads .env files for local development. Not needed in Docker (environment variables set via compose).
• `type: "module"`: Enables ESM imports. Required for top-level await and modern module syntax.
• Scripts:
• start / scrape: Run the scraper
• dev: Run with file watching (re-runs on source changes)
• build: Compile TypeScript to JavaScript
• typecheck: Verify types without emitting

See Also

• tsconfig.json.md — TypeScript configuration
• ../../references/docker-setup.md — Docker deployment

Playwright Configuration Template

Scraper-focused Playwright configuration. This differs from test-focused configs — it optimizes for data extraction rather than test assertions.

Template

/**
 * Playwright configuration for scraping.
 *
 * This config is used when running Playwright as a library (not as a test runner).
 * Import these settings in your browser utility module.
 *
 * Note: This is NOT a @playwright/test config. Scrapers use the `playwright`
 * package directly, not the test runner.
 */

export const playwrightConfig = {
  /** Browser launch options */
  launch: {
    headless: process.env.HEADLESS !== 'false',
    args: [
      '--no-sandbox',
      '--disable-setuid-sandbox',
      '--disable-dev-shm-usage',
      '--disable-gpu',
    ],
  },

  /** Browser context options */
  context: {
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
    viewport: { width: 1920, height: 1080 },
    locale: 'en-US',
    timezoneId: 'America/New_York',
  },

  /** Navigation defaults */
  navigation: {
    timeout: 30000,
    waitUntil: 'networkidle' as const,
  },

  /** Scraping behavior */
  scraping: {
    /** Delay between page navigations (ms) */
    requestDelay: parseInt(process.env.REQUEST_DELAY ?? '1000'),
    /** Maximum pages to scrape in a single run */
    maxPages: parseInt(process.env.MAX_PAGES ?? '10'),
    /** Maximum retry attempts per page */
    maxRetries: 3,
    /** Base delay for exponential backoff (ms) */
    retryBaseDelay: 1000,
  },

  /** Output configuration */
  output: {
    dataDir: process.env.OUTPUT_DIR ?? './data',
    screenshotDir: './screenshots',
    formats: ['json', 'csv'] as const,
  },
};

Usage in Code

import { chromium } from 'playwright';
import { playwrightConfig } from '../playwright.config';

const browser = await chromium.launch(playwrightConfig.launch);
const context = await browser.newContext(playwrightConfig.context);
const page = await context.newPage();
page.setDefaultTimeout(playwrightConfig.navigation.timeout);

Customization Notes

• Not a test config: This is a plain TypeScript module exporting configuration objects, not a @playwright/test config file. Scrapers don't use the test runner.
• User agent: The default mimics a standard Chrome browser. Adjust for the target site's expectations.
• Viewport: Set to 1920x1080 to ensure desktop layouts render fully. Some sites show different content at smaller viewports.
• Request delay: Controls politeness. Increase for sensitive sites, decrease for sites you control.
• Docker compatibility: The --disable-dev-shm-usage and --no-sandbox flags are required for Docker environments.

See Also

• ../../references/docker-setup.md — Docker-specific Playwright configuration
• ../../references/playwright-selectors.md — Selector strategies

TypeScript Configuration Template

Strict TypeScript configuration targeting ES2022 for modern Node.js with Playwright.

Template

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "bundler",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "outDir": "./dist",
    "rootDir": "./src",
    "declaration": true,
    "resolveJsonModule": true,
    "isolatedModules": true
  },
  "include": ["src/**/*.ts"],
  "exclude": ["node_modules", "dist", "data", "screenshots"]
}

Customization Notes

• ES2022 target: Provides top-level await, Array.at(), and Object.hasOwn(). Matches Node.js 18+ capabilities.
• Bundler module resolution: Works with both ESM imports and CommonJS interop. Use "node" if not using a bundler.
• Declaration files: Enable declaration: true if the scraper exports types for other tools. Remove if standalone.
• Strict mode: Keeps all strict checks enabled. Do not relax these.

See Also

• package.json.md — Package configuration

Example: E-Commerce Product Scraper

Complete walkthrough of building a multi-page product scraper for an online store. Demonstrates the full PageObject pattern with pagination, data validation, and Docker deployment.

Scenario

Target: An online electronics store with paginated product listings. Each product has a title, price, rating, and detail page link.

Project Structure

shop-scraper/
├── src/
│   ├── pages/
│   │   ├── BasePage.ts
│   │   └── ProductListingPage.ts
│   ├── components/
│   │   └── Pagination.ts
│   ├── schemas/
│   │   └── product.schema.ts
│   ├── utils/
│   │   ├── browser.ts
│   │   ├── retry.ts
│   │   └── storage.ts
│   ├── scrapers/
│   │   └── ShopScraper.ts
│   └── index.ts
├── data/
├── screenshots/
├── Dockerfile
├── docker-compose.yml
├── package.json
├── tsconfig.json
└── .env.example

Step 1: Define the Data Schema

// src/schemas/product.schema.ts
import { z } from 'zod';

export const ProductSchema = z.object({
  title: z.string().min(1, 'Title is required'),
  price: z
    .string()
    .transform(val => parseFloat(val.replace(/[^0-9.]/g, '')))
    .pipe(z.number().positive()),
  rating: z
    .string()
    .transform(val => parseFloat(val))
    .pipe(z.number().min(0).max(5))
    .optional(),
  url: z.string().min(1),
  imageUrl: z.string().url().nullable(),
});

export type Product = z.infer<typeof ProductSchema>;

Step 2: Build the Page Object

// src/pages/ProductListingPage.ts
import { Page, Locator } from 'playwright';
import { BasePage } from './BasePage';
import { Pagination } from '../components/Pagination';

export class ProductListingPage extends BasePage {
  readonly productCards: Locator;
  readonly pagination: Pagination;

  constructor(page: Page) {
    super(page);
    this.productCards = page.locator('.product-card');
    this.pagination = new Pagination(page);
  }

  async scrape(): Promise<Array<Record<string, string | null>>> {
    const items: Array<Record<string, string | null>> = [];
    const count = await this.productCards.count();

    for (let i = 0; i < count; i++) {
      const card = this.productCards.nth(i);

      try {
        items.push({
          title: (await card.locator('.product-title').textContent()) ?? '',
          price: (await card.locator('.product-price').textContent()) ?? '',
          rating: await card.locator('.star-rating').getAttribute('data-rating'),
          url: await card.locator('a.product-link').getAttribute('href'),
          imageUrl: await card.locator('img').getAttribute('src'),
        });
      } catch (error) {
        console.warn(`Failed to scrape product ${i}: ${error}`);
        await this.screenshot(`error-product-${i}`);
      }
    }

    return items;
  }

  async goToNextPage(): Promise<boolean> {
    if (!(await this.pagination.hasNextPage())) return false;
    await this.pagination.goToNext();
    return true;
  }
}

Step 3: Build the Scraper Runner

// src/scrapers/ShopScraper.ts
import { chromium } from 'playwright';
import { ProductListingPage } from '../pages/ProductListingPage';
import { ProductSchema } from '../schemas/product.schema';
import { writeJson, writeCsv } from '../utils/storage';
import { withRetry } from '../utils/retry';

export class ShopScraper {
  private readonly baseUrl = process.env.BASE_URL ?? 'https://shop.example.com';
  private readonly maxPages = parseInt(process.env.MAX_PAGES ?? '10');
  private readonly requestDelay = parseInt(process.env.REQUEST_DELAY ?? '1000');

  async run(): Promise<void> {
    const browser = await chromium.launch({ headless: true });
    const context = await browser.newContext();
    const page = await context.newPage();

    try {
      const listing = new ProductListingPage(page);
      await withRetry(() => listing.navigate(`${this.baseUrl}/products`));

      const allProducts: Array<Record<string, string | null>> = [];
      let currentPage = 1;

      do {
        console.log(`Scraping page ${currentPage}...`);
        const products = await listing.scrape();
        allProducts.push(...products);
        currentPage++;

        await new Promise(r => setTimeout(r, this.requestDelay));
      } while (currentPage <= this.maxPages && (await listing.goToNextPage()));

      // Validate
      const validated = allProducts
        .map(p => ProductSchema.safeParse(p))
        .filter(r => r.success)
        .map(r => r.data);

      console.log(`Validated ${validated.length}/${allProducts.length} products`);

      // Write output
      const timestamp = new Date().toISOString().split('T')[0];
      writeJson(validated, `products-${timestamp}`);
      writeCsv(validated, `products-${timestamp}`);
    } finally {
      await browser.close();
    }
  }
}

Step 4: Entry Point

// src/index.ts
import { ShopScraper } from './scrapers/ShopScraper';
import 'dotenv/config';

async function main() {
  console.log('Starting shop scraper...');
  const scraper = new ShopScraper();
  await scraper.run();
  console.log('Done.');
}

main().catch(error => {
  console.error('Fatal error:', error);
  process.exit(1);
});

Step 5: Run

# Local development
npm run dev

# With custom URL
BASE_URL=https://actual-shop.com npm run scrape

# Docker
docker compose up --build

# Docker with custom config
BASE_URL=https://actual-shop.com MAX_PAGES=50 docker compose up

Output

Starting shop scraper...
Scraping page 1...
Scraping page 2...
Scraping page 3...
Validated 72/75 products
Wrote 72 records to data/products-2026-02-04.json
Wrote 72 records to data/products-2026-02-04.csv
Done.

Key Patterns Demonstrated

1. PageObject encapsulation — ProductListingPage owns all selectors and extraction logic 2. Component composition — Pagination is a separate reusable component 3. Schema validation — ProductSchema catches 3 invalid records out of 75 4. Error resilience — Individual product failures don't abort the entire scrape 5. Configurable behavior — All parameters via environment variables 6. Docker-ready — Same code runs locally and in containers

Example: Multi-Page Pagination Handling

Strategies for handling different pagination patterns found across websites. Covers numbered pagination, "Load More" buttons, infinite scroll, and URL-based pagination.

Pattern 1: Numbered Pagination (Click-Based)

The most common pattern. A pagination bar with numbered links and Next/Previous buttons.

import { Page, Locator } from 'playwright';
import { BasePage } from './BasePage';

export class NumberedPaginationPage extends BasePage {
  readonly pagination: Locator;
  readonly nextButton: Locator;
  readonly items: Locator;

  constructor(page: Page) {
    super(page);
    this.pagination = page.locator('nav[aria-label="pagination"]');
    this.nextButton = page.locator('[aria-label="Next page"], a[rel="next"]');
    this.items = page.locator('.item-card');
  }

  async scrapeAllPages(maxPages = 10): Promise<Array<Record<string, string | null>>> {
    const allItems: Array<Record<string, string | null>> = [];
    let currentPage = 1;

    do {
      const items = await this.scrapeCurrentPage();
      allItems.push(...items);
      console.log(`Page ${currentPage}: ${items.length} items`);
      currentPage++;
    } while (currentPage <= maxPages && (await this.goToNextPage()));

    return allItems;
  }

  private async scrapeCurrentPage(): Promise<Array<Record<string, string | null>>> {
    const items: Array<Record<string, string | null>> = [];
    const count = await this.items.count();

    for (let i = 0; i < count; i++) {
      const card = this.items.nth(i);
      items.push({
        title: (await card.locator('h3').textContent()) ?? '',
        url: await card.locator('a').getAttribute('href'),
      });
    }

    return items;
  }

  private async goToNextPage(): Promise<boolean> {
    if (!(await this.nextButton.isVisible())) return false;

    const isDisabled = await this.nextButton.getAttribute('disabled');
    if (isDisabled !== null) return false;

    await this.nextButton.click();
    await this.waitForPageLoad();
    return true;
  }
}

Key points:

• Check both visibility and disabled attribute before clicking Next
• Use aria-label="Next page" or a[rel="next"] for resilient selectors
• Wait for networkidle after each page transition

Pattern 2: "Load More" Button

Content stays on one page; clicking a button appends more items.

export class LoadMorePage extends BasePage {
  readonly loadMoreButton: Locator;
  readonly items: Locator;

  constructor(page: Page) {
    super(page);
    this.loadMoreButton = page.locator('button:has-text("Load More"), button:has-text("Show More")');
    this.items = page.locator('.item-card');
  }

  async loadAllItems(maxClicks = 20): Promise<void> {
    let clicks = 0;

    while (clicks < maxClicks) {
      const isVisible = await this.loadMoreButton.isVisible();
      if (!isVisible) break;

      const countBefore = await this.items.count();
      await this.loadMoreButton.click();

      // Wait for new items to appear
      await this.page.waitForFunction(
        (prevCount) => {
          return document.querySelectorAll('.item-card').length > prevCount;
        },
        countBefore,
        { timeout: 10000 },
      ).catch(() => {
        console.log('No new items loaded, stopping');
      });

      const countAfter = await this.items.count();
      if (countAfter === countBefore) break; // No new items

      console.log(`Loaded ${countAfter - countBefore} more items (total: ${countAfter})`);
      clicks++;

      // Polite delay
      await new Promise(r => setTimeout(r, 1000));
    }
  }

  async scrapeAll(): Promise<Array<Record<string, string | null>>> {
    await this.loadAllItems();
    return this.scrapeCurrentPage();
  }

  private async scrapeCurrentPage(): Promise<Array<Record<string, string | null>>> {
    const items: Array<Record<string, string | null>> = [];
    const count = await this.items.count();

    for (let i = 0; i < count; i++) {
      const card = this.items.nth(i);
      items.push({
        title: (await card.locator('h3').textContent()) ?? '',
        url: await card.locator('a').getAttribute('href'),
      });
    }

    return items;
  }
}

Key points:

• Use waitForFunction to detect when new items actually appear
• Compare counts before/after to detect when loading is exhausted
• Set a maxClicks limit to prevent infinite loops

Pattern 3: Infinite Scroll

Content loads automatically as the user scrolls down.

export class InfiniteScrollPage extends BasePage {
  readonly items: Locator;
  readonly loadingSpinner: Locator;

  constructor(page: Page) {
    super(page);
    this.items = page.locator('.item-card');
    this.loadingSpinner = page.locator('.loading-spinner, [data-loading="true"]');
  }

  async scrollToLoadAll(maxScrolls = 30): Promise<void> {
    let scrolls = 0;
    let previousCount = 0;
    let stableCount = 0; // How many scrolls with no new items

    while (scrolls < maxScrolls && stableCount < 3) {
      const currentCount = await this.items.count();

      if (currentCount === previousCount) {
        stableCount++;
      } else {
        stableCount = 0;
      }

      previousCount = currentCount;
      console.log(`Scroll ${scrolls + 1}: ${currentCount} items`);

      // Scroll to bottom
      await this.page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));

      // Wait for loading spinner to appear and disappear
      try {
        await this.loadingSpinner.waitFor({ state: 'visible', timeout: 2000 });
        await this.loadingSpinner.waitFor({ state: 'hidden', timeout: 10000 });
      } catch {
        // Spinner may not appear if content loads instantly
      }

      // Brief wait for DOM to update
      await new Promise(r => setTimeout(r, 500));
      scrolls++;
    }

    console.log(`Scrolling complete: ${await this.items.count()} total items`);
  }

  async scrapeAll(): Promise<Array<Record<string, string | null>>> {
    await this.scrollToLoadAll();
    return this.scrapeVisibleItems();
  }

  private async scrapeVisibleItems(): Promise<Array<Record<string, string | null>>> {
    const items: Array<Record<string, string | null>> = [];
    const count = await this.items.count();

    for (let i = 0; i < count; i++) {
      const card = this.items.nth(i);
      items.push({
        title: (await card.locator('h3').textContent()) ?? '',
        url: await card.locator('a').getAttribute('href'),
      });
    }

    return items;
  }
}

Key points:

• Track stableCount — stop when 3 consecutive scrolls produce no new items
• Watch for a loading spinner to know when content is being fetched
• Use page.evaluate to scroll the actual browser window

Pattern 4: URL-Based Pagination

Pages accessed via URL parameters (e.g., ?page=2). No clicking needed.

export class UrlPaginationPage extends BasePage {
  readonly items: Locator;

  constructor(page: Page) {
    super(page);
    this.items = page.locator('.item-card');
  }

  async scrapeAllPages(
    baseUrl: string,
    maxPages = 10,
  ): Promise<Array<Record<string, string | null>>> {
    const allItems: Array<Record<string, string | null>> = [];

    for (let pageNum = 1; pageNum <= maxPages; pageNum++) {
      const url = `${baseUrl}?page=${pageNum}`;
      console.log(`Fetching ${url}`);

      await this.navigate(url);
      const items = await this.scrapeCurrentPage();

      if (items.length === 0) {
        console.log(`Page ${pageNum} is empty, stopping`);
        break;
      }

      allItems.push(...items);
      console.log(`Page ${pageNum}: ${items.length} items (total: ${allItems.length})`);

      // Polite delay between requests
      await new Promise(r => setTimeout(r, 1000));
    }

    return allItems;
  }

  private async scrapeCurrentPage(): Promise<Array<Record<string, string | null>>> {
    const items: Array<Record<string, string | null>> = [];
    const count = await this.items.count();

    for (let i = 0; i < count; i++) {
      const card = this.items.nth(i);
      items.push({
        title: (await card.locator('h3').textContent()) ?? '',
        url: await card.locator('a').getAttribute('href'),
      });
    }

    return items;
  }
}

Key points:

• Simplest approach — construct URLs directly, no click interactions
• Detect empty pages to know when to stop
• Works well for API-backed sites with predictable URL patterns

Choosing a Pagination Strategy

BasePage Template

Abstract base class for all page objects. Provides navigation, waiting, screenshot, and text extraction utilities.

Usage

Every page object in a scraper project extends BasePage. Do not instantiate it directly.

Template

import { Page } from 'playwright';

/**
 * Abstract base class for all page objects.
 *
 * Provides common navigation, waiting, and extraction methods.
 * All site-specific page objects extend this class.
 */
export abstract class BasePage {
  constructor(protected readonly page: Page) {}

  /**
   * Navigate to a URL and wait for the page to stabilize.
   */
  async navigate(url: string): Promise<void> {
    await this.page.goto(url, { waitUntil: 'networkidle' });
  }

  /**
   * Wait for the current page to finish loading.
   * Use after interactions that trigger navigation or content updates.
   */
  async waitForPageLoad(): Promise<void> {
    await this.page.waitForLoadState('networkidle');
  }

  /**
   * Save a full-page screenshot for debugging.
   * Screenshots are saved to the screenshots/ directory.
   */
  async screenshot(name: string): Promise<void> {
    await this.page.screenshot({
      path: `screenshots/${name}.png`,
      fullPage: true,
    });
  }

  /**
   * Extract text content from a single element.
   * Returns empty string if element not found.
   */
  async getText(selector: string): Promise<string> {
    const el = this.page.locator(selector);
    return (await el.textContent()) ?? '';
  }

  /**
   * Extract text content from all matching elements.
   */
  async getTexts(selector: string): Promise<string[]> {
    return this.page.locator(selector).allTextContents();
  }

  /**
   * Get an attribute value from a single element.
   */
  async getAttribute(selector: string, attr: string): Promise<string | null> {
    return this.page.locator(selector).getAttribute(attr);
  }

  /**
   * Check whether an element exists on the page.
   */
  async exists(selector: string): Promise<boolean> {
    return (await this.page.locator(selector).count()) > 0;
  }

  /**
   * Get the current page URL.
   */
  get currentUrl(): string {
    return this.page.url();
  }
}

Customization Notes

• Wait strategy: networkidle works for most server-rendered sites. For SPAs with streaming data, consider domcontentloaded plus explicit element waits.
• Screenshot path: Adjust the screenshots/ prefix if your project uses a different directory structure.
• Additional helpers: Add methods like getNumber(), getHref(), or getImageSrc() if multiple page objects need them.

Component Templates

Reusable UI components that page objects compose. Each component models a single UI pattern (pagination, data table, filter sidebar) and can be used across multiple page objects.

Pagination Component

import { Page, Locator } from 'playwright';

/**
 * Pagination component for navigating through paged results.
 *
 * Handles multiple pagination patterns:
 * - Numbered page links
 * - Next/Previous buttons
 * - aria-label based navigation
 */
export class Pagination {
  readonly nextButton: Locator;
  readonly prevButton: Locator;
  readonly currentPage: Locator;
  readonly pageLinks: Locator;

  constructor(
    private page: Page,
    scope?: Locator,
  ) {
    const root = scope ?? page;

    this.nextButton = root.locator(
      '[aria-label="Next page"], a[rel="next"], .pagination-next, button:has-text("Next")'
    );
    this.prevButton = root.locator(
      '[aria-label="Previous page"], a[rel="prev"], .pagination-prev, button:has-text("Previous")'
    );
    this.currentPage = root.locator(
      '[aria-current="page"], .pagination .active, .current-page'
    );
    this.pageLinks = root.locator(
      '.pagination a, .page-numbers a, nav[aria-label="pagination"] a'
    );
  }

  async hasNextPage(): Promise<boolean> {
    return this.nextButton.isVisible();
  }

  async hasPrevPage(): Promise<boolean> {
    return this.prevButton.isVisible();
  }

  async goToNext(): Promise<void> {
    await this.nextButton.click();
    await this.page.waitForLoadState('networkidle');
  }

  async goToPrev(): Promise<void> {
    await this.prevButton.click();
    await this.page.waitForLoadState('networkidle');
  }

  async getCurrentPageNumber(): Promise<number> {
    const text = await this.currentPage.textContent();
    return parseInt(text ?? '1', 10);
  }

  async getTotalPages(): Promise<number> {
    const links = await this.pageLinks.allTextContents();
    const numbers = links
      .map(t => parseInt(t.trim(), 10))
      .filter(n => !isNaN(n));
    return numbers.length > 0 ? Math.max(...numbers) : 1;
  }
}

DataTable Component

import { Page, Locator } from 'playwright';

/**
 * DataTable component for extracting tabular data.
 *
 * Handles standard HTML tables with thead/tbody structure.
 */
export class DataTable {
  readonly table: Locator;
  readonly headerCells: Locator;
  readonly bodyRows: Locator;

  constructor(
    private page: Page,
    tableSelector: string,
  ) {
    this.table = page.locator(tableSelector);
    this.headerCells = this.table.locator('thead th, thead td');
    this.bodyRows = this.table.locator('tbody tr');
  }

  /**
   * Get column headers as an array of strings.
   */
  async getHeaders(): Promise<string[]> {
    return this.headerCells.allTextContents();
  }

  /**
   * Get the number of data rows (excluding header).
   */
  async getRowCount(): Promise<number> {
    return this.bodyRows.count();
  }

  /**
   * Get a specific cell value by row and column index.
   */
  async getCellValue(row: number, col: number): Promise<string> {
    const cell = this.bodyRows.nth(row).locator('td').nth(col);
    return (await cell.textContent()) ?? '';
  }

  /**
   * Extract all rows as arrays of strings.
   */
  async extractRows(): Promise<string[][]> {
    const rows: string[][] = [];
    const rowCount = await this.getRowCount();

    for (let i = 0; i < rowCount; i++) {
      const cells = this.bodyRows.nth(i).locator('td');
      const cellTexts = await cells.allTextContents();
      rows.push(cellTexts);
    }

    return rows;
  }

  /**
   * Extract all rows as objects using headers as keys.
   */
  async extractAsObjects(): Promise<Record<string, string>[]> {
    const headers = await this.getHeaders();
    const rows = await this.extractRows();

    return rows.map(row => {
      const obj: Record<string, string> = {};
      headers.forEach((header, i) => {
        obj[header.trim()] = (row[i] ?? '').trim();
      });
      return obj;
    });
  }
}

Customization Notes

Pagination

• Selector fallbacks: The component uses comma-separated selectors to match common patterns. Narrow these down once you know the target site's structure.
• Scope parameter: Pass a parent Locator to scope pagination within a specific section (e.g., bottom pagination vs. top pagination).
• Infinite scroll: For infinite scroll pages, replace goToNext() with a scroll-based approach.

DataTable

• Non-standard tables: Some sites use div-based tables. Adjust selectors from thead/tbody/tr/td to the site's custom structure.
• Sortable tables: Add a sortBy(column: string) method that clicks header cells.
• Cell links: Extend extractRows() to capture href attributes from linked cells.

Data Schema Template

Zod schemas for validating scraped data. Every data type extracted by the scraper should have a corresponding schema.

Purpose

Schemas serve as a contract between the scraper and its consumers. They catch:

• Selector drift — When a site changes markup, fields extract as null or empty strings
• Type mismatches — Prices that aren't numbers, URLs that aren't valid
• Missing data — Required fields that didn't extract

Template

import { z } from 'zod';

/**
 * Schema for {{DataType}} records scraped from {{siteName}}.
 *
 * Adjust field types and constraints based on the actual data format.
 * Run a test scrape and review raw output before tightening constraints.
 */
export const {{DataType}}Schema = z.object({
  // Required string fields
  title: z.string().min(1, 'Title is required'),
  url: z.string().url('Must be a valid URL').or(z.string().startsWith('/')),

  // Optional string fields
  description: z.string().optional(),
  imageUrl: z.string().url().nullable().optional(),

  // Numeric fields (extracted as strings, then transformed)
  price: z
    .string()
    .transform((val) => {
      const cleaned = val.replace(/[^0-9.]/g, '');
      return parseFloat(cleaned);
    })
    .pipe(z.number().positive('Price must be positive')),

  // Rating (typically 1-5)
  rating: z
    .string()
    .transform((val) => parseFloat(val))
    .pipe(z.number().min(0).max(5))
    .optional(),

  // Date fields
  date: z.string().optional(),

  // Enum fields
  availability: z
    .enum(['in_stock', 'out_of_stock', 'pre_order'])
    .optional()
    .default('in_stock'),
});

/** TypeScript type inferred from the schema */
export type {{DataType}} = z.infer<typeof {{DataType}}Schema>;

/**
 * Validate a single record. Returns the validated data or null.
 */
export function validate{{DataType}}(
  data: unknown,
): {{DataType}} | null {
  const result = {{DataType}}Schema.safeParse(data);
  if (result.success) return result.data;

  console.warn(
    `Validation failed for {{DataType}}:`,
    result.error.issues.map((i) => `${i.path.join('.')}: ${i.message}`).join(', '),
  );
  return null;
}

/**
 * Validate an array of records. Returns only valid records.
 * Logs a summary of failures.
 */
export function validateBatch(
  data: unknown[],
): {{DataType}}[] {
  const results = data.map(validate{{DataType}});
  const valid = results.filter((r): r is {{DataType}} => r !== null);

  const failures = data.length - valid.length;
  if (failures > 0) {
    console.warn(
      `Schema validation: ${valid.length}/${data.length} passed, ${failures} failed`,
    );
  }

  return valid;
}

Common Field Patterns

Price Extraction

Prices come as strings with currency symbols. Use transform to clean and parse:

price: z.string()
  .transform(val => parseFloat(val.replace(/[^0-9.]/g, '')))
  .pipe(z.number().positive())

Relative URLs

Some sites use relative URLs. Accept both absolute and relative:

url: z.string().url().or(z.string().startsWith('/'))

Optional with Default

Fields that may be missing but have a sensible default:

currency: z.string().default('USD')
availability: z.enum(['in_stock', 'out_of_stock']).default('in_stock')

Array Fields

Tags, categories, or image galleries:

tags: z.array(z.string()).default([])
images: z.array(z.string().url()).default([])

Customization Notes

• Replace `{{DataType}}` with the actual type name (e.g., Product, Job, Article)
• Start loose, tighten later: Begin with z.string().optional() for most fields, then add constraints after reviewing real scraped data
• Transform chains: Use .transform().pipe() to convert string extractions to proper types
• Multiple schemas: Create one schema per page object's output type (e.g., ProductSchema, ReviewSchema)

Page Object Template

Site-specific page object with locators, extraction methods, and navigation. Extend this for each page type on the target site.

Usage

Create one page object per distinct page or view. Define locators in the constructor and data extraction in methods.

Template

import { Page, Locator } from 'playwright';
import { BasePage } from './BasePage';

/**
 * {{PageName}}Page — Scrapes data from {{url}}.
 *
 * Locators target {{description of what's on this page}}.
 * Adjust selectors after running agent-browser snapshot against the live site.
 */
export class {{PageName}}Page extends BasePage {
  // === Locators ===

  /** Container for each data item (card, row, list item) */
  readonly itemCards: Locator;

  /** Individual field locators within each item */
  readonly itemTitle: Locator;
  readonly itemPrice: Locator;
  readonly itemImage: Locator;
  readonly itemLink: Locator;

  // === Pagination (if applicable) ===
  readonly nextButton: Locator;

  constructor(page: Page) {
    super(page);

    // Item containers — adjust selector for the target site
    this.itemCards = page.locator('[data-testid="item-card"], .item-card, .product-card');

    // Field locators — scoped to page level; use card.locator() in scrape methods
    this.itemTitle = page.locator('[data-testid="item-title"]');
    this.itemPrice = page.locator('[data-testid="item-price"]');
    this.itemImage = page.locator('[data-testid="item-card"] img');
    this.itemLink = page.locator('[data-testid="item-card"] a');

    // Pagination
    this.nextButton = page.locator(
      '[aria-label="Next page"], a[rel="next"], button:has-text("Next")'
    );
  }

  /**
   * Scrape all items from the current page.
   *
   * Returns an array of extracted records. Fields that fail to extract
   * will be null rather than throwing.
   */
  async scrape(): Promise<Array<{{DataType}}>> {
    const items: Array<{{DataType}}> = [];
    const count = await this.itemCards.count();

    for (let i = 0; i < count; i++) {
      const card = this.itemCards.nth(i);

      items.push({
        title: (await card.locator('[data-testid="item-title"]').textContent()) ?? '',
        price: (await card.locator('[data-testid="item-price"]').textContent()) ?? '',
        imageUrl: await card.locator('img').getAttribute('src'),
        url: await card.locator('a').first().getAttribute('href'),
      });
    }

    return items;
  }

  /**
   * Check if there is a next page.
   */
  async hasNextPage(): Promise<boolean> {
    return this.nextButton.isVisible();
  }

  /**
   * Navigate to the next page. Returns false if no next page.
   */
  async goToNextPage(): Promise<boolean> {
    if (!(await this.hasNextPage())) return false;
    await this.nextButton.click();
    await this.waitForPageLoad();
    return true;
  }

  /**
   * Scrape all pages, following pagination until exhausted or limit reached.
   */
  async scrapeAllPages(maxPages = 10): Promise<Array<{{DataType}}>> {
    const allItems: Array<{{DataType}}> = [];
    let currentPage = 1;

    do {
      console.log(`Scraping page ${currentPage}...`);
      const items = await this.scrape();
      allItems.push(...items);
      currentPage++;
    } while (currentPage <= maxPages && (await this.goToNextPage()));

    console.log(`Scraped ${allItems.length} items across ${currentPage - 1} pages`);
    return allItems;
  }
}

Customization Notes

• Replace `{{PageName}}` with the actual page class name (e.g., ProductListing, SearchResults)
• Replace `{{DataType}}` with the TypeScript interface for extracted data
• Replace `{{url}}` with the target page URL
• Adjust selectors after analyzing the actual site with agent-browser or browser DevTools
• Remove pagination if the page doesn't paginate
• Add more fields by adding locators and extending the scrape() return object
• Scoped extraction: The scrape() method uses card.locator() (scoped within each item) rather than page-level locators to avoid cross-item contamination

Scraper Runner Template

Orchestrator that launches the browser, creates page objects, iterates through pages, collects data, validates with schemas, and writes output.

Template

import { chromium, Browser, Page } from 'playwright';
import { {{PageName}}Page } from '../pages/{{PageName}}Page';
import { {{SchemaName}} } from '../schemas/{{schemaFile}}';
import { writeJson, writeCsv } from '../utils/storage';
import { withRetry } from '../utils/retry';

/**
 * {{SiteName}}Scraper — Orchestrates scraping of {{description}}.
 *
 * Workflow:
 * 1. Launch headless browser
 * 2. Navigate to starting URL
 * 3. Scrape data across pages (with pagination)
 * 4. Validate extracted data against schemas
 * 5. Write validated data to output files
 * 6. Close browser and report results
 */
export class {{SiteName}}Scraper {
  private browser: Browser | null = null;
  private page: Page | null = null;

  private readonly baseUrl: string;
  private readonly maxPages: number;
  private readonly requestDelay: number;
  private readonly headless: boolean;
  private readonly outputDir: string;

  constructor() {
    this.baseUrl = process.env.BASE_URL ?? '{{defaultUrl}}';
    this.maxPages = parseInt(process.env.MAX_PAGES ?? '10');
    this.requestDelay = parseInt(process.env.REQUEST_DELAY ?? '1000');
    this.headless = process.env.HEADLESS !== 'false';
    this.outputDir = process.env.OUTPUT_DIR ?? './data';
  }

  /**
   * Main entry point. Runs the full scraping workflow.
   */
  async run(): Promise<void> {
    console.log(`Starting scrape of ${this.baseUrl}`);
    console.log(`Max pages: ${this.maxPages}, Delay: ${this.requestDelay}ms`);

    try {
      await this.setup();
      const data = await this.scrape();
      const validated = this.validate(data);
      await this.save(validated);
      this.report(data.length, validated.length);
    } catch (error) {
      console.error('Scrape failed:', error);
      if (this.page) {
        await this.page.screenshot({
          path: `screenshots/fatal-error-${Date.now()}.png`,
        });
      }
      throw error;
    } finally {
      await this.teardown();
    }
  }

  /**
   * Launch browser and create page.
   */
  private async setup(): Promise<void> {
    this.browser = await chromium.launch({
      headless: this.headless,
      args: ['--no-sandbox', '--disable-setuid-sandbox'],
    });

    const context = await this.browser.newContext({
      userAgent: 'Mozilla/5.0 (compatible; ScrapeBot/1.0)',
    });

    this.page = await context.newPage();
    this.page.setDefaultTimeout(30000);
  }

  /**
   * Execute the scraping logic.
   */
  private async scrape(): Promise<Array<Record<string, unknown>>> {
    if (!this.page) throw new Error('Browser not initialized');

    const pageObj = new {{PageName}}Page(this.page);

    // Navigate to starting URL with retry
    await withRetry(() => pageObj.navigate(this.baseUrl));

    // Scrape with pagination
    const allData: Array<Record<string, unknown>> = [];
    let currentPage = 1;

    do {
      console.log(`Scraping page ${currentPage}/${this.maxPages}...`);

      const items = await withRetry(() => pageObj.scrape());
      allData.push(...items);

      console.log(`  Found ${items.length} items (total: ${allData.length})`);
      currentPage++;

      // Delay between pages to be respectful
      if (this.requestDelay > 0) {
        await new Promise(r => setTimeout(r, this.requestDelay));
      }
    } while (currentPage <= this.maxPages && (await pageObj.goToNextPage()));

    return allData;
  }

  /**
   * Validate scraped data against schema.
   */
  private validate(
    data: Array<Record<string, unknown>>,
  ): Array<Record<string, unknown>> {
    const validated: Array<Record<string, unknown>> = [];
    let failures = 0;

    for (const item of data) {
      const result = {{SchemaName}}.safeParse(item);
      if (result.success) {
        validated.push(result.data);
      } else {
        failures++;
        console.warn(`Validation failed: ${result.error.issues[0]?.message}`);
      }
    }

    if (failures > 0) {
      console.warn(`${failures}/${data.length} items failed validation`);
    }

    return validated;
  }

  /**
   * Write validated data to output files.
   */
  private async save(data: Array<Record<string, unknown>>): Promise<void> {
    const timestamp = new Date().toISOString().split('T')[0];
    const filename = `{{siteName}}-${timestamp}`;

    writeJson(data, filename, this.outputDir);
    writeCsv(data as Record<string, unknown>[], filename, this.outputDir);
  }

  /**
   * Report scraping results.
   */
  private report(total: number, valid: number): void {
    console.log('\n=== SCRAPE COMPLETE ===');
    console.log(`Total extracted: ${total}`);
    console.log(`Valid records:   ${valid}`);
    console.log(`Failed:          ${total - valid}`);
    console.log(`Output:          ${this.outputDir}/`);
  }

  /**
   * Clean up browser resources.
   */
  private async teardown(): Promise<void> {
    if (this.browser) {
      await this.browser.close();
      this.browser = null;
      this.page = null;
    }
  }
}

Customization Notes

• Replace `{{placeholders}}` with actual names: SiteName, PageName, SchemaName, etc.
• Multiple page types: If the scraper visits different page types (listing → detail), add multiple page object instances in the scrape() method.
• Output format: The template writes both JSON and CSV. Remove whichever isn't needed.
• Error screenshots: Fatal errors save a timestamped screenshot for debugging.
• Request delay: Controlled via REQUEST_DELAY environment variable. Default 1000ms.
• Validation: The validate() method uses Zod's safeParse to avoid throwing on invalid records. Remove if --no-validation was used during scaffolding.

Agent-Browser Site Analysis Workflow

The agent-browser CLI tool enables AI agents to analyze websites by capturing accessibility tree snapshots. This document describes the complete workflow for using agent-browser to discover page structure, identify selectors, and generate PageObject classes.

Overview

Agent-browser works by: 1. Launching a browser session (like Playwright, but controlled via CLI) 2. Capturing accessibility tree snapshots (not screenshots or DOM dumps) 3. Returning structured JSON with element references and roles 4. Supporting interactions (click, type, scroll) to test dynamic behavior

The accessibility tree is approximately 93% smaller than the full DOM, making it efficient for AI agents to process without vision models.

Installation

If agent-browser is not already available, install it as a skill:

npx skills add vercel-labs/agent-browser

This adds the agent-browser CLI commands to the agent's environment. Verify installation by running agent-browser --help.

Core Commands

Navigation

# Open a URL in the browser
agent-browser open https://example.com/products

# Wait for the page to stabilize
agent-browser wait --load networkidle

Snapshots

# Full page snapshot with interactive element references
agent-browser snapshot -i --json

# Scoped snapshot (specific section)
agent-browser snapshot -i --json -s "main"
agent-browser snapshot -i --json -s "nav"
agent-browser snapshot -i --json -s "form"

# Text-only snapshot (no element refs)
agent-browser snapshot --json

Interactions

# Click an element by reference
agent-browser click @e3

# Type text into an input
agent-browser type @e5 "search query"

# Scroll the page
agent-browser scroll down 500

# Press a key
agent-browser press Enter

# Wait after interaction
agent-browser wait --load networkidle
agent-browser wait 2000  # explicit ms delay

Data Extraction

# Get text content of an element
agent-browser get text body --json
agent-browser get text "main" --json

# Get an attribute
agent-browser get attr "img.product" src --json

Session Management

# Close the browser
agent-browser close

Snapshot Output Format

Interactive snapshots return JSON with element references:

{
  "url": "https://example.com/products",
  "title": "Products - Example Store",
  "elements": [
    {
      "ref": "@e1",
      "role": "link",
      "name": "Home",
      "selector": "nav a:first-child"
    },
    {
      "ref": "@e2",
      "role": "searchbox",
      "name": "Search products",
      "selector": "input[type=\"search\"]"
    },
    {
      "ref": "@e3",
      "role": "button",
      "name": "Next Page",
      "selector": "button.pagination-next"
    },
    {
      "ref": "@e4",
      "role": "heading",
      "name": "Electronics",
      "selector": "h2.category-title"
    }
  ]
}

Element Properties

Important: Element refs (@e1) are stable within a single snapshot but may change between snapshots (after navigation or DOM mutations).

Complete Analysis Workflow

Step 1: Initial Reconnaissance

Navigate to the target page and capture the full structure:

agent-browser open https://example.com/products
agent-browser wait --load networkidle
agent-browser snapshot -i --json > full-snapshot.json

Step 2: Section Analysis

Capture scoped snapshots for focused analysis of different page regions:

# Navigation structure
agent-browser snapshot -i --json -s "nav" > nav-snapshot.json

# Main content area (where data lives)
agent-browser snapshot -i --json -s "main" > main-snapshot.json

# Footer (often has pagination)
agent-browser snapshot -i --json -s "footer" > footer-snapshot.json

# Any sidebar filters
agent-browser snapshot -i --json -s "aside" > sidebar-snapshot.json

Step 3: Semantic Grouping

From the snapshot data, the AI agent groups elements by purpose:

Step 4: Dynamic Behavior Discovery

Test interactions to understand page behavior:

# Test pagination
agent-browser click @e3  # Click "Next Page" button
agent-browser wait --load networkidle
agent-browser snapshot -i --json > after-pagination.json

# Compare: Did the URL change? Did content update?
# If elements changed, this confirms dynamic pagination

# Test search
agent-browser open https://example.com/products  # Reset
agent-browser wait --load networkidle
agent-browser type @e2 "laptop"
agent-browser press Enter
agent-browser wait --load networkidle
agent-browser snapshot -i --json > after-search.json

Step 5: Selector Validation

Verify discovered selectors work by interacting with them:

# Click a discovered element to confirm it's interactive
agent-browser click @e4

# Extract text to confirm content is readable
agent-browser get text ".product-card:first-child" --json

Step 6: Generate Page Objects

The AI agent uses the collected snapshots to:

1. Map elements to locators — Use the selector property from snapshots 2. Determine data fields — Headings, text content, and attributes become field names 3. Identify components — Pagination, tables, and repeated patterns become components 4. Create the scrape flow — Navigation order and pagination strategy

Mapping Snapshots to PageObjects

From Snapshot to Locator

{"ref": "@e4", "role": "heading", "name": "Product Title", "selector": "h2.product-title"}

Maps to:

readonly productTitle: Locator;
// In constructor:
this.productTitle = page.locator('h2.product-title');

From Roles to Methods

Limitations

1. Accessibility tree gaps — Some elements may not appear in the accessibility tree if they lack ARIA roles. Fall back to full-page snapshot or scoped CSS selectors.

2. Dynamic content — SPAs that load content lazily may not show all elements in the initial snapshot. Scroll and wait before re-snapshotting.

3. Shadow DOM — Elements inside shadow roots may not be discoverable via standard snapshots. Use Playwright's pierce selector engine as a fallback.

4. Element ref instability — Refs change between snapshots. Don't store refs across navigation; re-snapshot after each page change.

5. Rate limiting — Agent-browser opens a real browser session. Avoid rapid-fire commands; include waits between interactions.

See Also

• pageobject-pattern.md — How to structure generated page objects
• playwright-selectors.md — Selector resilience strategies
• ../SKILL.md — Generation Mode 1 overview

Anti-Pattern Catalog

Extended catalog of common web scraping mistakes, why they fail, and how to fix them.

1. Monolith Scraper

Pattern: All scraping logic in a single file — navigation, extraction, validation, and output all mixed together.

Why it fails:

• Impossible to reuse extraction logic for different pages
• One change to a selector requires reading the entire file
• Cannot test components in isolation
• Quickly exceeds 500+ lines and becomes unmaintainable

Fix: Split into PageObject classes (one per page), component classes (for reusable UI patterns), schemas (for validation), and a runner (for orchestration).

# Before (monolith)
scraper.ts (800 lines)

# After (structured)
src/pages/BasePage.ts
src/pages/ProductListingPage.ts
src/pages/ProductDetailPage.ts
src/components/Pagination.ts
src/schemas/product.schema.ts
src/scrapers/ShopScraper.ts

2. Sleep Waiter

Pattern: Using setTimeout, page.waitForTimeout(), or fixed delays instead of event-based waiting.

Why it fails:

• Too short: Element not loaded yet, extraction fails silently with empty strings
• Too long: Wastes time on every page load, scraper runs 5-10x slower than necessary
• Unreliable: Network conditions vary; what works locally breaks in Docker on slow networks

Fix: Use Playwright's built-in auto-wait and explicit wait conditions:

// Bad
await page.waitForTimeout(3000);

// Good — wait for network to be idle
await page.waitForLoadState('networkidle');

// Good — wait for specific element
await page.locator('.product-card').first().waitFor({ state: 'visible' });

// Good — wait for specific network response
await page.waitForResponse(resp => resp.url().includes('/api/products'));

3. Unvalidated Pipeline

Pattern: Extracting data and writing it directly to output without schema validation.

Why it fails:

• Selector drift (site changed markup) produces null/empty fields silently
• Malformed data propagates downstream causing failures in data consumers
• No way to detect partial failures (some fields extracted, others missed)
• Debugging requires comparing raw output against expected shape

Fix: Validate every record with a Zod schema before writing:

import { z } from 'zod';

const ProductSchema = z.object({
  title: z.string().min(1, 'Title is required'),
  price: z.number().positive('Price must be positive'),
  url: z.string().url('Must be a valid URL'),
});

// In scraper:
const raw = await page.scrapeProducts();
const validated = raw.map(item => {
  const result = ProductSchema.safeParse(item);
  if (!result.success) {
    console.warn(`Validation failed: ${result.error.message}`);
    return null;
  }
  return result.data;
}).filter(Boolean);

4. Selector Lottery

Pattern: Using fragile, position-dependent, or auto-generated selectors.

Why it fails:

• div > div > div > span breaks when any ancestor changes
• .css-1a2b3c (CSS-in-JS hash) changes on every build
• nth-child(3) breaks when items are reordered
• XPath like /html/body/div[2]/main/div[1] breaks on any structural change

Fix: Follow the selector resilience hierarchy:

// Bad — fragile positional selector
page.locator('div.container > div:nth-child(2) > span')

// Bad — CSS-in-JS hash
page.locator('.css-1a2b3c')

// Good — data attribute
page.locator('[data-testid="product-title"]')

// Good — semantic + aria
page.locator('[role="heading"][aria-level="2"]')

// Good — stable class name
page.locator('.product-card .product-title')

See playwright-selectors.md for the full priority hierarchy.

5. Silent Failure

Pattern: Catching errors and continuing without logging, screenshots, or metrics.

Why it fails:

• Scraper appears to succeed but returns empty or partial data
• No indication of which pages or items failed
• Cannot distinguish between "no data on page" and "selector broken"
• Hours of scraping wasted with no diagnostic information

Fix: Log failures, save debug screenshots, and track success metrics:

try {
  const title = await card.locator('.title').textContent();
  if (!title) throw new Error('Empty title');
  return title;
} catch (error) {
  console.error(`Failed to extract title for item ${index}: ${error}`);
  await this.page.screenshot({
    path: `screenshots/error-item-${index}-${Date.now()}.png`,
  });
  return null; // Return null, don't swallow silently
}

// After scraping, report metrics
console.log(`Scraped ${results.length} items, ${failures} failures`);

6. Unthrottled Crawler

Pattern: Making requests as fast as possible without delays between pages.

Why it fails:

• Gets IP blocked or rate-limited by the target site
• Triggers anti-bot protection (CAPTCHA, WAF blocks)
• Can overwhelm small sites, causing actual harm
• Results in inconsistent scraping (some pages blocked, others succeed)

Fix: Add configurable delays between requests:

const REQUEST_DELAY = parseInt(process.env.REQUEST_DELAY ?? '1000');

async function delay(ms: number): Promise<void> {
  return new Promise(resolve => setTimeout(resolve, ms));
}

// Between page navigations:
for (const url of urls) {
  await this.navigate(url);
  const data = await this.scrape();
  results.push(...data);
  await delay(REQUEST_DELAY);
}

7. Hardcoded Configuration

Pattern: Embedding URLs, selectors, page limits, and output paths as string literals in code.

Why it fails:

• Cannot run against different environments (staging vs production)
• Cannot adjust behavior without code changes
• Docker deployments require rebuilding images for config changes
• Team members with different setups cannot share the same code

Fix: Use environment variables with sensible defaults:

const config = {
  baseUrl: process.env.BASE_URL ?? 'https://example.com',
  maxPages: parseInt(process.env.MAX_PAGES ?? '10'),
  requestDelay: parseInt(process.env.REQUEST_DELAY ?? '1000'),
  outputDir: process.env.OUTPUT_DIR ?? './data',
  headless: process.env.HEADLESS !== 'false',
};

8. No Retry Logic

Pattern: Making a single attempt per request and treating any failure as fatal.

Why it fails:

• Transient network errors abort the entire scrape
• Occasional timeouts lose hours of accumulated data
• Rate-limit responses (429) could be waited out but instead cause failure

Fix: Implement exponential backoff with configurable limits:

async function withRetry<T>(fn: () => Promise<T>, maxAttempts = 3): Promise<T> {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      if (attempt === maxAttempts) throw error;
      const delay = Math.min(1000 * Math.pow(2, attempt - 1), 10000);
      console.warn(`Attempt ${attempt} failed, retrying in ${delay}ms...`);
      await new Promise(r => setTimeout(r, delay));
    }
  }
  throw new Error('Unreachable');
}

Quick Reference

See Also

• ../SKILL.md — Anti-patterns quick reference table
• playwright-selectors.md — Selector resilience strategies
• pageobject-pattern.md — Proper code structure

Docker Setup for Playwright Scrapers

Running scrapers in Docker ensures consistent browser environments, eliminates "works on my machine" issues, and simplifies deployment to servers and CI.

Official Playwright Docker Images

Microsoft maintains official Docker images with all browser dependencies pre-installed:

mcr.microsoft.com/playwright:v1.48.0-jammy

These images include:

• Chromium, Firefox, and WebKit browsers
• All system dependencies (fonts, libraries, codecs)
• Node.js runtime
• Non-root user pwuser (UID 1000)

Image Tags

Always pin the Playwright image version to match your package.json Playwright version. Mismatched versions cause browser launch failures.

Dockerfile

# Build stage
FROM mcr.microsoft.com/playwright:v1.48.0-jammy AS builder

WORKDIR /app

# Copy dependency manifests first (Docker cache optimization)
COPY package*.json ./
RUN npm ci

# Copy source and compile
COPY tsconfig.json ./
COPY src/ ./src/
RUN npx tsc

# Production stage
FROM mcr.microsoft.com/playwright:v1.48.0-jammy

WORKDIR /app

COPY package*.json ./
RUN npm ci --omit=dev

COPY --from=builder /app/dist ./dist

# Create output directories
RUN mkdir -p data screenshots

# Run as non-root
USER pwuser

CMD ["node", "dist/index.js"]

Build Decisions

• Multi-stage build — Keeps TypeScript compiler and dev dependencies out of the production image
• Dependency cache — Copy package*.json before source code so npm ci is cached when only code changes
• Non-root user — pwuser is included in the Playwright image; use it for security
• Output directories — Create data/ and screenshots/ in the image; mount volumes at runtime for persistence

Docker Compose

services:
  scraper:
    build: .
    environment:
      - NODE_ENV=production
      - BASE_URL=${BASE_URL:-https://example.com}
      - HEADLESS=true
      - MAX_PAGES=${MAX_PAGES:-10}
      - REQUEST_DELAY=${REQUEST_DELAY:-1000}
    volumes:
      - ./data:/app/data
      - ./screenshots:/app/screenshots
    # Resource limits prevent runaway browser processes
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: '2'

Volume Mounts

Environment Variables

Running

Build and Run

# Build the image
docker compose build

# Run the scraper
docker compose up

# Run with custom URL
BASE_URL=https://shop.example.com docker compose up

# Run in background
docker compose up -d

# View logs
docker compose logs -f scraper

One-Off Runs

# Run once and remove container
docker compose run --rm scraper

# Override command for debugging
docker compose run --rm scraper node -e "console.log('Browser check')"

Playwright Configuration for Docker

When running in Docker, Playwright needs specific configuration:

import { chromium } from 'playwright';

const browser = await chromium.launch({
  headless: true,
  args: [
    '--no-sandbox',
    '--disable-setuid-sandbox',
    '--disable-dev-shm-usage',  // Use /tmp instead of /dev/shm
    '--disable-gpu',
  ],
});

--disable-dev-shm-usage

Docker containers have a small /dev/shm (shared memory) by default (64MB). Chromium uses /dev/shm for shared memory, which can cause crashes. This flag tells Chromium to use /tmp instead. Alternatively, increase shared memory in docker-compose:

services:
  scraper:
    shm_size: '512mb'

CI Integration

GitHub Actions

jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker compose build
      - run: docker compose run --rm scraper
      - uses: actions/upload-artifact@v4
        with:
          name: scraped-data
          path: data/

Scheduled Runs

on:
  schedule:
    - cron: '0 6 * * *'  # Daily at 6 AM UTC

Troubleshooting

See Also

• ../assets/configs/dockerfile.md — Dockerfile template
• ../assets/configs/docker-compose.yml.md — Docker Compose template
• pageobject-pattern.md — PageObject pattern overview

PageObject Pattern for Web Scraping

The PageObject pattern, originally designed for test automation, adapts naturally to web scraping. The key difference: in testing, page objects abstract interactions to make assertions; in scraping, page objects abstract element access to extract data.

Core Concept

Each distinct page (or view) on the target site maps to one TypeScript class. The class encapsulates:

1. Locators — How to find elements on the page 2. Extraction methods — How to pull data from those elements 3. Navigation methods — How to move between pages or states

BasePage Design

Every page object extends a shared BasePage that provides common functionality:

import { Page } from 'playwright';

export abstract class BasePage {
  constructor(protected readonly page: Page) {}

  async navigate(url: string): Promise<void> {
    await this.page.goto(url, { waitUntil: 'networkidle' });
  }

  async waitForPageLoad(): Promise<void> {
    await this.page.waitForLoadState('networkidle');
  }

  async screenshot(name: string): Promise<void> {
    await this.page.screenshot({
      path: `screenshots/${name}.png`,
      fullPage: true,
    });
  }

  async getText(selector: string): Promise<string> {
    const el = this.page.locator(selector);
    return (await el.textContent()) ?? '';
  }

  async getTexts(selector: string): Promise<string[]> {
    const els = this.page.locator(selector);
    return els.allTextContents();
  }

  async getAttribute(selector: string, attr: string): Promise<string | null> {
    return this.page.locator(selector).getAttribute(attr);
  }

  async exists(selector: string): Promise<boolean> {
    return (await this.page.locator(selector).count()) > 0;
  }
}

Design Decisions

• protected readonly page — Page objects share the Playwright Page instance but don't expose it publicly
• networkidle — Default wait strategy for scraping; ensures dynamic content has loaded
• screenshot() — Debug tool; save screenshots on errors to diagnose selector failures
• Helper methods (getText, getTexts, getAttribute) reduce boilerplate in subclasses

Site-Specific Page Objects

Each page object defines locators as readonly properties in the constructor and provides typed extraction methods:

import { Page, Locator } from 'playwright';
import { BasePage } from './BasePage';

interface Product {
  title: string;
  price: string;
  imageUrl: string | null;
  url: string | null;
}

export class ProductListingPage extends BasePage {
  readonly productCards: Locator;
  readonly productTitle: Locator;
  readonly productPrice: Locator;
  readonly productImage: Locator;
  readonly productLink: Locator;

  constructor(page: Page) {
    super(page);
    this.productCards = page.locator('[data-testid="product-card"]');
    this.productTitle = page.locator('[data-testid="product-title"]');
    this.productPrice = page.locator('[data-testid="product-price"]');
    this.productImage = page.locator('[data-testid="product-card"] img');
    this.productLink = page.locator('[data-testid="product-card"] a');
  }

  async scrapeProducts(): Promise<Product[]> {
    const products: Product[] = [];
    const count = await this.productCards.count();

    for (let i = 0; i < count; i++) {
      const card = this.productCards.nth(i);
      products.push({
        title: (await card.locator('[data-testid="product-title"]').textContent()) ?? '',
        price: (await card.locator('[data-testid="product-price"]').textContent()) ?? '',
        imageUrl: await card.locator('img').getAttribute('src'),
        url: await card.locator('a').first().getAttribute('href'),
      });
    }

    return products;
  }
}

Key Rules

1. Locators in constructor — Define all locators when the object is created, not in methods 2. Methods return data — Scrape methods return typed objects, not void 3. No assertions — Page objects extract data; validation happens in schemas 4. No side effects — Page objects don't write files or log results

Component Composition

Reusable UI patterns become component classes that page objects use via composition:

export class ProductListingPage extends BasePage {
  readonly pagination: Pagination;
  readonly productCards: Locator;

  constructor(page: Page) {
    super(page);
    this.pagination = new Pagination(page);
    this.productCards = page.locator('.product-card');
  }

  async scrapeAllPages(): Promise<Product[]> {
    const allProducts: Product[] = [];

    do {
      const products = await this.scrapeCurrentPage();
      allProducts.push(...products);
    } while (await this.pagination.hasNextPage() && await this.pagination.goToNext());

    return allProducts;
  }
}

Components differ from page objects:

• They don't extend BasePage
• They receive a scope (parent locator or full page)
• They model a UI widget, not a full page
• Multiple components can exist on one page

Page Navigation Patterns

Single Page → Detail Pattern

When a listing page links to detail pages:

async scrapeWithDetails(): Promise<ProductDetail[]> {
  const links = await this.getProductLinks();
  const details: ProductDetail[] = [];

  for (const link of links) {
    await this.navigate(link);
    const detailPage = new ProductDetailPage(this.page);
    details.push(await detailPage.scrapeDetail());
    await this.page.goBack();
    await this.waitForPageLoad();
  }

  return details;
}

Multi-Step Flow Pattern

When scraping requires navigating through multiple distinct pages:

class ScraperFlow {
  async run(page: Page): Promise<Result[]> {
    const searchPage = new SearchPage(page);
    await searchPage.navigate('https://example.com/search');
    await searchPage.search('query');

    const resultsPage = new SearchResultsPage(page);
    const items = await resultsPage.scrapeResults();

    return items;
  }
}

Scraping vs Testing: Key Differences

File Organization

src/pages/
├── BasePage.ts              # Abstract base
├── ProductListingPage.ts    # One per page type
├── ProductDetailPage.ts
└── SearchResultsPage.ts

src/components/
├── Pagination.ts            # Reusable UI patterns
├── DataTable.ts
└── FilterSidebar.ts

See Also

• ../assets/templates/base-page.ts.md — BasePage template
• ../assets/templates/page-object.ts.md — PageObject template
• ../assets/templates/component.ts.md — Component template
• playwright-selectors.md — Selector strategies

Playwright Selector Strategies

Selector choice is the most critical factor in scraper reliability. A fragile selector breaks on every site update; a resilient selector survives layout changes, redesigns, and A/B tests.

Selector Priority Hierarchy

Use selectors in this order, from most to least resilient:

1. Data Attributes (Most Resilient)

page.locator('[data-testid="product-card"]')
page.locator('[data-product-id="123"]')
page.locator('[data-automation="price"]')

Why: Data attributes exist specifically for programmatic access. They survive CSS refactors, class name changes, and layout updates. Sites that use them rarely remove them.

2. ID Selectors

page.locator('#product-list')
page.locator('#search-results')

Why: IDs are unique per page and typically stable. Avoid IDs that look auto-generated (e.g., #a3f7b2).

3. Semantic HTML and ARIA

page.locator('nav[aria-label="main"]')
page.locator('[role="search"]')
page.locator('[aria-label="Next page"]')
page.locator('article')

Why: Semantic attributes reflect the page's structure and purpose. They change less often than visual styles.

4. Structured CSS Classes

page.locator('.product-card')
page.locator('.search-results__item')

Why: Meaningful class names (BEM, component-based) are reasonably stable. Avoid utility classes like .mt-4 or .flex.

5. Playwright Text Selectors

page.locator('button:has-text("Add to Cart")')
page.locator('a:has-text("Next")')
page.getByText('View Details')

Why: Text content is stable when the site's language doesn't change. Useful for buttons and links. Fragile for internationalized sites.

6. Playwright Role Selectors

page.getByRole('button', { name: 'Submit' })
page.getByRole('heading', { level: 2 })
page.getByRole('link', { name: 'Products' })

Why: Combines ARIA roles with accessible names. More resilient than raw CSS but may match multiple elements.

7. CSS Combinators (Use with Caution)

page.locator('.product-card h2')
page.locator('.results > .item')

Why: Depends on DOM structure. Breaks when nesting changes.

8. Positional Selectors (Least Resilient)

page.locator('.item').nth(0)
page.locator('tr:nth-child(2) td:nth-child(3)')

Why: Depends on element order. Breaks when items are reordered, added, or removed. Only use when iterating a known collection.

Playwright-Specific Selector Engines

Built-in Locator Methods

// Preferred API — more readable and type-safe
page.getByRole('button', { name: 'Submit' })
page.getByText('Product Title')
page.getByLabel('Search')
page.getByPlaceholder('Enter search term')
page.getByAltText('Product image')
page.getByTitle('Close dialog')
page.getByTestId('product-card')  // matches data-testid by default

Chaining Locators

// Scope a locator within another
const card = page.locator('.product-card').nth(i);
const title = card.locator('h2');
const price = card.locator('.price');

Filtering

// Filter by text
page.locator('.product-card').filter({ hasText: 'Sale' })

// Filter by child locator
page.locator('.product-card').filter({
  has: page.locator('.discount-badge'),
})

Resilience Strategies

1. Multiple Selector Fallback

When you're unsure which selector will work, use comma-separated CSS:

page.locator('[data-testid="next"], [aria-label="Next page"], a.next-page')

Playwright matches the first visible element from any of these selectors.

2. Schema-Driven Validation

Don't trust selectors blindly. Validate extracted data:

const price = await card.locator('.price').textContent();
const parsed = parseFloat(price?.replace(/[^0-9.]/g, '') ?? '');
if (isNaN(parsed)) {
  console.warn(`Invalid price for item ${i}: "${price}"`);
}

3. Conditional Selectors

Handle sites that use different markup for different states:

async getPrice(card: Locator): Promise<string> {
  const salePrice = card.locator('.sale-price');
  if (await salePrice.count() > 0) {
    return (await salePrice.textContent()) ?? '';
  }
  return (await card.locator('.regular-price').textContent()) ?? '';
}

4. Wait for Stability

Ensure elements are fully rendered before extracting:

await page.waitForLoadState('networkidle');
await page.locator('.product-card').first().waitFor({ state: 'visible' });

Debugging Selectors

Playwright Inspector

During development, use headed mode to see what selectors match:

const browser = await chromium.launch({ headless: false });

Count Check

Verify your selector matches the expected number of elements:

const count = await page.locator('.product-card').count();
console.log(`Found ${count} product cards`);

Screenshot on Failure

Save a screenshot whenever extraction fails:

try {
  const title = await card.locator('.title').textContent();
  if (!title) throw new Error('No title found');
} catch (error) {
  await page.screenshot({ path: `screenshots/error-${Date.now()}.png` });
  throw error;
}

Common Selector Patterns by Element Type

See ../data/selector-patterns.json for a comprehensive mapping of UI element types to common selectors, organized by:

• Product cards
• Pagination controls
• Data tables
• Search inputs
• Navigation menus
• Form elements

See Also

• pageobject-pattern.md — How selectors fit into page objects
• agent-browser-workflow.md — Automated selector discovery
• anti-patterns.md — Selector anti-patterns to avoid