Web Scraping

Web Scraping with Intelligent Strategy Selection

When This Skill Activates

Activate automatically when user requests:

• "Scrape [website]"
• "Extract data from [site]"
• "Get product information from [URL]"
• "Find all links/pages on [site]"
• "I'm getting blocked" or "Getting 403 errors" (loads strategies/anti-blocking.md)
• "Make this an Apify Actor" (loads apify/ subdirectory)
• "Productionize this scraper"

Input Parsing

Determine reconnaissance depth from user request:

Default is Standard mode. Escalate to Full if protection signals appear during any phase.

Adaptive Reconnaissance Workflow

This skill uses an adaptive phased workflow with quality gates. Each gate asks "Do I have enough?" — continue only when the answer is no.

See: strategies/framework-signatures.md for framework detection tables referenced throughout.

Phase 0: QUICK ASSESSMENT (curl, no browser)

Gather maximum intelligence with minimum cost — a single HTTP request.

Step 0a: Fetch raw HTML and headers

curl -s -D- -L "https://target.com/page" -o response.html

Step 0b: Check response headers

• Match headers against strategies/framework-signatures.md → Response Header Signatures table
• Note Server, X-Powered-By, X-Shopify-Stage, Set-Cookie (protection markers)
• Check HTTP status code (200 = accessible, 403 = protected, 3xx = redirects)

Step 0c: Check Known Major Sites table

• Match domain against strategies/framework-signatures.md → Known Major Sites
• If matched: use the specified data strategy, skip generic pattern scanning

Step 0d: Detect framework from HTML

• Search raw HTML for signatures in strategies/framework-signatures.md → HTML Signatures table
• Look for __NEXT_DATA__, __NUXT__, ld+json, /wp-content/, data-reactroot

Step 0e: Search for target data points

• For each data point the user wants: search raw HTML for that content
• Track which data points are found vs missing
• Check for sitemaps: curl -s https://[site]/robots.txt | grep -i Sitemap

Step 0f: Note protection signals

• 403/503 status, Cloudflare challenge HTML, CAPTCHA elements, cf-ray header
• Record for Phase 4 decision

See: strategies/cheerio-vs-browser-test.md for the Cheerio viability assessment

QUALITY GATE A: All target data points found in raw HTML + no protection signals?

→ YES: Skip to Phase 3 (Validate Findings). No browser needed.

→ NO: Continue to Phase 1.

Phase 1: BROWSER RECONNAISSANCE (only if Phase 0 needs it)

Launch browser only for data points missing from raw HTML or when JavaScript rendering is required.

Step 1a: Initialize browser session

• proxy_start() → Start traffic interception proxy
• interceptor_chrome_launch(url, stealthMode: true) → Launch Chrome with anti-detection
• interceptor_chrome_devtools_attach(target_id) → Attach DevTools bridge
• interceptor_chrome_devtools_screenshot() → Capture visual state

Step 1b: Capture traffic and rendered DOM

• proxy_list_traffic() → Review all traffic from page load
• proxy_search_traffic(query: "application/json") → Find JSON responses
• interceptor_chrome_devtools_list_network(resource_types: ["xhr", "fetch"]) → XHR/fetch calls
• interceptor_chrome_devtools_snapshot() → Accessibility tree (rendered DOM)

Step 1c: Search rendered DOM for missing data points

• For each data point NOT found in Phase 0: search rendered DOM
• Use framework-specific search strategy from strategies/framework-signatures.md → Framework → Search Strategy table
• Only search patterns relevant to the detected framework

Step 1d: Inspect discovered endpoints

• proxy_get_exchange(exchange_id) → Full request/response for promising endpoints
• Document: method, headers, auth, response structure, pagination

QUALITY GATE B: All target data points now covered (raw HTML + rendered DOM + traffic)?

→ YES: Skip to Phase 3 (Validate Findings). No deep scan needed.

→ NO: Continue to Phase 2 for missing data points only.

Phase 2: DEEP SCAN (only for missing data points)

Targeted investigation for data points not yet found. Only search for what's missing.

Step 2a: Test interactions for missing data

• proxy_clear_traffic() before each action → Isolate API calls
• humanizer_click(target_id, selector) → Trigger dynamic content loads
• humanizer_scroll(target_id, direction, amount) → Trigger lazy loading / infinite scroll
• humanizer_idle(target_id, duration_ms) → Wait for delayed content
• After each action: proxy_list_traffic() → Check for new API calls

Step 2b: Sniff APIs (framework-aware)

• Search only patterns relevant to detected framework:
• Next.js → proxy_list_traffic(url_filter: "/_next/data/")
• WordPress → proxy_list_traffic(url_filter: "/wp-json/")
• GraphQL → proxy_search_traffic(query: "graphql")
• Generic → proxy_list_traffic(url_filter: "/api/") + proxy_search_traffic(query: "application/json")
• Skip patterns that don't apply to the detected framework

Step 2c: Test pagination and filtering

• Only if pagination data is a missing data point or needed for coverage assessment
• proxy_clear_traffic() → click next page → proxy_list_traffic(url_filter: "page=")
• Document pagination type (URL-based, API offset, cursor, infinite scroll)

QUALITY GATE C: Enough data points covered for a useful report?

→ YES: Go to Phase 3.

→ NO: Document gaps, go to Phase 3 anyway (report will note missing data in self-critique).

Phase 3: VALIDATE FINDINGS

Every claimed extraction method must be verified. A data point is not "found" until the extraction path is specified and tested.

See: strategies/cheerio-vs-browser-test.md for validation methodology

Step 3a: Validate CSS selectors

• For each Cheerio/selector-based method: confirm the selector matches actual HTML
• Test against raw HTML (curl output) or rendered DOM (snapshot)
• Confirm selector extracts the correct value, not a different element

Step 3b: Validate JSON paths

• For each JSON extraction (e.g., __NEXT_DATA__, API response): confirm the path resolves
• Parse the JSON, follow the path, verify it returns the expected data type and value

Step 3c: Validate API endpoints

• For each discovered API: replay the request (curl or proxy_get_exchange)
• Confirm: response status 200, expected data structure, correct values
• Test pagination if claimed (at least page 1 and page 2)

Step 3d: Downgrade or re-investigate failures

• If a selector doesn't match: try alternative selectors, or downgrade to PARTIAL confidence
• If an API returns 403: note protection requirement, flag for Phase 4
• If a JSON path is wrong: re-examine the JSON structure, correct the path

Phase 4: PROTECTION TESTING (conditional)

See: strategies/proxy-escalation.md for complete skip/run decision logic

Skip Phase 4 when ALL true:

• No protection signals detected in Phases 0-2
• All data points have validated extraction methods
• User didn't request "full recon"

Run Phase 4 when ANY true:

• 403/challenge page observed during any phase
• Known high-protection domain
• High-volume or production intent
• User explicitly requested it

If running:

Step 4a: Test raw HTTP access

curl -s -o /dev/null -w "%{http_code}" "https://target.com/page"

• 200 → Cheerio viable, no browser needed for accessible endpoints
• 403/503 → Escalate to stealth browser

Step 4b: Test with stealth browser (if needed)

• Already running from Phase 1 — check if pages loaded without challenges
• interceptor_chrome_devtools_list_cookies(domain_filter: "cloudflare") → Protection cookies
• interceptor_chrome_devtools_list_storage_keys(storage_type: "local") → Fingerprint markers
• proxy_get_tls_fingerprints() → TLS fingerprint analysis

Step 4c: Test with upstream proxy (if needed)

• proxy_set_upstream("http://user:pass@proxy-provider:port")
• Re-test blocked endpoints through proxy
• Document minimum access level for each data point

Step 4d: Document protection profile

• What protections exist, what worked to bypass them, what production scrapers will need

Phase 5: REPORT + SELF-CRITIQUE

Generate the intelligence report, then critically review it for gaps.

See: reference/report-schema.md for complete report format

Step 5a: Generate report

• Follow reference/report-schema.md schema (Sections 1-6)
• Include Validated? status for every strategy (YES / PARTIAL / NO)
• Include all discovered endpoints with full specs

Step 5b: Self-critique

• Write Section 7 (Self-Critique) per reference/report-schema.md:
• Gaps: Data points not found — why, and what would find them
• Skipped steps: Which phases skipped, with quality gate reasoning
• Unvalidated claims: Anything marked PARTIAL or NO
• Assumptions: Things not verified (e.g., "consistent layout across categories")
• Staleness risk: Geo-dependent prices, A/B layouts, session-specific content
• Recommendations: Targeted next steps (not "re-run everything")

Step 5c: Fix gaps with targeted re-investigation

• If self-critique reveals fixable gaps: go back to the specific phase/step, not a full re-run
• Example: "Price selector untested" → run one curl + parse, don't re-launch browser
• Update report with results

Step 5d: Record session (if browser was used)

• proxy_session_start(name) → proxy_session_stop(session_id) → proxy_export_har(session_id, path)
• HAR file captures all traffic for replay. See strategies/session-workflows.md

---

IMPLEMENTATION (after reconnaissance)

After reconnaissance report is accepted, implement scraper iteratively.

Core Pattern: 1. Implement recommended approach (minimal code) 2. Test with small batch (5-10 items) 3. Validate data quality 4. Scale to full dataset or fallback 5. Handle blocking if encountered 6. Add robustness (error handling, retries, logging)

See: workflows/implementation.md for complete implementation patterns and code examples

PRODUCTIONIZATION (on request)

Convert scraper to production-ready Apify Actor.

Activation triggers: "Make this an Apify Actor", "Productionize this", "Deploy to Apify"

Core Pattern: 1. Confirm TypeScript preference (STRONGLY RECOMMENDED) 2. Initialize with apify create command (CRITICAL) 3. Port scraping logic to Actor format 4. Test locally and deploy

Note: During development, proxy-mcp provides reconnaissance and traffic analysis. For production Actors, use Crawlee crawlers (CheerioCrawler/PlaywrightCrawler) on Apify infrastructure.

See: workflows/productionization.md for complete workflow and apify/ for Actor development guides

Quick Reference

Common Patterns

Pattern 1: Sitemap-Based Scraping

import { RobotsFile, CheerioCrawler, Dataset } from 'crawlee';

// Auto-discover and parse sitemaps
const robots = await RobotsFile.find('https://example.com');
const urls = await robots.parseUrlsFromSitemaps();

const crawler = new CheerioCrawler({
    async requestHandler({ $, request }) {
        const data = {
            title: $('h1').text().trim(),
            // ... extract data
        };
        await Dataset.pushData(data);
    },
});

await crawler.addRequests(urls);
await crawler.run();

See examples/sitemap-basic.js for complete example.

Pattern 2: API-Based Scraping

import { gotScraping } from 'got-scraping';

const productIds = [123, 456, 789];

for (const id of productIds) {
    const response = await gotScraping({
        url: `https://api.example.com/products/${id}`,
        responseType: 'json',
    });

    console.log(response.body);
}

See examples/api-scraper.js for complete example.

Pattern 3: Hybrid (Sitemap + API)

// Get URLs from sitemap
const robots = await RobotsFile.find('https://shop.com');
const urls = await robots.parseUrlsFromSitemaps();

// Extract IDs from URLs
const productIds = urls
    .map(url => url.match(/\/products\/(\d+)/)?.[1])
    .filter(Boolean);

// Fetch data via API
for (const id of productIds) {
    const data = await gotScraping({
        url: `https://api.shop.com/v1/products/${id}`,
        responseType: 'json',
    });
    // Process data
}

See examples/hybrid-sitemap-api.js for complete example.

Directory Navigation

This skill uses progressive disclosure - detailed information is organized in subdirectories and loaded only when needed.

Workflows (Implementation Patterns)

For: Step-by-step workflow guides for each phase

• workflows/reconnaissance.md - Phase 1 interactive reconnaissance (CRITICAL)
• workflows/implementation.md - Phase 4 iterative implementation patterns
• workflows/productionization.md - Phase 5 Apify Actor creation workflow

Strategies (Deep Dives)

For: Detailed guides on specific scraping approaches

• strategies/framework-signatures.md - Framework detection lookup tables (Phase 0/1)
• strategies/cheerio-vs-browser-test.md - Cheerio vs Browser decision test with early exit
• strategies/proxy-escalation.md - Protection testing skip/run conditions (Phase 4)
• strategies/traffic-interception.md - Traffic interception via MITM proxy
• strategies/sitemap-discovery.md - Complete sitemap guide (4 patterns)
• strategies/api-discovery.md - Finding and using APIs
• strategies/dom-scraping.md - DOM scraping via DevTools bridge
• strategies/cheerio-scraping.md - HTTP-only scraping
• strategies/hybrid-approaches.md - Combining strategies
• strategies/anti-blocking.md - Multi-layer anti-detection (stealth, humanizer, proxies, TLS)
• strategies/session-workflows.md - Session recording, HAR export, replay

Examples (Runnable Code)

For: Working code to reference or execute

JavaScript Learning Examples (Simple standalone scripts):

• examples/sitemap-basic.js - Simple sitemap scraper
• examples/api-scraper.js - Pure API approach
• examples/traffic-interception-basic.js - Proxy-based reconnaissance
• examples/hybrid-sitemap-api.js - Combined approach
• examples/iterative-fallback.js - Try traffic interception→sitemap→API→DOM scraping

TypeScript Production Examples (Complete Actors):

• apify/examples/basic-scraper/ - Sitemap + Playwright
• apify/examples/anti-blocking/ - Fingerprinting + proxies
• apify/examples/hybrid-api/ - Sitemap + API (optimal)

Reference (Quick Lookup)

For: Quick patterns and troubleshooting

• reference/report-schema.md - Intelligence report format (Sections 1-7 + self-critique)
• reference/proxy-tool-reference.md - Proxy-MCP tool reference (all 80+ tools)
• reference/regex-patterns.md - Common URL regex patterns
• reference/fingerprint-patterns.md - Stealth mode + TLS fingerprint presets
• reference/anti-patterns.md - What NOT to do

Apify (Production Deployment)

For: Creating production Apify Actors

• apify/README.md - When and how to use Apify
• apify/typescript-first.md - Why TypeScript for actors
• apify/cli-workflow.md - apify create workflow (CRITICAL)
• apify/initialization.md - Complete setup guide
• apify/input-schemas.md - Input validation patterns
• apify/configuration.md - actor.json setup
• apify/deployment.md - Testing and deployment
• apify/templates/ - TypeScript boilerplate

Note: Each file is self-contained and can be read independently. Claude will navigate to specific files as needed.

Core Principles

1. Assess Before Committing Resources

Start cheap (curl), escalate only when needed:

• Phase 0 (curl) before Phase 1 (browser) before Phase 2 (deep scan)
• Quality gates skip phases when data is sufficient
• Never launch a browser if curl gives you everything

2. Detect First, Then Search Relevant Patterns

Use framework detection to focus searches:

• Match against strategies/framework-signatures.md before scanning
• Skip patterns that don't apply (no __NEXT_DATA__ on Amazon)
• Known major sites get direct strategy lookup

3. Validate, Don't Assume

Every claimed extraction method must be tested:

• "Found text in HTML" is not enough — need a working selector/path
• Phase 3 validates every finding before the report
• Unvalidated claims are marked PARTIAL or NO in the report

4. Iterative Implementation

Build incrementally:

• Small test batch first (5-10 items)
• Validate quality
• Scale or fallback
• Add robustness last

5. Production-Ready Code

When productionizing:

• Use TypeScript (strongly recommended)
• Use apify create (never manual setup)
• Add proper error handling
• Include logging and monitoring

---

Remember: Traffic interception first, sitemaps second, APIs third, DOM scraping last!

For detailed guidance on any topic, navigate to the relevant subdirectory file listed above.

Understanding AGENTS.md in Apify Templates

What is AGENTS.md?

AGENTS.md is official Apify documentation included in Actor templates (generated by apify create). It provides AI agent-specific guidance for Actor development, maintained by Apify.

Location

After running apify create my-scraper, you'll find:

my-scraper/
├── AGENTS.md          ← Official Apify AI agent guidance
├── .actor/
├── src/
└── ...

Scope Division

This Skill Provides: Strategy & Workflow

When and why:

• Phase 1: Interactive reconnaissance (Proxy-MCP traffic interception + stealth browser)
• Phase 2: Strategy discovery (sitemaps, APIs)
• Phase 3: Strategy recommendation (which approach to use)
• Phase 4: Iterative implementation
• Phase 5: Actor productionization workflow

Template selection:

• Decision tree: Cheerio vs Playwright vs Camoufox
• When to use each template based on site characteristics
• Performance trade-offs

Integration:

• How to convert scraper to Actor
• When to productionize
• Deployment workflow

AGENTS.md Provides: Implementation Details

How to implement:

• Do/Don't patterns for Actor code
• SDK usage best practices
• Concurrency settings (HTTP: 10-50, Browser: 1-5)
• Error handling patterns
• Safety and permission guidelines

Schema specifications:

• Input schema detailed structure and examples
• Output schema with template variables
• Dataset schema with views and transformations
• Key-value store schema with collections

Apify-specific patterns:

• When to use Actor.getValue() vs Dataset.getData()
• How to structure router patterns
• Retry strategies with exponential backoff
• Storage management patterns

When to Reference AGENTS.md

During Actor Development (Phase 5)

After `apify create`: 1. Read AGENTS.md for Do/Don't patterns 2. Use input schema examples from AGENTS.md 3. Reference output schema specifications 4. Follow SDK best practices

While coding:

• Check AGENTS.md for concurrency settings
• Verify proper error handling patterns
• Ensure proper dataset/key-value store usage

What This Skill Already Covers

Don't duplicate AGENTS.md for:

• Basic CLI workflow (apify create, apify run, apify push) - covered in cli-workflow.md
• TypeScript recommendation - covered in typescript-first.md
• Template selection logic - covered in productionization.md

Use AGENTS.md for:

• Detailed schema examples beyond what's in input-schemas.md
• Dataset transformation and view configuration
• Key-value store collection patterns
• Advanced SDK usage patterns

Key Content in AGENTS.md

1. Do/Don't Lists

Aligned with this skill:

• ✅ "use CheerioCrawler for static HTML (10x faster)" - matches our template selection
• ✅ "use PlaywrightCrawler only for JavaScript-heavy sites" - matches our decision tree
• ✅ "validate input early with proper error handling" - production best practice

Adds value:

• Concurrency settings: HTTP: 10-50, Browser: 1-5
• Retry strategies with exponential backoff
• Deprecated options to avoid (e.g., requestHandlerTimeoutMillis on CheerioCrawler v3.x)

2. Schema Specifications

What we cover (in input-schemas.md):

• Basic field types (string, number, boolean, array)
• Common patterns (startUrls, maxItems, proxy config)
• 6 complete examples

What AGENTS.md adds:

• Output schema structure and template variables
• Dataset schema with views, transformations, and display components
• Key-value store schema with collections and content type validation
• Complete examples with all properties

3. Safety and Permissions

AGENTS.md provides:

• Allowed without prompt: Actor.getValue(), Dataset.pushData(), apify run
• Ask first: npm installations, apify push, proxy changes, Dockerfile changes

This skill assumes:

• User controls deployment decisions
• Focus on workflow, not permission model

4. Resources and References

AGENTS.md links:

• Apify MCP tools (search-apify-docs, fetch-apify-docs)
• llms.txt and llms-full.txt
• Crawlee documentation
• Actor whitepaper

This skill links:

• Anthropic best practices
• Our own workflow documentation
• Apify CLI and platform docs

Integration Workflow

Recommended Approach

Phase 1-4: Use this skill → Reconnaissance, strategy, implementation, testing

Phase 5: Combine both → This skill for apify create workflow → AGENTS.md for implementation details

Example:

1. User: "Make this an Apify Actor"
2. Claude references: workflows/productionization.md (this skill)
3. Claude runs: apify create my-scraper
4. Claude notes: "Template includes AGENTS.md - reference for schema details"
5. User develops Actor
6. Claude references AGENTS.md for:
   - Input schema detailed examples
   - Dataset view configuration
   - SDK best practices

Quick Reference: When to Use Which

Best Practices

✅ DO:

• Reference AGENTS.md after apify create for implementation details
• Use this skill for strategy and workflow decisions
• Combine both: our workflow guidance + AGENTS.md implementation details
• Check AGENTS.md for Apify-specific patterns (concurrency, storage, SDK)
• Use AGENTS.md schema examples when configuring complex views

❌ DON'T:

• Duplicate AGENTS.md content in this skill (maintenance burden)
• Ignore AGENTS.md - it's official Apify guidance
• Reference AGENTS.md before apify create (it doesn't exist yet)
• Choose templates based on AGENTS.md (use our decision tree instead)

Summary

AGENTS.md and this skill are complementary:

• This skill: Strategy, workflow, when/why (reconnaissance → Actor creation)
• AGENTS.md: Implementation, how, Apify-specific patterns (schemas, SDK, Do/Don't)

Integration point: After apify create, reference AGENTS.md for implementation details while using this skill for overall workflow and strategy.

Result: Best of both worlds - strategic workflow from this skill + official Apify implementation guidance from AGENTS.md.

---

Back to Apify module: README.md

Apify CLI Workflow

Overview

CRITICAL: Always use `apify create` command when starting a new Actor.

This is THE recommended and ONLY proper way to initialize Apify Actors.

Why apify create is CRITICAL

✅ Auto-Generated Files

The apify create command generates:

• ✅ package.json with correct dependencies and scripts
• ✅ .actor/actor.json with proper structure
• ✅ .actor/input_schema.json template
• ✅ Dockerfile with correct base image
• ✅ tsconfig.json (for TypeScript templates)
• ✅ eslint.config.js for code quality
• ✅ .gitignore with Apify-specific entries
• ✅ storage/ directory structure
• ✅ README.md template
• ✅ Example source code

✅ Proper Tooling Setup

Automatically configures:

• ESLint for code quality
• TypeScript compilation (for TS templates)
• npm scripts (start, build, test)
• Apify SDK with correct version
• Crawlee with correct version

❌ What Happens Without apify create

Manual setup leads to:

• ❌ Missing ESLint configuration
• ❌ Incorrect dependencies/versions
• ❌ Poor project structure
• ❌ Missing .actor/ directory
• ❌ Incorrect Dockerfile
• ❌ More debugging time
• ❌ Deployment failures

Step-by-Step Workflow

Step 1: Install Apify CLI

# Check if already installed
apify --version

# If not installed
npm install -g apify-cli

# Verify installation
apify --version

Step 2: Login to Apify

# Login (required for push/deployment)
apify login

# This opens browser for authentication

Step 3: Create New Actor

# Create actor
apify create my-scraper

# You'll be prompted:
# → What type of Actor do you want to create?

Step 4: Choose Template

Choose based on site type (see ../workflows/productionization.md for decision tree):

Available TypeScript templates:

Selection guide:

• Static HTML → project_cheerio_crawler_ts (fastest)
• JavaScript/SPA → project_playwright_crawler_ts
• Being blocked → project_playwright_camoufox_crawler_ts

? What type of Actor do you want to create?
❯ project_cheerio_crawler_ts (TypeScript + Cheerio)
  project_playwright_crawler_ts (TypeScript + Playwright)
  project_playwright_camoufox_crawler_ts (TypeScript + Camoufox)
  project_puppeteer_crawler_ts (TypeScript + Puppeteer)

Step 5: Navigate to Project

cd my-scraper

# View generated structure
ls -la

Step 6: Review Generated Files

my-scraper/
├── .actor/
│   ├── actor.json                 ← Actor configuration
│   └── input_schema.json          ← Input validation
├── src/
│   └── main.ts                    ← Your code here
├── storage/                       ← Local storage
├── .dockerignore
├── .gitignore
├── .prettierrc
├── AGENTS.md                      ← AI agent guidance (Apify-maintained)
├── Dockerfile                     ← Production build
├── eslint.config.js               ← Code quality
├── package.json                   ← Dependencies & scripts
├── tsconfig.json                  ← TypeScript config
└── README.md                      ← Documentation

Important: The template includes AGENTS.md, official Apify documentation for AI agents working with Actors. This file provides:

• Do/Don't patterns for Actor development
• Input/output schema detailed specifications
• Dataset and key-value store schema patterns
• Safety and permission guidelines
• Apify SDK best practices

See agents-md-guide.md in this directory for how AGENTS.md complements this skill.

Step 7: Install Dependencies

npm install

Step 8: Develop Your Actor

Edit src/main.ts:

import { Actor } from 'apify';
import { PlaywrightCrawler, Dataset } from 'crawlee';

await Actor.main(async () => {
    const input = await Actor.getInput();

    const crawler = new PlaywrightCrawler({
        async requestHandler({ page, request }) {
            // Your scraping logic here
        },
    });

    await crawler.run(input.startUrls);
});

Step 9: Test Locally

# Run actor locally
apify run

# With specific input
apify run --input='{"startUrls":[{"url":"https://example.com"}]}'

# Debug mode
DEBUG=crawlee:* apify run

Step 10: Build (TypeScript Only)

# Compile TypeScript
npm run build

# Output in dist/ directory

Step 11: Push to Apify Platform

# Deploy to Apify
apify push

# With specific build tag
apify push --build-tag beta

# Force rebuild
apify push --force

Step 12: Call Your Actor

# Run actor on Apify platform
apify call my-scraper

# With input
apify call my-scraper --input='{"startUrls":[{"url":"https://example.com"}]}'

Complete CLI Command Reference

Project Management

# Create new actor
apify create [name]

# Initialize in existing directory
apify init

# Login/logout
apify login
apify logout

# Check login status
apify info

Development

# Run locally
apify run
apify run --purge           # Clear storage first
apify run --input-file=input.json

# Run specific actor
apify call [actor-id]
apify call [actor-id] --build=beta

Deployment

# Push to platform
apify push
apify push --build-tag [tag]
apify push --version-number [version]
apify push --wait-for-finish

# Pull actor from platform
apify pull [actor-id]

Storage Management

# Manage datasets
apify dataset ls
apify dataset get [id]

# Manage key-value stores
apify kv-store ls
apify kv-store get [id]

npm Scripts (Generated by apify create)

The CLI generates these useful scripts:

{
    "scripts": {
        "start": "npm run build && node dist/main.js",
        "build": "tsc",
        "test": "echo \"No tests yet\"",
        "lint": "eslint src",
        "lint:fix": "eslint src --fix"
    }
}

Usage:

npm start          # Build and run
npm run build      # Compile TypeScript
npm test           # Run tests
npm run lint       # Check code quality
npm run lint:fix   # Auto-fix linting issues

Development Workflow

Typical Development Cycle

# 1. Create actor
apify create my-scraper
cd my-scraper

# 2. Develop
# Edit src/main.ts

# 3. Test locally
apify run

# 4. Fix issues, repeat step 3

# 5. Lint code
npm run lint:fix

# 6. Push to platform
apify push

# 7. Test on platform
apify call my-scraper

# 8. Iterate
# Edit code, repeat from step 3

Common Issues

Issue: "Command not found: apify"

Solution:

npm install -g apify-cli

Issue: "Not logged in"

Solution:

apify login

Issue: Build fails

Solution:

# Check TypeScript errors
npm run build

# Fix errors in src/
# Then try again:
apify push

Anti-Pattern: Manual Creation

❌ DON'T Do This

# BAD: Manual setup
mkdir my-actor
cd my-actor
npm init -y
npm install apify crawlee
# ... missing tons of configuration

Why this is wrong:

• Missing .actor/ directory
• No input schema
• Incorrect Dockerfile
• No ESLint config
• No TypeScript setup
• Missing npm scripts
• Will fail deployment

✅ DO This Instead

# GOOD: Use CLI
apify create my-actor
cd my-actor
# Everything configured correctly!

Best Practices

✅ DO:

• Always use `apify create` (not manual setup)
• Choose appropriate template based on site type (see decision tree in productionization guide)
• Test locally first with apify run
• Use build tags for staging (--build-tag beta)
• Keep CLI updated (npm update -g apify-cli)
• Use `.env` file for local secrets
• Commit to git (except storage/, dist/)

❌ DON'T:

• Create actors manually - use CLI!
• Skip local testing - test before push
• Hardcode secrets - use environment variables
• Push without building (TypeScript actors)
• Ignore linting errors - fix them
• Skip version tags - use semantic versioning

Resources

• Apify CLI Docs
• CLI Reference
• Actor Development

Summary

The Apify CLI is THE way to create Actors

Key commands: 1. apify create - Create new actor (CRITICAL) 2. apify run - Test locally 3. apify push - Deploy to platform 4. apify call - Run on platform

Remember: Always use apify create, never manual setup!

Actor Configuration Patterns

Patterns for .actor/actor.json configuration.

Basic Structure

{
    "actorSpecification": 1,
    "name": "my-actor",
    "title": "My Actor",
    "description": "Short description",
    "version": "1.0",
    "meta": {
        "templateId": "project_playwright_crawler_ts"
    },
    "input": "./input_schema.json",
    "dockerfile": "./Dockerfile"
}

Essential Fields

Actor Identity

{
    "name": "my-scraper",
    "title": "My Scraper",
    "description": "Scrapes data from example.com",
    "version": "1.0.0"
}

Documentation

{
    "readme": "./README.md",
    "changelog": "./CHANGELOG.md"
}

Input/Output

{
    "input": "./input_schema.json",
    "storages": {
        "dataset": {
            "actorSpecification": 1,
            "title": "Scraped data",
            "views": {
                "overview": {
                    "title": "Overview",
                    "transformation": {
                        "fields": ["title", "price", "url"]
                    },
                    "display": {
                        "component": "table"
                    }
                }
            }
        }
    }
}

Resource Configuration

Memory Settings

{
    "defaultRunOptions": {
        "build": "latest",
        "timeoutSecs": 3600,
        "memoryMbytes": 4096
    }
}

Memory recommendations:

• 256-512 MB: Simple HTTP scrapers
• 1024 MB: Basic Playwright scrapers
• 2048 MB: Medium-scale Playwright
• 4096 MB: Large-scale or multiple browsers
• 8192+ MB: Very large datasets

Build Configuration

{
    "dockerfile": "./Dockerfile",
    "dockerContextDir": "./",
    "buildTag": "latest"
}

Environment Variables

Pattern 1: Public Variables

{
    "environmentVariables": {
        "LOG_LEVEL": "info",
        "MAX_RETRY": "3"
    }
}

Pattern 2: With Descriptions

{
    "environmentVariables": {
        "API_ENDPOINT": {
            "value": "https://api.example.com",
            "description": "API base URL"
        },
        "RATE_LIMIT": {
            "value": "60",
            "description": "Requests per minute"
        }
    }
}

Complete Examples

Pattern 1: Simple Scraper

{
    "actorSpecification": 1,
    "name": "simple-scraper",
    "title": "Simple Web Scraper",
    "description": "Scrapes product data from e-commerce sites",
    "version": "1.0.0",
    "meta": {
        "templateId": "project_playwright_crawler_ts"
    },
    "input": "./input_schema.json",
    "dockerfile": "./Dockerfile",
    "readme": "./README.md",
    "defaultRunOptions": {
        "build": "latest",
        "timeoutSecs": 3600,
        "memoryMbytes": 2048
    },
    "storages": {
        "dataset": {
            "actorSpecification": 1,
            "title": "Scraped products",
            "views": {
                "overview": {
                    "title": "Product overview",
                    "transformation": {
                        "fields": ["name", "price", "inStock", "url"]
                    },
                    "display": {
                        "component": "table"
                    }
                }
            }
        }
    }
}

Pattern 2: High-Performance Scraper

{
    "actorSpecification": 1,
    "name": "fast-scraper",
    "title": "High-Performance Scraper",
    "description": "Fast scraping with concurrent requests",
    "version": "2.0.0",
    "input": "./input_schema.json",
    "dockerfile": "./Dockerfile",
    "defaultRunOptions": {
        "build": "latest",
        "timeoutSecs": 7200,
        "memoryMbytes": 4096
    },
    "environmentVariables": {
        "MAX_CONCURRENCY": "10",
        "MAX_REQUESTS_PER_MINUTE": "120"
    }
}

Pattern 3: With Anti-Blocking

{
    "actorSpecification": 1,
    "name": "stealth-scraper",
    "title": "Anti-Blocking Scraper",
    "description": "Scraper with fingerprinting and proxies",
    "version": "1.5.0",
    "input": "./input_schema.json",
    "dockerfile": "./Dockerfile",
    "defaultRunOptions": {
        "build": "latest",
        "timeoutSecs": 3600,
        "memoryMbytes": 4096
    },
    "environmentVariables": {
        "USE_FINGERPRINTING": "true",
        "PROXY_GROUP": "RESIDENTIAL"
    }
}

Pattern 4: API-Based Scraper

{
    "actorSpecification": 1,
    "name": "api-scraper",
    "title": "API Data Scraper",
    "description": "Fetches data via REST API",
    "version": "1.0.0",
    "input": "./input_schema.json",
    "dockerfile": "./Dockerfile",
    "defaultRunOptions": {
        "build": "latest",
        "timeoutSecs": 1800,
        "memoryMbytes": 1024
    },
    "environmentVariables": {
        "API_TIMEOUT": "30000",
        "RATE_LIMIT": "60"
    }
}

Dataset Schema Configuration

Basic Table View

{
    "storages": {
        "dataset": {
            "actorSpecification": 1,
            "title": "Scraped data",
            "views": {
                "overview": {
                    "title": "Overview",
                    "transformation": {
                        "fields": ["title", "price", "url"]
                    },
                    "display": {
                        "component": "table",
                        "properties": {
                            "title": {
                                "label": "Product Name"
                            },
                            "price": {
                                "label": "Price ($)"
                            }
                        }
                    }
                }
            }
        }
    }
}

Multiple Views

{
    "storages": {
        "dataset": {
            "actorSpecification": 1,
            "title": "Product data",
            "views": {
                "overview": {
                    "title": "Product List",
                    "transformation": {
                        "fields": ["name", "price", "url"]
                    }
                },
                "detailed": {
                    "title": "Detailed View",
                    "transformation": {
                        "fields": ["name", "price", "description", "reviews", "url"]
                    }
                }
            }
        }
    }
}

Build Tags and Versions

Version Tagging

{
    "version": "1.2.3",
    "buildTag": "latest"
}

Versioning pattern:

• 1.0.0 - Major version (breaking changes)
• 1.1.0 - Minor version (new features)
• 1.1.1 - Patch version (bug fixes)

Build Tags

# Deploy with specific tag
apify push --build-tag beta

# Use in configuration
{
    "defaultRunOptions": {
        "build": "beta"
    }
}

Common tags:

• latest - Production release
• beta - Testing version
• dev - Development version

Dockerfile Configuration

Standard Playwright

FROM apify/actor-node-playwright-chrome:20

COPY package*.json ./
RUN npm install --production

COPY . ./

RUN npm run build

CMD npm start

With Custom Dependencies

FROM apify/actor-node-playwright-chrome:20

# Install system dependencies
RUN apt-get update && apt-get install -y \
    imagemagick \
    && rm -rf /var/lib/apt/lists/*

COPY package*.json ./
RUN npm install --production

COPY . ./
RUN npm run build

CMD npm start

Timeout Configuration

{
    "defaultRunOptions": {
        "timeoutSecs": 3600
    }
}

Timeout recommendations:

• 300s (5 min): Quick scrapers
• 1800s (30 min): Medium datasets
• 3600s (1 hour): Large datasets
• 7200s (2 hours): Very large datasets
• 86400s (24 hours): Maximum allowed

Best Practices

✅ DO:

• Set appropriate memory for task
• Use semantic versioning
• Configure dataset views
• Set reasonable timeouts
• Document environment variables
• Use build tags for staging

❌ DON'T:

• Don't set too little memory (causes OOM)
• Don't set excessive timeouts
• Don't hardcode secrets in config
• Don't skip version numbers
• Don't use latest for dependencies

Common Patterns

Low Memory (HTTP Only)

{
    "defaultRunOptions": {
        "memoryMbytes": 512,
        "timeoutSecs": 1800
    }
}

Standard Playwright

{
    "defaultRunOptions": {
        "memoryMbytes": 2048,
        "timeoutSecs": 3600
    }
}

High Performance

{
    "defaultRunOptions": {
        "memoryMbytes": 4096,
        "timeoutSecs": 7200
    },
    "environmentVariables": {
        "MAX_CONCURRENCY": "10"
    }
}

API Scraping

{
    "defaultRunOptions": {
        "memoryMbytes": 1024,
        "timeoutSecs": 1800
    }
}

Resources

• actor.json Docs
• Dataset Schema
• Docker Images

Actor Deployment Patterns

Testing and deployment workflows for Apify Actors.

Local Testing

Basic Run

# Run with default input
apify run

# Output shows:
# - Actor initialization
# - Logs
# - Results saved to ./storage/datasets/default/

With Custom Input

# Inline JSON
apify run --input='{"startUrls":[{"url":"https://example.com"}]}'

# From file
apify run --input-file=./test-input.json

# Different input file
apify run --input-file=./inputs/production.json

Clean Run

# Purge storage before running
apify run --purge

# Fresh start, no cached data

Debug Mode

# Enable debug logging
DEBUG=crawlee:* apify run

# Multiple debug namespaces
DEBUG=crawlee:*,apify:* apify run

# Playwright debug
DEBUG=pw:api apify run

TypeScript Build

Build Before Run

# Compile TypeScript
npm run build

# Output: dist/main.js

# Then run
npm start
# Or
node dist/main.js

Watch Mode (Development)

# Auto-rebuild on changes
npm run build -- --watch

# In another terminal
npm start

Build Errors

# Check TypeScript errors
npm run build

# Fix errors, then retry
# Common issues:
# - Type mismatches
# - Missing imports
# - Syntax errors

Deployment to Platform

First Deployment

# Push to Apify platform
apify push

# Process:
# 1. Uploads source code
# 2. Builds Docker image
# 3. Creates new Actor version
# 4. Sets as latest

With Build Tag

# Deploy to specific tag
apify push --build-tag beta

# Deploy to dev
apify push --build-tag dev

# Production release
apify push --build-tag latest

With Version

# Set version number
apify push --version-number 1.2.3

# Updates .actor/actor.json version field

Wait for Build

# Wait until build completes
apify push --wait-for-finish

# Useful in CI/CD pipelines

Force Rebuild

# Force rebuild even if no changes
apify push --force

# Use when:
# - Dependencies updated
# - Dockerfile changed
# - Build cache issues

Testing on Platform

Run Actor

# Run latest version
apify call my-actor

# Run specific build
apify call my-actor --build=beta

# With input
apify call my-actor --input='{"maxItems":10}'

# With input file
apify call my-actor --input-file=./input.json

Monitor Run

# Get run info
apify call my-actor --wait-for-finish

# Shows:
# - Run ID
# - Status
# - Duration
# - Results

Version Management

Semantic Versioning

# Major version (breaking changes)
apify push --version-number 2.0.0

# Minor version (new features)
apify push --version-number 1.1.0

# Patch version (bug fixes)
apify push --version-number 1.0.1

Build Tags

# Development
apify push --build-tag dev

# Staging/testing
apify push --build-tag beta

# Production
apify push --build-tag latest
apify push --build-tag v1.0.0

Tag Strategy

main branch    → --build-tag latest
develop branch → --build-tag dev
release/beta   → --build-tag beta
feature/*      → --build-tag feature-name

Complete Workflow Patterns

Pattern 1: Development Cycle

# 1. Make changes
vim src/main.ts

# 2. Build
npm run build

# 3. Test locally
apify run --purge

# 4. Fix issues, repeat 2-3

# 5. Lint code
npm run lint:fix

# 6. Deploy to dev
apify push --build-tag dev

# 7. Test on platform
apify call my-actor --build=dev

# 8. Deploy to production
apify push --build-tag latest --version-number 1.0.1

Pattern 2: Quick Test

# Quick test without build
apify run --input='{"startUrls":[{"url":"https://example.com"}],"maxItems":5}'

# Check ./storage/datasets/default/
cat storage/datasets/default/*.json

Pattern 3: CI/CD Deployment

#!/bin/bash
# deploy.sh

# Build TypeScript
npm run build

# Run tests
npm test

# Lint
npm run lint

# Push to platform
apify push --build-tag ${BUILD_TAG} --wait-for-finish

# Test deployment
apify call ${ACTOR_ID} --build=${BUILD_TAG} --wait-for-finish

Pattern 4: Staged Release

# 1. Deploy to beta
apify push --build-tag beta --version-number 1.1.0

# 2. Test beta
apify call my-actor --build=beta

# 3. Monitor for issues
# ... wait 24 hours ...

# 4. Promote to production
apify push --build-tag latest --version-number 1.1.0

Storage Inspection

View Results

# Local datasets
ls storage/datasets/default/
cat storage/datasets/default/000000001.json

# Pretty print JSON
cat storage/datasets/default/*.json | jq '.'

Key-Value Store

# View KV store
ls storage/key_value_stores/default/
cat storage/key_value_stores/default/INPUT.json

Request Queue

# View queue
ls storage/request_queues/default/

Troubleshooting Deployment

Build Fails

# Check build log
apify push

# Common issues:
# - TypeScript errors → run npm run build locally
# - Missing dependencies → check package.json
# - Dockerfile errors → test docker build locally

Actor Won't Start

# Check logs in Apify Console
# Or via CLI:
apify call my-actor --wait-for-finish

# Common issues:
# - Memory too low → increase in actor.json
# - Timeout → increase timeoutSecs
# - Missing environment variables

Build Too Slow

# Use faster base image
# In Dockerfile:
FROM apify/actor-node-playwright-chrome:20-bookworm-slim

# Skip optional dependencies
RUN npm install --production --no-optional

Deployment Fails

# Check auth
apify info

# Re-login if needed
apify logout
apify login

# Retry with force
apify push --force

Platform Commands

View Datasets

# List datasets
apify dataset ls

# Get dataset
apify dataset get <dataset-id>

# Download CSV
apify dataset get <dataset-id> --format csv > data.csv

View Runs

# List recent runs
apify actor calls my-actor

# Get specific run
apify run get <run-id>

# Abort run
apify run abort <run-id>

Manage Actor

# Get actor info
apify actor get my-actor

# Update actor
apify push

# Delete actor (careful!)
# Must be done via Console

Best Practices

✅ DO:

• Test locally before pushing
• Use semantic versioning
• Tag dev/beta/latest appropriately
• Run npm run build before testing TypeScript
• Use --purge for clean tests
• Wait for build to complete in CI/CD
• Monitor first runs after deployment

❌ DON'T:

• Don't push untested code
• Don't skip version numbers
• Don't use --force unnecessarily
• Don't deploy directly to latest without testing
• Don't ignore build warnings
• Don't commit secrets to git

Quick Reference

# Local development
apify run                        # Run locally
apify run --purge               # Clean run
apify run --input-file=input.json # Custom input
npm run build                    # Build TypeScript
npm start                        # Run built code

# Deployment
apify push                       # Deploy
apify push --build-tag beta     # Deploy to beta
apify push --version-number 1.0.0 # Set version
apify push --wait-for-finish    # Wait for build

# Testing
apify call my-actor             # Run on platform
apify call my-actor --build=beta # Run specific build

# Inspection
apify dataset ls                # List datasets
apify dataset get <id>          # Get dataset

Resources

• Deployment Docs
• Build Process
• CLI Reference

Anti-Blocking Scraper Example

Actor demonstrating fingerprinting and proxy usage for blocked sites.

What This Demonstrates

• Browser fingerprinting with fingerprintOptions
• Proxy configuration (residential proxies)
• Session management and rotation
• Blocking detection
• Error handling for blocked requests

Files

• src/main.ts - Main Actor code with anti-blocking
• .actor/actor.json - Actor configuration
• .actor/input_schema.json - Input schema

Usage

# Run locally (requires Apify proxies)
apify run --input='{"startUrls":[{"url":"https://example.com"}],"useFingerprinting":true}'

# Deploy
apify push

# Run on platform
apify call anti-blocking-scraper

Input

{
    "startUrls": [{"url": "https://example.com"}],
    "maxItems": 100,
    "useFingerprinting": true,
    "proxyGroup": "RESIDENTIAL"
}

Output

{
    "url": "https://example.com/page",
    "title": "Page Title",
    "content": "...",
    "sessionId": "session_abc123",
    "scrapedAt": "2025-01-15T10:30:00.000Z"
}

Pattern

1. Enable fingerprinting for realistic browser profile 2. Use residential proxies for IP rotation 3. Manage sessions (rotate after errors) 4. Detect blocking (Cloudflare, CAPTCHAs) 5. Retry with new session if blocked

Basic Sitemap Scraper Example

Simple Actor that scrapes URLs from a sitemap using Playwright.

What This Demonstrates

• Sitemap-based URL discovery with RobotsFile
• TypeScript Actor structure
• Basic Playwright scraping
• Typed input/output
• Error handling

Files

• src/main.ts - Main Actor code
• .actor/actor.json - Actor configuration
• .actor/input_schema.json - Input schema

Usage

# Run locally
apify run --input='{"sitemapUrl":"https://example.com/sitemap.xml","maxItems":10}'

# Deploy
apify push

# Run on platform
apify call basic-scraper

Input

{
    "sitemapUrl": "https://example.com/sitemap.xml",
    "urlPattern": "/products/.*",
    "maxItems": 100
}

Output

{
    "url": "https://example.com/product/1",
    "title": "Product Name",
    "description": "Product description",
    "scrapedAt": "2025-01-15T10:30:00.000Z"
}

Pattern

1. Parse sitemap URLs 2. Filter by regex pattern 3. Scrape each URL with Playwright 4. Save to dataset

Hybrid Sitemap + API Scraper Example

Actor demonstrating the optimal pattern: sitemap for URL discovery, API for data fetching.

What This Demonstrates

• Sitemap-based URL discovery
• ID extraction from URLs (regex)
• API-based data fetching (fast, reliable)
• TypeScript with got-scraping
• Hybrid approach (best of both worlds)

Files

• src/main.ts - Main Actor code
• .actor/actor.json - Actor configuration
• .actor/input_schema.json - Input schema

Usage

# Run locally
apify run --input='{"sitemapUrl":"https://example.com/sitemap.xml","apiBaseUrl":"https://api.example.com/products"}'

# Deploy
apify push

# Run on platform
apify call hybrid-api-scraper

Input

{
    "sitemapUrl": "https://example.com/sitemap.xml",
    "apiBaseUrl": "https://api.example.com/products",
    "idPattern": "/products/([^/]+)",
    "maxItems": 100
}

Output

{
    "id": "123",
    "url": "https://example.com/products/123",
    "name": "Product Name",
    "price": 99.99,
    "inStock": true,
    "scrapedAt": "2025-01-15T10:30:00.000Z"
}

Pattern

1. Parse sitemap to get all URLs instantly 2. Extract IDs from URLs using regex 3. Fetch data via API (10-100x faster than HTML) 4. Save structured JSON to dataset

Performance

For 1,000 products:

• Sitemap discovery: ~5 seconds
• API fetching: ~2-5 minutes
• Total: ~5 minutes vs ~45 minutes with pure Playwright

Actor Initialization Patterns

Quick reference for setting up Apify Actor development environment.

Prerequisites

# Node.js 16+ required
node --version

# npm 8+ required
npm --version

Installation

Install Apify CLI

# Global installation
npm install -g apify-cli

# Verify installation
apify --version

# Update to latest
npm update -g apify-cli

Authentication

Login to Apify Platform

# Interactive login (opens browser)
apify login

# Check login status
apify info

# View current user
apify info --json

# Logout
apify logout

API Token (Alternative)

# Set token via environment variable
export APIFY_TOKEN=your_token_here

# Or add to .env file
echo "APIFY_TOKEN=your_token" > .env

Project Initialization

Pattern 1: New Actor (Recommended)

# Create new actor with CLI
apify create my-actor

# CLI prompts:
? What type of Actor do you want to create?
❯ project_cheerio_crawler_ts (TypeScript + Cheerio) ← RECOMMENDED for static HTML
  project_playwright_crawler_ts (TypeScript + Playwright) ← For JavaScript-heavy
  project_playwright_camoufox_crawler_ts (TypeScript + Camoufox) ← Anti-bot
  project_puppeteer_crawler_ts (TypeScript + Puppeteer)

# Navigate to project
cd my-actor

# Install dependencies
npm install

# Run locally
apify run

Pattern 2: Initialize Existing Directory

# If you have existing code
cd existing-project

# Initialize as Actor
apify init

# Select template
# Add .actor/ configuration files
# Update package.json

Pattern 3: Clone Existing Actor

# Pull actor from platform
apify pull username/actor-name

# Or by ID
apify pull actor-id

# Makes local copy with all configuration

Generated Structure

TypeScript Actor (project_cheerio_crawler_ts or project_playwright_crawler_ts)

my-actor/
├── .actor/
│   ├── actor.json              # Actor configuration
│   └── input_schema.json       # Input validation
├── src/
│   └── main.ts                 # Main source file
├── storage/
│   ├── datasets/
│   ├── key_value_stores/
│   └── request_queues/
├── .dockerignore
├── .gitignore
├── .prettierrc
├── Dockerfile                  # Build configuration
├── eslint.config.js           # Linting rules
├── package.json               # Dependencies
├── tsconfig.json              # TypeScript config
└── README.md                  # Documentation

Environment Setup

Local Development

# Create .env file for secrets
cat > .env << 'EOF'
APIFY_TOKEN=your_token_here
PROXY_PASSWORD=your_proxy_password
EOF

# Add to .gitignore
echo ".env" >> .gitignore

Environment Variables

# Available in Actor runs
APIFY_TOKEN              # Authentication token
APIFY_IS_AT_HOME        # Running on platform?
APIFY_DEFAULT_DATASET_ID
APIFY_DEFAULT_KEY_VALUE_STORE_ID
APIFY_DEFAULT_REQUEST_QUEUE_ID

First Run

Local Testing

# Run with default input
apify run

# Run with custom input
apify run --input='{"startUrls":[{"url":"https://example.com"}]}'

# Run with input file
apify run --input-file=input.json

# Purge storage before run
apify run --purge

# Debug mode
DEBUG=crawlee:* apify run

Input File Example

{
  "startUrls": [
    { "url": "https://example.com" }
  ],
  "maxItems": 100,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Troubleshooting

Issue: "Command not found: apify"

# Reinstall CLI
npm install -g apify-cli

# Check PATH
echo $PATH

# Verify installation location
npm root -g

Issue: "Not logged in"

# Login
apify login

# Or use token
export APIFY_TOKEN=your_token

Issue: "Permission denied"

# Use sudo (Linux/Mac)
sudo npm install -g apify-cli

# Or use nvm to manage Node without sudo

Issue: Build Fails

# Check TypeScript errors
npm run build

# Clean install
rm -rf node_modules package-lock.json
npm install

# Update dependencies
npm update

Issue: Storage Not Created

# Ensure storage/ directory exists
mkdir -p storage/{datasets,key_value_stores,request_queues}

# Run with --purge once
apify run --purge

Quick Command Reference

# Project setup
apify create <name>              # New actor
apify init                       # Init existing dir
apify pull <actor-id>           # Clone actor

# Authentication
apify login                      # Login
apify logout                     # Logout
apify info                       # Check status

# Development
apify run                        # Run locally
apify run --purge               # Clear storage first
apify run --input-file=file.json # Custom input

# Deployment
apify push                       # Deploy to platform
apify push --build-tag beta     # Tag build
apify call <actor-id>           # Run on platform

Best Practices

✅ DO:

• Always use apify create for new actors
• Keep CLI updated (npm update -g apify-cli)
• Use .env for local secrets
• Test locally before pushing
• Use TypeScript templates (Cheerio for static, Playwright for JS-heavy)

❌ DON'T:

• Don't create actors manually
• Don't commit .env files
• Don't skip local testing
• Don't use old CLI versions
• Don't hardcode tokens

Next Steps

After initialization: 1. Review generated files 2. Customize input schema (.actor/input_schema.json) 3. Update actor config (.actor/actor.json) 4. Write scraping logic (src/main.ts) 5. Test locally (apify run) 6. Deploy (apify push)

See:

• input-schemas.md - Input schema patterns
• configuration.md - Actor configuration
• deployment.md - Deployment workflow

Input Schema Patterns

Patterns for defining Actor input validation in .actor/input_schema.json.

Schema Structure

{
    "title": "Actor Input",
    "type": "object",
    "schemaVersion": 1,
    "properties": {
        "fieldName": {
            "title": "Field Label",
            "type": "string",
            "description": "Help text",
            "editor": "textfield"
        }
    },
    "required": ["fieldName"]
}

Common Field Types

String Field

{
    "url": {
        "title": "URL",
        "type": "string",
        "description": "Website URL to scrape",
        "editor": "textfield",
        "pattern": "https?://.+",
        "example": "https://example.com"
    }
}

Number Field

{
    "maxItems": {
        "title": "Maximum items",
        "type": "integer",
        "description": "Max number of items to scrape",
        "editor": "number",
        "default": 100,
        "minimum": 1,
        "maximum": 10000
    }
}

Boolean Field

{
    "saveHtml": {
        "title": "Save HTML",
        "type": "boolean",
        "description": "Save raw HTML",
        "editor": "checkbox",
        "default": false
    }
}

Array of URLs

{
    "startUrls": {
        "title": "Start URLs",
        "type": "array",
        "description": "List of URLs to scrape",
        "editor": "requestListSources",
        "placeholderValue": [{"url": "https://example.com"}],
        "minItems": 1
    }
}

Select Dropdown

{
    "mode": {
        "title": "Scraping mode",
        "type": "string",
        "description": "Choose scraping strategy",
        "editor": "select",
        "enum": ["fast", "thorough", "balanced"],
        "enumTitles": ["Fast", "Thorough", "Balanced"],
        "default": "balanced"
    }
}

Object Field

{
    "proxyConfiguration": {
        "title": "Proxy configuration",
        "type": "object",
        "description": "Proxy settings",
        "editor": "proxy",
        "default": {"useApifyProxy": true}
    }
}

Text Area

{
    "customJs": {
        "title": "Custom JavaScript",
        "type": "string",
        "description": "Custom page function",
        "editor": "javascript",
        "prefill": "async ({ page }) => {\n    // Your code\n}"
    }
}

Hidden Field

{
    "version": {
        "title": "Version",
        "type": "string",
        "description": "Internal version",
        "editor": "hidden",
        "default": "1.0.0"
    }
}

Complete Examples

Pattern 1: Basic Scraper

{
    "title": "Basic Scraper Input",
    "type": "object",
    "schemaVersion": 1,
    "properties": {
        "startUrls": {
            "title": "Start URLs",
            "type": "array",
            "description": "URLs to scrape",
            "editor": "requestListSources",
            "minItems": 1
        },
        "maxItems": {
            "title": "Maximum items",
            "type": "integer",
            "description": "Max results",
            "editor": "number",
            "default": 100,
            "minimum": 1
        }
    },
    "required": ["startUrls"]
}

Pattern 2: E-commerce Scraper

{
    "title": "E-commerce Scraper Input",
    "type": "object",
    "schemaVersion": 1,
    "properties": {
        "startUrls": {
            "title": "Product URLs",
            "type": "array",
            "description": "Product pages to scrape",
            "editor": "requestListSources"
        },
        "maxItems": {
            "title": "Max products",
            "type": "integer",
            "description": "Maximum products to scrape",
            "editor": "number",
            "default": 1000
        },
        "includeReviews": {
            "title": "Include reviews",
            "type": "boolean",
            "description": "Scrape product reviews",
            "editor": "checkbox",
            "default": false
        },
        "minPrice": {
            "title": "Minimum price",
            "type": "number",
            "description": "Filter by minimum price",
            "editor": "number",
            "minimum": 0
        },
        "proxyConfiguration": {
            "title": "Proxy configuration",
            "type": "object",
            "description": "Proxy settings",
            "editor": "proxy"
        }
    },
    "required": ["startUrls"]
}

Pattern 3: Advanced Scraper with Options

{
    "title": "Advanced Scraper Input",
    "type": "object",
    "schemaVersion": 1,
    "properties": {
        "startUrls": {
            "title": "Start URLs",
            "type": "array",
            "description": "URLs to scrape",
            "editor": "requestListSources"
        },
        "mode": {
            "title": "Scraping mode",
            "type": "string",
            "description": "Choose strategy",
            "editor": "select",
            "enum": ["sitemap", "api", "playwright", "hybrid"],
            "enumTitles": ["Sitemap", "API", "Playwright", "Hybrid"],
            "default": "hybrid"
        },
        "maxConcurrency": {
            "title": "Max concurrency",
            "type": "integer",
            "description": "Parallel requests",
            "editor": "number",
            "default": 5,
            "minimum": 1,
            "maximum": 50
        },
        "maxRequestsPerMinute": {
            "title": "Max requests/min",
            "type": "integer",
            "description": "Rate limit",
            "editor": "number",
            "default": 60
        },
        "useFingerprinting": {
            "title": "Use fingerprinting",
            "type": "boolean",
            "description": "Anti-blocking",
            "editor": "checkbox",
            "default": false
        },
        "proxyConfiguration": {
            "title": "Proxy configuration",
            "type": "object",
            "description": "Proxy settings",
            "editor": "proxy"
        }
    },
    "required": ["startUrls", "mode"]
}

Pattern 4: API-based Scraper

{
    "title": "API Scraper Input",
    "type": "object",
    "schemaVersion": 1,
    "properties": {
        "apiUrl": {
            "title": "API URL",
            "type": "string",
            "description": "API endpoint",
            "editor": "textfield",
            "pattern": "https?://.+"
        },
        "apiKey": {
            "title": "API Key",
            "type": "string",
            "description": "Authentication key",
            "editor": "textfield",
            "isSecret": true
        },
        "pageSize": {
            "title": "Page size",
            "type": "integer",
            "description": "Items per page",
            "editor": "number",
            "default": 100
        },
        "maxPages": {
            "title": "Max pages",
            "type": "integer",
            "description": "Maximum pages to fetch",
            "editor": "number",
            "default": 10
        }
    },
    "required": ["apiUrl"]
}

Pattern 5: Sitemap + Playwright

{
    "title": "Sitemap Scraper Input",
    "type": "object",
    "schemaVersion": 1,
    "properties": {
        "sitemapUrl": {
            "title": "Sitemap URL",
            "type": "string",
            "description": "URL to sitemap.xml",
            "editor": "textfield",
            "example": "https://example.com/sitemap.xml"
        },
        "urlPattern": {
            "title": "URL pattern (regex)",
            "type": "string",
            "description": "Filter URLs by regex",
            "editor": "textfield",
            "example": "/products/.*"
        },
        "maxItems": {
            "title": "Maximum items",
            "type": "integer",
            "description": "Max URLs to scrape",
            "editor": "number",
            "default": 1000
        },
        "proxyConfiguration": {
            "title": "Proxy configuration",
            "type": "object",
            "editor": "proxy"
        }
    },
    "required": ["sitemapUrl"]
}

Pattern 6: With Custom Fields

{
    "title": "Custom Scraper Input",
    "type": "object",
    "schemaVersion": 1,
    "properties": {
        "startUrls": {
            "title": "Start URLs",
            "type": "array",
            "editor": "requestListSources"
        },
        "selectors": {
            "title": "Custom selectors",
            "type": "object",
            "description": "CSS selectors for data",
            "editor": "json",
            "prefill": "{\n  \"title\": \"h1\",\n  \"price\": \".price\"\n}"
        },
        "customFunction": {
            "title": "Custom function",
            "type": "string",
            "description": "Custom extraction logic",
            "editor": "javascript",
            "prefill": "async ({ page }) => {\n    return { title: await page.title() };\n}"
        }
    },
    "required": ["startUrls"]
}

Field Editors

Available editor types:

Validation Patterns

URL Validation

{
    "pattern": "^https?://.*",
    "example": "https://example.com"
}

Email Validation

{
    "pattern": "^[^@]+@[^@]+\\.[^@]+$",
    "example": "user@example.com"
}

Number Range

{
    "minimum": 1,
    "maximum": 1000,
    "default": 100
}

Required Array

{
    "type": "array",
    "minItems": 1
}

Secret Field

{
    "isSecret": true,
    "editor": "textfield"
}

TypeScript Usage

// Define matching interface
interface Input {
    startUrls: { url: string }[];
    maxItems?: number;
    proxyConfiguration?: object;
}

// Use in Actor
await Actor.main(async () => {
    const input = await Actor.getInput<Input>();

    if (!input?.startUrls) {
        throw new Error('startUrls is required');
    }
});

Best Practices

✅ DO:

• Provide clear description for each field
• Set sensible default values
• Use appropriate editor types
• Add example values
• Validate with pattern, minimum, maximum
• Mark secrets with isSecret: true

❌ DON'T:

• Don't use any type
• Don't skip descriptions
• Don't hardcode large defaults
• Don't forget required fields
• Don't expose secrets in prefill

Resources

• Input Schema Docs
• JSON Schema Spec
• Apify Editor Types

Apify Actor Development

Production-ready Actor creation with TypeScript and Apify CLI.

When to Use This Module

Load this module when user requests:

• "Make this an Apify Actor"
• "Productionize this scraper"
• "Deploy to Apify"
• "Create an actor"

Quick Start

# 1. Install Apify CLI
npm install -g apify-cli

# 2. Create actor (TypeScript recommended)
apify create my-scraper

# 3. Select template based on site type:
#    - project_cheerio_crawler_ts (static HTML, fastest)
#    - project_playwright_crawler_ts (JavaScript-heavy)

# 4. Develop and test
apify run

# 5. Deploy
apify push

Files in This Directory

Core Guides

1. typescript-first.md - Why TypeScript for Actors (STRONGLY RECOMMENDED) 2. cli-workflow.md - apify create workflow (CRITICAL - always use CLI) 3. initialization.md - Setup and authentication patterns 4. input-schemas.md - Input validation patterns (6 complete examples) 5. configuration.md - actor.json configuration patterns 6. deployment.md - Testing and deployment workflows 7. agents-md-guide.md - How to use AGENTS.md in templates (reference guide)

Templates

• templates/typescript-actor/ - Complete TypeScript actor template
• src/main.ts - Full-featured main file
• src/types.ts - Type definitions
• .actor/ - Configuration files
• package.json, tsconfig.json, Dockerfile

Examples

• examples/basic-scraper/ - Sitemap + Playwright scraper
• examples/anti-blocking/ - Fingerprinting + proxies
• examples/hybrid-api/ - Sitemap + API (optimal pattern)

TypeScript-First Philosophy

For production Actors, TypeScript is STRONGLY RECOMMENDED:

✅ Type safety (catch errors at compile time) ✅ IDE autocomplete for Apify/Crawlee APIs ✅ Better refactoring support ✅ Self-documenting code ✅ Industry standard for production

See typescript-first.md for details.

CLI-First Workflow

CRITICAL: Always use `apify create` command

❌ DON'T create actors manually ✅ DO use apify create command

The CLI auto-generates:

• Proper project structure
• TypeScript configuration
• ESLint setup
• .actor/ directory
• Dockerfile
• npm scripts

See cli-workflow.md for details.

Recommended Reading Order

1. cli-workflow.md - Start here (apify create) 2. typescript-first.md - Why TypeScript 3. agents-md-guide.md - Understanding AGENTS.md in templates 4. initialization.md - Complete setup 5. input-schemas.md - Define inputs 6. configuration.md - Configure actor 7. deployment.md - Deploy to platform

Scope and External Resources

This Skill's Scope

• When and why: Workflow, strategy selection, template decision tree
• Project structure: CLI usage, TypeScript setup, deployment patterns
• Integration: How to use scraping logic from Phase 1-4 in Actors

AGENTS.md Scope (in Templates)

• How: Implementation details, SDK patterns, best practices
• Schemas: Detailed input/output/dataset/key-value store specifications
• Do/Don't: Apify-specific anti-patterns and recommendations

When you run apify create, the template includes AGENTS.md with official Apify guidance. Read agents-md-guide.md to understand how AGENTS.md complements this skill.

---

Back to main skill: ../SKILL.md

TypeScript-First Actor Development

Overview

For production Apify Actors, TypeScript is STRONGLY RECOMMENDED over JavaScript.

Why TypeScript?

1. Type Safety

Catch errors at compile time, not runtime:

// TypeScript catches this error BEFORE deployment
interface Input {
    startUrls: { url: string }[];
    maxItems: number; // Expects number
}

const input = await Actor.getInput<Input>();
input.maxItems = "100"; // ❌ TypeScript error: Type 'string' is not assignable to type 'number'

// JavaScript fails AT RUNTIME in production
const input = await Actor.getInput();
input.maxItems = "100"; // ❌ Runtime error when used in math operation

2. IDE Autocomplete

TypeScript provides IntelliSense for all Apify/Crawlee APIs:

import { Actor } from 'apify';
import { PlaywrightCrawler } from 'crawlee';

// IDE shows all available methods and their parameters
await Actor.main(async () => {
    const input = await Actor.getInput(); // Autocomplete for Actor methods

    const crawler = new PlaywrightCrawler({
        // Autocomplete shows all config options
        maxConcurrency: 5,
        maxRequestsPerMinute: 60,
        // IDE warns if you use invalid options
    });
});

3. Self-Documenting Code

Types serve as inline documentation:

// Clear interface = instant understanding
interface Product {
    id: number;
    name: string;
    price: number;
    inStock: boolean;
    images: string[];
    metadata?: {
        brand: string;
        category: string;
    };
}

// Function signature is self-documenting
async function scrapeProduct(url: string): Promise<Product> {
    // Implementation
}

4. Better Refactoring

Rename variables/functions with confidence:

// Rename 'maxItems' to 'limit'
// TypeScript updates ALL usages automatically
// JavaScript might miss some references

5. Team Collaboration

New team members understand code faster:

// Clear types = less documentation needed
interface ActorInput {
    startUrls: { url: string }[];
    maxItems?: number; // Optional
    proxyConfiguration?: object;
}

// Anyone reading this knows exactly what to expect

Setting Up TypeScript Actor

Use apify create (Recommended)

# Create new actor with TypeScript template
apify create my-scraper

# Select appropriate TypeScript template:
# - project_cheerio_crawler_ts (static HTML, fastest)
# - project_playwright_crawler_ts (JavaScript-heavy sites)

This auto-generates:

• tsconfig.json - TypeScript configuration
• src/main.ts - TypeScript source
• package.json - Build scripts
• Type definitions for Apify/Crawlee

Generated Structure

my-scraper/
├── src/
│   ├── main.ts          ← TypeScript source
│   └── types.ts         ← Custom type definitions
├── .actor/
│   ├── actor.json
│   └── input_schema.json
├── tsconfig.json        ← TypeScript config
├── package.json         ← Build scripts
├── Dockerfile
└── README.md

TypeScript Patterns for Actors

Pattern 1: Typed Input

import { Actor } from 'apify';

// Define input interface
interface Input {
    startUrls: { url: string }[];
    maxItems?: number;
    proxyConfiguration?: object;
}

await Actor.main(async () => {
    // Get typed input
    const input = await Actor.getInput<Input>();

    if (!input) {
        throw new Error('Input is required');
    }

    // TypeScript knows input.startUrls exists and is an array
    console.log(`Processing ${input.startUrls.length} URLs`);

    // Optional chaining with type safety
    const limit = input.maxItems ?? 100;
});

Pattern 2: Typed Dataset Output

import { Dataset } from 'crawlee';

// Define output interface
interface Product {
    url: string;
    name: string;
    price: number;
    inStock: boolean;
}

// TypeScript ensures correct shape
await Dataset.pushData<Product>({
    url: 'https://...',
    name: 'Product Name',
    price: 99.99,
    inStock: true,
    // extraField: 'value' // ❌ TypeScript error
});

Pattern 3: Typed Request Handler

import { PlaywrightCrawler, Dataset } from 'crawlee';

interface Product {
    name: string;
    price: number;
}

const crawler = new PlaywrightCrawler({
    async requestHandler({ page, request, log }) {
        // Extract with type safety
        const product: Product = await page.evaluate(() => ({
            name: document.querySelector('h1')?.textContent ?? '',
            price: parseFloat(document.querySelector('.price')?.textContent ?? '0'),
        }));

        await Dataset.pushData<Product>(product);
    },
});

Pattern 4: Custom Types

// types.ts
export interface ScrapedProduct {
    id: number;
    name: string;
    price: number;
    url: string;
}

export interface ScraperConfig {
    maxConcurrency: number;
    requestsPerMinute: number;
}

// main.ts
import { ScrapedProduct, ScraperConfig } from './types';

const config: ScraperConfig = {
    maxConcurrency: 5,
    requestsPerMinute: 60,
};

Build Process

TypeScript actors require compilation:

# Build TypeScript to JavaScript
npm run build

# Output goes to dist/ directory

The build process is automatic when using apify push.

TypeScript Configuration

Example tsconfig.json:

{
    "compilerOptions": {
        "target": "ES2022",
        "module": "NodeNext",
        "moduleResolution": "NodeNext",
        "outDir": "dist",
        "strict": true,
        "esModuleInterop": true,
        "skipLibCheck": true,
        "resolveJsonModule": true
    },
    "include": ["src/**/*"],
    "exclude": ["node_modules", "dist"]
}

Common TypeScript Patterns

Null Safety

// Handle potentially null values safely
const price = document.querySelector('.price')?.textContent ?? 'N/A';

// TypeScript enforces null checks
if (element) {
    const text = element.textContent; // Safe
}

Enum for Constants

enum ScraperMode {
    FAST = 'fast',
    THOROUGH = 'thorough',
    BALANCED = 'balanced',
}

const mode: ScraperMode = ScraperMode.BALANCED;

Type Guards

function isValidProduct(data: any): data is Product {
    return (
        typeof data.name === 'string' &&
        typeof data.price === 'number' &&
        typeof data.inStock === 'boolean'
    );
}

const scraped = await page.evaluate(/* ... */);
if (isValidProduct(scraped)) {
    await Dataset.pushData(scraped); // Type-safe
}

Best Practices

✅ DO:

• Use TypeScript for all production Actors
• Define interfaces for input/output
• Enable strict mode in tsconfig.json
• Use type imports from Apify/Crawlee
• Document complex types with JSDoc
• Use enums for constant values
• Leverage IDE autocomplete

❌ DON'T:

• Use `any` type (defeats purpose of TypeScript)
• Disable strict checks (loses type safety)
• Skip type definitions for custom data
• Forget to compile before testing locally
• Ignore TypeScript errors (fix them!)

Migration from JavaScript

If you have existing JavaScript actor:

1. Rename .js files to .ts 2. Add type annotations gradually 3. Fix TypeScript errors 4. Add tsconfig.json 5. Update build scripts 6. Test thoroughly

Or better: Create new TypeScript actor with apify create and port logic.

Resources

• Apify TypeScript Docs
• TypeScript Handbook
• Crawlee TypeScript Examples

Summary

TypeScript = Better Actors

Key benefits: 1. Catch errors before deployment 2. IDE autocomplete for all APIs 3. Self-documenting code 4. Easier refactoring 5. Better team collaboration 6. Industry standard

Always use TypeScript templates when creating new actors! (Choose Cheerio for static HTML, Playwright for JS-heavy sites)

Scraping Examples

Runnable code examples demonstrating different scraping patterns.

Files in This Directory

1. traffic-interception-basic.js - Proxy-based reconnaissance (MCP tool sequence) 2. sitemap-basic.js - Get URLs from sitemap, scrape with CheerioCrawler 3. api-scraper.js - Pure API-based scraping (fastest) 4. hybrid-sitemap-api.js - Sitemap URLs + API data (best approach) 5. iterative-fallback.js - Try multiple approaches automatically

How to Run

Prerequisites

npm install crawlee got-scraping

Run an Example

node sitemap-basic.js
node api-scraper.js
node hybrid-sitemap-api.js
node iterative-fallback.js

Note: traffic-interception-basic.js is a reference for proxy-mcp MCP tool calls, not a standalone Node.js script. It documents the tool call sequence used during interactive reconnaissance with Claude.

Example Selection Guide

Modifying Examples

Each example is fully commented and can be adapted to your needs:

1. Change the baseUrl variable 2. Adjust selectors to match your target site 3. Modify data extraction logic 4. Adjust concurrency/rate limits

Performance Comparison

Running these examples on a 1000-page e-commerce site:

Next Steps

After understanding these examples: 1. Read strategy guides in ../strategies/ 2. Check reference materials in ../reference/ 3. For production deployment, see ../apify/

---

Back to main skill: ../SKILL.md

Quick Reference

Fast lookup guides for common patterns and troubleshooting.

Files

1. proxy-tool-reference.md - Proxy-MCP tool reference (all 80+ tools) 2. regex-patterns.md - URL filtering patterns for sitemaps 3. fingerprint-patterns.md - Stealth mode + TLS fingerprint presets 4. anti-patterns.md - What NOT to do

Quick Links

• Need proxy-mcp tool info? → proxy-tool-reference.md
• Need to filter sitemap URLs? → regex-patterns.md
• Need anti-detection patterns? → fingerprint-patterns.md
• Want to avoid mistakes? → anti-patterns.md

Back to main skill: ../SKILL.md

Common Regex Patterns for URL Filtering

Quick reference for filtering sitemap URLs with regex.

Product Pages

// Basic product pattern
/\/products\/[a-z0-9-]+$/i

// Product with numeric ID
/\/products\/(\d+)/

// Product with slug
/\/products\/([a-z0-9-]+)$/i

// Exclude category pages
/\/products\/[^\/]+$/
// Matches: /products/shoe-123
// Skips: /products/shoes/running

// Specific category
/\/products\/electronics\/[^\/]+$/

Blog Posts

// Blog with date
/\/blog\/\d{4}\/\d{2}\/[a-z0-9-]+/i
// Matches: /blog/2025/10/my-post

// Blog without date
/\/blog\/[a-z0-9-]+$/i

// WordPress pattern
/\/\d{4}\/\d{2}\/[a-z0-9-]+/

Multiple Patterns

// Products OR deals
/(\/products\/[^\/]+|\/deals\/[^\/]+)/

// Multiple categories
/\/(electronics|clothing|books)\/[^\/]+$/

Exclude Patterns

// Exclude pages
/^(?!.*(about|contact|help)).*$/

// Exclude file extensions
/^(?!.*\.(pdf|jpg|png)).*$/

Usage with RequestList

import { RequestList } from 'crawlee';

const requestList = await RequestList.open(null, [{
    requestsFromUrl: 'https://site.com/sitemap.xml',
    regex: /\/products\/[^\/]+$/,
}]);

Testing Patterns

Test your regex before running:

const pattern = /\/products\/[^\/]+$/;
const urls = [
    'https://shop.com/products/shoe-123',      // ✓ Match
    'https://shop.com/products/shoes/running', // ✗ No match (has /)
    'https://shop.com/products',               // ✗ No match (no product)
];

urls.forEach(url => {
    console.log(`${url}: ${pattern.test(url) ? '✓' : '✗'}`);
});