Playwright Scraper Skill

Name: Playwright Scraper Skill
Author: bighardperson

bighardperson/computer-science-skills-collection

8 installs
33 repo stars
Updated April 26, 2026
bighardperson/computer-science-skills-collection

Playwright-scraper-skill is a skill that scrapes web pages with Playwright, using stealth techniques to bypass anti-bot and Cloudflare protection.

About

Playwright-scraper-skill is a Playwright-based web scraping skill with anti-bot protection. It provides a simple script for dynamic JavaScript sites and a stealth script that hides automation markers and mimics human behavior to bypass Cloudflare. A developer uses it to fetch content from sites that block basic fetchers, choosing the method by the target's anti-bot level.

Playwright web scraping with anti-bot stealth and Cloudflare bypass
Two scripts: simple for dynamic sites, stealth for protected sites
Configurable via env vars: user-agent, wait time, headful, screenshots

Playwright Scraper Skill by the numbers

8 all-time installs (skills.sh)
Ranked #1,522 of 2,719 Automation & Workflows skills by installs in the Skillselion catalog
Data as of Jul 30, 2026 (Skillselion catalog sync)

At a glance

playwright-scraper-skill capabilities & compatibility

Capabilities: web scraping · browser automation
Works with: playwright · chrome
Use cases: web scraping · web search

From the docs

What playwright-scraper-skill says it does

A Playwright-based web scraping OpenClaw Skill with anti-bot protection.

SKILL.md

Hide automation markers (`navigator.webdriver = false`)

SKILL.md

npx skills add https://github.com/bighardperson/computer-science-skills-collection --skill playwright-scraper-skill

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/bighardperson/computer-science-skills-collection/playwright-scraper-skill.svg)](https://skillselion.com/skills/bighardperson/computer-science-skills-collection/playwright-scraper-skill)

Installs	8
repo stars	★ 33
Last updated	April 26, 2026
Repository	bighardperson/computer-science-skills-collection ↗

What it does

Scrape dynamic or anti-bot-protected websites using Playwright simple or stealth scripts.

Who is it for?

Fetching content from dynamic or Cloudflare-protected sites that block basic fetchers

Skip if: Repositories or source-tree code search, and simple static sites where web_fetch suffices

When should I use this skill?

A target site returns 403 or a Cloudflare challenge to basic fetching

What you get

Page content returned as JSON, optionally with a screenshot and saved HTML.

scraped page content JSON
optional screenshot and HTML

By the numbers

2 bundled scraper scripts
100% success rate reported on Discuss.com.hk with stealth

Files

SKILL.mdMarkdownGitHub ↗

Playwright Scraper Skill

A Playwright-based web scraping OpenClaw Skill with anti-bot protection. Choose the best approach based on the target website's anti-bot level.

---

🎯 Use Case Matrix

Target Website	Anti-Bot Level	Recommended Method	Script
Regular Sites	Low	web_fetch tool	N/A (built-in)
Dynamic Sites	Medium	Playwright Simple	`scripts/playwright-simple.js`
Cloudflare Protected	High	Playwright Stealth ⭐	`scripts/playwright-stealth.js`
YouTube	Special	deep-scraper	Install separately
Reddit	Special	reddit-scraper	Install separately

---

📦 Installation

cd playwright-scraper-skill
npm install
npx playwright install chromium

---

🚀 Quick Start

1️⃣ Simple Sites (No Anti-Bot)

Use OpenClaw's built-in web_fetch tool:

# Invoke directly in OpenClaw
Hey, fetch me the content from https://example.com

---

2️⃣ Dynamic Sites (Requires JavaScript)

Use Playwright Simple:

node scripts/playwright-simple.js "https://example.com"

Example output:

{
  "url": "https://example.com",
  "title": "Example Domain",
  "content": "...",
  "elapsedSeconds": "3.45"
}

---

3️⃣ Anti-Bot Protected Sites (Cloudflare etc.)

Use Playwright Stealth:

node scripts/playwright-stealth.js "https://m.discuss.com.hk/#hot"

Features:

Hide automation markers (navigator.webdriver = false)
Realistic User-Agent (iPhone, Android)
Random delays to mimic human behavior
Screenshot and HTML saving support

---

4️⃣ YouTube Video Transcripts

Use deep-scraper (install separately):

# Install deep-scraper skill
npx clawhub install deep-scraper

# Use it
cd skills/deep-scraper
node assets/youtube_handler.js "https://www.youtube.com/watch?v=VIDEO_ID"

---

📖 Script Descriptions

`scripts/playwright-simple.js`

Use Case: Regular dynamic websites
Speed: Fast (3-5 seconds)
Anti-Bot: None
Output: JSON (title, content, URL)

`scripts/playwright-stealth.js` ⭐

Use Case: Sites with Cloudflare or anti-bot protection
Speed: Medium (5-20 seconds)
Anti-Bot: Medium-High (hides automation, realistic UA)
Output: JSON + Screenshot + HTML file
Verified: 100% success on Discuss.com.hk

---

🎓 Best Practices

1. Try web_fetch First

If the site doesn't have dynamic loading, use OpenClaw's web_fetch tool—it's fastest.

2. Need JavaScript? Use Playwright Simple

If you need to wait for JavaScript rendering, use playwright-simple.js.

3. Getting Blocked? Use Stealth

If you encounter 403 or Cloudflare challenges, use playwright-stealth.js.

4. Special Sites Need Specialized Skills

YouTube → deep-scraper
Reddit → reddit-scraper
Twitter → bird skill

---

🔧 Customization

All scripts support environment variables:

# Set screenshot path
SCREENSHOT_PATH=/path/to/screenshot.png node scripts/playwright-stealth.js URL

# Set wait time (milliseconds)
WAIT_TIME=10000 node scripts/playwright-simple.js URL

# Enable headful mode (show browser)
HEADLESS=false node scripts/playwright-stealth.js URL

# Save HTML
SAVE_HTML=true node scripts/playwright-stealth.js URL

# Custom User-Agent
USER_AGENT="Mozilla/5.0 ..." node scripts/playwright-stealth.js URL

---

📊 Performance Comparison

Method	Speed	Anti-Bot	Success Rate (Discuss.com.hk)
web_fetch	⚡ Fastest	❌ None	0%
Playwright Simple	🚀 Fast	⚠️ Low	20%
Playwright Stealth	⏱️ Medium	✅ Medium	100% ✅
Puppeteer Stealth	⏱️ Medium	✅ Medium-High	~80%
Crawlee (deep-scraper)	🐢 Slow	❌ Detected	0%
Chaser (Rust)	⏱️ Medium	❌ Detected	0%

---

🛡️ Anti-Bot Techniques Summary

Lessons learned from our testing:

✅ Effective Anti-Bot Measures

1. Hide `navigator.webdriver` — Essential 2. Realistic User-Agent — Use real devices (iPhone, Android) 3. Mimic Human Behavior — Random delays, scrolling 4. Avoid Framework Signatures — Crawlee, Selenium are easily detected 5. Use `addInitScript` (Playwright) — Inject before page load

❌ Ineffective Anti-Bot Measures

1. Only changing User-Agent — Not enough 2. Using high-level frameworks (Crawlee) — More easily detected 3. Docker isolation — Doesn't help with Cloudflare

---

🔍 Troubleshooting

Issue: 403 Forbidden

Solution: Use playwright-stealth.js

Issue: Cloudflare Challenge Page

Solution: 1. Increase wait time (10-15 seconds) 2. Try headless: false (headful mode sometimes has higher success rate) 3. Consider using proxy IPs

Issue: Blank Page

Solution: 1. Increase waitForTimeout 2. Use waitUntil: 'networkidle' or 'domcontentloaded' 3. Check if login is required

---

📝 Memory & Experience

2026-02-07 Discuss.com.hk Test Conclusions

✅ Pure Playwright + Stealth succeeded (5s, 200 OK)
❌ Crawlee (deep-scraper) failed (403)
❌ Chaser (Rust) failed (Cloudflare)
❌ Puppeteer standard failed (403)

Best Solution: Pure Playwright + anti-bot techniques (framework-independent)

---

🚧 Future Improvements

[ ] Add proxy IP rotation
[ ] Implement cookie management (maintain login state)
[ ] Add CAPTCHA handling (2captcha / Anti-Captcha)
[ ] Batch scraping (parallel URLs)
[ ] Integration with OpenClaw's browser tool

---

📚 References

Contributing Guide

Thank you for considering contributing to playwright-scraper-skill!

🐛 Reporting Issues

If you find a bug or have a feature suggestion:

1. Check Issues to see if it already exists 2. If not, create a new Issue 3. Provide the following information:

Problem description
Steps to reproduce
Expected vs actual behavior
Environment (Node.js version, OS)
Error messages (if any)

💡 Feature Requests

1. Create an Issue with [Feature Request] in the title 2. Explain:

The desired feature
Use cases
Why this feature would be useful

🔧 Submitting Code

Setting Up Development Environment

# Fork the repo and clone
git clone https://github.com/YOUR_USERNAME/playwright-scraper-skill.git
cd playwright-scraper-skill

# Install dependencies
npm install
npx playwright install chromium

# Test
node scripts/playwright-simple.js https://example.com

Contribution Workflow

1. Create a new branch:

   git checkout -b feature/my-new-feature

2. Make your changes

3. Test your changes:

   npm test
   node scripts/playwright-stealth.js <test-URL>

4. Commit:

   git add .
   git commit -m "Add: brief description of changes"

5. Push and create a Pull Request:

   git push origin feature/my-new-feature

Commit Message Guidelines

Use clear commit messages:

Add: new feature
Fix: issue description
Update: existing feature
Refactor: code refactoring
Docs: documentation update
Test: add or modify tests

Example:

Fix: playwright-stealth.js screenshot timeout issue

- Increase timeout parameter to 10 seconds
- Add try-catch error handling
- Update documentation

📝 Documentation

If your changes affect usage:

Update SKILL.md (full documentation)
Update README.md (quick reference)
Update examples/README.md (if adding new examples)
Update CHANGELOG.md (record changes)

✅ Checklist

Before submitting a PR, confirm:

[ ] Code runs properly
[ ] Doesn't break existing functionality
[ ] Updated relevant documentation
[ ] Clear commit messages
[ ] No sensitive information (API keys, personal paths, etc.)

🎯 Priority Areas

Currently welcoming contributions in:

1. New anti-bot techniques — Improve success rates 2. Support more websites — Test and share success cases 3. Performance optimization — Speed up scraping 4. Error handling — Better error messages and recovery 5. Documentation improvements — Clearer explanations and examples

🚫 Unaccepted Contributions

Adding complex dependencies (keep it lightweight)
Features violating privacy or laws
Breaking existing API changes (unless well justified)

📞 Contact

Have questions? Feel free to:

Create an Issue for discussion
Ask in Pull Request comments

---

Thank you for your contribution! 🙏

#!/bin/bash
# 範例：爬取 Discuss.com.hk 熱門話題

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SKILL_DIR="$(dirname "$SCRIPT_DIR")"

echo "🕷️  Discuss.com.hk 爬蟲範例"
echo ""
echo "使用 Playwright Stealth（已驗證成功）"
echo ""

cd "$SKILL_DIR" && \
WAIT_TIME=10000 \
SCREENSHOT_PATH=/tmp/discuss-hk.png \
SAVE_HTML=true \
node scripts/playwright-stealth.js "https://m.discuss.com.hk/#hot"

Usage Examples

Basic Usage

1. Quick Scrape (Example.com)

node scripts/playwright-simple.js https://example.com

Output:

{
  "title": "Example Domain",
  "url": "https://example.com/",
  "content": "Example Domain\n\nThis domain is for use...",
  "metaDescription": "",
  "elapsedSeconds": "3.42"
}

---

2. Anti-Bot Protected Site (Discuss.com.hk)

node scripts/playwright-stealth.js "https://m.discuss.com.hk/#hot"

Output:

{
  "title": "香港討論區 discuss.com.hk",
  "url": "https://m.discuss.com.hk/#hot",
  "htmlLength": 186345,
  "contentPreview": "...",
  "cloudflare": false,
  "screenshot": "./screenshot-1770467444364.png",
  "data": {
    "links": [
      {
        "text": "區議員周潔瑩疑消防通道違泊 道歉稱急於搬貨",
        "href": "https://m.discuss.com.hk/index.php?action=thread&tid=32148378..."
      }
    ]
  },
  "elapsedSeconds": "19.59"
}

---

Advanced Usage

3. Custom Wait Time

WAIT_TIME=15000 node scripts/playwright-stealth.js <URL>

4. Show Browser (Debug Mode)

HEADLESS=false node scripts/playwright-stealth.js <URL>

5. Save Screenshot and HTML

SCREENSHOT_PATH=/tmp/my-page.png \
SAVE_HTML=true \
node scripts/playwright-stealth.js <URL>

6. Custom User-Agent

USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
node scripts/playwright-stealth.js <URL>

---

Integration Examples

Using in Shell Scripts

#!/bin/bash
# Run from playwright-scraper-skill directory

URL="https://example.com"
OUTPUT_FILE="result.json"

echo "🕷️  Starting scrape: $URL"

node scripts/playwright-stealth.js "$URL" > "$OUTPUT_FILE"

if [ $? -eq 0 ]; then
  echo "✅ Success! Results saved to: $OUTPUT_FILE"
else
  echo "❌ Failed"
  exit 1
fi

Batch Scraping Multiple URLs

#!/bin/bash

URLS=(
  "https://example.com"
  "https://example.org"
  "https://example.net"
)

for url in "${URLS[@]}"; do
  echo "Scraping: $url"
  node scripts/playwright-stealth.js "$url" > "output_$(date +%s).json"
  sleep 5  # Avoid IP blocking
done

---

Calling from Node.js

const { spawn } = require('child_process');

function scrape(url) {
  return new Promise((resolve, reject) => {
    const proc = spawn('node', [
      'scripts/playwright-stealth.js',
      url
    ]);
    
    let output = '';
    
    proc.stdout.on('data', (data) => {
      output += data.toString();
    });
    
    proc.on('close', (code) => {
      if (code === 0) {
        try {
          // Extract JSON (last line)
          const lines = output.trim().split('\n');
          const json = JSON.parse(lines[lines.length - 1]);
          resolve(json);
        } catch (e) {
          reject(e);
        }
      } else {
        reject(new Error(`Exit code: ${code}`));
      }
    });
  });
}

// Usage
(async () => {
  const result = await scrape('https://example.com');
  console.log(result.title);
})();

---

Common Scenarios

Scraping News Articles

node scripts/playwright-stealth.js "https://news.example.com/article/123"

Scraping E-commerce Products

WAIT_TIME=10000 \
SAVE_HTML=true \
node scripts/playwright-stealth.js "https://shop.example.com/product/456"

Scraping Forum Posts

node scripts/playwright-stealth.js "https://forum.example.com/thread/789"

---

Troubleshooting

Issue: Page Not Fully Loaded

Solution: Increase wait time

WAIT_TIME=20000 node scripts/playwright-stealth.js <URL>

Issue: Still Blocked by Cloudflare

Solution: Use headful mode + manual wait

HEADLESS=false \
WAIT_TIME=30000 \
node scripts/playwright-stealth.js <URL>

Issue: Requires Login

Solution: Manually login first, export cookies, then load (Future feature, currently not supported)

---

Performance Tips

1. Parallel scraping: Use Promise.all() or shell & 2. Delay requests: sleep 5 to avoid IP blocking 3. Use proxies: Rotate IPs (future feature) 4. Cache results: Avoid duplicate scraping

---

For more information, see SKILL.md

Installation Guide

📦 Quick Installation

1. Clone or Download the Skill

# Method 1: Using git clone (if public repo)
git clone https://github.com/waisimon/playwright-scraper-skill.git
cd playwright-scraper-skill

# Method 2: Download ZIP and extract
# After downloading, enter the directory
cd playwright-scraper-skill

2. Install Dependencies

# Install Playwright (recommended)
npm install

# Install browser (Chromium)
npx playwright install chromium

3. Test

# Quick test
node scripts/playwright-simple.js https://example.com

# Test Stealth version
node scripts/playwright-stealth.js https://example.com

---

🔧 Advanced Installation

Using with OpenClaw

If you're using OpenClaw, you can place this skill in the skills directory:

# Assuming your OpenClaw workspace is at ~/.openclaw/workspace
cp -r playwright-scraper-skill ~/.openclaw/workspace/skills/

# Then you can invoke it in OpenClaw

---

✅ Verify Installation

Run the example script:

# Discuss.com.hk example (verified working)
bash examples/discuss-hk.sh

If you see output similar to this, installation is successful:

🕷️  Starting Playwright Stealth scraper...
📱 Navigating to: https://m.discuss.com.hk/#hot
📡 HTTP Status: 200
✅ Scraping complete!

---

🐛 Common Issues

Issue: Playwright not found

Error message: Error: Cannot find module 'playwright'

Solution:

npm install
npx playwright install chromium

Issue: Browser launch failed

Error message: browserType.launch: Executable doesn't exist

Solution:

npx playwright install chromium

Issue: Permission errors

Error message: Permission denied

Solution:

chmod +x scripts/*.js
chmod +x examples/*.sh

---

📝 System Requirements

Node.js: v18+ recommended
OS: macOS / Linux / Windows
Disk Space: ~500MB (including Chromium)
RAM: 2GB+ recommended

---

🚀 Next Steps

After installation, check out:

README.md — Quick reference
SKILL.md — Full documentation
examples/ — Example scripts

{
  "name": "playwright-scraper-skill",
  "version": "1.2.0",
  "description": "基於 Playwright 的網頁爬蟲 OpenClaw Skill",
  "main": "scripts/playwright-stealth.js",
  "scripts": {
    "simple": "node scripts/playwright-simple.js",
    "stealth": "node scripts/playwright-stealth.js",
    "test": "bash test.sh"
  },
  "keywords": [
    "scraper",
    "playwright",
    "puppeteer",
    "cloudflare",
    "anti-detection"
  ],
  "author": "多米",
  "license": "MIT",
  "dependencies": {
    "playwright": "^1.40.0"
  }
}

Playwright Scraper Skill 🕷️

![License: MIT](https://opensource.org/licenses/MIT) ![Node.js](https://nodejs.org/) ![Playwright](https://playwright.dev/)

基於 Playwright 的網頁爬蟲 OpenClaw Skill。支援反爬保護，已驗證成功爬取 Discuss.com.hk 等複雜網站。

📦 安裝方法： 查看 INSTALL.md

📚 完整文件： 查看 SKILL.md

💡 使用範例： 查看 examples/README.md

---

✨ 特色

✅ 純 Playwright — 現代、強大、易用
✅ 反爬保護 — 隱藏自動化特徵、真實 UA
✅ 已驗證 — Discuss.com.hk 100% 成功
✅ 簡單易用 — 一行命令搞定
✅ 可自訂 — 支援環境變數配置

---

🚀 快速開始

安裝

npm install
npx playwright install chromium

使用

# 快速爬取
node scripts/playwright-simple.js https://example.com

# 反爬保護版（推薦）
node scripts/playwright-stealth.js "https://m.discuss.com.hk/#hot"

---

📖 兩種模式

模式	適用場景	速度	反爬能力
Simple	一般動態網站	快（3-5秒）	無
Stealth ⭐	有反爬保護的網站	中（5-20秒）	中高

Simple 模式

適合沒有反爬保護的網站：

node scripts/playwright-simple.js <URL>

Stealth 模式（推薦）

適合有 Cloudflare 或反爬保護的網站：

node scripts/playwright-stealth.js <URL>

反爬技巧：

隱藏 navigator.webdriver
真實 User-Agent（iPhone）
模擬真人行為
支援截圖和 HTML 儲存

---

🎯 自訂參數

所有腳本都支援環境變數：

# 顯示瀏覽器
HEADLESS=false node scripts/playwright-stealth.js <URL>

# 自訂等待時間（毫秒）
WAIT_TIME=10000 node scripts/playwright-stealth.js <URL>

# 儲存截圖
SCREENSHOT_PATH=/tmp/page.png node scripts/playwright-stealth.js <URL>

# 儲存 HTML
SAVE_HTML=true node scripts/playwright-stealth.js <URL>

# 自訂 User-Agent
USER_AGENT="Mozilla/5.0 ..." node scripts/playwright-stealth.js <URL>

---

📊 測試結果

網站	結果	時間
Discuss.com.hk	✅ 200 OK	5-20 秒
Example.com	✅ 200 OK	3-5 秒
Cloudflare 保護網站	✅ 多數成功	10-30 秒

---

📁 檔案結構

playwright-scraper-skill/
├── scripts/
│   ├── playwright-simple.js       # 簡單版
│   └── playwright-stealth.js      # Stealth 版 ⭐
├── examples/
│   ├── discuss-hk.sh              # Discuss.com.hk 範例
│   └── README.md                  # 更多範例
├── SKILL.md                       # 完整文件
├── INSTALL.md                     # 安裝指南
├── README.md                      # 本檔案
├── CONTRIBUTING.md                # 貢獻指南
├── CHANGELOG.md                   # 版本記錄
└── package.json                   # npm 配置

---

💡 使用建議

1. 先試 web_fetch — OpenClaw 內建工具最快 2. 動態網站用 Simple — 沒有反爬保護時 3. 反爬網站用 Stealth ⭐ — 主力工具 4. 特殊網站用專用 skill — YouTube、Reddit 等

---

🐛 故障排除

被 403 擋住？

使用 Stealth 模式：

node scripts/playwright-stealth.js <URL>

Cloudflare 挑戰？

增加等待時間 + 有頭模式：

HEADLESS=false WAIT_TIME=30000 node scripts/playwright-stealth.js <URL>

找不到 Playwright？

重新安裝：

npm install
npx playwright install chromium

更多問題查看 INSTALL.md

---

🤝 貢獻

歡迎貢獻！查看 CONTRIBUTING.md

---

📄 授權

MIT License - 查看 LICENSE

---

🔗 相關連結

Playwright 官方文檔
完整文件 (SKILL.md)
安裝指南 (INSTALL.md)
使用範例 (examples/)

Playwright Scraper Skill 🕷️

![License: MIT](https://opensource.org/licenses/MIT) ![Node.js](https://nodejs.org/) ![Playwright](https://playwright.dev/)

[中文文檔](README_ZH.md) | English

A Playwright-based web scraping OpenClaw Skill with anti-bot protection. Successfully tested on complex websites like Discuss.com.hk.

📦 Installation: See INSTALL.md

📚 Full Documentation: See SKILL.md

💡 Examples: See examples/README.md

---

✨ Features

✅ Pure Playwright — Modern, powerful, easy to use
✅ Anti-Bot Protection — Hides automation, realistic UA
✅ Verified — 100% success on Discuss.com.hk
✅ Simple to Use — One-line commands
✅ Customizable — Environment variable support

---

🚀 Quick Start

Installation

npm install
npx playwright install chromium

Usage

# Quick scraping
node scripts/playwright-simple.js https://example.com

# Stealth mode (recommended)
node scripts/playwright-stealth.js "https://m.discuss.com.hk/#hot"

---

📖 Two Modes

Mode	Use Case	Speed	Anti-Bot
Simple	Regular dynamic sites	Fast (3-5s)	None
Stealth ⭐	Sites with anti-bot	Medium (5-20s)	Medium-High

Simple Mode

For sites without anti-bot protection:

node scripts/playwright-simple.js <URL>

Stealth Mode (Recommended)

For sites with Cloudflare or anti-bot protection:

node scripts/playwright-stealth.js <URL>

Anti-Bot Techniques:

Hide navigator.webdriver
Realistic User-Agent (iPhone)
Human-like behavior simulation
Screenshot and HTML saving support

---

🎯 Customization

All scripts support environment variables:

# Show browser
HEADLESS=false node scripts/playwright-stealth.js <URL>

# Custom wait time (milliseconds)
WAIT_TIME=10000 node scripts/playwright-stealth.js <URL>

# Save screenshot
SCREENSHOT_PATH=/tmp/page.png node scripts/playwright-stealth.js <URL>

# Save HTML
SAVE_HTML=true node scripts/playwright-stealth.js <URL>

# Custom User-Agent
USER_AGENT="Mozilla/5.0 ..." node scripts/playwright-stealth.js <URL>

---

📊 Test Results

Website	Result	Time
Discuss.com.hk	✅ 200 OK	5-20s
Example.com	✅ 200 OK	3-5s
Cloudflare Protected	✅ Mostly successful	10-30s

---

📁 File Structure

playwright-scraper-skill/
├── scripts/
│   ├── playwright-simple.js       # Simple mode
│   └── playwright-stealth.js      # Stealth mode ⭐
├── examples/
│   ├── discuss-hk.sh              # Discuss.com.hk example
│   └── README.md                  # More examples
├── SKILL.md                       # Full documentation
├── INSTALL.md                     # Installation guide
├── README.md                      # This file
├── README_ZH.md                   # Chinese documentation
├── CONTRIBUTING.md                # Contribution guide
├── CHANGELOG.md                   # Version history
└── package.json                   # npm config

---

💡 Best Practices

1. Try web_fetch first — OpenClaw's built-in tool is fastest 2. Use Simple for dynamic sites — When no anti-bot protection 3. Use Stealth for protected sites ⭐ — Main workhorse 4. Use specialized skills — For YouTube, Reddit, etc.

---

🐛 Troubleshooting

Getting 403 blocked?

Use Stealth mode:

node scripts/playwright-stealth.js <URL>

Cloudflare challenge?

Increase wait time + headful mode:

HEADLESS=false WAIT_TIME=30000 node scripts/playwright-stealth.js <URL>

Playwright not found?

Reinstall:

npm install
npx playwright install chromium

More issues? See INSTALL.md

---

🤝 Contributing

Contributions welcome! See CONTRIBUTING.md

---

📄 License

MIT License - See LICENSE

---

🔗 Links

Playwright Official Docs
Full Documentation (SKILL.md)
Installation Guide (INSTALL.md)
Examples (examples/)

#!/usr/bin/env node
/**
 * Playwright Simple Scraper
 * 適用：一般動態網站，無反爬保護
 * 速度：快（3-5 秒）
 * 
 * Usage: node playwright-simple.js <URL>
 */

const { chromium } = require('playwright');

const url = process.argv[2];
const waitTime = parseInt(process.env.WAIT_TIME || '3000');
const screenshotPath = process.env.SCREENSHOT_PATH;

if (!url) {
    console.error('❌ 請提供 URL');
    console.error('用法: node playwright-simple.js <URL>');
    process.exit(1);
}

(async () => {
    console.log('🚀 啟動 Playwright 簡單版爬蟲...');
    const startTime = Date.now();
    
    const browser = await chromium.launch({ 
        headless: process.env.HEADLESS !== 'false' 
    });
    const page = await browser.newPage();
    
    console.log(`📱 導航到: ${url}`);
    await page.goto(url, { waitUntil: 'domcontentloaded' });
    
    console.log(`⏳ 等待 ${waitTime}ms...`);
    await page.waitForTimeout(waitTime);
    
    // 擷取基本資訊
    const result = await page.evaluate(() => {
        return {
            title: document.title,
            url: window.location.href,
            content: document.body.innerText.substring(0, 5000),
            metaDescription: document.querySelector('meta[name="description"]')?.content || '',
        };
    });
    
    // 截圖（如果指定）
    if (screenshotPath) {
        await page.screenshot({ path: screenshotPath });
        console.log(`📸 截圖已儲存: ${screenshotPath}`);
    }
    
    const elapsed = ((Date.now() - startTime) / 1000).toFixed(2);
    result.elapsedSeconds = elapsed;
    
    console.log('\n✅ 爬取完成！');
    console.log(JSON.stringify(result, null, 2));
    
    await browser.close();
})();

#!/usr/bin/env node
/**
 * Playwright Stealth Scraper
 * 適用：有 Cloudflare 或反爬保護的網站
 * 速度：中等（5-10 秒）
 * 反爬能力：中（隱藏自動化、真實 UA）
 * 
 * Usage: node playwright-stealth.js <URL>
 * 
 * 環境變數：
 * - HEADLESS=false        顯示瀏覽器
 * - WAIT_TIME=10000       等待時間（毫秒）
 * - SCREENSHOT_PATH=...   截圖路徑
 * - SAVE_HTML=true        儲存 HTML
 * - USER_AGENT=...        自訂 User-Agent
 */

const { chromium } = require('playwright');
const fs = require('fs');
const path = require('path');

const url = process.argv[2];
const waitTime = parseInt(process.env.WAIT_TIME || '5000');
const headless = process.env.HEADLESS !== 'false';
const screenshotPath = process.env.SCREENSHOT_PATH || `./screenshot-${Date.now()}.png`;
const saveHtml = process.env.SAVE_HTML === 'true';

// 預設 User-Agent（iPhone）
const defaultUA = 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1';
const userAgent = process.env.USER_AGENT || defaultUA;

if (!url) {
    console.error('❌ 請提供 URL');
    console.error('用法: node playwright-stealth.js <URL>');
    process.exit(1);
}

(async () => {
    console.log('🕷️  啟動 Playwright Stealth 爬蟲...');
    console.log(`🔒 反爬模式: ${headless ? '無頭' : '有頭'}`);
    const startTime = Date.now();
    
    const browser = await chromium.launch({
        headless: headless,
        args: [
            '--no-sandbox',
            '--disable-setuid-sandbox',
            '--disable-blink-features=AutomationControlled',
            '--disable-features=IsolateOrigins,site-per-process',
        ],
    });
    
    const context = await browser.newContext({
        userAgent: userAgent,
        locale: 'zh-HK',
        viewport: { width: 375, height: 812 }, // iPhone size
        extraHTTPHeaders: {
            'Accept-Language': 'zh-HK,zh-TW;q=0.9,zh;q=0.8,en;q=0.7',
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
        },
    });
    
    // 隱藏自動化特徵
    await context.addInitScript(() => {
        Object.defineProperty(navigator, 'webdriver', {
            get: () => false,
        });
        
        window.chrome = { runtime: {} };
        
        // Mock permissions
        const originalQuery = window.navigator.permissions.query;
        window.navigator.permissions.query = (parameters) => (
            parameters.name === 'notifications' ?
                Promise.resolve({ state: Notification.permission }) :
                originalQuery(parameters)
        );
    });
    
    const page = await context.newPage();
    
    console.log(`📱 導航到: ${url}`);
    try {
        const response = await page.goto(url, {
            waitUntil: 'domcontentloaded',
            timeout: 30000,
        });
        
        console.log(`📡 HTTP Status: ${response.status()}`);
        
        if (response.status() === 403) {
            console.log('⚠️  收到 403，但繼續嘗試...');
        }
        
    } catch (error) {
        console.error(`❌ 導航失敗: ${error.message}`);
    }
    
    console.log(`⏳ 等待 ${waitTime}ms 讓內容載入...`);
    await page.waitForTimeout(waitTime);
    
    // 檢查 Cloudflare
    const cloudflare = await page.evaluate(() => {
        return document.body.innerText.includes('Checking your browser') ||
               document.body.innerText.includes('Just a moment') ||
               document.querySelector('iframe[src*="challenges.cloudflare.com"]') !== null;
    });
    
    if (cloudflare) {
        console.log('🛡️  偵測到 Cloudflare 挑戰，等待額外 10 秒...');
        await page.waitForTimeout(10000);
    }
    
    // 擷取資訊
    const result = await page.evaluate(() => {
        return {
            title: document.title,
            url: window.location.href,
            htmlLength: document.documentElement.outerHTML.length,
            contentPreview: document.body.innerText.substring(0, 1000),
        };
    });
    
    result.cloudflare = cloudflare;
    
    // 截圖
    try {
        await page.screenshot({ path: screenshotPath, fullPage: false, timeout: 10000 });
        console.log(`📸 截圖已儲存: ${screenshotPath}`);
        result.screenshot = screenshotPath;
    } catch (error) {
        console.log(`⚠️  截圖失敗: ${error.message}`);
        result.screenshot = null;
    }
    
    // 儲存 HTML（如果需要）
    if (saveHtml) {
        const htmlPath = screenshotPath.replace(/\.[^.]+$/, '.html');
        const html = await page.content();
        fs.writeFileSync(htmlPath, html);
        console.log(`📄 HTML 已儲存: ${htmlPath}`);
        result.htmlFile = htmlPath;
    }
    
    // 嘗試提取結構化資料（依網站調整）
    const customData = await page.evaluate(() => {
        // 範例：提取所有連結
        const links = Array.from(document.querySelectorAll('a[href*="tid="]'))
            .slice(0, 10)
            .map(a => ({
                text: a.innerText.trim().substring(0, 100),
                href: a.href,
            }));
        
        return { links };
    });
    
    result.data = customData;
    
    const elapsed = ((Date.now() - startTime) / 1000).toFixed(2);
    result.elapsedSeconds = elapsed;
    
    console.log('\n✅ 爬取完成！');
    console.log(JSON.stringify(result, null, 2));
    
    await browser.close();
})();

#!/bin/bash
# 簡單測試腳本

set -e

echo "🧪 Playwright Scraper Skill 測試"
echo ""

# 測試 1: Playwright Simple
echo "📝 測試 1: Playwright Simple (Example.com)"
node scripts/playwright-simple.js https://example.com > /tmp/test-simple.json
if grep -q "Example Domain" /tmp/test-simple.json; then
  echo "✅ Simple 模式正常"
else
  echo "❌ Simple 模式失敗"
  exit 1
fi
echo ""

# 測試 2: Playwright Stealth
echo "📝 測試 2: Playwright Stealth (Example.com)"
node scripts/playwright-stealth.js https://example.com > /tmp/test-stealth.json
if grep -q "Example Domain" /tmp/test-stealth.json; then
  echo "✅ Stealth 模式正常"
else
  echo "❌ Stealth 模式失敗"
  exit 1
fi
echo ""

# 測試 3: 環境變數
echo "📝 測試 3: 環境變數 (WAIT_TIME)"
WAIT_TIME=1000 node scripts/playwright-simple.js https://example.com > /tmp/test-env.json
if grep -q "Example Domain" /tmp/test-env.json; then
  echo "✅ 環境變數正常"
else
  echo "❌ 環境變數失敗"
  exit 1
fi
echo ""

# 清理
rm -f /tmp/test-*.json screenshot-*.png

echo "✅ 所有測試通過！"

Related skills

Agent BrowserGive their coding agent reliable, high-fidelity control over any website or Electron desktop app.591k39.6k

Lark ApprovalLet their AI coding agent create, read, update, and approve items in Lark (Feishu) approval workflows without leaving the coding environment.494k

Lark EventHandle Feishu/Lark bot events, webhooks, and subscription callbacks in agent-driven backend code.493k

Lark Workflow Meeting SummaryAutomatically generate structured meeting summaries and action items from Lark/Feishu calls without manual note-taking.492k

Lark Workflow Standup ReportAutomatically generate and post daily standup reports in Lark/Feishu from their workflow and activity data.492k

Twitter AutomationAutomate X/Twitter posts, scheduling, and engagement via X API integration441k669

FAQ

When should I use the stealth script?

When you hit a 403 or Cloudflare challenge; the stealth script hides automation and uses realistic user agents.

What output does it produce?

JSON with title, content, and URL, plus optional screenshot and HTML file for the stealth script.

Automation & Workflowsintegrationsbackend

About

Playwright Scraper Skill by the numbers

playwright-scraper-skill capabilities & compatibility

What playwright-scraper-skill says it does

Add your badge

What it does

Who is it for?

When should I use this skill?

What you get

By the numbers

Files

Playwright Scraper Skill

🎯 Use Case Matrix

📦 Installation

🚀 Quick Start

1️⃣ Simple Sites (No Anti-Bot)

2️⃣ Dynamic Sites (Requires JavaScript)

3️⃣ Anti-Bot Protected Sites (Cloudflare etc.)

4️⃣ YouTube Video Transcripts

📖 Script Descriptions

scripts/playwright-simple.js

scripts/playwright-stealth.js ⭐

🎓 Best Practices

1. Try web_fetch First

2. Need JavaScript? Use Playwright Simple

3. Getting Blocked? Use Stealth

4. Special Sites Need Specialized Skills

🔧 Customization

📊 Performance Comparison

🛡️ Anti-Bot Techniques Summary

✅ Effective Anti-Bot Measures

❌ Ineffective Anti-Bot Measures

🔍 Troubleshooting

Issue: 403 Forbidden

Issue: Cloudflare Challenge Page

Issue: Blank Page

📝 Memory & Experience

2026-02-07 Discuss.com.hk Test Conclusions

🚧 Future Improvements

📚 References

Changelog

[1.2.0] - 2026-02-07

🔄 Major Changes

[1.1.0] - 2026-02-07

✅ Added

🔧 Improvements

[1.0.0] - 2026-02-07

✅ Initial Release

Future Plans

Contributing Guide

🐛 Reporting Issues

💡 Feature Requests

🔧 Submitting Code

Setting Up Development Environment

Contribution Workflow

Commit Message Guidelines

📝 Documentation

✅ Checklist

🎯 Priority Areas

🚫 Unaccepted Contributions

📞 Contact

Usage Examples

Basic Usage

1. Quick Scrape (Example.com)

2. Anti-Bot Protected Site (Discuss.com.hk)

Advanced Usage

3. Custom Wait Time

4. Show Browser (Debug Mode)

5. Save Screenshot and HTML

6. Custom User-Agent

Integration Examples

Using in Shell Scripts

Batch Scraping Multiple URLs

Calling from Node.js

Common Scenarios

Scraping News Articles

Scraping E-commerce Products

Scraping Forum Posts

Troubleshooting

Issue: Page Not Fully Loaded

`scripts/playwright-simple.js`

`scripts/playwright-stealth.js` ⭐