Web Content Fetcher

Name: Web Content Fetcher
Author: shino369

shino369/claude-code-personal-workspace

68 installs
3 repo stars
Updated January 13, 2026
shino369/claude-code-personal-workspace

Helps with marketing & seo tasks.

About

web-content-fetcher is a Claude Code skill for marketing & seo. It helps solo builders move faster with AI-assisted development.

web-content-fetcher
Marketing & SEO
AI-coding skill

Web Content Fetcher by the numbers

68 all-time installs (skills.sh)
Ranked #1,234 of 1,879 Marketing & SEO skills by installs in the Skillselion catalog
Data as of Jul 27, 2026 (Skillselion catalog sync)

npx skills add https://github.com/shino369/claude-code-personal-workspace --skill web-content-fetcher

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/shino369/claude-code-personal-workspace/web-content-fetcher.svg)](https://skillselion.com/skills/shino369/claude-code-personal-workspace/web-content-fetcher)

Installs	68
repo stars	★ 3
Last updated	January 13, 2026
Repository	shino369/claude-code-personal-workspace ↗

What it does

Helps with marketing & seo tasks.

Files

SKILL.mdMarkdownGitHub ↗

Web Content Fetcher

Expert knowledge for fetching and parsing web content, handling size limitations, JavaScript-rendered sites, and extracting clean article content from HTML.

Overview

This skill provides battle-tested strategies for fetching web content in Claude Code, addressing critical challenges:

1. WebFetch tool limitation: ~50KB max content size 2. Read tool limitation: 256KB max file size per call 3. JavaScript-rendered content: Twitter/X, React SPAs require special handling

Four-tier approach:

Tier 1: WebFetch for small content (< 50KB)
Tier 2: curl + Task agent for any size static content ⭐ DEFAULT
Tier 3: Scripts for edge cases (EUC-JP encoding, complex HTML)
Tier 4: Playwright for JavaScript-rendered sites (Twitter/X, React SPAs)

Quick Tool Limitations

Tool	Max Size	Best For	Limitation
WebFetch	~50KB	Small articles, first attempt	Prompt length constraints
Read	256KB	Reading fetched files	Large files need pagination
curl	Unlimited	Any size content, raw downloads	No parsing
Task agent	Unlimited	Extraction from any size HTML	Handles pagination automatically

Tiered Fetching Strategy

Tier 1: Small Content - WebFetch Direct

Use when: Simple articles, API responses, first attempt on unknown content

Quick example:

WebFetch tool with url and extraction prompt

If fails: "Prompt too long" error → Switch to Tier 2

Tier 2: Any Size Content - curl + Task Agent ⭐ RECOMMENDED

Use when: News articles, blog posts, WebFetch fails, any content of any size

Quick workflow:

# 1. Create directory
mkdir -p output/tasks/YYYYMMDD_taskname/original

# 2. Fetch HTML
curl -s "URL" > output/tasks/YYYYMMDD_taskname/original/raw_html.html

# 3. Extract with Task agent
# Prompt: "Read HTML at [path], extract article content, save to fetched_content.md"

Why default: Handles any size, AI-powered extraction, no script setup needed.

For detailed workflow: See references/workflows.md

Tier 3: Script-Based Extraction - Edge Cases Only

Use when: Task agent struggles with complex HTML or special encoding requirements (EUC-JP)

Available scripts:

scripts/extract_article.js - Standard HTML with Mozilla Readability
scripts/extract_eucjp.js - Japanese EUC-JP encoded sites (4gamer.net)

Quick example:

node .claude/skills/web-content-fetcher/scripts/extract_article.js raw_html.html > fetched_content.md

Note: Try Task agent (Tier 2) first. Scripts only when Task agent explicitly fails.

For script details: See references/advanced-fetching.md

Tier 4: JavaScript-Rendered Content - Playwright

Use when: Twitter/X, React SPAs, dynamic sites, static fetch returns empty/error

Quick example:

node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://x.com/user/status/123456789" \
  --output fetched_content.md

Common sites: Twitter/X, Threads, Instagram, React/Vue/Angular SPAs, AJAX-loaded content

Image Extraction:

Automatically extracts images if exist (Twitter/X, Threads, blogs, etc.)
Captures image URLs with alt text from main content area
Filters out small images (icons/logos) automatically
Output includes structured Media section with URLs for downloading

Setup required (one-time):

cd .claude/skills/web-content-fetcher
pnpm add -D playwright --save-catalog-name=dev
pnpm exec playwright install chromium

For Playwright details: See references/advanced-fetching.md

Decision Tree

Use this flowchart to choose the right approach:

Need to fetch web content?
│
├─ JavaScript-rendered site (Twitter/X, React SPA, dynamic content)?
│  └─ Use Playwright script (Tier 4)
│     └─ See: references/advanced-fetching.md
│
├─ Size unknown or small expected content?
│  └─ Try WebFetch (Tier 1)
│     ├─ Success? → Done ✓
│     └─ "Prompt too long" error? → Use Tier 2
│
├─ Any other case (medium/large content, WebFetch failed)?
│  └─ Use curl + Task agent (Tier 2) ⭐ DEFAULT
│     ├─ Success? → Done ✓
│     └─ Task agent struggles? → Use Tier 3
│        └─ See: references/advanced-fetching.md
│
└─ Edge cases (Task agent fails, special encoding)?
   └─ Use curl + script (Tier 3)
      └─ See: references/advanced-fetching.md

Standard Workflow Quick Reference

Most common pattern (works for 90% of cases):

# 1. Setup
mkdir -p output/tasks/20260111_taskname/original

# 2. Fetch
curl -s "URL" > output/tasks/20260111_taskname/original/raw_html.html

# 3. Extract (Task agent prompt)
"Read HTML file at [path], extract main article content,
remove navigation/ads/sidebars/comments/footer,
save clean markdown to: [path]/fetched_content.md"

# 4. Use
Read: output/tasks/20260111_taskname/original/fetched_content.md

For detailed workflows and patterns: See references/workflows.md

Common Use Cases

Translation with images:

1. Fetch content (Tier 2 or Tier 4 for JS sites) 2. Extract to original/fetched_content.md with readable format (includes Media section with image URLs) 3. Download images using URLs from Media section 4. Pass to /translate command

Analysis:

1. Fetch content (Tier 2 or Tier 4) 2. Extract to original/fetched_content.md with readable format 3. Read and analyze

Social media posts (Twitter/X, Threads, Instagram):

1. Use Playwright (Tier 4) directly 2. Automatically extracts text, images, and metadata 3. Outputs to fetched_content.md with Media section in a readable format 4. Download images from extracted URLs 5. Use for translation/analysis

For all patterns: See references/workflows.md

When Things Go Wrong

Common issues and quick fixes:

Issue	Quick Fix	Details
WebFetch "Prompt too long"	Switch to Tier 2	troubleshooting.md
Read tool file too large	Use Task agent	troubleshooting.md
Garbled Japanese text	EUC-JP encoding issue	troubleshooting.md
JavaScript required	Use Playwright (Tier 4)	troubleshooting.md
Anti-bot protection	Add user agent	troubleshooting.md
Authentication required	Use curl with headers/cookies	troubleshooting.md

For complete troubleshooting guide: See references/troubleshooting.md

Best Practices

1. Always use task directories: output/tasks/YYYYMMDD_taskname/ 2. Default to Tier 2: curl + Task agent works for nearly all cases 3. Keep raw HTML: Save to original/raw_html.html for reference (except Tier 4) 4. Use Playwright for JS sites: Twitter/X, React SPAs, dynamic content 5. Clean content format: Always save extracted content as markdown 6. Descriptive naming: Use date prefix (YYYYMMDD_) and descriptive task names 7. Try simple first: WebFetch → curl + Task agent → Scripts (only if needed) 8. Task agent for extraction: Let AI handle pagination and parsing complexity

Reference Files

This skill uses progressive disclosure for efficiency. Core information is in this file. Detailed guides are in reference files:

[references/workflows.md](references/workflows.md) - Standard workflows, common patterns, directory structure, content extraction priorities
[references/advanced-fetching.md](references/advanced-fetching.md) - Tier 3 script-based extraction, Tier 4 Playwright details, setup instructions
[references/troubleshooting.md](references/troubleshooting.md) - Common issues, solutions, quick reference commands

Read reference files when you need detailed guidance for specific scenarios.

Quick Start Examples

Simple article:

mkdir -p output/tasks/20260111_article/original
curl -s "https://example.com/article" > output/tasks/20260111_article/original/raw_html.html
# Task agent: Extract to fetched_content.md

Twitter/X post:

mkdir -p output/tasks/20260111_twitter/original
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://x.com/user/status/123" \
  --output output/tasks/20260111_twitter/original/fetched_content.md

Threads post:

mkdir -p output/tasks/20260111_threads/original
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://www.threads.com/@user/post/ABC123" \
  --output output/tasks/20260111_threads/original/fetched_content.md

Output format with images (works for all sites):

# Title

Main content text here...

### Media

1. Image description or alt text
   - URL: https://example.com/image1.jpg
2. Another image
   - URL: https://example.com/image2.jpg

Note:

Images are automatically extracted from the main content area
Small images (< 100x100px) like icons/logos are filtered out
Image formats: .jpg, .png, .webp, .gif, .svg - all preserved in URLs
When downloading, respect the original file extension

Japanese site (potential encoding issue):

mkdir -p output/tasks/20260111_japanese/original
curl -s "https://4gamer.net/..." > output/tasks/20260111_japanese/original/raw_html.html
# Task agent with encoding detection, or use extract_eucjp.js if needed

Advanced Fetching Methods

This reference covers advanced fetching methods for edge cases: script-based extraction (Tier 3) and JavaScript-rendered content (Tier 4).

When to use this guide: Only when standard Task agent approach (Tier 2) fails or for special requirements.

Tier 3: Script-Based Extraction
extract_article.js - Standard HTML
extract_eucjp.js - Japanese EUC-JP Encoding
Tier 4: JavaScript-Rendered Content
Playwright Setup
fetch_js_content.js Usage
Site-Specific Features

Tier 3: Script-Based Extraction

Use only when: Task agent struggles with complex HTML, need precise control, or have special encoding requirements.

Note: Task agent (Tier 2) handles 99% of cases including large files and complex HTML. Use scripts only when Task agent explicitly fails or for special encoding requirements.

extract_article.js - Standard HTML

Purpose: Extract article content using Mozilla's Readability algorithm (same as Firefox Reader View)

Requirements (one-time setup):

cd .claude/skills/web-content-fetcher
pnpm install @mozilla/readability jsdom

Usage:

node .claude/skills/web-content-fetcher/scripts/extract_article.js path/to/raw_html.html > output.md

Best For:

Standard UTF-8 encoded HTML
Complex page structures Task agent struggles with
Consistent extraction using battle-tested algorithm

Complete Workflow Example:

# 1. Setup (one-time)
cd .claude/skills/web-content-fetcher
pnpm install @mozilla/readability jsdom

# 2. Create task directory
mkdir -p output/tasks/20260111_article/original

# 3. Fetch HTML
curl -s "https://example.com/article" > output/tasks/20260111_article/original/raw_html.html

# 4. Extract with script
node .claude/skills/web-content-fetcher/scripts/extract_article.js \
  output/tasks/20260111_article/original/raw_html.html \
  > output/tasks/20260111_article/original/fetched_content.md

extract_eucjp.js - Japanese EUC-JP Encoding

Purpose: Extract content from EUC-JP encoded Japanese websites (4gamer.net, older Japanese sites)

Requirements (one-time setup):

cd .claude/skills/web-content-fetcher
pnpm install @mozilla/readability jsdom iconv-lite

Usage:

node .claude/skills/web-content-fetcher/scripts/extract_eucjp.js path/to/raw_html.html > output.md

Best For:

Japanese websites with EUC-JP encoding (4gamer.net, government sites)
When you see garbled Japanese characters (mojibake/文字化け)
Sites that haven't migrated to UTF-8

How to Identify EUC-JP Pages:

1. Extracted content shows garbled Japanese characters 2. HTML meta tag shows: <meta charset="EUC-JP"> or charset=euc-jp 3. Common on older Japanese sites (4gamer.net, academic sites, government sites)

Complete Workflow Example:

# 1. Setup (one-time)
cd .claude/skills/web-content-fetcher
pnpm install @mozilla/readability jsdom iconv-lite

# 2. Create task directory
mkdir -p output/tasks/20260111_4gamer_article/original

# 3. Fetch HTML
curl -s "https://www.4gamer.net/games/999/G999999/..." > output/tasks/20260111_4gamer_article/original/raw_html.html

# 4. Extract with EUC-JP script
node .claude/skills/web-content-fetcher/scripts/extract_eucjp.js \
  output/tasks/20260111_4gamer_article/original/raw_html.html \
  > output/tasks/20260111_4gamer_article/original/fetched_content.md

Important: Always try Task agent (Tier 2) first with encoding detection instructions. Only use this script if Task agent cannot properly handle the encoding.

Tier 4: JavaScript-Rendered Content

Use when: Website requires JavaScript to load content (Twitter/X, React SPAs, dynamic websites), static fetch returns error messages or empty content.

Playwright Setup

One-time installation:

cd .claude/skills/web-content-fetcher
pnpm add -D playwright --save-catalog-name=dev
pnpm exec playwright install chromium

Requirements:

~200MB disk space for Chromium browser
Internet connection for initial download

fetch_js_content.js Usage

Script: scripts/fetch_js_content.js

Basic Usage:

node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js <url> [options]

Options:

--output <file>: Output file path (default: stdout)
--selector <sel>: CSS selector to wait for (auto-detected for Twitter/X)
--timeout <ms>: Page load timeout (default: 30000)

Best For:

Twitter/X posts (automatic tweet extraction)
React/Vue/Angular single-page applications
Content loaded dynamically via AJAX
Sites that show "JavaScript required" error

Supported Sites and Examples

Twitter/X Posts

Automatic tweet extraction with metadata (author, timestamp, quoted tweets, media descriptions).

# Twitter/X example
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://x.com/user/status/123456789" \
  --output output/tasks/20260111_twitter/original/fetched_content.md

Complete Workflow:

# 1. Create task directory
mkdir -p output/tasks/20260111_twitter_post/original

# 2. Fetch with Playwright (no raw HTML needed)
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://x.com/user/status/123456789" \
  --output output/tasks/20260111_twitter_post/original/fetched_content.md

# 3. Use extracted content

React/Vue/Angular SPAs

# Generic React SPA
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://example-react-app.com/article" \
  --selector ".main-content" \
  --timeout 60000 \
  --output fetched_content.md

Dynamic Content Sites

# Site with AJAX-loaded content
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://dynamic-site.com/page" \
  --selector "#content-loaded" \
  --output fetched_content.md

Site-Specific Features

Twitter/X:

Automatically extracts tweet text, author, timestamp
Includes quoted tweets if present
Extracts media descriptions (alt text)
No need to specify selector

Generic Sites:

Extracts main content area (article, main, [role="main"])
Waits for page load and JavaScript execution
Can specify custom selector for precise extraction

How It Works

1. Launches headless Chromium browser 2. Navigates to URL and waits for JavaScript to execute 3. Auto-detects site type (Twitter/X has custom extraction logic) 4. Waits for content to load (specified selector or default) 5. Extracts clean content to markdown 6. Closes browser and returns result

Performance Notes

Comparison with Static Fetch:

Playwright: 3-10 seconds (browser launch + JS execution)
curl: <1 second (direct HTTP request)

Resource Usage:

~200MB disk space (Chromium browser)
More memory (headless browser process)
CPU usage during JavaScript execution

When to Use:

Only when JavaScript is truly required
Content doesn't appear with static fetch (curl)
Site shows "JavaScript required" or similar error

Pros:

Handles JavaScript rendering
Auto-detects site types
Extracts structured content

Cons:

Slower than static fetching
Requires Chromium browser installation
Higher resource usage

Advanced Usage

Custom Selector for Specific Content:

# Wait for and extract specific element
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://example.com/page" \
  --selector ".article-body" \
  --output content.md

Longer Timeout for Slow Sites:

# Increase timeout to 2 minutes
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://slow-loading-site.com/page" \
  --timeout 120000 \
  --output content.md

Output to Stdout for Piping:

# Output to stdout (default)
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://x.com/user/status/123" | grep "keyword"

Limitations

Authentication: This script doesn't handle login flows. For authenticated content:

Use browser extensions to export cookies
Use official APIs if available
Manual login + cookie export

Rate Limiting: Respect site rate limits and robots.txt. Consider:

Adding delays between requests
Using official APIs when available
Checking terms of service

Dynamic Content: Some sites load content asynchronously after initial render. If content is missing:

Increase --timeout value
Specify exact --selector to wait for
Check if content requires user interaction (scrolling, clicking)

Troubleshooting Guide

This reference provides solutions to common issues when fetching web content and quick reference commands for various scenarios.

Common Issues
Quick Reference Commands

Common Issues

Issue: WebFetch returns "Prompt too long"

Cause: Content exceeds WebFetch's ~50KB limit

Solution: Switch to Tier 2 (curl + Task agent)

# Instead of WebFetch, use:
mkdir -p output/tasks/YYYYMMDD_task/original
curl -s "URL" > output/tasks/YYYYMMDD_task/original/raw_html.html
# Then extract with Task agent

Issue: Read tool says file too large (> 256KB)

Cause: Fetched HTML file exceeds 256KB limit for single Read operation

Solution: Use Task agent which handles pagination automatically

Task agent prompt: "Read the HTML file at [path] and extract the main article content.
Save clean markdown to fetched_content.md"

Alternative: Use Grep to search specific content without reading entire file

Grep pattern="article title" path="raw_html.html"

Issue: Extracted content is messy or incomplete

Symptoms:

Navigation menus mixed with content
Missing paragraphs or sections
Excessive formatting or HTML remnants

Solutions:

1. Improve Task agent prompt (be more specific):

"Read the HTML file at [path]. Extract ONLY the main article content including:
- Article title (h1)
- Body paragraphs
- Subheadings (h2-h6)
EXCLUDE: navigation, sidebars, ads, comments, footer, related articles.
Save clean markdown to fetched_content.md"

2. Check if page uses JavaScript rendering (may need Playwright - Tier 4)

3. Manually inspect raw_html.html to understand page structure:

# Check page structure
grep -i "<article" raw_html.html
grep -i "class=\"content" raw_html.html

4. Provide specific HTML structure details to Task agent:

"The main content is in <article class='post-content'>. Extract only from that section."

Issue: Japanese text shows garbled characters (mojibake/文字化け)

Symptoms:

Japanese characters appear as �, �, or random symbols
Text is unreadable despite being from a Japanese site

Cause: Website uses EUC-JP encoding instead of UTF-8 (common on older Japanese sites like 4gamer.net)

Solution 1 (Try first): Use Task agent with encoding instructions:

Task agent prompt: "Read the HTML file at [path]. This file uses EUC-JP encoding.
Detect the encoding, properly decode the content, then extract the article content.
Save the clean markdown to fetched_content.md"

Solution 2 (If Task agent fails): Use EUC-JP extraction script:

node .claude/skills/web-content-fetcher/scripts/extract_eucjp.js raw_html.html > fetched_content.md

Common sites requiring special encoding handling:

4gamer.net
Older Japanese gaming/news sites
Japanese government and academic sites

How to identify EUC-JP encoding:

1. Check HTML meta tag in raw file:

grep -i "charset" raw_html.html
# Look for: <meta charset="EUC-JP"> or charset=euc-jp

2. If extracted content is garbled, encoding is likely the issue

Issue: Page has anti-bot protection

Symptoms:

HTTP 403 Forbidden error
Cloudflare challenge page
"Access denied" message

Solutions:

1. Add user agent:

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "URL" > raw_html.html

2. Add common browser headers:

curl -H "Accept-Language: en-US,en;q=0.9" \
     -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
     -H "Accept-Encoding: gzip, deflate, br" \
     -A "Mozilla/5.0" \
     "URL" > raw_html.html

3. Use cookies if you have session:

curl -b "session=abc123" "URL" > raw_html.html

4. Use Playwright (for complex bot detection):

node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js "URL" --output content.md

Issue: Content requires authentication

Symptoms:

HTTP 401 Unauthorized
Login page returned instead of content
"Members only" message

Solutions:

Bearer Token Authentication:

curl -H "Authorization: Bearer YOUR_TOKEN" "URL" > raw_html.html

Cookie-based Authentication:

# Export cookies from browser, then:
curl -b "session=YOUR_SESSION_COOKIE; auth=YOUR_AUTH_COOKIE" "URL" > raw_html.html

Basic Authentication:

curl -u username:password "URL" > raw_html.html

Multiple Headers:

curl -H "Authorization: Bearer TOKEN" \
     -H "X-API-Key: KEY" \
     "URL" > raw_html.html

Issue: Page redirects to different domain

Symptoms:

Fetched content is redirect page or error
Content is from different domain than expected

Solution: Use -L flag to follow redirects:

curl -L "URL" > raw_html.html

Check redirect chain:

curl -I -L "URL"
# Shows all HTTP headers and redirects

Issue: Timeout or slow connection

Symptoms:

curl hangs or takes very long
"Operation timed out" error

Solution: Increase timeout and show progress:

# Increase timeout to 60 seconds
curl --max-time 60 "URL" > raw_html.html

With progress bar (remove -s flag):

curl -L --max-time 60 "URL" > raw_html.html

Check connection first:

curl -I --max-time 10 "URL"
# Returns headers quickly to test connectivity

Issue: Page requires JavaScript / Shows "JavaScript not available"

Symptoms:

curl or WebFetch returns error message like "JavaScript is not available"
Fetched HTML contains empty state objects <div id="root"></div> with no content
Page is a React/Vue/Angular SPA or social media site
Content appears in browser but not in fetched HTML

Solution: Use Playwright script (Tier 4):

# Twitter/X example
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://x.com/user/status/123456789" \
  --output fetched_content.md

# Generic SPA with longer timeout
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://react-app.com/page" \
  --selector ".main-content" \
  --timeout 60000 \
  --output fetched_content.md

Common sites requiring JavaScript:

Twitter/X (x.com)
Modern single-page applications (React/Vue/Angular)
Dynamic dashboards
Content loaded via AJAX after page load

Setup required (one-time):

cd .claude/skills/web-content-fetcher
pnpm add -D playwright --save-catalog-name=dev
pnpm exec playwright install chromium

Issue: Playwright timeout errors

Symptoms:

"Timeout 30000ms exceeded" error
Page loads but content not found

Solutions:

1. Increase timeout:

node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "URL" \
  --timeout 120000 \
  --output content.md

2. Specify exact selector to wait for:

node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "URL" \
  --selector "#main-content" \
  --output content.md

3. Check if content requires user interaction (scrolling, clicking):

Script doesn't handle interaction
Consider official API instead

Issue: Downloaded file is HTML when expecting binary

Symptoms:

Expected PDF/image but got HTML
File shows "404 Not Found" or error page

Solution: Check HTTP status before saving:

# Check status code first
STATUS=$(curl -s -o raw_html.html -w "%{http_code}" "URL")
if [ "$STATUS" -eq 200 ]; then
  echo "Success"
else
  echo "Failed with status: $STATUS"
  cat raw_html.html  # Show error page
fi

Or use fail flag:

# Exit on HTTP error
curl -f "URL" > raw_html.html || echo "Download failed"

Quick Reference Commands

Basic Fetch Operations

Simple fetch:

curl -s "URL" > raw_html.html

Fetch with redirects:

curl -s -L "URL" > raw_html.html

Fetch with user agent:

curl -s -A "Mozilla/5.0" "URL" > raw_html.html

Fetch with timeout:

curl -s --max-time 30 "URL" > raw_html.html

Error Handling

Fetch with error handling:

curl -f -s -L --max-time 30 -A "Mozilla/5.0" "URL" > raw_html.html || echo "Fetch failed"

Check HTTP status:

curl -w "%{http_code}" -o raw_html.html "URL"

Get headers only:

curl -I "URL"

Silent fetch with status:

HTTP_CODE=$(curl -s -w "%{http_code}" -o raw_html.html "URL")
echo "Status: $HTTP_CODE"

Authentication

Bearer token:

curl -H "Authorization: Bearer TOKEN" "URL" > raw_html.html

Cookie authentication:

curl -b "session=COOKIE" "URL" > raw_html.html

Basic authentication:

curl -u username:password "URL" > raw_html.html

JavaScript-Rendered Content

Fetch Twitter/X post:

node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://x.com/user/status/123456789" \
  --output twitter_post.md

Fetch React SPA:

node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://react-app.com/page" \
  --selector ".main-content" \
  --timeout 60000 \
  --output content.md

Content Inspection

Check encoding:

grep -i "charset" raw_html.html | head -5

Check for JavaScript requirement:

grep -i "javascript" raw_html.html | head -5

Check main content areas:

grep -i "<article" raw_html.html
grep -i "class=\"content" raw_html.html
grep -i "<main" raw_html.html

Count HTML size:

wc -c raw_html.html  # Size in bytes

Debugging

Fetch and show headers:

curl -v "URL" > raw_html.html 2>&1 | grep "< HTTP"

Test connectivity:

curl -I --max-time 10 "URL"

Show redirect chain:

curl -I -L "URL"

Check final URL after redirects:

curl -Ls -o /dev/null -w "%{url_effective}" "URL"

Standard Workflows and Patterns

This reference provides detailed step-by-step workflows, common patterns, and best practices for organizing fetched web content.

Standard Workflow (Tier 2 - Recommended)
Directory Structure Best Practice
Content Extraction Priorities
Common Patterns

Standard Workflow (Tier 2 - Recommended)

This workflow using curl + Task agent works for nearly all cases.

Step 1: Create Task Directory

mkdir -p output/tasks/YYYYMMDD_descriptive_name/original

Convention: YYYYMMDD_ prefix based on current date

Example:

mkdir -p output/tasks/20260111_article_translation/original

Step 2: Fetch Raw Content

curl -s "URL" > output/tasks/YYYYMMDD_taskname/original/raw_html.html

Common Options:

-s: Silent mode (no progress bar)
-L: Follow redirects
-A "Mozilla/5.0": Set user agent if needed
--max-time 30: Set timeout

Example:

curl -s -L "https://example.com/article" > output/tasks/20260111_article_translation/original/raw_html.html

Step 3: Extract Clean Content

Use Task agent (general-purpose subagent type):

Prompt: "Read the HTML file at output/tasks/YYYYMMDD_taskname/original/raw_html.html
and extract the main article content (title, body, headings).
Remove navigation, ads, sidebars, comments, footer.
Save clean markdown to: output/tasks/YYYYMMDD_taskname/original/fetched_content.md"

Why Task agent?:

Automatically handles large files (>256KB) with pagination
AI-powered smart extraction
Cleans HTML tags and structures content
No script creation needed

Step 4: Use Extracted Content

Read: output/tasks/YYYYMMDD_taskname/original/fetched_content.md

Now proceed with translation, analysis, or other tasks.

Directory Structure Best Practice

Always organize fetched content in task directories:

output/tasks/YYYYMMDD_descriptive_name/
├── original/                    # Raw and cleaned source content
│   ├── raw_html.html           # Original HTML (keep for reference)
│   └── fetched_content.md      # Cleaned article content
├── tmp/                        # Intermediate processing files
│   └── ...
└── [output_type]/              # Final deliverables
    ├── translated/             # For translation tasks
    ├── analyzed/               # For analysis tasks
    └── ...

Key principles:

Date prefix: Use YYYYMMDD_ for chronological sorting
Descriptive names: Clearly indicate task purpose
Preserve raw: Keep raw_html.html for debugging and re-processing (not needed for Tier 4 Playwright)
Clean output: Always save extracted content to fetched_content.md
Organize by type: Use subdirectories for final deliverables

Content Extraction Priorities

When extracting content (via Task agent or scripts), focus on:

Include

1. Article title (h1) 2. Main body content (paragraphs) 3. Subheadings (h2-h6) 4. Publication date/metadata (if relevant to the task) 5. Inline links (if important for context)

Exclude

1. Navigation menus 2. Sidebars 3. Advertisement blocks 4. Comment sections 5. Footer content 6. Scripts and styles 7. Related article suggestions 8. Cookie banners 9. Social media widgets

Format

1. Output as clean markdown 2. Preserve heading hierarchy (h1 → #, h2 → ##, etc.) 3. Convert HTML links to markdown format: [text](url) 4. Decode HTML entities (& → &, < → <) 5. Use proper line breaks between paragraphs (blank line) 6. Preserve code blocks if present 7. Keep list formatting (ordered and unordered)

Common Patterns

Pattern 1: Fetch for Translation

Use case: Fetching web content to translate to another language

# 1. Create task directory
mkdir -p output/tasks/20260111_translate_article/original

# 2. Fetch raw HTML
curl -s "https://example.com/article" > output/tasks/20260111_translate_article/original/raw_html.html

# 3. Extract clean content (via Task agent)

Task agent prompt:

Read the HTML file at output/tasks/20260111_translate_article/original/raw_html.html
and extract the main article content. Save clean markdown to:
output/tasks/20260111_translate_article/original/fetched_content.md

# 4. Pass to translation workflow
# Use /translate command or translation agent with fetched_content.md

Pattern 2: Fetch for Analysis

Use case: Fetching web content to analyze or summarize

# 1. Create task directory
mkdir -p output/tasks/20260111_analyze_article/original

# 2. Fetch raw HTML
curl -s "https://example.com/research-paper" > output/tasks/20260111_analyze_article/original/raw_html.html

# 3. Extract clean content (via Task agent)

Task agent prompt:

Read the HTML file at output/tasks/20260111_analyze_article/original/raw_html.html
and extract the main article content. Save clean markdown to:
output/tasks/20260111_analyze_article/original/fetched_content.md

# 4. Read and analyze

Then read fetched_content.md and perform analysis.

Pattern 3: Fetch Series of Articles

Use case: Fetching multiple related articles or pages

# 1. Create task directory
mkdir -p output/tasks/20260111_series_analysis/original

# 2. Fetch multiple pages
for i in {1..5}; do
  curl -s "https://example.com/article/part-$i" > output/tasks/20260111_series_analysis/original/page$i.html
done

# 3. Extract each page (via Task agent for each file)

Task agent prompt (run for each page):

Read the HTML file at output/tasks/20260111_series_analysis/original/page[N].html
and extract the main article content. Save clean markdown to:
output/tasks/20260111_series_analysis/original/page[N].md

# 4. Combine all extracted content
cat output/tasks/20260111_series_analysis/original/page*.md > output/tasks/20260111_series_analysis/combined.md

Pattern 4: Fetch with Authentication

Use case: Fetching content behind authentication/login

Bearer Token:

curl -H "Authorization: Bearer YOUR_TOKEN" "https://api.example.com/article" > raw_html.html

Cookie-based Authentication:

curl -b "session=YOUR_SESSION_COOKIE" "https://example.com/members/article" > raw_html.html

Basic Authentication:

curl -u username:password "https://example.com/protected/article" > raw_html.html

Then extract with Task agent as usual.

Pattern 5: Fetch JavaScript-Rendered Content

Use case: Twitter/X, React SPAs, dynamic sites requiring JavaScript

# 1. Create task directory
mkdir -p output/tasks/20260111_twitter_post/original

# 2. Fetch with Playwright script (no need to save raw HTML)
node .claude/skills/web-content-fetcher/scripts/fetch_js_content.js \
  "https://x.com/user/status/123456789" \
  --output output/tasks/20260111_twitter_post/original/fetched_content.md

# 3. Use fetched content directly

Then read fetched_content.md for translation/analysis.

/**
 * Tests for extract_article.js - Integration tests using real JSDOM
 */
import { vi } from 'vitest';
import fs from 'fs';
import path from 'path';
import { extractArticle, htmlToMarkdown, main } from '../extract_article.js';

describe('extract_article', () => {
  const testDir = 'test_html_files';

  beforeAll(() => {
    if (!fs.existsSync(testDir)) {
      fs.mkdirSync(testDir, { recursive: true });
    }
  });

  afterAll(() => {
    if (fs.existsSync(testDir)) {
      fs.rmSync(testDir, { recursive: true, force: true });
    }
  });

  describe('extractArticle', () => {
    test('should throw error for missing file', () => {
      const nonExistentFile = path.join(testDir, 'nonexistent.html');
      expect(() => extractArticle(nonExistentFile)).toThrow(
        `File not found: ${nonExistentFile}`
      );
    });

    test('should throw error for HTML without readable content', () => {
      const htmlFile = path.join(testDir, 'empty.html');
      fs.writeFileSync(htmlFile, '<html><body></body></html>', 'utf-8');
      expect(() => extractArticle(htmlFile)).toThrow(
        'Failed to extract article content'
      );
    });

    test('should extract article from valid HTML file', () => {
      const htmlFile = path.join(testDir, 'article.html');
      const htmlContent = `
        <!DOCTYPE html>
        <html>
          <head><title>Test Article</title></head>
          <body>
            <article>
              <h1>Test Article</h1>
              <p>This is the first paragraph with enough content.</p>
              <p>This is the second paragraph with more content.</p>
              <p>This is the third paragraph to ensure readability works.</p>
            </article>
          </body>
        </html>
      `;
      fs.writeFileSync(htmlFile, htmlContent, 'utf-8');
      const result = extractArticle(htmlFile);
      expect(result).toContain('Test Article');
      expect(result).toContain('first paragraph');
    });
  });

  describe('htmlToMarkdown', () => {
    test('should convert h1 to markdown', () => {
      expect(htmlToMarkdown('<h1>Heading 1</h1>')).toContain('# Heading 1');
    });

    test('should convert h2 to markdown', () => {
      expect(htmlToMarkdown('<h2>Heading 2</h2>')).toContain('## Heading 2');
    });

    test('should convert h3 to markdown', () => {
      expect(htmlToMarkdown('<h3>Heading 3</h3>')).toContain('### Heading 3');
    });

    test('should convert h4 to markdown', () => {
      expect(htmlToMarkdown('<h4>Heading 4</h4>')).toContain('#### Heading 4');
    });

    test('should convert h5 to markdown', () => {
      expect(htmlToMarkdown('<h5>Heading 5</h5>')).toContain('##### Heading 5');
    });

    test('should convert h6 to markdown', () => {
      expect(htmlToMarkdown('<h6>Heading 6</h6>')).toContain(
        '###### Heading 6'
      );
    });

    test('should convert paragraphs to markdown', () => {
      expect(htmlToMarkdown('<p>Paragraph text</p>')).toContain(
        'Paragraph text'
      );
    });

    test('should convert br tags to newlines', () => {
      const result = htmlToMarkdown('<div>Line 1<br/>Line 2<br>Line 3</div>');
      expect(result).toContain('Line 1');
      expect(result).toContain('Line 2');
      expect(result).toContain('Line 3');
    });

    test('should convert links with href to markdown', () => {
      expect(
        htmlToMarkdown('<a href="https://example.com">Link Text</a>')
      ).toContain('[Link Text](https://example.com)');
    });

    test('should handle links without href', () => {
      expect(htmlToMarkdown('<a>Text only</a>')).toContain('Text only');
    });

    test('should handle links with empty text', () => {
      expect(htmlToMarkdown('<a href="https://example.com"></a>')).toBe('');
    });

    test('should handle links with whitespace-only text', () => {
      expect(htmlToMarkdown('<a href="https://example.com">  </a>')).toBe('');
    });

    test('should convert strong to markdown', () => {
      expect(htmlToMarkdown('<strong>Bold</strong>')).toContain('**Bold**');
    });

    test('should convert b to markdown', () => {
      expect(htmlToMarkdown('<b>Bold</b>')).toContain('**Bold**');
    });

    test('should convert em to markdown', () => {
      expect(htmlToMarkdown('<em>Italic</em>')).toContain('*Italic*');
    });

    test('should convert i to markdown', () => {
      expect(htmlToMarkdown('<i>Italic</i>')).toContain('*Italic*');
    });

    test('should convert code to markdown', () => {
      expect(htmlToMarkdown('<code>const x = 1;</code>')).toContain(
        '`const x = 1;`'
      );
    });

    test('should convert blockquote to markdown', () => {
      expect(htmlToMarkdown('<blockquote>Quoted text</blockquote>')).toContain(
        '> Quoted text'
      );
    });

    test('should convert ul lists to markdown', () => {
      const result = htmlToMarkdown('<ul><li>Item 1</li><li>Item 2</li></ul>');
      expect(result).toContain('- Item 1');
      expect(result).toContain('- Item 2');
    });

    test('should convert ol lists to markdown', () => {
      const result = htmlToMarkdown('<ol><li>First</li><li>Second</li></ol>');
      expect(result).toContain('- First');
      expect(result).toContain('- Second');
    });

    test('should handle ul with non-li children', () => {
      const result = htmlToMarkdown(
        '<ul><li>Item</li><div>Not an li</div></ul>'
      );
      expect(result).toContain('- Item');
    });

    test('should handle nested elements via default case', () => {
      expect(htmlToMarkdown('<div><span>Nested text</span></div>')).toContain(
        'Nested text'
      );
    });

    test('should handle empty text nodes', () => {
      expect(htmlToMarkdown('<p>  </p>')).toBe('');
    });

    test('should handle non-element non-text nodes', () => {
      // Test with processing instruction, CDATA, and comments
      expect(
        htmlToMarkdown('<div>Text<?xml?><![CDATA[data]]><!-- comment --></div>')
      ).toContain('Text');
    });

    test('should clean up multiple spaces', () => {
      expect(
        htmlToMarkdown('<p>Text   with    many    spaces</p>')
      ).not.toMatch(/  +/);
    });

    test('should clean up multiple newlines', () => {
      expect(
        htmlToMarkdown('<p>Line 1</p><p>Line 2</p><p>Line 3</p>')
      ).not.toMatch(/\n{4,}/);
    });

    test('should trim result', () => {
      const result = htmlToMarkdown('<p>Content</p>');
      expect(result).toBe(result.trim());
    });
  });

  describe('main', () => {
    let originalArgv;
    let originalExit;

    beforeEach(() => {
      originalArgv = process.argv;
      originalExit = process.exit;
    });

    afterEach(() => {
      process.argv = originalArgv;
      process.exit = originalExit;
    });

    test('should exit with error when no arguments provided', () => {
      process.argv = ['node', 'extract_article.js'];
      let exitCode;
      process.exit = vi.fn((code) => {
        exitCode = code;
      });
      const consoleErrorSpy = vi
        .spyOn(console, 'error')
        .mockImplementation(() => {});

      main();

      expect(exitCode).toBe(1);
      expect(consoleErrorSpy).toHaveBeenCalledWith(
        expect.stringContaining('Usage')
      );
      consoleErrorSpy.mockRestore();
    });

    test('should handle file not found error', () => {
      process.argv = ['node', 'extract_article.js', 'nonexistent.html'];
      let exitCode;
      process.exit = vi.fn((code) => {
        exitCode = code;
      });
      const consoleErrorSpy = vi
        .spyOn(console, 'error')
        .mockImplementation(() => {});

      main();

      expect(exitCode).toBe(1);
      expect(consoleErrorSpy).toHaveBeenCalledWith(
        expect.stringContaining('Error extracting')
      );
      consoleErrorSpy.mockRestore();
    });

    test('should output article when successful', () => {
      const htmlFile = path.join(testDir, 'cli_test.html');
      const htmlContent = `
        <!DOCTYPE html>
        <html>
          <head><title>CLI Test</title></head>
          <body>
            <article>
              <h1>CLI Test</h1>
              <p>This is enough content for extraction to work.</p>
              <p>Multiple paragraphs help Readability parse correctly.</p>
              <p>Third paragraph for good measure.</p>
            </article>
          </body>
        </html>
      `;
      fs.writeFileSync(htmlFile, htmlContent, 'utf-8');
      process.argv = ['node', 'extract_article.js', htmlFile];
      const consoleLogSpy = vi
        .spyOn(console, 'log')
        .mockImplementation(() => {});

      main();

      expect(consoleLogSpy).toHaveBeenCalled();
      const output = consoleLogSpy.mock.calls[0][0];
      expect(output).toContain('CLI Test');
      consoleLogSpy.mockRestore();
    });
  });
});

/**
 * Tests for extract_eucjp.js - Integration tests using real JSDOM
 */
import { vi } from 'vitest';
import fs from 'fs';
import path from 'path';
import iconv from 'iconv-lite';
import { JSDOM } from 'jsdom';
import {
  extractArticle,
  htmlToMarkdown,
  main,
  extractWithFallback,
} from '../extract_eucjp.js';

describe('extract_eucjp', () => {
  const testDir = 'test_eucjp_files';

  beforeAll(() => {
    if (!fs.existsSync(testDir)) {
      fs.mkdirSync(testDir, { recursive: true });
    }
  });

  afterAll(() => {
    if (fs.existsSync(testDir)) {
      fs.rmSync(testDir, { recursive: true, force: true });
    }
  });

  describe('extractWithFallback', () => {
    test('should extract from #main selector', () => {
      const html =
        '<html><head><title>Main Test</title></head><body><div id="main"><p>Main content</p></div></body></html>';
      const dom = new JSDOM(html);

      const result = extractWithFallback(dom);

      expect(result).toContain('# Main Test');
      expect(result).toContain('Main content');
    });

    test('should extract from article selector', () => {
      const html =
        '<html><head><title>Article Test</title></head><body><article><p>Article content</p></article></body></html>';
      const dom = new JSDOM(html);

      const result = extractWithFallback(dom);

      expect(result).toContain('# Article Test');
      expect(result).toContain('Article content');
    });

    test('should extract from articleBody class selector', () => {
      const html =
        '<html><head><title>ArticleBody Test</title></head><body><div class="articleBody"><p>Body content</p></div></body></html>';
      const dom = new JSDOM(html);

      const result = extractWithFallback(dom);

      expect(result).toContain('# ArticleBody Test');
      expect(result).toContain('Body content');
    });

    test('should throw error when no fallback selectors found', () => {
      const html =
        '<html><head><title>No Content</title></head><body><div>No selectors</div></body></html>';
      const dom = new JSDOM(html);

      expect(() => extractWithFallback(dom)).toThrow(
        'Failed to extract article content'
      );
    });
  });

  describe('extractArticle', () => {
    test('should throw error for missing file', () => {
      const nonExistentFile = path.join(testDir, 'nonexistent.html');

      expect(() => extractArticle(nonExistentFile)).toThrow(
        `File not found: ${nonExistentFile}`
      );
    });

    test('should throw error when no fallback selectors found', () => {
      const htmlFile = path.join(testDir, 'no_fallback.html');
      const htmlContent =
        '<html><head><title>No Content</title></head><body></body></html>';

      const buffer = iconv.encode(htmlContent, 'eucjp');
      fs.writeFileSync(htmlFile, buffer);

      expect(() => extractArticle(htmlFile)).toThrow(
        'Failed to extract article content'
      );
    });

    test('should extract article from EUC-JP encoded HTML file', () => {
      const htmlFile = path.join(testDir, 'article_eucjp.html');
      const htmlContent = `
        <!DOCTYPE html>
        <html>
          <head><title>テスト記事</title></head>
          <body>
            <article>
              <h1>テスト記事</h1>
              <p>これは記事の内容です。十分な長さのコンテンツが必要です。</p>
              <p>これは別の段落です。Readabilityが動作するために必要です。</p>
              <p>さらに追加の段落を含めます。</p>
            </article>
          </body>
        </html>
      `;

      const buffer = iconv.encode(htmlContent, 'eucjp');
      fs.writeFileSync(htmlFile, buffer);

      const result = extractArticle(htmlFile);

      expect(result).toContain('テスト記事');
      expect(result).toContain('記事の内容');
    });
  });

  describe('htmlToMarkdown', () => {
    test('should convert h1 to markdown', () => {
      const html = '<h1>見出し1</h1>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('# 見出し1');
    });

    test('should convert h2 to markdown', () => {
      const html = '<h2>見出し2</h2>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('## 見出し2');
    });

    test('should convert h3 to markdown', () => {
      const html = '<h3>見出し3</h3>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('### 見出し3');
    });

    test('should convert h4 to markdown', () => {
      const html = '<h4>見出し4</h4>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('#### 見出し4');
    });

    test('should convert paragraphs to markdown', () => {
      const html = '<p>段落テキスト</p>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('段落テキスト');
    });

    test('should convert br tags', () => {
      const html = '<p>行1<br>行2</p>';
      const result = htmlToMarkdown(html);
      expect(result).toBeTruthy();
    });

    test('should convert links with href', () => {
      const html = '<a href="https://example.jp">リンク</a>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('[リンク](https://example.jp)');
    });

    test('should handle links without href', () => {
      const html = '<a>テキストのみ</a>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('テキストのみ');
    });

    test('should convert strong to markdown', () => {
      const html = '<strong>太字</strong>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('**太字**');
    });

    test('should convert b to markdown', () => {
      const html = '<b>太字</b>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('**太字**');
    });

    test('should convert em to markdown', () => {
      const html = '<em>イタリック</em>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('*イタリック*');
    });

    test('should convert i to markdown', () => {
      const html = '<i>イタリック</i>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('*イタリック*');
    });

    test('should handle nested elements', () => {
      const html = '<div><span>ネストされたテキスト</span></div>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('ネストされたテキスト');
    });

    test('should handle empty text nodes', () => {
      const html = '<p>  </p>';
      const result = htmlToMarkdown(html);
      expect(result).toBe('');
    });

    test('should handle non-element non-text nodes', () => {
      const html = '<!-- comment --><p>テキスト</p>';
      const result = htmlToMarkdown(html);
      expect(result).toContain('テキスト');
    });

    test('should clean up multiple spaces', () => {
      const html = '<p>テキスト   が    あります</p>';
      const result = htmlToMarkdown(html);
      expect(result).not.toMatch(/  +/);
    });

    test('should clean up multiple newlines', () => {
      const html = '<p>行1</p><p>行2</p><p>行3</p>';
      const result = htmlToMarkdown(html);
      expect(result).not.toMatch(/\n{4,}/);
    });

    test('should trim result', () => {
      const html = '<p>コンテンツ</p>';
      const result = htmlToMarkdown(html);
      expect(result).toBe(result.trim());
    });
  });

  describe('main', () => {
    let originalArgv;
    let originalExit;

    beforeEach(() => {
      originalArgv = process.argv;
      originalExit = process.exit;
    });

    afterEach(() => {
      process.argv = originalArgv;
      process.exit = originalExit;
    });

    test('should exit with error when no arguments provided', () => {
      process.argv = ['node', 'extract_eucjp.js'];
      let exitCode;
      process.exit = vi.fn((code) => {
        exitCode = code;
      });
      const consoleErrorSpy = vi
        .spyOn(console, 'error')
        .mockImplementation(() => {});

      main();

      expect(exitCode).toBe(1);
      expect(consoleErrorSpy).toHaveBeenCalledWith(
        expect.stringContaining('Usage')
      );
      consoleErrorSpy.mockRestore();
    });

    test('should handle file not found error', () => {
      process.argv = ['node', 'extract_eucjp.js', 'nonexistent.html'];
      let exitCode;
      process.exit = vi.fn((code) => {
        exitCode = code;
      });
      const consoleErrorSpy = vi
        .spyOn(console, 'error')
        .mockImplementation(() => {});

      main();

      expect(exitCode).toBe(1);
      expect(consoleErrorSpy).toHaveBeenCalledWith(
        expect.stringContaining('Error extracting')
      );
      consoleErrorSpy.mockRestore();
    });

    test('should output article when successful', () => {
      const htmlFile = path.join(testDir, 'cli_test.html');
      const htmlContent = `
        <!DOCTYPE html>
        <html>
          <head><title>CLI Test</title></head>
          <body>
            <article>
              <h1>CLI Test</h1>
              <p>これは十分なコンテンツです。</p>
              <p>複数の段落がReadabilityの動作を助けます。</p>
              <p>3番目の段落も追加します。</p>
            </article>
          </body>
        </html>
      `;
      const buffer = iconv.encode(htmlContent, 'eucjp');
      fs.writeFileSync(htmlFile, buffer);
      process.argv = ['node', 'extract_eucjp.js', htmlFile];
      const consoleLogSpy = vi
        .spyOn(console, 'log')
        .mockImplementation(() => {});

      main();

      expect(consoleLogSpy).toHaveBeenCalled();
      const output = consoleLogSpy.mock.calls[0][0];
      expect(output).toContain('CLI Test');
      consoleLogSpy.mockRestore();
    });
  });
});

/**
 * Integration tests for fetch_js_content.js
 *
 * These tests use actual Playwright browser automation to verify the script works end-to-end.
 * They are slower than unit tests but optimized to share test data and minimize browser launches.
 */

import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { fetchContent } from '../fetch_js_content.js';
import { writeFileSync, unlinkSync, mkdirSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';

describe('fetch_js_content.js - Integration Tests', () => {
  let testHtmlPath;
  let testDir;

  beforeAll(() => {
    // Create a temporary directory for test files
    testDir = join(tmpdir(), `fetch-js-test-${Date.now()}`);
    mkdirSync(testDir, { recursive: true });

    // Create a simple test HTML file
    testHtmlPath = join(testDir, 'test.html');
    const testHtml = `
      <!DOCTYPE html>
      <html>
      <head>
        <title>Test Page</title>
      </head>
      <body>
        <h1>Test Article Title</h1>
        <article>
          <p>This is the main content of the test article.</p>
          <p>It has multiple paragraphs.</p>
        </article>
      </body>
      </html>
    `;
    writeFileSync(testHtmlPath, testHtml, 'utf-8');
  });

  afterAll(() => {
    // Cleanup test files
    try {
      unlinkSync(testHtmlPath);
    } catch (_error) {
      // Ignore cleanup errors
    }
  });

  describe('fetchContent', () => {
    it('should fetch and extract content with custom options', async () => {
      const fileUrl = `file:///${testHtmlPath.replace(/\\/g, '/')}`;

      // Test basic fetch, custom selector, and formatting in one test
      const result = await fetchContent(fileUrl, {
        selector: 'article',
        timeout: 5000,
      });

      // Verify content extraction
      expect(result).toContain('Test Article Title');
      expect(result).toContain('main content');
      expect(result).toContain('multiple paragraphs');

      // Verify markdown formatting
      expect(result).toMatch(/^# /); // Starts with h1
      expect(result).toContain('\n\n'); // Has paragraph breaks
    }, 10000);
  });

  describe('Error handling', () => {
    it('should handle various error conditions', async () => {
      // Test invalid URL
      await expect(
        fetchContent('not-a-valid-url', { timeout: 5000 })
      ).rejects.toThrow();

      // Test non-existent file
      await expect(
        fetchContent('file:///this/file/does/not/exist.html', { timeout: 5000 })
      ).rejects.toThrow();

      // Test timeout with non-existent selector
      const fileUrl = `file:///${testHtmlPath.replace(/\\/g, '/')}`;
      await expect(
        fetchContent(fileUrl, {
          selector: '.does-not-exist',
          timeout: 100,
        })
      ).rejects.toThrow();
    }, 20000);
  });

  describe('Browser lifecycle', () => {
    it('should properly close browser on error and allow subsequent calls', async () => {
      const fileUrl = `file:///${testHtmlPath.replace(/\\/g, '/')}`;

      // First call fails with timeout
      try {
        await fetchContent(fileUrl, {
          selector: '.does-not-exist',
          timeout: 100,
        });
      } catch (_error) {
        // Expected to fail
      }

      // Browser should be closed properly, next call should work
      const result = await fetchContent(fileUrl, { timeout: 5000 });
      expect(result).toContain('Test Article Title');
    }, 10000);
  });
});

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import {
  validateUrl,
  validateOutputPath,
  validateTimeout,
  getSiteConfig,
  formatTwitterMarkdown,
  formatGenericMarkdown,
  extractTwitterContent,
  extractGenericContent,
  fetchContent,
  main,
} from '../fetch_js_content.js';

describe('fetch_js_content.js', () => {
  describe('validateUrl', () => {
    it('should accept valid http URLs', () => {
      expect(() => validateUrl('http://example.com')).not.toThrow();
      expect(() => validateUrl('http://example.com/path')).not.toThrow();
      expect(() => validateUrl('http://localhost:8080')).not.toThrow();
    });

    it('should accept valid https URLs', () => {
      expect(() => validateUrl('https://example.com')).not.toThrow();
      expect(() => validateUrl('https://x.com/user/status/123')).not.toThrow();
      expect(() =>
        validateUrl('https://example.com:443/path?query=value')
      ).not.toThrow();
    });

    it('should accept valid file:// URLs', () => {
      expect(() => validateUrl('file:///path/to/file.html')).not.toThrow();
      expect(() =>
        validateUrl('file://localhost/path/to/file.html')
      ).not.toThrow();
    });

    it('should reject invalid URL formats', () => {
      expect(() => validateUrl('not-a-url')).toThrow('Invalid URL format');
      expect(() => validateUrl('just text')).toThrow('Invalid URL format');
      expect(() => validateUrl('')).toThrow('Invalid URL format');
    });

    it('should reject unsafe protocols', () => {
      expect(() => validateUrl('javascript:alert(1)')).toThrow(
        'Invalid protocol: javascript:'
      );
      expect(() =>
        validateUrl('data:text/html,<script>alert(1)</script>')
      ).toThrow('Invalid protocol: data:');
      expect(() => validateUrl('ftp://example.com')).toThrow(
        'Invalid protocol: ftp:'
      );
    });

    it('should reject file:// URLs to sensitive system paths', () => {
      expect(() => validateUrl('file:///etc/passwd')).toThrow(
        'Access to sensitive system directories'
      );
      expect(() => validateUrl('file:///sys/kernel')).toThrow(
        'Access to sensitive system directories'
      );
      expect(() => validateUrl('file:///proc/cpuinfo')).toThrow(
        'Access to sensitive system directories'
      );
      expect(() => validateUrl('file:///C:/Windows/System32/config')).toThrow(
        'Access to sensitive system directories'
      );
    });

    it('should handle case-insensitive path checks for Windows', () => {
      expect(() => validateUrl('file:///C:/WINDOWS/SYSTEM32/file')).toThrow(
        'Access to sensitive system directories'
      );
    });

    it('should allow file:// URLs to non-sensitive paths', () => {
      expect(() =>
        validateUrl('file:///home/user/documents/file.html')
      ).not.toThrow();
      expect(() => validateUrl('file:///tmp/test.html')).not.toThrow();
    });
  });

  describe('validateOutputPath', () => {
    it('should accept paths within current directory', () => {
      const result = validateOutputPath('output.md');
      expect(result).toContain('output.md');
      expect(result.startsWith(process.cwd())).toBe(true);
    });

    it('should accept paths in subdirectories', () => {
      const result = validateOutputPath('subdir/output.md');
      expect(result).toContain('subdir');
      expect(result).toContain('output.md');
      expect(result.startsWith(process.cwd())).toBe(true);
    });

    it('should resolve relative paths', () => {
      const result = validateOutputPath('./test/output.md');
      expect(result.startsWith(process.cwd())).toBe(true);
    });

    it('should reject path traversal attempts', () => {
      expect(() => validateOutputPath('../../../etc/passwd')).toThrow(
        'Output path must be within current directory'
      );
      expect(() => validateOutputPath('../../outside/file.md')).toThrow(
        'Output path must be within current directory'
      );
    });

    it('should reject absolute paths outside current directory', () => {
      expect(() => validateOutputPath('/etc/passwd')).toThrow(
        'Output path must be within current directory'
      );
      expect(() =>
        validateOutputPath('C:\\Windows\\System32\\file.txt')
      ).toThrow('Output path must be within current directory');
    });

    it('should handle complex path traversal with valid segments', () => {
      const result = validateOutputPath('sub/../output.md');
      expect(result.startsWith(process.cwd())).toBe(true);
    });

    it('should reject paths that resolve outside current directory', () => {
      const outsidePath =
        process.cwd().split(/[/\\]/).slice(0, -2).join('/') + '/outside.md';
      expect(() => validateOutputPath(outsidePath)).toThrow(
        'Output path must be within current directory'
      );
    });

    it('should accept Windows absolute paths within current directory on Windows', () => {
      // This test simulates Windows behavior by testing a path that would be valid on Windows
      const isWindows = process.platform === 'win32';
      if (isWindows) {
        const cwd = process.cwd();
        const windowsPath = cwd + '\\subdir\\file.txt';
        const result = validateOutputPath(windowsPath);
        expect(result).toContain('subdir');
        expect(result).toContain('file.txt');
      } else {
        // On non-Windows, this should be rejected as suspicious
        expect(() => validateOutputPath('C:\\Users\\test\\file.txt')).toThrow(
          'Output path must be within current directory'
        );
      }
    });

    it('should accept Unix absolute paths within current directory', () => {
      const cwd = process.cwd();
      const isWindows = process.platform === 'win32';

      if (!isWindows) {
        // On Unix, test an absolute path within cwd
        const unixPath = cwd + '/subdir/file.txt';
        const result = validateOutputPath(unixPath);
        expect(result).toBe(unixPath);
      } else {
        // On Windows, Unix-style absolute paths like /etc/passwd should be rejected
        expect(() => validateOutputPath('/etc/passwd')).toThrow(
          'Output path must be within current directory'
        );
      }
    });

    it('should accept absolute paths within current directory', () => {
      // This test covers the return path for absolute paths within cwd
      const cwd = process.cwd();
      const isWindows = process.platform === 'win32';

      // Create an absolute path within current directory
      if (isWindows) {
        // Use Windows-style path
        const absolutePath = cwd + '\\nested\\dir\\file.txt';
        const result = validateOutputPath(absolutePath);
        expect(result.toLowerCase()).toContain('nested');
        expect(result.toLowerCase()).toContain('file.txt');
      } else {
        // Use Unix-style path
        const absolutePath = cwd + '/nested/dir/file.txt';
        const result = validateOutputPath(absolutePath);
        expect(result).toBe(absolutePath);
      }
    });
  });

  describe('validateTimeout', () => {
    it('should accept valid timeout values', () => {
      expect(validateTimeout(1000)).toBe(1000);
      expect(validateTimeout(30000)).toBe(30000);
      expect(validateTimeout(300000)).toBe(300000);
      expect(validateTimeout(150000)).toBe(150000);
    });

    it('should accept minimum timeout', () => {
      expect(validateTimeout(1000)).toBe(1000);
    });

    it('should accept maximum timeout', () => {
      expect(validateTimeout(300000)).toBe(300000);
    });

    it('should reject timeout below minimum', () => {
      expect(() => validateTimeout(500)).toThrow(
        'Timeout must be between 1000 and 300000 milliseconds'
      );
      expect(() => validateTimeout(999)).toThrow(
        'Timeout must be between 1000 and 300000 milliseconds'
      );
      expect(() => validateTimeout(0)).toThrow(
        'Timeout must be between 1000 and 300000 milliseconds'
      );
      expect(() => validateTimeout(-1000)).toThrow(
        'Timeout must be between 1000 and 300000 milliseconds'
      );
    });

    it('should reject timeout above maximum', () => {
      expect(() => validateTimeout(300001)).toThrow(
        'Timeout must be between 1000 and 300000 milliseconds'
      );
      expect(() => validateTimeout(600000)).toThrow(
        'Timeout must be between 1000 and 300000 milliseconds'
      );
      expect(() => validateTimeout(999999999)).toThrow(
        'Timeout must be between 1000 and 300000 milliseconds'
      );
    });

    it('should reject non-numeric timeouts', () => {
      expect(() => validateTimeout(NaN)).toThrow(
        'Timeout must be between 1000 and 300000 milliseconds'
      );
      expect(() => validateTimeout(Infinity)).toThrow(
        'Timeout must be between 1000 and 300000 milliseconds'
      );
      expect(() => validateTimeout(-Infinity)).toThrow(
        'Timeout must be between 1000 and 300000 milliseconds'
      );
    });

    it('should handle string inputs that convert to valid numbers', () => {
      // Note: This tests the behavior when parseInt is used before calling validateTimeout
      expect(validateTimeout(parseInt('30000', 10))).toBe(30000);
    });

    it('should reject string inputs that convert to NaN', () => {
      expect(() => validateTimeout(parseInt('invalid', 10))).toThrow(
        'Timeout must be between 1000 and 300000 milliseconds'
      );
    });
  });

  describe('getSiteConfig', () => {
    it('should detect Twitter/X URLs with x.com', () => {
      const config = getSiteConfig('https://x.com/user/status/123');

      expect(config.name).toBe('Twitter/X');
      expect(config.waitSelector).toBe('article[data-testid="tweet"]');
      expect(config.extractFunction).toBe(extractTwitterContent);
    });

    it('should detect Twitter/X URLs with twitter.com', () => {
      const config = getSiteConfig('https://twitter.com/user/status/456');

      expect(config.name).toBe('Twitter/X');
      expect(config.waitSelector).toBe('article[data-testid="tweet"]');
    });

    it('should use custom selector for Twitter when provided', () => {
      const config = getSiteConfig(
        'https://x.com/user/status/123',
        '.custom-selector'
      );

      expect(config.name).toBe('Twitter/X');
      expect(config.waitSelector).toBe('.custom-selector');
    });

    it('should return generic config for non-Twitter URLs', () => {
      const config = getSiteConfig('https://example.com');

      expect(config.name).toBe('Generic');
      expect(config.waitSelector).toBe('body');
      expect(config.extractFunction).toBe(extractGenericContent);
    });

    it('should use custom selector for generic sites when provided', () => {
      const config = getSiteConfig('https://example.com', '.main-content');

      expect(config.name).toBe('Generic');
      expect(config.waitSelector).toBe('.main-content');
    });
  });

  describe('formatTwitterMarkdown', () => {
    it('should format basic tweet data', () => {
      const tweetData = {
        userName: 'Test User\n@testuser',
        timestamp: '2026-01-11T12:00:00.000Z',
        tweetText: 'This is a test tweet',
        quotedTweet: null,
        media: [],
      };

      const markdown = formatTwitterMarkdown(tweetData);

      expect(markdown).toContain('# Twitter/X Post');
      expect(markdown).toContain('**Author:** Test User');
      expect(markdown).toContain('**Timestamp:** 2026-01-11T12:00:00.000Z');
      expect(markdown).toContain('## Tweet Content');
      expect(markdown).toContain('This is a test tweet');
    });

    it('should format tweet with quoted tweet', () => {
      const tweetData = {
        userName: 'User',
        timestamp: '2026-01-11T12:00:00.000Z',
        tweetText: 'Main tweet',
        quotedTweet: 'Quoted tweet content',
        media: [],
      };

      const markdown = formatTwitterMarkdown(tweetData);

      expect(markdown).toContain('### Quoted Tweet');
      expect(markdown).toContain('> Quoted tweet content');
    });

    it('should format tweet with media descriptions', () => {
      const tweetData = {
        userName: 'User',
        timestamp: '2026-01-11T12:00:00.000Z',
        tweetText: 'Tweet with media',
        quotedTweet: null,
        media: [
          { url: 'https://example.com/img1.jpg', alt: 'Image 1 description' },
          { url: 'https://example.com/img2.jpg', alt: 'Image 2 description' },
        ],
      };

      const markdown = formatTwitterMarkdown(tweetData);

      expect(markdown).toContain('### Media');
      expect(markdown).toContain('1. Image 1 description');
      expect(markdown).toContain('URL: https://example.com/img1.jpg');
      expect(markdown).toContain('2. Image 2 description');
      expect(markdown).toContain('URL: https://example.com/img2.jpg');
    });

    it('should handle tweet with both quoted tweet and media', () => {
      const tweetData = {
        userName: 'User',
        timestamp: '2026-01-11T12:00:00.000Z',
        tweetText: 'Complex tweet',
        quotedTweet: 'Quoted',
        media: [{ url: 'https://example.com/img.jpg', alt: 'Media' }],
      };

      const markdown = formatTwitterMarkdown(tweetData);

      expect(markdown).toContain('### Quoted Tweet');
      expect(markdown).toContain('### Media');
    });

    it('should handle empty media array', () => {
      const tweetData = {
        userName: 'User',
        timestamp: '2026-01-11T12:00:00.000Z',
        tweetText: 'Tweet',
        quotedTweet: null,
        media: [],
      };

      const markdown = formatTwitterMarkdown(tweetData);

      expect(markdown).not.toContain('### Media');
    });
  });

  describe('formatGenericMarkdown', () => {
    it('should format basic content', () => {
      const content = {
        title: 'Test Article',
        content: 'This is the article content.\nMultiple lines here.',
      };

      const markdown = formatGenericMarkdown(content);

      expect(markdown).toContain('# Test Article');
      expect(markdown).toContain('This is the article content.');
      expect(markdown).toContain('Multiple lines here.');
    });

    it('should handle content with special characters', () => {
      const content = {
        title: 'Article with "quotes" & symbols',
        content: 'Content with <html> & special chars',
      };

      const markdown = formatGenericMarkdown(content);

      expect(markdown).toContain('# Article with "quotes" & symbols');
      expect(markdown).toContain('Content with <html> & special chars');
    });

    it('should handle empty content', () => {
      const content = {
        title: 'Empty',
        content: '',
      };

      const markdown = formatGenericMarkdown(content);

      expect(markdown).toContain('# Empty');
      expect(markdown).toMatch(/# Empty\n\n\n$/);
    });

    it('should format content with media', () => {
      const content = {
        title: 'Article with Images',
        content: 'This article has images.',
        media: [
          { url: 'https://example.com/img1.jpg', alt: 'First image' },
          { url: 'https://example.com/img2.jpg', alt: 'Second image' },
        ],
      };

      const markdown = formatGenericMarkdown(content);

      expect(markdown).toContain('# Article with Images');
      expect(markdown).toContain('This article has images.');
      expect(markdown).toContain('### Media');
      expect(markdown).toContain('1. First image');
      expect(markdown).toContain('URL: https://example.com/img1.jpg');
      expect(markdown).toContain('2. Second image');
      expect(markdown).toContain('URL: https://example.com/img2.jpg');
    });

    it('should handle content with empty media array', () => {
      const content = {
        title: 'No Images',
        content: 'Content without images',
        media: [],
      };

      const markdown = formatGenericMarkdown(content);

      expect(markdown).toContain('# No Images');
      expect(markdown).toContain('Content without images');
      expect(markdown).not.toContain('### Media');
    });

    it('should handle content without media field', () => {
      const content = {
        title: 'Legacy Content',
        content: 'Old format without media field',
      };

      const markdown = formatGenericMarkdown(content);

      expect(markdown).toContain('# Legacy Content');
      expect(markdown).toContain('Old format without media field');
      expect(markdown).not.toContain('### Media');
    });
  });

  describe('extractTwitterContent', () => {
    let mockPage;

    beforeEach(() => {
      mockPage = {
        evaluate: vi.fn(),
      };
    });

    it('should extract complete tweet data', async () => {
      mockPage.evaluate.mockResolvedValue({
        userName: 'Test User\n@testuser',
        tweetText: 'Tweet content',
        timestamp: '2026-01-11T12:00:00.000Z',
        quotedTweet: null,
        media: [],
      });

      const result = await extractTwitterContent(mockPage);

      expect(mockPage.evaluate).toHaveBeenCalledOnce();
      expect(result).toContain('# Twitter/X Post');
      expect(result).toContain('Test User');
      expect(result).toContain('Tweet content');
    });

    it('should handle extraction errors', async () => {
      mockPage.evaluate.mockRejectedValue(new Error('Evaluation failed'));

      await expect(extractTwitterContent(mockPage)).rejects.toThrow(
        'Failed to extract Twitter content: Evaluation failed'
      );
    });

    it('should extract tweet with all fields', async () => {
      mockPage.evaluate.mockResolvedValue({
        userName: 'User',
        tweetText: 'Main',
        timestamp: '2026-01-11T12:00:00.000Z',
        quotedTweet: 'Quoted',
        media: [
          { url: 'https://example.com/img1.jpg', alt: 'Media 1' },
          { url: 'https://example.com/img2.jpg', alt: 'Media 2' },
        ],
      });

      const result = await extractTwitterContent(mockPage);

      expect(result).toContain('### Quoted Tweet');
      expect(result).toContain('### Media');
      expect(result).toContain('1. Media 1');
      expect(result).toContain('URL: https://example.com/img1.jpg');
      expect(result).toContain('2. Media 2');
      expect(result).toContain('URL: https://example.com/img2.jpg');
    });
  });

  describe('extractGenericContent', () => {
    let mockPage;

    beforeEach(() => {
      mockPage = {
        evaluate: vi.fn(),
      };
    });

    it('should extract generic page content', async () => {
      mockPage.evaluate.mockResolvedValue({
        title: 'Page Title',
        content: 'Page content here',
      });

      const result = await extractGenericContent(mockPage);

      expect(mockPage.evaluate).toHaveBeenCalledOnce();
      expect(result).toContain('# Page Title');
      expect(result).toContain('Page content here');
    });

    it('should handle extraction errors', async () => {
      mockPage.evaluate.mockRejectedValue(new Error('Evaluation failed'));

      await expect(extractGenericContent(mockPage)).rejects.toThrow(
        'Failed to extract generic content: Evaluation failed'
      );
    });

    it('should handle minimal content', async () => {
      mockPage.evaluate.mockResolvedValue({
        title: 'Untitled',
        content: 'No content found',
      });

      const result = await extractGenericContent(mockPage);

      expect(result).toContain('# Untitled');
      expect(result).toContain('No content found');
    });
  });

  describe('Edge cases', () => {
    describe('formatTwitterMarkdown edge cases', () => {
      it('should handle undefined quotedTweet', () => {
        const tweetData = {
          userName: 'User',
          timestamp: '2026-01-11T12:00:00.000Z',
          tweetText: 'Tweet',
          quotedTweet: undefined,
          media: [],
        };

        const markdown = formatTwitterMarkdown(tweetData);

        expect(markdown).not.toContain('### Quoted Tweet');
      });

      it('should handle null quotedTweet', () => {
        const tweetData = {
          userName: 'User',
          timestamp: '2026-01-11T12:00:00.000Z',
          tweetText: 'Tweet',
          quotedTweet: null,
          media: [],
        };

        const markdown = formatTwitterMarkdown(tweetData);

        expect(markdown).not.toContain('### Quoted Tweet');
      });

      it('should handle undefined media', () => {
        const tweetData = {
          userName: 'User',
          timestamp: '2026-01-11T12:00:00.000Z',
          tweetText: 'Tweet',
          quotedTweet: null,
          media: undefined,
        };

        const markdown = formatTwitterMarkdown(tweetData);

        expect(markdown).not.toContain('### Media');
      });

      it('should handle null media', () => {
        const tweetData = {
          userName: 'User',
          timestamp: '2026-01-11T12:00:00.000Z',
          tweetText: 'Tweet',
          quotedTweet: null,
          media: null,
        };

        const markdown = formatTwitterMarkdown(tweetData);

        expect(markdown).not.toContain('### Media');
      });
    });

    describe('getSiteConfig edge cases', () => {
      it('should handle URL with x.com in path but not domain', () => {
        const config = getSiteConfig('https://example.com/x.com/page');

        expect(config.name).toBe('Twitter/X');
      });

      it('should handle case-sensitive Twitter URLs', () => {
        const config = getSiteConfig('https://X.COM/user/status/123');

        // Note: includes() is case-sensitive, so this would be Generic
        // This test documents current behavior
        expect(config.name).toBe('Generic');
      });

      it('should handle null custom selector', () => {
        const config = getSiteConfig('https://example.com', null);

        expect(config.waitSelector).toBe('body');
      });

      it('should handle empty string custom selector', () => {
        const config = getSiteConfig('https://example.com', '');

        // Empty string is falsy, so should use default
        expect(config.waitSelector).toBe('body');
      });
    });

    describe('formatGenericMarkdown edge cases', () => {
      it('should handle very long titles', () => {
        const longTitle = 'A'.repeat(1000);
        const content = {
          title: longTitle,
          content: 'Content',
        };

        const markdown = formatGenericMarkdown(content);

        expect(markdown).toContain(`# ${longTitle}`);
      });

      it('should handle newlines in title', () => {
        const content = {
          title: 'Title\nWith\nNewlines',
          content: 'Content',
        };

        const markdown = formatGenericMarkdown(content);

        expect(markdown).toContain('# Title\nWith\nNewlines');
      });
    });
  });

  describe('fetchContent', () => {
    // fetchContent is integration-tested through main(), and the core logic
    // is unit-tested through getSiteConfig, extractTwitterContent, and extractGenericContent.
    // Testing fetchContent directly would require mocking Playwright's browser automation,
    // which is better covered by integration tests or manual testing.

    it('should be exported as a function', () => {
      expect(typeof fetchContent).toBe('function');
    });
  });

  describe('main', () => {
    let originalArgv, originalExit, exitCode, consoleErrorSpy;

    beforeEach(() => {
      originalArgv = process.argv;
      originalExit = process.exit;
      exitCode = null;

      process.exit = vi.fn((code) => {
        exitCode = code;
        throw new Error(`process.exit(${code})`);
      });

      consoleErrorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});
    });

    afterEach(() => {
      process.argv = originalArgv;
      process.exit = originalExit;
      vi.restoreAllMocks();
    });

    it('should exit with error when no URL provided', async () => {
      process.argv = ['node', 'script.js'];

      try {
        await main();
      } catch (_error) {
        // Expected to throw due to process.exit
      }

      expect(exitCode).toBe(1);
      expect(consoleErrorSpy).toHaveBeenCalledWith('Error: URL is required');
      expect(consoleErrorSpy).toHaveBeenCalledWith(
        expect.stringContaining('Usage:')
      );
    });

    it('should be exported as a function', () => {
      expect(typeof main).toBe('function');
    });
  });
});

#!/usr/bin/env node
/**
 * Article Content Extractor using Mozilla Readability
 *
 * Usage:
 *   node extract_article.js <html_file>
 *
 * Output:
 *   Clean article content in markdown format to stdout
 *
 * Requirements:
 *   npm install @mozilla/readability jsdom
 */

import fs from 'fs';
import { Readability } from '@mozilla/readability';
import { JSDOM } from 'jsdom';
import { runIfMain } from '#utils/module-runner.js';

/**
 * Convert HTML to markdown-like text format
 */
export function htmlToMarkdown(html) {
  const dom = new JSDOM(html);
  const doc = dom.window.document;
  const result = [];

  function processNode(node, depth = 0) {
    if (node.nodeType === 3) {
      // Text node
      const text = node.textContent.trim();
      if (text) {
        result.push(text);
      }
      return;
    }

    if (node.nodeType !== 1) {
      return;
    } // Not an element

    const tagName = node.tagName.toLowerCase();

    switch (tagName) {
      case 'h1':
        result.push(`\n# ${node.textContent.trim()}\n`);
        break;
      case 'h2':
        result.push(`\n## ${node.textContent.trim()}\n`);
        break;
      case 'h3':
        result.push(`\n### ${node.textContent.trim()}\n`);
        break;
      case 'h4':
        result.push(`\n#### ${node.textContent.trim()}\n`);
        break;
      case 'h5':
        result.push(`\n##### ${node.textContent.trim()}\n`);
        break;
      case 'h6':
        result.push(`\n###### ${node.textContent.trim()}\n`);
        break;
      case 'p':
        result.push(`\n${node.textContent.trim()}\n`);
        break;
      case 'br':
        result.push('\n');
        break;
      case 'a': {
        const href = node.getAttribute('href');
        const text = node.textContent.trim();
        if (href && text) {
          result.push(`[${text}](${href})`);
        } else {
          result.push(text);
        }
        break;
      }
      case 'strong':
      case 'b':
        result.push(`**${node.textContent.trim()}**`);
        break;
      case 'em':
      case 'i':
        result.push(`*${node.textContent.trim()}*`);
        break;
      case 'code':
        result.push(`\`${node.textContent.trim()}\``);
        break;
      case 'blockquote':
        result.push(`\n> ${node.textContent.trim()}\n`);
        break;
      case 'ul':
      case 'ol':
        result.push('\n');
        Array.from(node.children).forEach((li) => {
          if (li.tagName.toLowerCase() === 'li') {
            result.push(`- ${li.textContent.trim()}\n`);
          }
        });
        result.push('\n');
        break;
      default:
        // Process children
        Array.from(node.childNodes).forEach((child) =>
          processNode(child, depth + 1)
        );
    }
  }

  processNode(doc.body);
  return result
    .join(' ')
    .replace(/ +/g, ' ')
    .replace(/\n\n\n+/g, '\n\n')
    .trim();
}

/**
 * Extract article from HTML file
 */
export function extractArticle(htmlFilePath) {
  // Read HTML file
  if (!fs.existsSync(htmlFilePath)) {
    throw new Error(`File not found: ${htmlFilePath}`);
  }

  const htmlContent = fs.readFileSync(htmlFilePath, 'utf-8');

  // Parse with JSDOM
  const dom = new JSDOM(htmlContent);
  const reader = new Readability(dom.window.document);
  const article = reader.parse();

  if (!article) {
    throw new Error('Failed to extract article content');
  }

  // Convert to markdown format
  const markdown = htmlToMarkdown(article.content);

  return `# ${article.title}\n\n${markdown}`;
}

// Main execution
export function main() {
  if (process.argv.length !== 3) {
    console.error('Usage: node extract_article.js <html_file>');
    process.exit(1);
  }

  const htmlFile = process.argv[2];

  try {
    const articleContent = extractArticle(htmlFile);
    console.log(articleContent);
    // No need for process.exit(0) - Node exits naturally on success
  } catch (error) {
    console.error(`Error extracting article: ${error.message}`);
    process.exit(1); // Exit with error code
  }
}

// Run main function only if executed directly (not imported)
runIfMain(import.meta.url, main);

#!/usr/bin/env node
/**
 * Extract article from EUC-JP encoded HTML (common for Japanese sites like 4gamer)
 *
 * Usage:
 *   node extract_eucjp.js <html_file>
 *
 * Output:
 *   Clean article content in markdown format to stdout
 *
 * Requirements:
 *   npm install @mozilla/readability jsdom iconv-lite
 */

import fs from 'fs';
import iconv from 'iconv-lite';
import { Readability } from '@mozilla/readability';
import { JSDOM } from 'jsdom';
import { runIfMain } from '#utils/module-runner.js';

/**
 * Convert HTML to markdown-like text format
 */
/* istanbul ignore next - JSDOM-dependent, tested via integration */
export function htmlToMarkdown(html) {
  const dom = new JSDOM(html);
  const doc = dom.window.document;
  const result = [];

  function processNode(node) {
    if (node.nodeType === 3) {
      // Text node
      const text = node.textContent.trim();
      if (text) {
        result.push(text);
      }
      return;
    }

    if (node.nodeType !== 1) {
      return;
    } // Not an element

    const tagName = node.tagName.toLowerCase();

    switch (tagName) {
      case 'h1':
        result.push(`\n# ${node.textContent.trim()}\n`);
        break;
      case 'h2':
        result.push(`\n## ${node.textContent.trim()}\n`);
        break;
      case 'h3':
        result.push(`\n### ${node.textContent.trim()}\n`);
        break;
      case 'h4':
        result.push(`\n#### ${node.textContent.trim()}\n`);
        break;
      case 'p':
        result.push(`\n${node.textContent.trim()}\n`);
        break;
      case 'br':
        result.push('\n');
        break;
      case 'a': {
        const href = node.getAttribute('href');
        const text = node.textContent.trim();
        if (href && text) {
          result.push(`[${text}](${href})`);
        } else {
          result.push(text);
        }
        break;
      }
      case 'strong':
      case 'b':
        result.push(`**${node.textContent.trim()}**`);
        break;
      case 'em':
      case 'i':
        result.push(`*${node.textContent.trim()}*`);
        break;
      default:
        Array.from(node.childNodes).forEach((child) => processNode(child));
    }
  }

  processNode(doc.body);
  return result
    .join(' ')
    .replace(/ +/g, ' ')
    .replace(/\n\n\n+/g, '\n\n')
    .trim();
}

/**
 * Extract content using fallback selectors when Readability fails
 */
export function extractWithFallback(dom) {
  const doc = dom.window.document;
  const mainContent =
    doc.querySelector('article') ||
    doc.querySelector('.articleBody') ||
    doc.querySelector('#main');

  if (mainContent) {
    return `# ${doc.title}\n\n${htmlToMarkdown(mainContent.innerHTML)}`;
  } else {
    throw new Error('Failed to extract article content');
  }
}

/**
 * Extract article from EUC-JP encoded HTML file
 */
export function extractArticle(htmlFilePath) {
  if (!fs.existsSync(htmlFilePath)) {
    throw new Error(`File not found: ${htmlFilePath}`);
  }

  // Read file as buffer and decode from EUC-JP
  const htmlBuffer = fs.readFileSync(htmlFilePath);
  const htmlContent = iconv.decode(htmlBuffer, 'eucjp');

  // Parse with JSDOM
  const dom = new JSDOM(htmlContent);
  const reader = new Readability(dom.window.document);
  const article = reader.parse();

  if (!article) {
    return extractWithFallback(dom);
  }

  const markdown = htmlToMarkdown(article.content);
  return `# ${article.title}\n\n${markdown}`;
}

// Main execution
export function main() {
  if (process.argv.length !== 3) {
    console.error('Usage: node extract_eucjp.js <html_file>');
    process.exit(1);
  }

  const htmlFile = process.argv[2];

  try {
    const articleContent = extractArticle(htmlFile);
    console.log(articleContent);
    // No need for process.exit(0) - Node exits naturally on success
  } catch (error) {
    console.error(`Error extracting article: ${error.message}`);
    process.exit(1); // Exit with error code
  }
}

// Run main function only if executed directly (not imported)
runIfMain(import.meta.url, main);

#!/usr/bin/env node

/**
 * Fetch JavaScript-rendered content using Playwright
 *
 * This script uses a headless browser to render JavaScript and extract content
 * from dynamic websites like Twitter/X, React apps, and SPAs.
 *
 * Usage:
 *   node fetch_js_content.js <url> [options]
 *
 * Options:
 *   --output <file>    Output file path (default: stdout)
 *   --selector <sel>   CSS selector to wait for (default: auto-detect)
 *   --timeout <ms>     Page load timeout in milliseconds (default: 30000)
 *   --full-page        Extract full page content instead of article only
 *
 * Examples:
 *   node fetch_js_content.js https://x.com/user/status/123456789
 *   node fetch_js_content.js https://example.com --output content.md
 *   node fetch_js_content.js https://example.com --selector ".main-content" --timeout 60000
 */

import { chromium } from 'playwright';
import { writeFileSync } from 'fs';
import { parseArgs } from 'node:util';
import { resolve, isAbsolute, sep } from 'path';
import { runIfMain } from '#utils/module-runner.js';

/**
 * Validate URL to prevent SSRF attacks
 * @param {string} url - The URL to validate
 * @throws {Error} If URL is invalid or uses unsafe protocol
 */
export function validateUrl(url) {
  try {
    const urlObj = new URL(url);
    const allowedProtocols = ['http:', 'https:', 'file:'];

    if (!allowedProtocols.includes(urlObj.protocol)) {
      throw new Error(
        `Invalid protocol: ${urlObj.protocol}. Only http:, https:, and file: are allowed.`
      );
    }

    // Additional check for file:// URLs - prevent access to sensitive locations
    if (urlObj.protocol === 'file:') {
      const path = urlObj.pathname.toLowerCase();
      // Block common sensitive paths (basic protection)
      // Note: Windows paths in URLs look like /C:/Windows/System32
      const blockedPaths = [
        '/etc/',
        '/sys/',
        '/proc/',
        '/windows/system32/',
        ':/windows/system32/',
      ];
      if (blockedPaths.some((blocked) => path.includes(blocked))) {
        throw new Error(
          'Access to sensitive system directories is not allowed.'
        );
      }
    }
  } catch (error) {
    if (error.message.includes('Invalid URL')) {
      throw new Error(`Invalid URL format: ${url}`);
    }
    throw error;
  }
}

/**
 * Validate and sanitize output file path to prevent path traversal
 * @param {string} filePath - The file path to validate
 * @returns {string} Resolved safe file path
 * @throws {Error} If path is unsafe
 */
export function validateOutputPath(filePath) {
  const currentDir = process.cwd();

  // Detect Windows-style absolute paths (C:\... or C:/...) even on non-Windows systems
  // This is important for cross-platform security validation
  const windowsAbsolutePathPattern = /^[A-Za-z]:[/\\]/;
  if (windowsAbsolutePathPattern.test(filePath)) {
    const resolvedPath = resolve(filePath);

    // If after resolution, the path no longer matches the Windows pattern,
    // it means we're on a non-Windows system where it was treated as a relative path.
    // This is suspicious behavior and should be rejected.
    /* c8 ignore next 4 -- Platform-specific: only executed on non-Windows systems */
    if (!windowsAbsolutePathPattern.test(resolvedPath)) {
      throw new Error(
        `Output path must be within current directory. Attempted: ${filePath}`
      );
    }

    /* c8 ignore start -- Platform-specific: Windows path validation only executed on Windows systems */
    // We're on Windows, validate that the absolute path is within current directory
    const normalizedResolved = resolvedPath.split(sep).join('/').toLowerCase();
    const normalizedCwd = currentDir.split(sep).join('/').toLowerCase();

    if (!normalizedResolved.startsWith(normalizedCwd + '/')) {
      throw new Error(
        `Output path must be within current directory. Attempted: ${resolvedPath}`
      );
    }
    return resolvedPath;
    /* c8 ignore stop */
  }

  // Check if the input path is absolute (Unix-style: /etc/passwd)
  /* c8 ignore next 12 -- Platform-specific: Unix absolute path handling, primarily executed on Unix systems */
  if (isAbsolute(filePath)) {
    const resolvedPath = resolve(filePath);
    // Normalize both paths to ensure proper comparison across platforms
    const normalizedResolved = resolvedPath.split(sep).join('/').toLowerCase();
    const normalizedCwd = currentDir.split(sep).join('/').toLowerCase();

    if (!normalizedResolved.startsWith(normalizedCwd + '/')) {
      throw new Error(
        `Output path must be within current directory. Attempted: ${resolvedPath}`
      );
    }
    return resolvedPath;
  }

  // For relative paths, resolve and check
  const resolvedPath = resolve(filePath);
  if (!resolvedPath.startsWith(currentDir)) {
    throw new Error(
      `Output path must be within current directory. Attempted: ${resolvedPath}`
    );
  }

  return resolvedPath;
}

/**
 * Validate timeout value
 * @param {number} timeout - Timeout in milliseconds
 * @returns {number} Validated timeout
 * @throws {Error} If timeout is invalid
 */
export function validateTimeout(timeout) {
  const MIN_TIMEOUT = 1000; // 1 second
  const MAX_TIMEOUT = 300000; // 5 minutes

  if (isNaN(timeout) || timeout < MIN_TIMEOUT || timeout > MAX_TIMEOUT) {
    throw new Error(
      `Timeout must be between ${MIN_TIMEOUT} and ${MAX_TIMEOUT} milliseconds.`
    );
  }

  return timeout;
}

/**
 * Detect site type and return appropriate selectors
 * @param {string} url - The URL to fetch
 * @param {string} [customSelector] - Custom CSS selector to wait for
 * @returns {object} Site configuration
 */
export function getSiteConfig(url, customSelector = null) {
  if (url.includes('x.com') || url.includes('twitter.com')) {
    return {
      name: 'Twitter/X',
      waitSelector: customSelector || 'article[data-testid="tweet"]',
      extractFunction: extractTwitterContent,
    };
  }

  // Default config for generic sites
  return {
    name: 'Generic',
    waitSelector: customSelector || 'body',
    extractFunction: extractGenericContent,
  };
}

/**
 * Extract Twitter/X tweet content
 * @param {import('playwright').Page} page - Playwright page object
 * @returns {Promise<string>} Formatted markdown content
 */
export async function extractTwitterContent(page) {
  try {
    /* eslint-disable no-undef -- Code runs in browser context where document is defined */
    /* c8 ignore start -- Browser context code */
    const tweetData = await page.evaluate(() => {
      // Extract all tweet text elements at once
      const tweetTextElements = Array.from(
        document.querySelectorAll('[data-testid="tweetText"]')
      );
      const tweetText = tweetTextElements[0]
        ? tweetTextElements[0].innerText
        : 'Tweet text not found';

      // Extract author info
      const userNameElement = document.querySelector(
        '[data-testid="User-Name"]'
      );
      const userName = userNameElement ? userNameElement.innerText : 'Unknown';

      // Extract timestamp
      const timeElement = document.querySelector('time');
      const timestamp = timeElement
        ? timeElement.getAttribute('datetime')
        : 'Unknown';

      // Extract quoted tweet if present (will be the second tweet text element)
      const quotedTweet =
        tweetTextElements.length > 1 ? tweetTextElements[1].innerText : null;

      // Extract media alt texts and URLs if present
      const mediaElements = Array.from(
        document.querySelectorAll('[data-testid="tweetPhoto"] img')
      );
      const media = mediaElements.map((img) => ({
        url: img.src,
        alt: img.alt || 'Image',
      }));

      return {
        userName,
        tweetText,
        timestamp,
        quotedTweet,
        media,
      };
    });
    /* c8 ignore stop */
    /* eslint-enable no-undef */

    return formatTwitterMarkdown(tweetData);
  } catch (error) {
    throw new Error(`Failed to extract Twitter content: ${error.message}`);
  }
}

/**
 * Format Twitter data as markdown
 * @param {object} tweetData - Tweet data object
 * @returns {string} Formatted markdown
 */
export function formatTwitterMarkdown(tweetData) {
  let markdown = `# Twitter/X Post\n\n`;
  markdown += `**Author:** ${tweetData.userName}\n`;
  markdown += `**Timestamp:** ${tweetData.timestamp}\n\n`;
  markdown += `## Tweet Content\n\n`;
  markdown += `${tweetData.tweetText}\n\n`;

  if (tweetData.quotedTweet) {
    markdown += `### Quoted Tweet\n\n`;
    markdown += `> ${tweetData.quotedTweet}\n\n`;
  }

  if (tweetData.media && tweetData.media.length > 0) {
    markdown += `### Media\n\n`;
    tweetData.media.forEach((item, i) => {
      markdown += `${i + 1}. ${item.alt}\n`;
      markdown += `   - URL: ${item.url}\n`;
    });
    markdown += `\n`;
  }

  return markdown;
}

/**
 * Extract generic page content
 * @param {import('playwright').Page} page - Playwright page object
 * @returns {Promise<string>} Formatted markdown content
 */
export async function extractGenericContent(page) {
  try {
    /* eslint-disable no-undef -- Code runs in browser context where document is defined */
    /* c8 ignore start -- Browser context code */
    const content = await page.evaluate(() => {
      // Try to find main content area
      const mainSelectors = [
        'main',
        'article',
        '[role="main"]',
        '.content',
        '#content',
        '.main-content',
        'body',
      ];

      let mainElement = null;
      for (const selector of mainSelectors) {
        mainElement = document.querySelector(selector);
        if (mainElement) break;
      }

      if (!mainElement) {
        mainElement = document.body;
      }

      // Extract title
      const title =
        document.querySelector('h1')?.innerText || document.title || 'Untitled';

      // Extract text content
      const textContent =
        mainElement.innerText || mainElement.textContent || 'No content found';

      // Extract images from main content area
      const imageElements = Array.from(mainElement.querySelectorAll('img'));
      const media = imageElements
        .filter((img) => {
          // Filter out tiny images (likely icons/logos)
          const width = img.width || img.naturalWidth || 0;
          const height = img.height || img.naturalHeight || 0;
          return width > 100 && height > 100;
        })
        .map((img) => ({
          url: img.src,
          alt: img.alt || 'Image',
        }));

      return {
        title,
        content: textContent,
        media,
      };
    });
    /* c8 ignore stop */
    /* eslint-enable no-undef */

    return formatGenericMarkdown(content);
  } catch (error) {
    throw new Error(`Failed to extract generic content: ${error.message}`);
  }
}

/**
 * Format generic content as markdown
 * @param {object} content - Content object with title, content, and optional media
 * @returns {string} Formatted markdown
 */
export function formatGenericMarkdown(content) {
  let markdown = `# ${content.title}\n\n`;
  markdown += `${content.content}\n`;

  if (content.media && content.media.length > 0) {
    markdown += `\n### Media\n\n`;
    content.media.forEach((item, i) => {
      markdown += `${i + 1}. ${item.alt}\n`;
      markdown += `   - URL: ${item.url}\n`;
    });
    markdown += `\n`;
  }

  return markdown;
}

/**
 * Fetch and extract content from a URL
 * @param {string} url - The URL to fetch
 * @param {object} options - Fetch options
 * @param {string} [options.selector] - CSS selector to wait for
 * @param {number} [options.timeout=30000] - Page load timeout in ms
 * @param {number} [options.waitDelay=2000] - Additional delay after content loads in ms
 * @returns {Promise<string>} Extracted markdown content
 */
/* c8 ignore start -- Browser automation code, tested through integration tests */
export async function fetchContent(url, options = {}) {
  const { selector = null, timeout = 30000, waitDelay = null } = options;

  // Auto-detect delay: file:// URLs need less wait time
  const defaultDelay = url.startsWith('file://') ? 500 : 2000;
  const actualDelay = waitDelay !== null ? waitDelay : defaultDelay;

  let browser = null;

  try {
    // Get site-specific configuration
    const siteConfig = getSiteConfig(url, selector);

    // Launch browser
    browser = await chromium.launch({
      headless: true,
      args: ['--no-sandbox', '--disable-dev-shm-usage'],
    });

    // Create browser context with viewport and user agent
    const context = await browser.newContext({
      viewport: { width: 1920, height: 1080 },
      userAgent:
        'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
    });

    // Create new page
    const page = await context.newPage();

    // Navigate to URL
    await page.goto(url, {
      waitUntil: 'domcontentloaded',
      timeout: timeout,
    });

    // Wait for content to load
    await page.waitForSelector(siteConfig.waitSelector, { timeout: timeout });

    // Small additional delay to ensure content is fully rendered
    // Shorter delay for local files, longer for remote sites
    await page.waitForTimeout(actualDelay);

    // Extract content using site-specific function
    const markdown = await siteConfig.extractFunction(page);

    return markdown;
  } finally {
    if (browser) {
      await browser.close();
    }
  }
}
/* c8 ignore stop */

/**
 * Main function
 */
/* c8 ignore start -- CLI entry point, tested through integration tests */
export async function main() {
  // Parse command line arguments
  const { values, positionals } = parseArgs({
    args: process.argv.slice(2),
    options: {
      output: { type: 'string', short: 'o' },
      selector: { type: 'string', short: 's' },
      timeout: { type: 'string', short: 't', default: '30000' },
      'full-page': { type: 'boolean', default: false },
    },
    allowPositionals: true,
  });

  const url = positionals[0];
  const outputFile = values.output;
  const selector = values.selector;

  if (!url) {
    console.error('Error: URL is required');
    console.error('Usage: node fetch_js_content.js <url> [options]');
    process.exit(1);
  }

  try {
    // Validate URL to prevent SSRF
    validateUrl(url);

    // Validate timeout
    const timeout = validateTimeout(parseInt(values.timeout, 10));

    // Validate output path if provided
    let validatedOutputFile = null;
    if (outputFile) {
      validatedOutputFile = validateOutputPath(outputFile);
    }

    console.error(`Fetching content from: ${url}`);

    // Get site-specific configuration for logging
    const siteConfig = getSiteConfig(url, selector);
    console.error(`Detected site type: ${siteConfig.name}`);
    console.error('Launching headless browser...');
    console.error(`Navigating to URL...`);
    console.error(`Waiting for content selector: ${siteConfig.waitSelector}`);
    console.error('Extracting content...');

    // Fetch content
    const markdown = await fetchContent(url, { selector, timeout });

    // Output result
    if (validatedOutputFile) {
      writeFileSync(validatedOutputFile, markdown, 'utf-8');
      console.error(`Content saved to: ${validatedOutputFile}`);
    } else {
      console.log(markdown);
    }

    console.error('Fetch completed successfully!');
  } catch (error) {
    console.error(`Error: ${error.message}`);
    process.exit(1);
  }
}
/* c8 ignore stop */

// Run main function only when called directly
runIfMain(import.meta.url, main);

Related skills

Seo AuditRun structured SEO audits on their SaaS site or content hub and receive a prioritized action plan.173k42.2k

CopywritingGenerate, rewrite, or strengthen persuasive website and landing-page copy that converts visitors into users.164k42.2k

Viral Short FormQuickly generate high-retention hooks, scripts, and outlines for TikTok, Reels, YouTube Shorts, and carousels.132k67

Viral HooksWrite and critique viral hooks for short-form video opening sequences.123k67

Viral Captions And CtasOptimize social media captions and CTAs for viral short-form video reach and saves.123k67

Viral Youtube ShortsWrite and diagnose YouTube Shorts for Shorts Feed and long-form funnel.123k67

Marketing & SEOcontent

About

Web Content Fetcher by the numbers

Add your badge

What it does

Files

Web Content Fetcher

Overview

Quick Tool Limitations

Tiered Fetching Strategy

Tier 1: Small Content - WebFetch Direct

Tier 2: Any Size Content - curl + Task Agent ⭐ RECOMMENDED

Tier 3: Script-Based Extraction - Edge Cases Only

Tier 4: JavaScript-Rendered Content - Playwright

Decision Tree

Standard Workflow Quick Reference

Common Use Cases

When Things Go Wrong

Best Practices

Reference Files

Quick Start Examples

Advanced Fetching Methods

Table of Contents

Tier 3: Script-Based Extraction

extract_article.js - Standard HTML

extract_eucjp.js - Japanese EUC-JP Encoding

Tier 4: JavaScript-Rendered Content

Playwright Setup

fetch_js_content.js Usage

Supported Sites and Examples

Twitter/X Posts

React/Vue/Angular SPAs

Dynamic Content Sites

Site-Specific Features

How It Works

Performance Notes

Advanced Usage

Limitations

Troubleshooting Guide

Table of Contents

Common Issues

Issue: WebFetch returns "Prompt too long"

Issue: Read tool says file too large (> 256KB)

Issue: Extracted content is messy or incomplete

Issue: Japanese text shows garbled characters (mojibake/文字化け)

Issue: Page has anti-bot protection

Issue: Content requires authentication

Issue: Page redirects to different domain

Issue: Timeout or slow connection

Issue: Page requires JavaScript / Shows "JavaScript not available"

Issue: Playwright timeout errors

Issue: Downloaded file is HTML when expecting binary

Quick Reference Commands

Basic Fetch Operations

Error Handling

Authentication

JavaScript-Rendered Content

Content Inspection

Debugging

Standard Workflows and Patterns

Table of Contents

Standard Workflow (Tier 2 - Recommended)

Step 1: Create Task Directory

Step 2: Fetch Raw Content

Step 3: Extract Clean Content

Step 4: Use Extracted Content

Directory Structure Best Practice

Content Extraction Priorities

Include

Exclude

Format

Common Patterns

Pattern 1: Fetch for Translation

Pattern 2: Fetch for Analysis

Pattern 3: Fetch Series of Articles

Pattern 4: Fetch with Authentication

Pattern 5: Fetch JavaScript-Rendered Content

Related skills

This week in AI coding