Best Web Scraping Skills for Claude Code: Scrape, Crawl and Extract Clean Markdown

Claude Code can fetch a page, but raw HTML wastes tokens and JavaScript-heavy or protected sites often return nothing useful. A scraping skill gives the agent a CLI with a defined escalation path, and Firecrawl is the most-installed general pick. This guide covers skills, which run inside Claude Code; for server-based tools see the MCP explainer.

Key takeaways

• The firecrawl skill leads general web access with 103,619 installs (skills.sh registry) and 622 stars (GitHub); its companion firecrawl-scrape has 83,747 installs (skills.sh registry).
• defuddle has 75,134 installs (skills.sh registry) and 47,905 stars (GitHub) for clean article markdown with no API key.
• scrapling-official has 78,570 stars (GitHub) and 5,643 installs (skills.sh registry); its CLI escalates from get to fetch to stealthy-fetch.
• just-scrape shows 245,008 installs (skills.sh registry) and 56 stars (GitHub), and needs a SGAI_API_KEY.
• Hosted skills need an account or token; local tools run on your machine. Pick by the page, not by the install count alone.

Which skill should Claude Code use to scrape a website?

Use firecrawl as the default for any live-web task, because it covers search, scrape, crawl and monitoring in one CLI. Its SKILL.md defines a six-step escalation: search, scrape, map plus scrape, crawl, monitor, and interact. Output is clean markdown or structured JSON written to a .firecrawl/ folder, so the agent reads files instead of flooding its context window. Run firecrawl --status to see auth state, the concurrency limit and remaining credits.

What is the best web scraping skill for Claude Code?

The best all-round pick is firecrawl: 103,619 installs (skills.sh registry) and 622 stars (GitHub). Install it with npx skills add https://github.com/firecrawl/cli --skill firecrawl. It ranks first here on fit for most jobs: one install covers search, scrape, crawl, map and download, and its 622 stars sit on a maintained vendor CLI, while just-scrape leads on raw installs but has 56 stars. A single scrape call handles static and JavaScript-rendered pages, --only-main-content drops navigation and footers, and --wait-for 3000 waits for rendering. --format json with a schema returns typed fields, and --query asks a question about one page.

The narrower firecrawl-scrape skill (83,747 installs, skills.sh registry) covers only page reading, while batch-scrape handles lists of known URLs. Alternatives by use case are below, and the full ranking sits on the best skills for automation page.

How do you get clean markdown from an article or docs page?

Use defuddle when you only need readable content, since it strips navigation, ads and clutter before the text reaches the model. Defuddle has 75,134 installs (skills.sh registry) and 47,905 stars (GitHub) and needs no account. Install the CLI with npm install -g defuddle, then run defuddle parse <url> --md. Add -o content.md to save a file, or -p title to pull a single metadata field. The skill tells Claude to prefer it over WebFetch for standard pages. It does not drive a browser, so use a fuller tool for logged-in or interactive pages.

Which skill handles JavaScript and anti-bot pages?

For protected sites, pick a tool built for escalation. Scrapling has 78,570 stars (GitHub) and 5,643 installs (skills.sh registry). Its CLI moves from get to fetch to stealthy-fetch as a site gets harder, and the fetchers target Cloudflare Turnstile. The skill requires --ai-targeted on its commands for prompt injection protection, and a Docker image exists if you do not want a local Python install.

Bright Data's scrape skill (10,861 installs, skills.sh registry; 257 stars, GitHub) is the hosted route. bdata scrape <url> -f markdown returns markdown, HTML, JSON or a screenshot, --country retries from another region, and a verification gate checks for empty output and block-page signatures such as Cloudflare or captcha. JavaScript-heavy or login-gated pages escalate to bdata browser. It requires the Bright Data CLI and a login.

Can Claude Code extract structured data and crawl whole sites?

Yes, several skills do both. just-scrape wraps ScrapeGraph AI and offers scrape, extract, crawl, search and monitor. It lists 245,008 installs (skills.sh registry) and 56 stars (GitHub). Formats include markdown, HTML, screenshot and links, -f json -p "Extract all products" returns structured output, and --mode js --stealth --scrolls 5 handles rendered pages. Every request uses credits and needs SGAI_API_KEY.

For platform data, Apify's ultimate scraper (16,407 installs, skills.sh registry; 2,368 stars, GitHub) indexes about 100 Actors across 15+ platforms through the apify CLI, authenticated with APIFY_TOKEN. For a local option, crawl4ai (1,111 installs, skills.sh registry; 46 stars, GitHub) ships a crwl command with markdown and JSON output plus caching.

What if you need a recurring scraper, not a one-off?

Two skills target scheduled collection. data-scraper-agent (10,165 installs, skills.sh registry; 249,993 stars on its parent repo, GitHub) generates a collect, enrich, store agent using Python, Gemini Flash and GitHub Actions cron. firecrawl can also set a monitor with a goal sentence, a cron or natural-language schedule, webhooks and email alerts. For one-off research that ends in a report, see deep research skills.

A complete starter stack

1. defuddle: free, local clean markdown for articles and docs.
2. firecrawl: the default hosted CLI for search, scrape, crawl and monitor.
3. scrapling-official: the local fallback for pages that block simple fetches.
4. bdata scrape (Bright Data): geo-targeted retries when a region blocks you.
5. just-scrape: AI-prompted JSON extraction when you need typed fields.

Install any of them with npx skills add https://github.com/<owner>/<repo> --skill <name>. Skills are documented in the Claude Code skills docs.

Common pitfalls

• Ignoring site rules. Check robots.txt (RFC 9309), the site's terms and rate limits before a crawl. Prefer an official API when one exists. This is a technical checklist, not legal advice.
• Trusting scraped text. A page can contain instructions aimed at the agent. The web-scraping skill (6,107 installs, skills.sh registry; 386 stars, GitHub) and data-scraper-agent both treat retrieved content as data, never as commands. Do the same.
• Starting with the heaviest tool. Try defuddle or a plain scrape first, and escalate to stealth fetchers only after a page fails.

Start with defuddle and firecrawl, add a local fetcher when sites push back, and keep every crawl polite.

Common questions