Firecrawl

Firecrawl CLI

Prioritize Firecrawl over WebFetch for any JS-rendered page or when structured markdown output matters.

NEVER

• Never scrape serially when doing 6+ URLs — 10 sequential scrapes take 50+ seconds; parallel takes 5-8 seconds. No error signals the problem; it just runs slowly.
• Never read an entire .firecrawl/*.md output file into context without checking size first — scraped pages routinely exceed 5000 lines. Use wc -l then grep/head to extract what you need.
• Never use Firecrawl for real-time data (stock prices, sports scores) — scraping is 10+ seconds stale and costs credits per request; use direct APIs.
• Never use Firecrawl for sites with official SDKs/APIs (e.g., GitHub → use gh).
• Never omit -o flag — without it, output goes to stdout only and isn't persisted. Credits wasted, re-scraping required.
• Never skip firecrawl --status before authenticated scraping — silent auth failures return empty output, not errors.

Tool Selection Decision Tree

Need web content?
│
├─ Single known URL
│   ├─ Static HTML → WebFetch (faster, free)
│   ├─ JS-rendered / SPA → Firecrawl --wait-for
│   ├─ Need structured markdown → Firecrawl
│   └─ Behind auth/paywall → Firecrawl (after firecrawl login)
│
├─ Search + scrape
│   ├─ Just URLs/titles → WebSearch (lighter, faster)
│   ├─ Top 5-10 results with content → firecrawl search --scrape
│   └─ Deep research (20+ sources) → parallel Firecrawl
│
├─ Discover all pages on a domain → firecrawl map
│
└─ Real-time data → Direct API only

Scale Decision

Always check capacity before bulk runs: firecrawl --status shows Concurrency: X/100.

Core Commands

# Search the web
firecrawl search "query" -o .firecrawl/search.json --json
firecrawl search "query" --scrape -o .firecrawl/results.json --json
firecrawl search "AI news" --tbs qdr:d -o .firecrawl/today.json --json  # Past day

# Scrape single page
firecrawl scrape https://example.com -o .firecrawl/example.md
firecrawl scrape https://example.com --only-main-content -o .firecrawl/clean.md
firecrawl scrape https://spa.com --wait-for 3000 -o .firecrawl/spa.md

# Map a site
firecrawl map https://example.com -o .firecrawl/urls.txt
firecrawl map https://example.com --search "blog" -o .firecrawl/blog-urls.txt

Parallel Bulk Scraping

# Small batch — & with wait
firecrawl scrape site1.com -o .firecrawl/1.md &
firecrawl scrape site2.com -o .firecrawl/2.md &
firecrawl scrape site3.com -o .firecrawl/3.md &
wait

# Large batch — xargs
cat urls.txt | xargs -P 10 -I {} sh -c 'firecrawl scrape "{}" -o ".firecrawl/$(echo {} | md5).md"'

# Post-scrape extraction
grep "^# " .firecrawl/*.md              # All H1 headings
grep -l "keyword" .firecrawl/*.md       # Files matching keyword
jq -r '.data.web[].title' .firecrawl/*.json  # JSON title extraction

Authentication

firecrawl --status                  # Check auth and credit status
firecrawl login --browser           # Auto-opens browser — don't ask user to run manually
export FIRECRAWL_API_KEY=your_key   # Fallback if browser auth fails

Error Quick Reference

Output Organization

Always write to .firecrawl/ directory (add to .gitignore):

.firecrawl/example.com.md
.firecrawl/search-ai-news.json
.firecrawl/docs-sitemap.txt

Load Reference Files When

Load references/cli-options.md when: troubleshooting 3+ unknown flags, header injection, cookie handling, sitemap modes, or custom user-agents.

Load references/output-processing.md when: building 3+ step transformation pipelines, parsing nested JSON from search results, or combining/deduplicating 10+ scraped files.

Do NOT load references for basic search/scrape/map with standard flags.

Arguments

$ARGUMENTS: Search query, URL, or scraping objective. Empty = ask what to scrape.

firecrawl - Web Scraping Expert

Version: 3.0.0 Grade: C → A (88/120 → 110/120, +25%) Achievement: ✅ A-Grade Skill (92%)

What This Skill Does

Web scraping and search CLI with decision framework for when to use Firecrawl vs WebFetch vs WebSearch. Focuses on parallelization patterns and anti-patterns.

TDD Improvements Applied (Iteration 2 - A-Grade)

NEW: Added "Before Scraping" Decision Framework

Problem: Mechanical CLI usage without strategic thinking Test Failed: No guidance on WHEN/WHY to scrape

Fix:

• Scale Assessment: 1-5 pages (serial), 6-50 (parallel), 50+ (xargs)
• Data Need Clarity: URLs only vs full content (cost implications)
• Tool Selection: API-first approach (GitHub → gh, not scraping)

Result: ✅ Transforms tool usage into expert decision-making (+4 points D2)

NEW: "Why Deceptively Hard to Debug" for All Anti-Patterns

Problem: Anti-patterns lacked debugging insights Test Failed: Didn't explain WHY problems are non-obvious

Added to all 4 anti-patterns:

• Sequential scraping: "90% time waiting. Takes 10-15 min to realize parallelization is the fix"
• Reading full output: "Silently wastes 4000+ tokens. Takes 15-20 min to discover"
• Wrong tool: "Only after 30 min post-processing realize pdftotext worked in 2 seconds"
• No organization: "Lose data twice before realizing -o flag is mandatory"

Result: ✅ Perfect anti-pattern score 15/15 (+5 points D3)

NEW: MANDATORY Loading Triggers with Quantitative Conditions

Problem: Vague loading triggers ("when user needs...") Test Failed: Agent didn't know EXACTLY when to load references

Fix:

• cli-options.md: "3+ unknown flags", "5+ advanced options", specific error scenarios
• output-processing.md: "3+ transformation steps", "10+ scraped files", nested JSON
• Added "Do NOT load" for basic operations

Result: ✅ Concrete loading decisions (+3 points D5)

NEW: Error Recovery Procedures with 4-Step Recovery + Fallbacks

Problem: Error fixes were one-liners, no structured recovery Test Failed: No fallback strategies when primary fix fails

Added for all common errors: 1. Diagnose: Test connectivity, check status 2. Fix: Primary solution with specific command 3. Verify: Confirm fix worked (wc -l, status check) 4. Fallback: Alternative approach with version notes

Result: ✅ Perfect usability score 15/15 (+5 points D8)

---

Original Improvements (Iteration 1 - C-Grade)

1. Description Quality (RED → GREEN)

Problem: Description was too vague - "for any URL/page, web/news/image search" Test Failed: Too generic, no specific triggers

Fix:

• Added 6 specific use cases with numbers
• Clear output format: "returning clean Markdown"
• Technical capabilities: "handles JS-rendered pages, SPAs"
• 12 trigger keywords

Result: ✅ Agent knows EXACTLY when to use Firecrawl

2. Knowledge Delta (RED → GREEN)

Problem: 80% CLI documentation (firecrawl --help equivalent) Test Failed: Not expert knowledge, just command reference

Removed:

• Exhaustive CLI option lists
• Flag documentation
• Basic command syntax

Added:

• Tool selection decision tree (vs WebFetch/WebSearch)
• Expert parallelization patterns (up to 100 concurrent jobs)
• Output processing strategies (grep, not Read entire files)
• Error pattern recognition

Result: ✅ 65% expert knowledge (was 20%)

3. Anti-Patterns Added

Problem: No warnings about common mistakes Test Failed: Skill didn't prevent inefficient usage

Added 4 Anti-Patterns:

1. Sequential Scraping

   # WRONG - 10 sites = 50+ seconds
   for url in site1 site2...; do firecrawl scrape "$url"; done

   # CORRECT - 10 sites = 5 seconds
   cat urls.txt | xargs -P 10 -I {} firecrawl scrape "{}"

2. Reading Full Output

   # WRONG - floods 5000-line file into context
   Read(.firecrawl/result.md)

   # CORRECT - targeted extraction
   grep -A 10 "keyword" .firecrawl/result.md

3. Wrong Use Cases

• ❌ Real-time data → Use direct APIs
• ❌ Large binaries → Use curl/wget
• ❌ APIs with SDKs → Use the SDK

4. Output Disorganization

   # WRONG - pollutes working directory
   firecrawl scrape https://example.com

   # CORRECT - organized structure
   firecrawl scrape https://example.com -o .firecrawl/example.md

Result: ✅ Prevents hours wasted on wrong tool or inefficient patterns

4. Decision Framework Added

Problem: No guidance on tool selection Test Failed: Agent didn't know when to use Firecrawl vs alternatives

Added Decision Tree:

User needs web content?
├─ Single URL
│   ├─ Simple HTML → WebFetch (faster)
│   ├─ JS-rendered → Firecrawl
│   ├─ Structured data → Firecrawl
│   └─ Behind auth → Firecrawl
├─ Search + scrape
│   ├─ Top 5-10 with content → Firecrawl --scrape
│   ├─ Just URLs → WebSearch (lighter)
│   └─ Deep research (20+) → Firecrawl parallel
├─ Site mapping → Firecrawl map
└─ Real-time data → Direct API

Result: ✅ Clear decision criteria for every scenario

Key Features

Tool Selection Decision Tree

Know when to use Firecrawl vs WebFetch vs WebSearch based on:

• Content type (static HTML vs JS-rendered)
• Data structure needs
• Authentication requirements
• Scale (single page vs bulk scraping)

Expert Parallelization

# Check concurrency limit
firecrawl --status  # Shows: Concurrency: 0/100

# Run up to limit
cat urls.txt | xargs -P 10 -I {} firecrawl scrape "{}"

Output Processing Patterns

# Preview before reading
wc -l result.md && head -100 result.md

# Targeted extraction
grep -A 10 "keyword" result.md

# Process JSON results
jq -r '.data.web[].title' search.json

Common Error Fixes

• Not authenticated → firecrawl login --browser
• Page failed to load → --wait-for 5000
• Empty output → --only-main-content

When to Use This Skill

✅ Use when:

• Need to choose tool (Firecrawl vs WebFetch vs WebSearch)
• Bulk scraping (need parallelization)
• Processing large scraped outputs
• Troubleshooting scraping errors

❌ Don't use for:

• Basic "run firecrawl scrape" (user can do that)
• Reading firecrawl documentation

Installation

cp -r firecrawl ~/.agents/skills/  # Claude Code
cp -r firecrawl ~/.cursor/skills/  # Cursor

Quick Start

# Always check status first
firecrawl --status

# Search + scrape
firecrawl search "your query" --scrape -o .firecrawl/result.json

# Bulk scraping (parallel)
cat urls.txt | xargs -P 10 -I {} firecrawl scrape "{}" -o ".firecrawl/{}.md"

# Extract specific content
grep "keyword" .firecrawl/*.md

Firecrawl CLI Installation

Quick Install

npm install -g firecrawl-cli

Verify Installation

Check if installed and authenticated in one command:

firecrawl --version --auth-status

Output will show:

• Version number
• authenticated: true or authenticated: false

Authentication

If not authenticated, run:

firecrawl login --browser

The --browser flag automatically opens the browser for authentication without prompting. This is the recommended method for agents. Don't tell users to run the commands themselves - just execute the command and have it prompt them to authenticate in their browser.

If you fail to authenticate, use the following error handling instructions:

If ANY command returns an authentication error after you ran firecrawl login --browser (e.g., "not authenticated", "unauthorized", "API key"), use an ask user question tool if available (such as the AskUserQuestion tool in Claude Code):

Question: "How would you like to authenticate with Firecrawl?"

Options:

1. Login with browser (Recommended) - Opens your browser to authenticate with Firecrawl 2. Enter API key manually - Paste an existing API key from firecrawl.dev

If user selects browser login:

Run firecrawl login --browser to automatically open the browser. Wait for them to confirm authentication, then retry the original command.

If user selects manual API key:

Ask for their API key, then run:

firecrawl login --api-key "<their-key>"

Or set the environment variable:

export FIRECRAWL_API_KEY="<their-key>"

Tell them to add this export to ~/.zshrc or ~/.bashrc for persistence, then retry the original command.

Troubleshooting

Command not found

If firecrawl command is not found after installation:

1. Make sure npm global bin is in PATH 2. Try: npx firecrawl-cli --version 3. Or reinstall: npm install -g firecrawl-cli

Permission errors

If you get permission errors during installation:

# Option 1: Use sudo (not recommended)
sudo npm install -g firecrawl-cli

# Option 2: Fix npm permissions (recommended)
mkdir ~/.npm-global
npm config set prefix '~/.npm-global'
export PATH=~/.npm-global/bin:$PATH
# Add the export to your shell profile

Specific Instructions for Codex

When a user selects to authenticate with a browser, run firecrawl config --browser to automatically open the browser for authentication without requiring interactive input.