Web Research

Web Research with x402 APIs

Access Exa (neural search) and Firecrawl (web scraping) through x402-protected endpoints.

Setup

See rules/getting-started.md for installation and wallet setup.

Quick Reference

When to Use What

See rules/when-to-use.md for detailed guidance.

Exa Neural Search

Semantic search that understands meaning, not just keywords:

npx agentcash@latest fetch https://stableenrich.dev/api/exa/search -m POST -b '{
  "query": "startups building AI agents for customer support",
  "numResults": 10,
  "type": "neural"
}'

Options:

• query - Search query (required)
• numResults - Number of results (default: 10, max: 25)
• type - "neural" (semantic) or "keyword" (traditional)
• includeDomains - Only search these domains
• excludeDomains - Skip these domains
• startPublishedDate / endPublishedDate - Date range filter
• category - Filter by content type: "company", "research paper", "news", "pdf", "github", "tweet", "personal site", "linkedin profile", "financial report"
• Tip: Use category: "linkedin profile" for people/profile discovery

Returns: List of URLs with titles, snippets, and relevance scores.

Find Similar Pages

Find pages semantically similar to a reference URL:

npx agentcash@latest fetch https://stableenrich.dev/api/exa/find-similar -m POST -b '{
  "url": "https://example.com/article-i-like",
  "numResults": 10
}'

Great for:

• Finding competitor products
• Discovering related content
• Expanding research sources

Extract Text Content

Get clean, structured text from URLs:

npx agentcash@latest fetch https://stableenrich.dev/api/exa/contents -m POST -b '{
  "urls": [
    "https://example.com/article1",
    "https://example.com/article2"
  ]
}'

Options:

• urls - Array of URLs to extract
• text - Include full text (default: true)
• highlights - Include key highlights

Cheapest option ($0.002) when you already have URLs and just need the content.

Direct Answers

Get factual answers to questions:

npx agentcash@latest fetch https://stableenrich.dev/api/exa/answer -m POST -b '{"query": "What is the population of Tokyo?"}'

Returns a direct answer with source citations. Best for:

• Factual questions
• Quick lookups
• Verification of claims

Firecrawl Scrape

Scrape a single page to clean markdown:

npx agentcash@latest fetch https://stableenrich.dev/api/firecrawl/scrape -m POST -b '{"url": "https://example.com/page-to-scrape"}'

Options:

• url - Page to scrape (required)
• formats - Output formats: ["markdown", "html", "links"]
• onlyMainContent - Skip nav/footer/ads (default: true)
• waitFor - Wait ms for JS to render

Advantages over WebFetch:

• Handles JavaScript-rendered content
• Bypasses common blocking
• Extracts main content only
• LLM-optimized markdown output

Firecrawl Search

Web search with automatic scraping of results:

npx agentcash@latest fetch https://stableenrich.dev/api/firecrawl/search -m POST -b '{
  "query": "best practices for react server components",
  "limit": 5
}'

Options:

• query - Search query (required)
• limit - Number of results (default: 5)
• scrapeOptions - Options passed to scraper

Returns search results with full scraped content for each.

Cloudflare Website Crawl

Crawl multiple pages from a website with browser rendering. Async two-step pattern.

Step 1: Start the crawl (paid, $0.10)

npx agentcash@latest fetch https://stableenrich.dev/api/cloudflare/crawl -m POST -b '{
  "url": "https://example.com",
  "limit": 10,
  "depth": 1,
  "formats": ["markdown"]
}'

Returns 202 with {"token": "jwt..."}.

Step 2: Poll for results (SIWX, free)

npx agentcash@latest fetch "https://stableenrich.dev/api/cloudflare/jobs?token=JWT_TOKEN"

Poll every 3-5 seconds until complete.

Parameters:

• url (required) — starting URL
• limit (default 10, max 25) — max pages
• depth (default 1, max 3) — max link depth
• formats — ["markdown", "html", "json"]
• render (default false) — execute JavaScript
• options.includePatterns / excludePatterns — URL wildcards

Good for: crawling docs sites, scraping multiple pages, building sitemaps.

Workflows

Deep Research

• [ ] (Optional) Check balance: npx agentcash@latest balance
• [ ] Search broadly with Exa
• [ ] Find related sources with find-similar
• [ ] Extract content from top sources
• [ ] Synthesize findings

npx agentcash@latest fetch https://stableenrich.dev/api/exa/search -m POST -b '{"query": "AI agents in healthcare 2024", "numResults": 15}'

npx agentcash@latest fetch https://stableenrich.dev/api/exa/find-similar -m POST -b '{"url": "https://best-article-found.com"}'

npx agentcash@latest fetch https://stableenrich.dev/api/exa/contents -m POST -b '{"urls": ["url1", "url2", "url3"]}'

Blocked Site Scraping

• [ ] Try WebFetch first (free)
• [ ] If blocked/empty, use Firecrawl with waitFor for JS-heavy sites

npx agentcash@latest fetch https://stableenrich.dev/api/firecrawl/scrape -m POST -b '{"url": "https://blocked-site.com/article", "waitFor": 3000}'

Cost Optimization

• Use Exa contents ($0.002) when you already have URLs
• Use WebSearch/WebFetch first (free) and fall back to x402 endpoints
• Batch URL extraction - pass multiple URLs to Exa contents
• Limit results - request only as many as needed

Getting Started

Setup

1. Install the agentcash CLI:

   npm install -g agentcash

2. Check wallet:

   npx agentcash@latest balance

3. Fund wallet (if needed):

• Redeem invite: npx agentcash@latest redeem YOUR_CODE
• Or run npx agentcash@latest accounts to get Base or Solana deposit links and wallet addresses

Troubleshooting

When to Use x402 Web Research

Decision Tree

Need web info?
|
+-- Simple keyword search? --> Use WebSearch (free)
|
+-- Semantic/conceptual search? --> Use Exa search ($0.01)
|
+-- Need to scrape a specific URL?
|   |
|   +-- WebFetch works? --> Use WebFetch (free)
|   |
|   +-- Blocked/JS-heavy? --> Use Firecrawl scrape ($0.0126)
|
+-- Find similar pages? --> Use Exa find-similar ($0.01)
|
+-- Already have URLs, need text? --> Use Exa contents ($0.002)
|
+-- Quick factual answer? --> Use Exa answer ($0.01)

Exa vs WebSearch

Use Exa when:

• Query is conceptual ("companies solving X problem")
• Need semantic similarity
• Researching a topic in depth
• Standard search isn't finding what you need

Use WebSearch when:

• Query is straightforward keywords
• Quick fact check
• Don't need deep results
• Budget is a concern

Firecrawl vs WebFetch

Use Firecrawl when:

• WebFetch returns blocked/403/empty
• Site uses heavy JavaScript
• Need clean markdown, not HTML
• Content is behind soft paywall

Use WebFetch when:

• Site works with simple fetch
• HTML format is fine
• Cost is a concern
• Testing before x402 call

Exa Search vs Firecrawl Search

Use Exa search to find relevant URLs, then selectively extract content.

Use Firecrawl search when you want search results AND their full content in one call.

Cost Comparison

For a typical research task (find 10 sources, read 3):

Recommendation

1. Start free: Try WebSearch + WebFetch first 2. Fall back to x402: Use Exa/Firecrawl when free tools fail 3. Exa for discovery: Use Exa search to find URLs 4. Cheapest extraction: Use Exa contents ($0.002/call) for known URLs 5. Firecrawl for tough sites: Use when JS rendering or bypass needed