Web Research Workflow

Web Research Workflow

Unified approach for web content research that automatically selects the right tool for each situation.

Quick Decision Tree

URL to research
     │
     ▼
┌─────────────────┐
│ 1. Try WebFetch │ ← Fast, free, no overhead
│    (always try) │
└─────────────────┘
     │
Content OK? ──Yes──► Parse and return
     │
     No (empty/partial/<500 chars)
     │
     ▼
┌───────────────────────┐
│ 2. TAVILY_API_KEY set?│
└───────────────────────┘
     │          │
    Yes         No ──► Skip to step 3
     │
     ▼
┌───────────────────────────┐
│ Tavily search/extract/    │ ← Raw markdown, batch URLs
│ crawl/research            │
└───────────────────────────┘
     │
Content OK? ──Yes──► Parse and return
     │
     No (JS-rendered/auth-required)
     │
     ▼
┌─────────────────────┐
│ 3. Use agent-browser │ ← Full browser, last resort
└─────────────────────┘
     │
├─ SPA (react/vue/angular) ──► wait --load networkidle
├─ Login required ──► auth flow + state save
├─ Dynamic content ──► wait --text "Expected"
└─ Multi-page ──► crawl pattern

Tavily Enhanced Research

When TAVILY_API_KEY is set, Tavily provides a powerful middle tier between WebFetch and agent-browser. It returns raw markdown content, supports batch URL extraction, and offers semantic search with relevance scoring. If TAVILY_API_KEY is not set, the 3-tier tree collapses to 2-tier (WebFetch → agent-browser) automatically.

Load details: Read("${CLAUDE_SKILL_DIR}/rules/tool-selection.md") for when-to-use-what tables, escalation heuristics, SPA detection patterns, and cost awareness.

Load details: Read("${CLAUDE_SKILL_DIR}/references/tavily-api.md") for Search, Extract, Map, Crawl, and Research endpoint examples and options.

Browser Patterns

For content requiring JavaScript rendering, authentication, or multi-page crawling, fall back to agent-browser.

Load details: Read("${CLAUDE_SKILL_DIR}/rules/browser-patterns.md") for auto-fallback, authentication flow, multi-page research patterns, best practices, and troubleshooting.

Competitive Monitoring

Track competitor websites for changes in pricing, features, positioning, and content.

Load details: Read("${CLAUDE_SKILL_DIR}/rules/monitoring-competitor.md") for snapshot capture, structured data extraction, and change classification.

Load details: Read("${CLAUDE_SKILL_DIR}/rules/monitoring-change-detection.md") for diff detection, structured comparison, Tavily site discovery, and CI automation.

Change Classification

Integration with Agents

This skill is used by:

• web-research-analyst - Primary user
• market-intelligence - Competitor research
• product-strategist - Deep competitive analysis

Related Skills

• browser-content-capture - Detailed browser patterns
• agent-browser - CLI reference

---

Version: 1.3.0 (March 2026)

Tavily API Reference

Search (Semantic Web Search)

Returns relevance-scored results with raw markdown content:

curl -s -X POST 'https://api.tavily.com/search' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TAVILY_API_KEY" \
  -d '{
    "query": "your search query",
    "search_depth": "advanced",
    "max_results": 5,
    "include_raw_content": "markdown"
  }' | python3 -m json.tool

Options:

• search_depth: "basic" (fast) or "advanced" (thorough, 2x cost)
• topic: "general" (default) or "news" or "finance"
• include_domains: ["example.com"] — restrict to specific sites
• exclude_domains: ["reddit.com"] — filter out sites
• days: 3 — limit to recent results (news/finance)
• include_raw_content: "markdown" — get full page content

Extract (Batch URL Content)

Extract raw content from up to 20 URLs at once:

curl -s -X POST 'https://api.tavily.com/extract' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TAVILY_API_KEY" \
  -d '{
    "urls": [
      "https://docs.example.com/guide",
      "https://competitor.com/pricing"
    ]
  }' | python3 -m json.tool

Returns markdown content for each URL. Use when you have specific URLs and need full content.

Map (Site Discovery)

Discover all URLs on a site before extracting:

curl -s -X POST 'https://api.tavily.com/map' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TAVILY_API_KEY" \
  -d '{
    "url": "https://docs.example.com",
    "max_depth": 2,
    "limit": 50
  }' | python3 -m json.tool

Useful for documentation sites and competitor sitemaps. Combine with extract for full crawl.

Crawl (Multi-Page Content Extraction)

Crawl an entire site and extract content from all discovered pages in one call. Replaces the manual map+extract two-step workflow:

curl -s -X POST 'https://api.tavily.com/crawl' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TAVILY_API_KEY" \
  -d '{
    "url": "https://docs.example.com",
    "max_depth": 2,
    "limit": 50,
    "include_raw_content": "markdown"
  }' | python3 -m json.tool

Options:

• max_depth: 2 — how many link-hops from the seed URL
• limit: 50 — max pages to crawl
• include_raw_content: "markdown" — get full page content (not just snippets)
• exclude_paths: ["/blog/*"] — skip certain URL patterns
• include_paths: ["/docs/*"] — restrict to certain URL patterns

When to use Crawl vs Map+Extract:

• Use Crawl when you want content from an entire site section (docs, changelog, pricing)
• Use Map when you only need URL discovery without content
• Use Extract when you already have specific URLs

Research (Deep Research — Beta)

Multi-step research agent that searches, reads, and synthesizes across multiple sources. Returns a comprehensive report with citations:

curl -s -X POST 'https://api.tavily.com/research' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TAVILY_API_KEY" \
  -d '{
    "query": "comparison of vector databases for RAG in 2026",
    "max_results": 10,
    "report_type": "research_report"
  }' | python3 -m json.tool

Options:

• report_type: "research_report" (default) | "outline_report" | "detailed_report"
• max_results: number of sources to analyze (more = deeper but slower)

Note: The /research endpoint is in beta. Falls back gracefully to /search + /extract if unavailable. Best for deep competitive analysis, market research, and technical comparisons.

Rule Categories

1. Tool Selection (tools) — HIGH — decision tree

Unified decision tree for web research tool selection: WebFetch, Tavily, agent-browser.

• tool-selection.md — When-to-use-what tables, escalation heuristics, SPA detection, cost awareness, graceful degradation

2. Browser Patterns (browser) — MEDIUM — 4 patterns

SPA detection, authentication flows, multi-page research, best practices, and troubleshooting.

• browser-patterns.md — Auto-fallback, auth flow, multi-page research, waits, rate limits, troubleshooting

3. Competitive Monitoring (monitoring) — HIGH — 2 rules

Track competitor page changes with snapshot capture, diff detection, and change classification.

• monitoring-competitor.md — Snapshot capture, structured data extraction, change classification, alerting
• monitoring-change-detection.md — Diff detection, structured comparison, site discovery, automated workflows

References

• references/tavily-api.md — Tavily Search, Extract, Map, Crawl, Research endpoint examples and options

[Rule Name]

[Brief description — 1-2 sentences.]

Incorrect:

// Bad pattern

Correct:

// Good pattern

Key rules:

• [Rule 1]
• [Rule 2]
• [Rule 3]

Reference: [link]

Browser Patterns

Pattern 1: Auto-Fallback

Try WebFetch first, fall back to browser if needed:

# Step 1: Try WebFetch
WebFetch(url="https://example.com", prompt="Extract main content")

# If result is empty, partial, or contains "Loading..." indicators:
# Step 2: Fall back to browser
agent-browser open https://example.com
agent-browser wait --load networkidle
agent-browser get text body

Pattern 2: Authentication Flow

For login-protected content:

# 1. Navigate to login
agent-browser open https://app.example.com/login
agent-browser snapshot -i

# 2. Fill credentials (use refs from snapshot)
agent-browser fill @e1 "$EMAIL"
agent-browser fill @e2 "$PASSWORD"
agent-browser click @e3  # Submit button

# 3. Wait for redirect
agent-browser wait --url "**/dashboard"

# 4. Save session for reuse
agent-browser state save /tmp/session-example.json

# 5. Later: restore session
agent-browser state load /tmp/session-example.json
agent-browser open https://app.example.com/protected-page

Pattern 3: Multi-Page Research

For documentation sites or multi-page content:

# 1. Get navigation links
agent-browser open https://docs.example.com
agent-browser snapshot -i

# 2. Extract all doc links
LINKS=$(agent-browser eval "JSON.stringify(
  Array.from(document.querySelectorAll('nav a'))
    .map(a => a.href)
    .filter(h => h.includes('/docs/'))
)")

# 3. Iterate with rate limiting
for link in $(echo "$LINKS" | jq -r '.[]' | head -20); do
  agent-browser open "$link"
  agent-browser wait --load networkidle
  agent-browser get text article > "/tmp/doc-$(basename $link).txt"
  sleep 2  # Rate limit
done

Best Practices

Use Appropriate Waits

# For SPAs with API calls
agent-browser wait --load networkidle

# For specific content
agent-browser wait --text "Expected content"

# For elements
agent-browser wait @e5

Respect Rate Limits

# Add delays between requests
sleep 2

# Use session isolation for parallel work
agent-browser --session site1 open https://site1.com
agent-browser --session site2 open https://site2.com

Cache Results

# Save extracted content to avoid re-scraping
agent-browser get text body > /tmp/cache/example-com.txt

Troubleshooting

Incorrect — Browser without waiting for content:

agent-browser open https://spa-app.com
agent-browser get text body
# Returns empty - content not loaded yet

Correct — Wait for network idle before extracting:

agent-browser open https://spa-app.com
agent-browser wait --load networkidle
agent-browser get text body
# Content fully loaded

Change Detection and Site Discovery

Detect differences between snapshots and discover competitor pages for monitoring.

Incorrect -- manual visual comparison:

# Open two browser tabs and visually compare
# No structured diff, easy to miss changes
# No record of what changed or when

Correct -- automated diff with structured comparison:

# Text diff between latest two snapshots
LATEST=$(ls -t .competitive-intel/snapshots/competitor-pricing-*.txt | head -1)
PREVIOUS=$(ls -t .competitive-intel/snapshots/competitor-pricing-*.txt | head -2 | tail -1)

diff -u "$PREVIOUS" "$LATEST" > \
  .competitive-intel/diffs/competitor-pricing-$(date +%Y%m%d).diff

# Structured JSON comparison for pricing changes
LATEST_JSON=$(ls -t .competitive-intel/snapshots/competitor-pricing-*.json | head -1)
PREVIOUS_JSON=$(ls -t .competitive-intel/snapshots/competitor-pricing-*.json | head -2 | tail -1)

jq -s '
  .[0] as $old | .[1] as $new |
  {
    price_changes: [
      $new[] | . as $tier |
      ($old[] | select(.name == $tier.name)) as $old_tier |
      select($old_tier.price \!= $tier.price) |
      {name: .name, old_price: $old_tier.price, new_price: .price}
    ],
    new_tiers: [$new[] | select(.name as $n | $old | map(.name) | index($n) | not)],
    removed_tiers: [$old[] | select(.name as $n | $new | map(.name) | index($n) | not)]
  }
' "$PREVIOUS_JSON" "$LATEST_JSON"

Site discovery with Tavily (when TAVILY_API_KEY is set):

# Crawl competitor site -- discovers URLs and extracts content
# Use Tavily crawl API with include_paths filter
# Save results to .competitive-intel/snapshots/ directory
# See web-research-workflow SKILL.md for full Tavily crawl examples

Automated monitoring via CI:

# .github/workflows/competitive-monitor.yml
name: Competitive Monitor
on:
  schedule:
    - cron: '0 9 * * *'  # Daily at 9 AM
jobs:
  monitor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm install -g agent-browser
      - run: agent-browser install
      - run: ./scripts/run-competitive-monitor.sh
      - uses: actions/upload-artifact@v4
        with:
          name: competitive-intel
          path: .competitive-intel/

Key rules:

• Run structured JSON diffs, not just text diffs, to detect price and feature changes precisely
• Use Tavily crawl for initial site discovery when API key is available, fall back to manual URL lists
• Store diffs with timestamps for change history and trend analysis
• Automate monitoring in CI with daily schedules for consistent coverage

Competitor Page Monitoring

Capture point-in-time snapshots of competitor pages for pricing, features, and positioning tracking.

Incorrect -- manual one-off checks with no history:

# Check competitor pricing once, no record kept
# No snapshot, no history, no comparison baseline

Correct -- structured snapshot capture with change classification:

# Create snapshots directory
mkdir -p .competitive-intel/snapshots

# Capture text content
agent-browser open https://competitor.com/pricing
agent-browser wait --load networkidle
agent-browser get text body > \
  .competitive-intel/snapshots/competitor-pricing-$(date +%Y%m%d).txt

# Extract structured pricing data
agent-browser eval "JSON.stringify(
  Array.from(document.querySelectorAll('.pricing-tier')).map(tier => ({
    name: tier.querySelector('h3')?.innerText,
    price: tier.querySelector('.price')?.innerText,
    features: Array.from(tier.querySelectorAll('li')).map(li => li.innerText)
  }))
)" > .competitive-intel/snapshots/competitor-pricing-$(date +%Y%m%d).json

agent-browser close

Change classification:

Config file for monitoring targets:

{
  "monitors": [
    {
      "name": "Competitor A Pricing",
      "url": "https://competitor-a.com/pricing",
      "frequency": "daily",
      "selectors": {
        "pricing": ".pricing-tier",
        "features": ".feature-list li"
      },
      "alerts": {
        "price_change": "critical",
        "new_feature": "high",
        "copy_change": "low"
      }
    }
  ],
  "storage": ".competitive-intel",
  "retention_days": 90
}

Key rules:

• Always save both text and structured (JSON) snapshots for each capture
• Use consistent date-stamped filenames for chronological comparison
• Classify changes by severity to prioritize strategic response
• Store findings in knowledge graph for cross-session persistence

Tool Selection Rules

When to Use Tavily over WebFetch

• WebFetch returned <500 chars (likely incomplete)
• You need raw markdown content (not Haiku-summarized)
• Batch extracting content from multiple URLs
• Semantic search with relevance scoring
• Site discovery/crawling (map API)

When to Skip Tavily and Go to agent-browser

• Content requires JavaScript rendering (SPAs)
• Authentication/login is required
• Interactive elements need clicking
• Content is behind CAPTCHAs

Scenario-Based Selection

Detection Heuristics

Content likely needs browser if WebFetch returns:

• Empty or very short content (< 500 chars)
• Contains <noscript> tags
• Contains "Loading...", "Please wait", "JavaScript required"
• Contains only <div id="root"></div> or <div id="app"></div>
• Returns 403/401 (may need auth)

SPA Detection

Known patterns that always need browser:

# URL patterns suggesting SPA
app.* | dashboard.* | portal.* | console.*

# Framework indicators in initial HTML
"__NEXT_DATA__"     → Next.js (may work with WebFetch)
"window.__NUXT__"   → Nuxt.js (may work with WebFetch)
"ng-app"            → Angular (needs browser)
"data-reactroot"    → React (needs browser)
"data-v-"           → Vue (needs browser)

Escalation Heuristic

# Auto-escalate from WebFetch to Tavily
CONTENT=$(WebFetch url="$URL" prompt="Extract main content")
CHAR_COUNT=${#CONTENT}

if [ "$CHAR_COUNT" -lt 500 ] && [ -n "$TAVILY_API_KEY" ]; then
  # WebFetch returned thin content — try Tavily extract
  RESULT=$(curl -s -X POST 'https://api.tavily.com/extract' \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $TAVILY_API_KEY" \
    -d "{\"urls\":[\"$URL\"]}")
  # Parse result...
fi

Cost Awareness

Graceful Degradation

If TAVILY_API_KEY is not set, the 3-tier tree collapses to the original 2-tier (WebFetch → agent-browser). No configuration needed — agents check for the env var before attempting Tavily calls.

Incorrect — Always using browser for everything:

agent-browser open https://github.com/owner/repo/blob/main/README.md
agent-browser wait --load networkidle
agent-browser get text body
# Slow, unnecessary - static content

Correct — Use WebFetch for static content:

WebFetch(
  url="https://github.com/owner/repo/blob/main/README.md",
  prompt="Extract README content"
)
# Fast, no browser needed