Brightdata

Customization

Before executing, check for user customizations at: ~/.claude/PAI/USER/SKILLCUSTOMIZATIONS/BrightData/

If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.

🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

You MUST send this notification BEFORE doing anything else when this skill is invoked.

1. Send voice notification:

   curl -s -X POST http://localhost:31337/notify \
     -H "Content-Type: application/json" \
     -d '{"message": "Running the WORKFLOWNAME workflow in the BrightData skill to ACTION"}' \
     > /dev/null 2>&1 &

2. Output text notification:

   Running the **WorkflowName** workflow in the **BrightData** skill to ACTION...

This is not optional. Execute this curl command immediately upon skill invocation.

Workflow Routing

When executing a workflow, output this notification directly:

Running the **WorkflowName** workflow in the **Brightdata** skill to ACTION...

Route to the appropriate workflow based on the request.

When user requests scraping/fetching a single URL: Examples: "scrape this URL", "fetch this page", "get content from [URL]", "pull content from this site", "retrieve [URL]", "can't access this site", "this site is blocking me", "use Bright Data to fetch" → READ: Workflows/FourTierScrape.md → EXECUTE: Four-tier progressive scraping workflow (WebFetch → Curl → Browser Automation → Bright Data MCP)

When user requests crawling multiple pages from a site: Examples: "crawl this site", "crawl all pages under /docs", "spider this domain", "map this website", "get all pages from", "crawl [URL]", "scrape the whole site", "extract all pages" → READ: Workflows/Crawl.md → EXECUTE: Crawl workflow (Light Crawl for <50 pages, Full Crawl via Bright Data Crawl API for larger sites)

---

When to Activate This Skill

Direct Scraping Requests (Categories 1-4)

• "scrape this URL", "scrape [URL]", "scrape this page"
• "fetch this URL", "fetch [URL]", "fetch this page", "fetch content from"
• "pull content from [URL]", "pull this page", "pull from this site"
• "get content from [URL]", "retrieve [URL]", "retrieve this page"
• "do scraping on [URL]", "run scraper on [URL]"
• "basic scrape", "quick scrape", "simple fetch"
• "comprehensive scrape", "deep scrape", "full content extraction"

Access & Bot Detection Issues (Categories 5-7)

• "can't access this site", "site is blocking me", "getting blocked"
• "bot detection", "CAPTCHA", "access denied", "403 error"
• "need to bypass bot detection", "get around blocking"
• "this URL won't load", "can't fetch this page"
• "use Bright Data", "use the scraper", "use advanced scraping"

Result-Oriented Requests (Category 8)

• "get me the content from [URL]"
• "extract text from [URL]"
• "download this page content"
• "convert [URL] to markdown"
• "need the HTML from this site"

Crawling Requests (Categories 9-11)

• "crawl this site", "crawl [URL]", "spider this domain"
• "map this website", "get all pages from [URL]", "scrape the whole site"
• "crawl all pages under /docs", "extract all pages from", "site crawl"
• "get every page on this site", "full site extraction"
• "crawl depth 3", "crawl up to 50 pages"

Use Case Indicators

• User needs web content for research or analysis
• Standard methods (WebFetch) are failing
• Site has bot detection or rate limiting
• Need reliable content extraction
• Converting web pages to structured format (markdown)
• User needs multiple pages from a site, not just one
• User wants to map a site's structure or extract a section

---

Core Capabilities

Progressive Escalation Strategy: 1. Tier 1: WebFetch - Fast, simple, built-in Claude Code tool 2. Tier 2: Customized Curl - Chrome-like browser headers to bypass basic bot detection 3. Tier 3: agent-browser - Headless browser automation via agent-browser Rust CLI daemon for JavaScript-heavy sites. Playwright is banned across PAI. 4. Tier 4: Bright Data MCP - Professional scraping service that handles CAPTCHA and advanced bot detection

Key Features:

• Automatic fallback between tiers
• Preserves content in markdown format
• Handles bot detection and CAPTCHA
• Works with any URL
• Efficient resource usage (only escalates when needed)

---

Workflow Overview

FourTierScrape.md - Complete URL content scraping with four-tier fallback strategy

• When to use: Any single URL content retrieval request
• Process: Start with WebFetch → If fails, use curl with Chrome headers → If fails, use Browser Automation → If fails, use Bright Data MCP
• Output: URL content in markdown format

Crawl.md - Multi-page crawling with link discovery and site mapping

• When to use: Crawling multiple pages from a site, mapping site structure, extracting a section
• Process: Light Crawl (MCP scrape_batch + link extraction loop, up to 50 pages) or Full Crawl (Bright Data Crawl API for entire sites)
• Output: Site map + page contents in markdown, with crawl stats and cost summary

---

Extended Context

Integration Points:

• WebFetch Tool - Built-in Claude Code tool for basic URL fetching
• Bash Tool - For executing curl commands with custom headers
• Browser Automation - agent-browser headless daemon for JavaScript rendering
• Bright Data MCP - mcp__Brightdata__scrape_as_markdown and scrape_batch for advanced scraping
• Bright Data Crawl API - HTTP POST to api.brightdata.com/datasets/v3/trigger for full-site crawls

When Each Tier Is Used:

• Tier 1 (WebFetch): Simple sites, public content, no bot detection
• Tier 2 (Curl): Sites with basic user-agent checking, simple bot detection
• Tier 3 (agent-browser): Sites requiring JavaScript execution, dynamic content loading
• Tier 4 (Bright Data): Sites with CAPTCHA, advanced bot detection, residential proxy requirements

Configuration: No configuration required - all tools are available by default in Claude Code

---

Examples

Example 1: Simple Public Website

User: "Scrape https://example.com"

Skill Response: 1. Routes to three-tier-scrape.md 2. Attempts Tier 1 (WebFetch) 3. Success → Returns content in markdown 4. Total time: <5 seconds

Example 2: Site with JavaScript Requirements

User: "Can't access this site https://dynamic-site.com"

Skill Response: 1. Routes to four-tier-scrape.md 2. Attempts Tier 1 (WebFetch) → Fails (blocked) 3. Attempts Tier 2 (Curl with Chrome headers) → Fails (JavaScript required) 4. Attempts Tier 3 (agent-browser) → Success 5. Returns content in markdown 6. Total time: ~15-20 seconds

Example 3: Site with Advanced Bot Detection

User: "Scrape https://protected-site.com"

Skill Response: 1. Routes to four-tier-scrape.md 2. Attempts Tier 1 (WebFetch) → Fails (blocked) 3. Attempts Tier 2 (Curl) → Fails (advanced detection) 4. Attempts Tier 3 (agent-browser) → Fails (CAPTCHA) 5. Attempts Tier 4 (Bright Data MCP) → Success 6. Returns content in markdown 7. Total time: ~30-40 seconds

Example 4: Explicit Bright Data Request

User: "Use Bright Data to fetch https://difficult-site.com"

Skill Response: 1. Routes to four-tier-scrape.md 2. User explicitly requested Bright Data 3. Goes directly to Tier 4 (Bright Data MCP) → Success 4. Returns content in markdown 5. Total time: ~5-10 seconds

---

Related Documentation:

• ~/.claude/PAI/DOCUMENTATION/Skills/SkillSystem.md - Canonical structure guide
• ~/.claude/ - Overall PAI philosophy

Last Updated: 2026-02-22

Gotchas

• 4-tier escalation: WebFetch → curl → agent-browser → Bright Data proxy. Always start at Tier 1 and escalate only when blocked. Playwright is banned across PAI.
• Bright Data proxy has usage costs. Don't use Tier 4 for sites accessible via Tier 1-3.
• CAPTCHA-solving introduces latency. Allow extra time for Tier 4 responses.
• Credentials in `~/.claude/.env` — BRIGHTDATA_API_KEY.

Execution Log

After completing any workflow, append a single JSONL entry:

echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","skill":"BrightData","workflow":"WORKFLOW_USED","input":"8_WORD_SUMMARY","status":"ok|error","duration_s":SECONDS}' >> ~/.claude/PAI/MEMORY/SKILLS/execution.jsonl

Replace WORKFLOW_USED with the workflow executed, 8_WORD_SUMMARY with a brief input description, and SECONDS with approximate wall-clock time. Log status: "error" if the workflow failed.

Crawl Workflow

Purpose: Crawl multiple pages from a website — following links, extracting content, and returning structured results. Two modes: Light Crawl (agent-driven, using MCP batch scraping) and Full Crawl (Bright Data Crawl API for entire sites).

---

Prerequisites

• Starting URL (required)
• Crawl scope: specific section, depth limit, or full site
• Output format preference (markdown, HTML, JSON)
• For Full Crawl: Bright Data API key (set in environment)

---

Mode Selection

Choose based on scale:

Decision logic:

• User says "crawl this section" or "get all pages under /docs" → Light Crawl
• User says "crawl the entire site" or "map this whole site" → Full Crawl
• User says "crawl" without scope → Ask how many pages or what section, then choose

---

Light Crawl (Agent-Driven)

Uses MCP scrape_batch (up to 10 URLs per call) with iterative link discovery.

Step 1: Scrape the Starting URL

Use the FourTierScrape workflow to fetch the starting URL. Extract all internal links from the page content.

Scrape starting URL → Extract all <a href="..."> links → Filter to same-domain only

Link filtering rules:

• Same domain only (no external links unless user requests it)
• Respect URL path scope if user specified one (e.g., only links under /docs/)
• Deduplicate URLs (normalize trailing slashes, query params)
• Skip: anchors (#), mailto:, tel:, javascript:, static assets (.css, .js, .png, .jpg, .svg, .pdf)

Step 2: Batch Scrape Discovered Links

Use mcp__Brightdata__scrape_as_markdown or scrape_batch to fetch pages in batches of up to 10.

For each batch of up to 10 unvisited URLs:
  1. Call scrape_batch with the URLs
  2. Extract new internal links from results
  3. Add new links to the queue (if not already visited)
  4. Store page content in results collection
  5. Repeat until: queue empty OR page limit reached OR depth limit reached

Depth tracking:

• Starting URL = depth 0
• Links found on starting URL = depth 1
• Links found on depth 1 pages = depth 2
• Default max depth: 3 (override with user preference)

Page limit:

• Default: 30 pages
• User can specify: "crawl up to 100 pages"
• Hard cap: 50 pages per Light Crawl (for cost/time control)

Step 3: Compile Results

Assemble all crawled pages into a structured output:

## Crawl Results: [domain]
**Pages crawled:** [N]
**Depth reached:** [N]
**Starting URL:** [URL]

### Site Map
- [URL 1] (depth 0)
  - [URL 2] (depth 1)
  - [URL 3] (depth 1)
    - [URL 4] (depth 2)

### Page Contents

#### [URL 1]
[markdown content]

#### [URL 2]
[markdown content]
...

Light Crawl Error Handling

• If a page fails all 4 tiers → log it as failed, continue crawling other pages
• If >50% of pages fail → warn user, suggest Full Crawl instead
• If rate limited → add 2-second delay between batches

---

Full Crawl (Bright Data Crawl API)

For large-scale site crawling. Uses Bright Data's dedicated Crawl API via HTTP POST.

Step 1: Configure the Crawl

Build the API request based on user requirements:

curl -X POST "https://api.brightdata.com/datasets/v3/trigger?dataset_id=CRAWL_DATASET_ID&format=json" \
  -H "Authorization: Bearer ${BRIGHT_DATA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '[{
    "url": "[STARTING_URL]",
    "crawl_depth": [DEPTH],
    "url_filter": "[REGEX_PATTERN]",
    "format": "markdown"
  }]'

Key parameters:

• url — Starting URL for the crawl
• crawl_depth — How many link hops to follow (default: 3)
• url_filter — Regex to restrict which URLs to crawl (e.g., "https://example\\.com/docs/.*")
• format — Output format: markdown, html, json, ld_json
• include_errors — Set to true for detailed error logs

Step 2: Monitor Progress

The API returns a snapshot_id. Poll for completion:

curl -X GET "https://api.brightdata.com/datasets/v3/progress/${SNAPSHOT_ID}" \
  -H "Authorization: Bearer ${BRIGHT_DATA_API_KEY}"

Status values: running, ready, failed

• Poll every 10 seconds
• Timeout after 5 minutes for small sites, 15 minutes for large
• Report progress to user: "Crawling in progress... X pages collected so far"

Step 3: Retrieve Results

Once status is ready:

curl -X GET "https://api.brightdata.com/datasets/v3/snapshot/${SNAPSHOT_ID}?format=json" \
  -H "Authorization: Bearer ${BRIGHT_DATA_API_KEY}"

Results come as an array of page objects, each with URL and content.

Step 4: Present Results

Same structured output format as Light Crawl, plus:

• Total pages crawled
• Total cost estimate ($1.50/1K pages)
• Any errors encountered
• Site map derived from crawled URLs

Full Crawl Error Handling

• API auth failure → check BRIGHT_DATA_API_KEY environment variable
• Timeout → offer to increase wait time or try Light Crawl for smaller scope
• Partial failure → deliver what was crawled, list failed URLs

---

Output & Verification

For both modes:

1. Present site map — hierarchical URL tree showing crawl structure 2. Present content — each page's content in the requested format 3. Summary stats — pages crawled, depth, time taken, cost 4. Quality check — flag any pages that returned empty or error content 5. Offer export — if large result set, offer to write to a file

---

Cost Considerations

Always confirm with user before:

• Light Crawl exceeding 20 pages
• Any Full Crawl (involves Bright Data API costs)

---

Examples

Example 1: Crawl a Documentation Section

User: "Crawl all the pages under https://docs.example.com/api/"

1. Mode: Light Crawl (section-scoped) 2. Start URL: https://docs.example.com/api/ 3. URL filter: only links matching /api/ path 4. Depth: 3 5. Page limit: 30 6. Scrape starting page → find 12 links under /api/ 7. Batch scrape 10, then 2 → find 5 more sub-pages 8. Batch scrape 5 → no new links 9. Total: 20 pages crawled in ~90 seconds 10. Return structured content + site map

Example 2: Map an Entire Site

User: "Crawl the entire site at https://smallbusiness.com"

1. Mode: Full Crawl (entire site) 2. Confirm with user: "This will use the Bright Data Crawl API. Estimated cost depends on site size. Proceed?" 3. POST to Crawl API with depth 3, no URL filter 4. Poll for completion 5. Retrieve results (e.g., 250 pages) 6. Present site map + content summary 7. Cost: ~$0.38

Example 3: Competitive Research Crawl

User: "Crawl competitor.com and get all their product pages"

1. Mode: Full Crawl with URL filter 2. URL filter regex: "https://competitor\\.com/products/.*" 3. Depth: 2 (products are usually 1-2 levels deep) 4. Crawl API handles bot detection automatically 5. Return structured product page content

---

Related Workflows

• FourTierScrape.md — Single-page scraping (used internally by Light Crawl for the starting URL and as fallback)

Four-Tier URL Content Scraping

Voice Notification

curl -s -X POST http://localhost:31337/notify \
  -H "Content-Type: application/json" \
  -d '{"message": "Running the FourTierScrape workflow in the BrightData skill to scrape URL content"}' \
  > /dev/null 2>&1 &

Running FourTierScrape in BrightData...

---

Purpose: Progressive escalation strategy to retrieve URL content using four fallback tiers

When to Use:

• User requests scraping or fetching content from any URL
• Standard methods are failing or blocked
• Site has bot detection or access restrictions
• Need reliable content extraction in markdown format

Prerequisites:

• URL to scrape (provided by user)
• WebFetch tool (built-in)
• Bash tool for curl commands
• Browser automation capability (agent-browser — Playwright is banned)
• Bright Data MCP available

---

Workflow Steps

Step 1: Tier 1 - WebFetch (Fast & Simple)

Description: Attempt to fetch URL using Claude Code's built-in WebFetch tool

Actions:

Use WebFetch tool with:
- URL: [user-provided URL]
- Prompt: "Extract all content from this page and convert to markdown"

Expected Outcomes:

• Success: Content retrieved in markdown format → Skip to Step 4 (Output)
• Failure: WebFetch blocked, timeout, or error → Proceed to Step 2 (Tier 2)

Typical Success Cases:

• Public websites without bot detection
• Simple content sites
• Sites with permissive access policies

Typical Failure Cases:

• Sites with user-agent filtering
• Sites with basic bot detection
• Sites requiring specific headers

---

Step 2: Tier 2 - Customized Curl (Chrome-like Headers)

Description: Use curl with comprehensive Chrome browser headers to bypass basic bot detection

Actions:

curl -L -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" \
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8" \
  -H "Accept-Language: en-US,en;q=0.9" \
  -H "Accept-Encoding: gzip, deflate, br" \
  -H "DNT: 1" \
  -H "Connection: keep-alive" \
  -H "Upgrade-Insecure-Requests: 1" \
  -H "Sec-Fetch-Dest: document" \
  -H "Sec-Fetch-Mode: navigate" \
  -H "Sec-Fetch-Site: none" \
  -H "Sec-Fetch-User: ?1" \
  -H "Cache-Control: max-age=0" \
  --compressed \
  "[URL]"

Header Explanation:

• User-Agent: Latest Chrome on macOS (most common, least suspicious)
• Accept headers: Legitimate browser accept patterns
• *Sec-Fetch- headers:** Chrome's security headers (critical for bypassing detection)
• DNT: Do Not Track (common privacy setting)
• --compressed: Handle gzip/br encoding like real browsers

Expected Outcomes:

• Success: HTML content retrieved → Convert to markdown → Skip to Step 5 (Output)
• Failure: Still blocked, CAPTCHA, or JavaScript required → Proceed to Step 3 (Tier 3)

Typical Success Cases:

• Sites with basic user-agent checking
• Sites with simple header validation
• Sites without JavaScript rendering requirements

Typical Failure Cases:

• Sites with CAPTCHA
• Sites requiring JavaScript execution
• Sites with advanced fingerprinting
• Sites with IP-based rate limiting

---

Step 3: Tier 3 - Browser Automation (agent-browser)

Description: Use headless browser automation via agent-browser (Rust CLI daemon) to handle JavaScript-heavy sites. Playwright is banned across PAI.

Actions:

# Use agent-browser to navigate and extract rendered content
agent-browser --session scrape open "<url>"
agent-browser --session scrape text

What Browser Automation Provides:

• Real browser execution (Chrome/Firefox)
• Full JavaScript rendering and execution
• DOM manipulation and dynamic content loading
• Cookie/session handling
• Screenshot and PDF capabilities
• Network request interception
• Proper browser fingerprinting

Expected Outcomes:

• Success: Content extracted from fully rendered page → Convert to markdown → Skip to Step 5 (Output)
• Failure: CAPTCHA or advanced bot detection → Proceed to Step 4 (Tier 4)

Typical Success Cases:

• Single-page applications (SPAs)
• Sites with heavy JavaScript frameworks (React, Vue, Angular)
• Sites with dynamic content loading
• Sites requiring cookies/sessions
• Sites with complex DOM structures

Typical Failure Cases:

• Sites with CAPTCHA challenges
• Sites with advanced bot detection that fingerprint browser automation
• Sites requiring residential IP addresses
• Sites with aggressive rate limiting based on datacenter IPs

---

Step 4: Tier 4 - Bright Data MCP (Professional Scraping)

Description: Use Bright Data MCP's professional scraping service with bot detection bypass

Actions:

Use mcp__Brightdata__scrape_as_markdown tool with:
- URL: [user-provided URL]

What Bright Data Provides:

• Residential proxy network (real IP addresses)
• Automatic CAPTCHA solving
• JavaScript rendering (headless browser)
• Anti-bot detection bypass
• Automatic retry logic
• Content extraction and markdown conversion

Expected Outcomes:

• Success: Content retrieved in markdown format → Proceed to Step 5 (Output)
• Failure: Extremely rare - site may be completely inaccessible or down

Typical Success Cases:

• Sites with CAPTCHA challenges
• Sites with advanced bot detection and fingerprinting
• Sites requiring residential IP addresses
• Sites with aggressive rate limiting
• Any site that blocked Tiers 1, 2, and 3

Typical Failure Cases:

• Site is completely down
• Site requires authentication (login)
• Site has legal restrictions (e.g., paywall, geographic restrictions)

---

Step 5: Output & Verification

Description: Present retrieved content to user with tier information

Actions:

• Present content in markdown format
• Indicate which tier was successful
• Provide any warnings or notes about content quality

Verification:

• Content is readable and properly formatted
• Content matches expected URL
• No major sections missing

Example Output:

Successfully retrieved content from [URL] using Tier [1/2/3/4]

[Content in markdown format...]

---

Outputs

Primary Output:

• URL content in markdown format
• Includes title, headers, paragraphs, links, images (as markdown)

Metadata:

• Which tier was successful
• Any warnings or notes
• Execution time

Where outputs are stored:

• Returned directly to user in conversation
• No persistent storage (unless user requests it)

---

Decision Logic

START
  ↓
Attempt Tier 1 (WebFetch)
  ↓
Success? → Yes → Return content ✓
  ↓
  No
  ↓
Attempt Tier 2 (Curl + Chrome Headers)
  ↓
Success? → Yes → Return content ✓
  ↓
  No
  ↓
Attempt Tier 3 (Browser Automation)
  ↓
Success? → Yes → Return content ✓
  ↓
  No
  ↓
Attempt Tier 4 (Bright Data MCP)
  ↓
Success? → Yes → Return content ✓
  ↓
  No
  ↓
Report failure + suggest alternatives

---

Error Handling

If Tier 1 Fails:

• Log failure reason (blocked, timeout, error)
• Automatically proceed to Tier 2
• No user intervention required

If Tier 2 Fails:

• Log failure reason
• Automatically proceed to Tier 3
• No user intervention required

If Tier 3 Fails:

• Log failure reason
• Automatically proceed to Tier 4
• No user intervention required

If Tier 4 Fails:

• Report to user that site is inaccessible
• Suggest alternatives:
• Check if URL is correct
• Check if site requires authentication
• Check if site has geographic restrictions
• Try accessing manually in browser to verify site is up

---

Optimization Notes

When to Skip Tiers:

• If user explicitly requests "use Bright Data" → Skip directly to Tier 4
• If user explicitly requests "use browser" → Skip to Tier 3
• If previous scrape of same domain failed at Tier 1 → Start at Tier 2
• If URL is known SPA or JavaScript-heavy → Consider starting at Tier 3
• If URL is known difficult site with CAPTCHA → Consider starting at Tier 4

Cost Considerations:

• Tier 1: Free (built-in)
• Tier 2: Free (built-in)
• Tier 3: Free (local browser automation)
• Tier 4: Uses Bright Data credits (minimal cost per scrape)
• Always try cheaper tiers first unless user specifies otherwise

Performance:

• Tier 1: ~2-5 seconds
• Tier 2: ~3-7 seconds
• Tier 3: ~10-20 seconds
• Tier 4: ~5-15 seconds
• Total worst-case: ~40 seconds for all four attempts

---

Related Workflows

• None (this is the primary workflow for brightdata skill)

Future Enhancements:

• Add caching layer to avoid re-scraping same URLs
• Add batch scraping for multiple URLs
• Add domain-specific optimizations (known difficult sites)
• Add custom header profiles for different site types

---

Examples

Example 1: Public Site (Tier 1 Success)

Input: https://example.com

Process: 1. Attempt Tier 1 (WebFetch) 2. Success in 3 seconds 3. Return content

Output:

Successfully retrieved content from https://example.com using Tier 1 (WebFetch)

# Example Domain

This domain is for use in illustrative examples...

Example 2: JavaScript-Heavy Site (Tier 3 Success)

Input: https://spa-site.com

Process: 1. Attempt Tier 1 (WebFetch) → Blocked (403) 2. Attempt Tier 2 (Curl) → Returns empty (JavaScript required) 3. Attempt Tier 3 (Browser Automation) → Success in 15 seconds 4. Return content

Output:

Successfully retrieved content from https://spa-site.com using Tier 3 (Browser Automation)

Note: This site requires JavaScript rendering. Content was retrieved using agent-browser.

# SPA Site Content

[Content retrieved successfully...]

Example 3: Protected Site with CAPTCHA (Tier 4 Success)

Input: https://protected-site.com

Process: 1. Attempt Tier 1 (WebFetch) → Blocked (403) 2. Attempt Tier 2 (Curl) → Blocked (bot detection) 3. Attempt Tier 3 (Browser Automation) → Blocked (CAPTCHA) 4. Attempt Tier 4 (Bright Data) → Success in 12 seconds 5. Return content

Output:

Successfully retrieved content from https://protected-site.com using Tier 4 (Bright Data MCP)

Note: This site has advanced bot detection and CAPTCHA. Content was retrieved using professional scraping service.

# Protected Site Content

[Content retrieved successfully...]

Example 4: Explicit Bright Data Request

Input: "Use Bright Data to fetch https://any-site.com"

Process: 1. User explicitly requested Bright Data 2. Skip directly to Tier 4 3. Success in 8 seconds 4. Return content

Output:

Retrieved content from https://any-site.com using Tier 4 (Bright Data MCP) as requested

[Content...]

---

Last Updated: 2025-11-23